NLP analysis -- Movie Reviews from Rotten Tomatos

Posted on Feb 21, 2017

Contributed by Tingting Chang. She is currently in the NYC Data Science Academy 12 week full time Data Science Bootcamp program taking place between Jan 9th to March 31th, 2017. This post is based on her second class project - Web Scraping (due on the 6th week of the program).

About Tingting Chang: GitHub | LinkedIn





It has became super common for people looking at reviews before they go to the theater to see a movie. Some website provides movie ranking, rating and reviews for coming movies as well as the old movies. Rotten tomato is one of famous one and it grabs all the reviews from other movie critics websites.  However, it brings a question: are these ranking, rating, critics true? Do they have any connection between them? I scraped all the movie reviews, rankings, ratings for the top 100 movies in 2016. After using NLTK package to tokenize data and remove all the stop words, along with punctuations, I was left with about 13467 reviews with only words for all 100 movies. From there, I used sentiment analysis, ft-idf, topic model(LDA) to analysis the reviews.

This is how my data looks like:

Screen Shot 2017-02-20 at 7.40.54 PM

ranking, rating for all top 100 movies in 2016


Screen Shot 2017-02-20 at 7.41.38 PM

all the movie reviews data for 100 movies


The bar plotting for every movie's the number of reviews.   I picked the 20 movies with the most reviews count and other 20 movies with the least number of reviews.

Screen Shot 2017-02-12 at 6.59.01 PM


Screen Shot 2017-02-20 at 7.48.51 PM

Screen Shot 2017-02-20 at 7.49.09 PM

20 movies with the least movie reviews count


I used sentiment analysis function build in NLTK package to calculate sentiment score for every review:

Screen Shot 2017-02-20 at 7.58.50 PM

sentiment score for every review

For example, the first review "one disney best movies past years" has score 0.6369 which is pretty positive. Another negative review "animal antics display movie also gently broaches human issues stereotyping prejudice racism something " has score -0.4588 due to some negative words "stereotyping prejudice racism".

I also computed the average sentiment score for every movie:

Screen Shot 2017-02-20 at 8.35.33 PM

average sentiment score for every movie in 2016

Then I use box plot try to find out the relationship between the movie rating and sentiment mean compound score.

Screen Shot 2017-02-12 at 9.33.55 PM

box plot for sentiment mean score and movie rating

But there is not any specific relationship between the sentiment mean score and movie rating for every movie except that the most of movies have positive sentiment score.

In addition, I used another box plot for movie ranking and sentiment mean compound score. Then I found out that rank 2.0 has a negative sentiment score which was pretty interesting. I went to website and try to find out why.

Screen Shot 2017-02-12 at 8.10.59 PM

box plot for sentiment mean score and movie ranking

Screen Shot 2017-02-20 at 8.45.10 PM



For this movie: Things to Come ,the Rotten Tomato gives 100% rating and ranking 11th. However, you can see from the website that it only has 73% audience scores. I did some researches and I learned that some company would hire some reviewers to write fake reviews for a movie. Moreover, the significant reviewers' reviews will have a larger effect on ranking movie. Nevertheless, their taste can not 100% represent audience's taste. Therefore, it will be normal if movie ranking, rating does not match the sentiment score from movie reviews.


Also, I tried some EDA analysis to find out whether there was any relationship between movie ranking, rating, review counts and sentiment scores. Unfortunately, there is not any obvious relationship between them.

Screen Shot 2017-02-12 at 9.35.18 PM

Screen Shot 2017-02-12 at 10.17.40 PMScreen Shot 2017-02-12 at 9.35.57 PM


Screen Shot 2017-02-12 at 9.36.51 PM





TF, or Term Frequency, measures the count of words in a document. By using TF-IDF functions in NLTK packages, I gained words frequency for every words in the document:

Screen Shot 2017-02-20 at 9.11.41 PM

TF: words frequency for first two reviews

The next step of TF-IDF is the IDF or inverse document frequency. Targets the words only appear in certain documents instead of in all the documents, and give them higher frequency. Now we have got a weight for every tokened word in every document and we can determine the important words from the unimportant ones.

Screen Shot 2017-02-20 at 9.12.10 PM

IDF: inverse document frequency for every tokened word



Topic Model


Topic Modelling, as the name suggests, it is a process to derive hidden patterns exhibited by a text corpus. I built 9 topics for all the reviews and printed out top 20 frequent words for every topic:



Topic Model: top 20 words for 9 topics


For better visualization, I use WordCloud package in python to visualize the top 500 frequent words in every topic.


top 500 words for the 1st topic


top 500 words for the 2nd topic


top 500 words for the 3rd topic


top 500 words for the 4th topic


top 500 words for the 5th topic


top 500 words for the 6th topic



top 500 words for the 7th topic



top 500 words for the 8th topic


top 500 words for the 9th topic

From above plots, we can see the word "fan" shows up in three topics. I have not found out why this is happened.



About Author


Tingting Chang

Tingting Chang got her master degree in Computer Science from the George Washington University. She is a self-starter and hardworking data scientist well equipped with data analytics skills to obtain actionable insights from massive datasets without losing sight...
View all posts by Tingting Chang >

Related Articles

Leave a Comment

Google January 22, 2021
Google Very handful of sites that occur to be in depth beneath, from our point of view are undoubtedly properly worth checking out.
Google October 19, 2019
Google Very few internet sites that transpire to be in depth below, from our point of view are undoubtedly well worth checking out.
Google October 8, 2019
Google The data mentioned within the write-up are a number of the most effective obtainable.
Lynda April 2, 2018
can I have the source code of how you scrapped the data ? and also for the topic modeling ?
bath body works promo code February 28, 2017
I Just stopped by to say your post is fantastic. The clarity inside your post is simply magnificent and i can assume you’re an specialist on this topic. Fine together with your permission let me to grab your RSS feed to keep as much as date with forthcoming post. Thanks a million and please continue the enjoyable work.
online school tips February 26, 2017
Nice post. I learn something totally new and challenging on websites I stumbleupon everyday. It's always useful to read articles from other writers and practice a little something from their web sites.
educational sites February 26, 2017
Hi there, for all time i used to check website posts here in the early hours in the dawn, as i enjoy to gain knowledge of more and more.
online learning February 26, 2017
Asking questions are actually good thing if you are not understanding anything entirely, except this paragraph presents good understanding even.

View Posts by Categories

Our Recent Popular Posts

View Posts by Tags

#python #trainwithnycdsa 2019 airbnb Alex Baransky alumni Alumni Interview Alumni Reviews Alumni Spotlight alumni story Alumnus API Application artist aws beautiful soup Best Bootcamp Best Data Science 2019 Best Data Science Bootcamp Best Data Science Bootcamp 2020 Best Ranked Big Data Book Launch Book-Signing bootcamp Bootcamp Alumni Bootcamp Prep Bundles California Cancer Research capstone Career Career Day citibike clustering Coding Course Demo Course Report D3.js data Data Analyst data science Data Science Academy Data Science Bootcamp Data science jobs Data Science Reviews Data Scientist Data Scientist Jobs data visualization Deep Learning Demo Day Discount dplyr employer networking feature engineering Finance Financial Data Science Flask gbm Get Hired ggplot2 googleVis Hadoop higgs boson Hiring hiring partner events Hiring Partners Industry Experts Instructor Blog Instructor Interview Job Job Placement Jobs Jon Krohn JP Morgan Chase Kaggle Kickstarter lasso regression Lead Data Scienctist Lead Data Scientist leaflet linear regression Logistic Regression machine learning Maps matplotlib Medical Research Meet the team meetup Networking neural network Neural networks New Courses nlp NYC NYC Data Science nyc data science academy NYC Open Data NYCDSA NYCDSA Alumni Online Online Bootcamp Online Training Open Data painter pandas Part-time Portfolio Development prediction Prework Programming PwC python Python Data Analysis python machine learning python scrapy python web scraping python webscraping Python Workshop R R language R Programming R Shiny r studio R Visualization R Workshop R-bloggers random forest Ranking recommendation recommendation system regression Remote remote data science bootcamp Scrapy scrapy visualization seaborn Selenium sentiment analysis Shiny Shiny Dashboard Spark Special Special Summer Sports statistics streaming Student Interview Student Showcase SVM Switchup Tableau team TensorFlow Testimonial tf-idf Top Data Science Bootcamp twitter visualization web scraping Weekend Course What to expect word cloud word2vec XGBoost yelp