NLP Data analysis -- Movie Reviews from Rotten Tomatos

Posted on Feb 21, 2017
The skills the author demoed here can be learned through taking Data Science with Machine Learning bootcamp with NYC Data Science Academy.
Contributed by Tingting Chang. She is currently in the NYC Data Science Academy 12 week full time Data Science Bootcamp program taking place between Jan 9th to March 31th, 2017. This post is based on her second class project - Web Scraping (due on the 6th week of the program).
About Tingting Chang: GitHub | LinkedIn

Introduction

Data shows it has became super common for people looking at reviews before they go to the theater to see a movie. Some website provides movie ranking, rating and reviews for coming movies as well as the old movies.

Rotten tomato is one of famous one and it grabs all the reviews from other movie critics websites.  However, it brings a question: are these ranking, rating, critics true? Do they have any connection between them? I scraped all the movie reviews, rankings, ratings for the top 100 movies in 2016. After using NLTK package to tokenize data and remove all the stop words, along with punctuations, I was left with about 13467 reviews with only words for all 100 movies. From there, I used sentiment analysis, ft-idf, topic model(LDA) to analysis the reviews.

Data

This is how my data looks like:

NLP Data analysis -- Movie Reviews from Rotten Tomatos

ranking, rating for all top 100 movies in 2016

 

NLP Data analysis -- Movie Reviews from Rotten Tomatos

all the movie reviews data for 100 movies

 

The bar plotting for every movie's the number of reviews.   I picked the 20 movies with the most reviews count and other 20 movies with the least number of reviews.

NLP Data analysis -- Movie Reviews from Rotten Tomatos

 

Screen Shot 2017-02-20 at 7.48.51 PM

NLP Data analysis -- Movie Reviews from Rotten Tomatos

20 movies with the least movie reviews count

 

I used sentiment analysis function build in NLTK package to calculate sentiment score for every review:

Screen Shot 2017-02-20 at 7.58.50 PM

sentiment score for every review

For example, the first review "one disney best movies past years" has score 0.6369 which is pretty positive. Another negative review "animal antics display movie also gently broaches human issues stereotyping prejudice racism something " has score -0.4588 due to some negative words "stereotyping prejudice racism".

I also computed the average sentiment score for every movie:

Screen Shot 2017-02-20 at 8.35.33 PM

average sentiment score for every movie in 2016

 

Then I use box plot try to find out the relationship between the movie rating and sentiment mean compound score.

Screen Shot 2017-02-12 at 9.33.55 PM

box plot for sentiment mean score and movie rating

But there is not any specific relationship between the sentiment mean score and movie rating for every movie except that the most of movies have positive sentiment score.

In addition, I used another box plot for movie ranking and sentiment mean compound score. Then I found out that rank 2.0 has a negative sentiment score which was pretty interesting. I went to website and try to find out why.

Screen Shot 2017-02-12 at 8.10.59 PM

box plot for sentiment mean score and movie ranking

Screen Shot 2017-02-20 at 8.45.10 PM

 

 

For this movie: Things to Come ,the Rotten Tomato gives 100% rating and ranking 11th. However, you can see from the website that it only has 73% audience scores. I did some researches and I learned that some company would hire some reviewers to write fake reviews for a movie. Moreover, the significant reviewers' reviews will have a larger effect on ranking movie. Nevertheless, their taste can not 100% represent audience's taste. Therefore, it will be normal if movie ranking, rating does not match the sentiment score from movie reviews.

 

Also, I tried some EDA analysis to find out whether there was any relationship between movie ranking, rating, review counts and sentiment scores. Unfortunately, there is not any obvious relationship between them.

Screen Shot 2017-02-12 at 9.35.18 PM

Screen Shot 2017-02-12 at 10.17.40 PMScreen Shot 2017-02-12 at 9.35.57 PM

 

Screen Shot 2017-02-12 at 9.36.51 PM

 

 

TF-IDF

 

TF, or Term Frequency, measures the count of words in a document. By using TF-IDF functions in NLTK packages, I gained words frequency for every words in the document:

Screen Shot 2017-02-20 at 9.11.41 PM

TF: words frequency for first two reviews

The next step of TF-IDF is the IDF or inverse document frequency. Targets the words only appear in certain documents instead of in all the documents, and give them higher frequency. Now we have got a weight for every tokened word in every document and we can determine the important words from the unimportant ones.

Screen Shot 2017-02-20 at 9.12.10 PM

IDF: inverse document frequency for every tokened word

 

 

Topic Model

 

Topic Modelling, as the name suggests, it is a process to derive hidden patterns exhibited by a text corpus. I built 9 topics for all the reviews and printed out top 20 frequent words for every topic:

topic_model_1

topic_model_2

Topic Model: top 20 words for 9 topics

 

For better visualization, I use WordCloud package in python to visualize the top 500 frequent words in every topic.

pic_topic_0

 1st topic

pic_topic_1

 2nd topic

pic_topic_2

 3rd topic

pic_topic_3

4th topic

pic_topic_4

5th topic

pic_topic_5

 6th topic

 

pic_topic_6

 7th topic

 

pic_topic_7

 8th topic

pic_topic_8

 9th topic

From above plots, we can see the word "fan" shows up in three topics. I have not found out why this is happened.

 

 

About Author

Tingting Chang

Tingting Chang got her master degree in Computer Science from the George Washington University. She is a self-starter and hardworking data scientist well equipped with data analytics skills to obtain actionable insights from massive datasets without losing sight...
View all posts by Tingting Chang >

Related Articles

Leave a Comment

Google August 31, 2021
Google Here are some links to web pages that we link to for the reason that we assume they are worth visiting.
Google July 19, 2021
Google Sites of interest we've a link to.
Google January 22, 2021
Google Very handful of sites that occur to be in depth beneath, from our point of view are undoubtedly properly worth checking out.
Google October 19, 2019
Google Very few internet sites that transpire to be in depth below, from our point of view are undoubtedly well worth checking out.
Google October 8, 2019
Google The data mentioned within the write-up are a number of the most effective obtainable.
Lynda April 2, 2018
can I have the source code of how you scrapped the data ? and also for the topic modeling ?
bath body works promo code February 28, 2017
I Just stopped by to say your post is fantastic. The clarity inside your post is simply magnificent and i can assume you’re an specialist on this topic. Fine together with your permission let me to grab your RSS feed to keep as much as date with forthcoming post. Thanks a million and please continue the enjoyable work.
online school tips February 26, 2017
Nice post. I learn something totally new and challenging on websites I stumbleupon everyday. It's always useful to read articles from other writers and practice a little something from their web sites.
educational sites February 26, 2017
Hi there, for all time i used to check website posts here in the early hours in the dawn, as i enjoy to gain knowledge of more and more.
online learning February 26, 2017
Asking questions are actually good thing if you are not understanding anything entirely, except this paragraph presents good understanding even.

View Posts by Categories


Our Recent Popular Posts


View Posts by Tags

#python #trainwithnycdsa 2019 2020 Revenue 3-points agriculture air quality airbnb airline alcohol Alex Baransky algorithm alumni Alumni Interview Alumni Reviews Alumni Spotlight alumni story Alumnus ames dataset ames housing dataset apartment rent API Application artist aws bank loans beautiful soup Best Bootcamp Best Data Science 2019 Best Data Science Bootcamp Best Data Science Bootcamp 2020 Best Ranked Big Data Book Launch Book-Signing bootcamp Bootcamp Alumni Bootcamp Prep boston safety Bundles cake recipe California Cancer Research capstone car price Career Career Day citibike classic cars classpass clustering Coding Course Demo Course Report covid 19 credit credit card crime frequency crops D3.js data data analysis Data Analyst data analytics data for tripadvisor reviews data science Data Science Academy Data Science Bootcamp Data science jobs Data Science Reviews Data Scientist Data Scientist Jobs data visualization database Deep Learning Demo Day Discount disney dplyr drug data e-commerce economy employee employee burnout employer networking environment feature engineering Finance Financial Data Science fitness studio Flask flight delay gbm Get Hired ggplot2 googleVis H20 Hadoop hallmark holiday movie happiness healthcare frauds higgs boson Hiring hiring partner events Hiring Partners hotels housing housing data housing predictions housing price hy-vee Income Industry Experts Injuries Instructor Blog Instructor Interview insurance italki Job Job Placement Jobs Jon Krohn JP Morgan Chase Kaggle Kickstarter las vegas airport lasso regression Lead Data Scienctist Lead Data Scientist leaflet league linear regression Logistic Regression machine learning Maps market matplotlib Medical Research Meet the team meetup methal health miami beach movie music Napoli NBA netflix Networking neural network Neural networks New Courses NHL nlp NYC NYC Data Science nyc data science academy NYC Open Data nyc property NYCDSA NYCDSA Alumni Online Online Bootcamp Online Training Open Data painter pandas Part-time performance phoenix pollutants Portfolio Development precision measurement prediction Prework Programming public safety PwC python Python Data Analysis python machine learning python scrapy python web scraping python webscraping Python Workshop R R Data Analysis R language R Programming R Shiny r studio R Visualization R Workshop R-bloggers random forest Ranking recommendation recommendation system regression Remote remote data science bootcamp Scrapy scrapy visualization seaborn seafood type Selenium sentiment analysis sentiment classification Shiny Shiny Dashboard Spark Special Special Summer Sports statistics streaming Student Interview Student Showcase SVM Switchup Tableau teachers team team performance TensorFlow Testimonial tf-idf Top Data Science Bootcamp Top manufacturing companies Transfers tweets twitter videos visualization wallstreet wallstreetbets web scraping Weekend Course What to expect whiskey whiskeyadvocate wildfire word cloud word2vec XGBoost yelp youtube trending ZORI