NLP analysis -- Movie Reviews from Rotten Tomatos

Avatar
Posted on Feb 21, 2017

Contributed by Tingting Chang. She is currently in the NYC Data Science Academy 12 week full time Data Science Bootcamp program taking place between Jan 9th to March 31th, 2017. This post is based on her second class project - Web Scraping (due on the 6th week of the program).

About Tingting Chang: GitHub | LinkedIn

 

 

Introduction

 

It has became super common for people looking at reviews before they go to the theater to see a movie. Some website provides movie ranking, rating and reviews for coming movies as well as the old movies. Rotten tomato is one of famous one and it grabs all the reviews from other movie critics websites.  However, it brings a question: are these ranking, rating, critics true? Do they have any connection between them? I scraped all the movie reviews, rankings, ratings for the top 100 movies in 2016. After using NLTK package to tokenize data and remove all the stop words, along with punctuations, I was left with about 13467 reviews with only words for all 100 movies. From there, I used sentiment analysis, ft-idf, topic model(LDA) to analysis the reviews.

This is how my data looks like:

Screen Shot 2017-02-20 at 7.40.54 PM

ranking, rating for all top 100 movies in 2016

 

Screen Shot 2017-02-20 at 7.41.38 PM

all the movie reviews data for 100 movies

 

The bar plotting for every movie's the number of reviews.   I picked the 20 movies with the most reviews count and other 20 movies with the least number of reviews.

Screen Shot 2017-02-12 at 6.59.01 PM

 

Screen Shot 2017-02-20 at 7.48.51 PM

Screen Shot 2017-02-20 at 7.49.09 PM

20 movies with the least movie reviews count

 

I used sentiment analysis function build in NLTK package to calculate sentiment score for every review:

Screen Shot 2017-02-20 at 7.58.50 PM

sentiment score for every review

For example, the first review "one disney best movies past years" has score 0.6369 which is pretty positive. Another negative review "animal antics display movie also gently broaches human issues stereotyping prejudice racism something " has score -0.4588 due to some negative words "stereotyping prejudice racism".

I also computed the average sentiment score for every movie:

Screen Shot 2017-02-20 at 8.35.33 PM

average sentiment score for every movie in 2016

 

Then I use box plot try to find out the relationship between the movie rating and sentiment mean compound score.

Screen Shot 2017-02-12 at 9.33.55 PM

box plot for sentiment mean score and movie rating

But there is not any specific relationship between the sentiment mean score and movie rating for every movie except that the most of movies have positive sentiment score.

In addition, I used another box plot for movie ranking and sentiment mean compound score. Then I found out that rank 2.0 has a negative sentiment score which was pretty interesting. I went to website and try to find out why.

Screen Shot 2017-02-12 at 8.10.59 PM

box plot for sentiment mean score and movie ranking

Screen Shot 2017-02-20 at 8.45.10 PM

 

 

For this movie: Things to Come ,the Rotten Tomato gives 100% rating and ranking 11th. However, you can see from the website that it only has 73% audience scores. I did some researches and I learned that some company would hire some reviewers to write fake reviews for a movie. Moreover, the significant reviewers' reviews will have a larger effect on ranking movie. Nevertheless, their taste can not 100% represent audience's taste. Therefore, it will be normal if movie ranking, rating does not match the sentiment score from movie reviews.

 

Also, I tried some EDA analysis to find out whether there was any relationship between movie ranking, rating, review counts and sentiment scores. Unfortunately, there is not any obvious relationship between them.

Screen Shot 2017-02-12 at 9.35.18 PM

Screen Shot 2017-02-12 at 10.17.40 PMScreen Shot 2017-02-12 at 9.35.57 PM

 

Screen Shot 2017-02-12 at 9.36.51 PM

 

 

TF-IDF

 

TF, or Term Frequency, measures the count of words in a document. By using TF-IDF functions in NLTK packages, I gained words frequency for every words in the document:

Screen Shot 2017-02-20 at 9.11.41 PM

TF: words frequency for first two reviews

The next step of TF-IDF is the IDF or inverse document frequency. Targets the words only appear in certain documents instead of in all the documents, and give them higher frequency. Now we have got a weight for every tokened word in every document and we can determine the important words from the unimportant ones.

Screen Shot 2017-02-20 at 9.12.10 PM

IDF: inverse document frequency for every tokened word

 

 

Topic Model

 

Topic Modelling, as the name suggests, it is a process to derive hidden patterns exhibited by a text corpus. I built 9 topics for all the reviews and printed out top 20 frequent words for every topic:

topic_model_1

topic_model_2

Topic Model: top 20 words for 9 topics

 

For better visualization, I use WordCloud package in python to visualize the top 500 frequent words in every topic.

pic_topic_0

top 500 words for the 1st topic

pic_topic_1

top 500 words for the 2nd topic

pic_topic_2

top 500 words for the 3rd topic

pic_topic_3

top 500 words for the 4th topic

pic_topic_4

top 500 words for the 5th topic

pic_topic_5

top 500 words for the 6th topic

 

pic_topic_6

top 500 words for the 7th topic

 

pic_topic_7

top 500 words for the 8th topic

pic_topic_8

top 500 words for the 9th topic

From above plots, we can see the word "fan" shows up in three topics. I have not found out why this is happened.

 

 

About Author

Avatar

Tingting Chang

Tingting Chang got her master degree in Computer Science from the George Washington University. She is a self-starter and hardworking data scientist well equipped with data analytics skills to obtain actionable insights from massive datasets without losing sight...
View all posts by Tingting Chang >

Related Articles

Leave a Comment

Your email address will not be published. Required fields are marked *

Avatar
Lynda April 2, 2018
can I have the source code of how you scrapped the data ? and also for the topic modeling ?
Avatar
bath body works promo code February 28, 2017
I Just stopped by to say your post is fantastic. The clarity inside your post is simply magnificent and i can assume you’re an specialist on this topic. Fine together with your permission let me to grab your RSS feed to keep as much as date with forthcoming post. Thanks a million and please continue the enjoyable work.
Avatar
online school tips February 26, 2017
Nice post. I learn something totally new and challenging on websites I stumbleupon everyday. It's always useful to read articles from other writers and practice a little something from their web sites.
Avatar
educational sites February 26, 2017
Hi there, for all time i used to check website posts here in the early hours in the dawn, as i enjoy to gain knowledge of more and more.
Avatar
online learning February 26, 2017
Asking questions are actually good thing if you are not understanding anything entirely, except this paragraph presents good understanding even.

View Posts by Categories


Our Recent Popular Posts


View Posts by Tags