Scraping IMDB for Insights

Avatar
Posted on Feb 17, 2020

Introduction

A daunting task in today’s day and age is to discover what are some of the best movies that are currently out there? What are people saying about these movies and what is that makes these movies good? This is a common question I find that I’m always asking and to alleviate some of the hindrance, I tend to check out the internet and search what are some good movies (surprise surprise). One website that constantly appears is the Internet Movie Database, or IMDB for short. It's a database that has been around since 1990 and it's a place where information about movies, television programs/shows and much more. One key feature I think is the most important their ratings are stored. I frequently like to watch movies as a pastime and I always wondered what makes a movie good? What is it about particular movies that makes it attractive for users such as myself? To gain insight on these questions, I decided to scrape the IMDB top rated movies to help me understand more about these questions.

Research Questions for Motivation

The reason I chose to scrape IMDB for my web scraping project is that it’s a place where many users go to determine whether a particular movie may be worthwhile their time to view, and similarly what others are saying about it. As of January 2020, there is a reported 83 million users on this website, so it’s pretty big deal. In addition to what was stated earlier, ultimately, I wanted to see what correlations can be drawn that exist among the data I collect such as genre vs. gross, what trends can be drawn based on the data given, and what might be some of the favorable genres, and would length of the movie have an effect on how much the user likes the movie.

Data Extraction

The several categories I decided to scrape are:

  1. Title of the movie
  2. Genre
  3. Gross
  4. Reviews
  5. Length of the movies (in minutes)
  6. Rating
  7. Score
  8. Year

I decided these are the most important pieces of information to scrape since they contain a lot of information, and moreover the information I need in order to perform my analysis. Most of the data was prepared in a fashion where it was accessible. From this point, I decided to perform some data cleaning to make the data even more accessible for my own use.

Gross vs. Gender

Initially, I had a hypothesis that perhaps certain genres might do significantly by far when compared to others, and that movies pertaining to these genres that do great generate higher profits simply because they're favorable. However, this is not the case. I took the average gross of all the genres available and found that Action/Adventure/Drama/Sci-fi is a genre many users enjoy. On the other side of the same coin, I found that other related genres/ genres not related to Action or Adventure generally do good too. Since this information is not readily available at all on the IMDB website, I found this to pretty cool to visualize on average what genres earned on average.

Movie Ratings Over Time

It is worth mentioning that the first color movie produced was in the year 1918. I wanted to see as time went on, how dramatic were the increases in ratings? When did the most ratings begin to occur? I saw that starting from 90's that's when popularity in movie heavily increased and inevitably more reviews were output. This makes sense in the context of this website since IMDB was established in 1990.

Gross vs Time

Here, I wanted to see how much gross has been generated over time. Since popularity of movies increased dramatically roughly around 1990, it makes sense to see the data displayed in a fashion as such where there's an increase over time among the different genres.

Rating vs Genre

I wanted to visualize what the average rating among all the different genres are. Generally, with the exception of a few, most movies receive a IMDB rating of 7.5+ out of 10.

Mins vs. Genre

I wanted to visualize how the length of time, in minutes, correlates to the different genres listed. For instance, we can see that most of the movies lie anywhere from 100 - 160 (1 hour 40 minutes - 2 hours 40 minutes respectively). Initially, I thought that on average certain genres may or may not be favorable due to fact it may be too long/short. Let's say for instance, one might think that western movies may last longer in terms of minutes compared to other genres but it turns out that it generally is about the same length of time as most of the other genres available.

Word Cloud

There were a lot of words used in the data I scraped from the ratings. Here are the most common words that were used among all the different movies.

Future Work

I would like to be able to scrape director information and determine what directors make the most successful movies. I would like to also like to incorporate whether different genders play a role on the ratings themselves. I would also like a way to discover sarcasm which may need further analysis such as a sentiment analysis. Finally, since there was limited time in doing this project, I plan to incorporate newly learned methods such as Machine Learning into this project.

About Author

Avatar

Abzal Bacchus

Hello! I am Abzal Bacchus and I am a Data Scientist Fellow at the New York City Data Science Academy. Feel free to contact me: www.linkedin.com/in/abzalb
View all posts by Abzal Bacchus >

Related Articles

Leave a Comment

No comments found.

View Posts by Categories


Our Recent Popular Posts


View Posts by Tags

#python #trainwithnycdsa 2019 airbnb Alex Baransky alumni Alumni Interview Alumni Reviews Alumni Spotlight alumni story Alumnus API Application artist aws beautiful soup Best Bootcamp Best Data Science 2019 Best Data Science Bootcamp Best Data Science Bootcamp 2020 Best Ranked Big Data Book Launch Book-Signing bootcamp Bootcamp Alumni Bootcamp Prep Bundles California Cancer Research capstone Career Career Day citibike clustering Coding Course Demo Course Report D3.js data Data Analyst data science Data Science Academy Data Science Bootcamp Data science jobs Data Science Reviews Data Scientist Data Scientist Jobs data visualization Deep Learning Demo Day Discount dplyr employer networking feature engineering Finance Financial Data Science Flask gbm Get Hired ggplot2 googleVis Hadoop higgs boson Hiring hiring partner events Hiring Partners Industry Experts Instructor Blog Instructor Interview Job Job Placement Jobs Jon Krohn JP Morgan Chase Kaggle Kickstarter lasso regression Lead Data Scienctist Lead Data Scientist leaflet linear regression Logistic Regression machine learning Maps matplotlib Medical Research Meet the team meetup Networking neural network Neural networks New Courses nlp NYC NYC Data Science nyc data science academy NYC Open Data NYCDSA NYCDSA Alumni Online Online Bootcamp Online Training Open Data painter pandas Part-time Portfolio Development prediction Prework Programming PwC python python machine learning python scrapy python web scraping python webscraping Python Workshop R R language R Programming R Shiny r studio R Visualization R Workshop R-bloggers random forest Ranking recommendation recommendation system regression Remote remote data science bootcamp Scrapy scrapy visualization seaborn Selenium sentiment analysis Shiny Shiny Dashboard Spark Special Special Summer Sports statistics streaming Student Interview Student Showcase SVM Switchup Tableau team TensorFlow Testimonial tf-idf Top Data Science Bootcamp twitter visualization web scraping Weekend Course What to expect word cloud word2vec XGBoost yelp