Data: The Best Movies? Scraping IMDB for Insights

Posted on Feb 17, 2020
The skills I demoed here can be learned through taking Data Science with Machine Learning bootcamp with NYC Data Science Academy.

LinkedIn |Β  GitHub |Β  Email | Data | Web App

Introduction

A daunting task in today’s day and age is to discover what are some of the best movies that are currently out there? What are people saying about these movies and what is that makes these movies good? This is a common question I find that I’m always asking and to alleviate some of the hindrance, I tend to check out the internet and search what are some good movies (surprise surprise). One website that constantly appears is the Internet Movie Data Base, or IMDB for short.

It's a database that has been around since 1990 and it's a place where information about movies, television programs/shows and much more. One key feature I think is the most important their ratings are stored. I frequently like to watch movies as a pastime and I always wondered what makes a movie good? What is it about particular movies that makes it attractive for users such as myself? To gain insight on these questions, I decided to scrape the IMDB top rated movies to help me understand more about these questions.

 

Research Questions for Motivation

The reason I chose to scrape IMDB for my web scraping project is that it’s a place where many users go to determine whether a particular movie may be worthwhile their time to view, and similarly what others are saying about it. As of January 2020, there is a reported 83 million users on this website, so it’s pretty big deal.

In addition to what was stated earlier, ultimately, I wanted to see what correlations can be drawn that exist among the data I collect such as genre vs. gross, what trends can be drawn based on the data given, and what might be some of the favorable genres, and would length of the movie have an effect on how much the user likes the movie.

Data Extraction

The several categories I decided to scrape are:

  1. Title of the movie
  2. Genre
  3. Gross
  4. Reviews
  5. Length of the movies (in minutes)
  6. Rating
  7. Score
  8. Year

 

I decided these are the most important pieces of information to scrape since they contain a lot of information, and moreover the information I need in order to perform my analysis. Most of the data was prepared in a fashion where it was accessible. From this point, I decided to perform some data cleaning to make the data even more accessible for my own use.

Gross vs. Gender

Initially, I had a hypothesis that perhaps certain genres might do significantly by far when compared to others, and that movies pertaining to these genres that do great generate higher profits simply because they're favorable. However, this is not the case.

I took the average gross of all the genres available and found that Action/Adventure/Drama/Sci-fi is a genre many users enjoy. On the other side of the same coin, I found that other related genres/ genres not related to Action or Adventure generally do good too. Since this information is not readily available at all on the IMDB website, I found this to pretty cool to visualize on average what genres earned on average.

Movie Ratings Over Time

It is worth mentioning that the first color movie produced was in the year 1918. I wanted to see as time went on, how dramatic were the increases in ratings? When did the most ratings begin to occur? I saw that starting from 90's that's when popularity in movie heavily increased and inevitably more reviews were output. This makes sense in the context of this website since IMDB was established in 1990.

Gross vs Time

Here, I wanted to see how much gross has been generated over time. Since popularity of movies increased dramatically roughly around 1990, it makes sense to see the data displayed in a fashion as such where there's an increase over time among the different genres.

Rating vs Genre

I wanted to visualize what the average rating among all the different genres are. Generally, with the exception of a few, most movies receive a IMDB rating of 7.5+ out of 10.

Mins vs. Genre

 

I wanted to visualize how the length of time, in minutes, correlates to the different genres listed. For instance, we can see that most of the movies lie anywhere from 100 - 160 (1 hour 40 minutes - 2 hours 40 minutes respectively). Initially, I thought that on average certain genres may or may not be favorable due to fact it may be too long/short. Let's say for instance, one might think that western movies may last longer in terms of minutes compared to other genres but it turns out that it generally is about the same length of time as most of the other genres available.

Word Cloud

There were a lot of words used in the data I scraped from the ratings. Here are the most common words that were used among all the different movies.

Future Work

I would like to be able to scrape director information and determine what directors make the most successful movies. I would like to also like to incorporate whether different genders play a role on the ratings themselves. I would also like a way to discover sarcasm which may need further analysis such as a sentiment analysis. Finally, since there was limited time in doing this project, I plan to incorporate newly learned methods such as Machine Learning into this project.

About Author

Abzal Bacchus

Hello! I am Abzal Bacchus and I am a Data Scientist Fellow at the New York City Data Science Academy. Feel free to contact me: www.linkedin.com/in/abzalb
View all posts by Abzal Bacchus >

Related Articles

Leave a Comment

No comments found.

View Posts by Categories


Our Recent Popular Posts


View Posts by Tags

#python #trainwithnycdsa 2019 2020 Revenue 3-points agriculture air quality airbnb airline alcohol Alex Baransky algorithm alumni Alumni Interview Alumni Reviews Alumni Spotlight alumni story Alumnus ames dataset ames housing dataset apartment rent API Application artist aws bank loans beautiful soup Best Bootcamp Best Data Science 2019 Best Data Science Bootcamp Best Data Science Bootcamp 2020 Best Ranked Big Data Book Launch Book-Signing bootcamp Bootcamp Alumni Bootcamp Prep boston safety Bundles cake recipe California Cancer Research capstone car price Career Career Day citibike classic cars classpass clustering Coding Course Demo Course Report covid 19 credit credit card crime frequency crops D3.js data data analysis Data Analyst data analytics data for tripadvisor reviews data science Data Science Academy Data Science Bootcamp Data science jobs Data Science Reviews Data Scientist Data Scientist Jobs data visualization database Deep Learning Demo Day Discount disney dplyr drug data e-commerce economy employee employee burnout employer networking environment feature engineering Finance Financial Data Science fitness studio Flask flight delay gbm Get Hired ggplot2 googleVis H20 Hadoop hallmark holiday movie happiness healthcare frauds higgs boson Hiring hiring partner events Hiring Partners hotels housing housing data housing predictions housing price hy-vee Income Industry Experts Injuries Instructor Blog Instructor Interview insurance italki Job Job Placement Jobs Jon Krohn JP Morgan Chase Kaggle Kickstarter las vegas airport lasso regression Lead Data Scienctist Lead Data Scientist leaflet league linear regression Logistic Regression machine learning Maps market matplotlib Medical Research Meet the team meetup methal health miami beach movie music Napoli NBA netflix Networking neural network Neural networks New Courses NHL nlp NYC NYC Data Science nyc data science academy NYC Open Data nyc property NYCDSA NYCDSA Alumni Online Online Bootcamp Online Training Open Data painter pandas Part-time performance phoenix pollutants Portfolio Development precision measurement prediction Prework Programming public safety PwC python Python Data Analysis python machine learning python scrapy python web scraping python webscraping Python Workshop R R Data Analysis R language R Programming R Shiny r studio R Visualization R Workshop R-bloggers random forest Ranking recommendation recommendation system regression Remote remote data science bootcamp Scrapy scrapy visualization seaborn seafood type Selenium sentiment analysis sentiment classification Shiny Shiny Dashboard Spark Special Special Summer Sports statistics streaming Student Interview Student Showcase SVM Switchup Tableau teachers team team performance TensorFlow Testimonial tf-idf Top Data Science Bootcamp Top manufacturing companies Transfers tweets twitter videos visualization wallstreet wallstreetbets web scraping Weekend Course What to expect whiskey whiskeyadvocate wildfire word cloud word2vec XGBoost yelp youtube trending ZORI