Data: The Best Movies? Scraping IMDB for Insights
The skills I demoed here can be learned through taking Data Science with Machine Learning bootcamp with NYC Data Science Academy.
LinkedIn | GitHub | Email | Data | Web App
Introduction
A daunting task in today’s day and age is to discover what are some of the best movies that are currently out there? What are people saying about these movies and what is that makes these movies good? This is a common question I find that I’m always asking and to alleviate some of the hindrance, I tend to check out the internet and search what are some good movies (surprise surprise). One website that constantly appears is the Internet Movie Data Base, or IMDB for short.
It's a database that has been around since 1990 and it's a place where information about movies, television programs/shows and much more. One key feature I think is the most important their ratings are stored. I frequently like to watch movies as a pastime and I always wondered what makes a movie good? What is it about particular movies that makes it attractive for users such as myself? To gain insight on these questions, I decided to scrape the IMDB top rated movies to help me understand more about these questions.
Research Questions for Motivation
The reason I chose to scrape IMDB for my web scraping project is that it’s a place where many users go to determine whether a particular movie may be worthwhile their time to view, and similarly what others are saying about it. As of January 2020, there is a reported 83 million users on this website, so it’s pretty big deal.
In addition to what was stated earlier, ultimately, I wanted to see what correlations can be drawn that exist among the data I collect such as genre vs. gross, what trends can be drawn based on the data given, and what might be some of the favorable genres, and would length of the movie have an effect on how much the user likes the movie.
Data Extraction
The several categories I decided to scrape are:
- Title of the movie
- Genre
- Gross
- Reviews
- Length of the movies (in minutes)
- Rating
- Score
- Year
I decided these are the most important pieces of information to scrape since they contain a lot of information, and moreover the information I need in order to perform my analysis. Most of the data was prepared in a fashion where it was accessible. From this point, I decided to perform some data cleaning to make the data even more accessible for my own use.
Gross vs. Gender
Initially, I had a hypothesis that perhaps certain genres might do significantly by far when compared to others, and that movies pertaining to these genres that do great generate higher profits simply because they're favorable. However, this is not the case.
I took the average gross of all the genres available and found that Action/Adventure/Drama/Sci-fi is a genre many users enjoy. On the other side of the same coin, I found that other related genres/ genres not related to Action or Adventure generally do good too. Since this information is not readily available at all on the IMDB website, I found this to pretty cool to visualize on average what genres earned on average.
Movie Ratings Over Time
It is worth mentioning that the first color movie produced was in the year 1918. I wanted to see as time went on, how dramatic were the increases in ratings? When did the most ratings begin to occur? I saw that starting from 90's that's when popularity in movie heavily increased and inevitably more reviews were output. This makes sense in the context of this website since IMDB was established in 1990.
Gross vs Time
Here, I wanted to see how much gross has been generated over time. Since popularity of movies increased dramatically roughly around 1990, it makes sense to see the data displayed in a fashion as such where there's an increase over time among the different genres.
Rating vs Genre
I wanted to visualize what the average rating among all the different genres are. Generally, with the exception of a few, most movies receive a IMDB rating of 7.5+ out of 10.
Mins vs. Genre
I wanted to visualize how the length of time, in minutes, correlates to the different genres listed. For instance, we can see that most of the movies lie anywhere from 100 - 160 (1 hour 40 minutes - 2 hours 40 minutes respectively). Initially, I thought that on average certain genres may or may not be favorable due to fact it may be too long/short. Let's say for instance, one might think that western movies may last longer in terms of minutes compared to other genres but it turns out that it generally is about the same length of time as most of the other genres available.
Word Cloud
There were a lot of words used in the data I scraped from the ratings. Here are the most common words that were used among all the different movies.
Future Work
I would like to be able to scrape director information and determine what directors make the most successful movies. I would like to also like to incorporate whether different genders play a role on the ratings themselves. I would also like a way to discover sarcasm which may need further analysis such as a sentiment analysis. Finally, since there was limited time in doing this project, I plan to incorporate newly learned methods such as Machine Learning into this project.
About Author
