Scraping TV Data From Metacritic

Posted on Aug 10, 2017
The skills the author demoed here can be learned through taking Data Science with Machine Learning bootcamp with NYC Data Science Academy.

Introduction

I don't discriminate when it comes to television - I love everything from bad reality shows to prestige dramas.  But over the past few years, I've felt overwhelmed by the sheer number of shows being heralded as "must-see TV". I know I should watch Atlanta and Mr. Robot, but who has the time? I'm not alone here - data from the Financial Times, in an article entitled "Watch it While it Lasts: Our Golden Age of Television," wrote,  "We used to complain that there was never anything good on TV. Now we complain about the opposite."

In addition, new companies like Netflix are spending billions of dollars each year on original content, which may contribute to the feeling that there is constantly something new to watch. 

The Data

I scraped TV data from Metacritic, a website that aggregates reviews of media products. Most notably, it aggregates both qualitative and quantitative critic reviews into a single "Metascore". It also contains detailed information about each product, though the TV section was not launched until 2005. Metacritic does have data on TV shows from prior years, but their listings are incomplete. I scraped several layers of data, including:

  1. Overall TV Rankings: From Metacritic's main TV page, I scraped each show's overall rank, name, and the link to the individual show page.
  2. TV Show Details: I scraped details for each individual television show, as well as a link to another “Details and Credits” page.
  3. TV Show Details, Continued: I scraped additional details for each television show.

From this process, I obtained 13 variables for over 2,000 shows, including Critic Rating, User Rating, Rating Description, Network, Season and Series Premiere Dates, and Genre(s).

Research Questions

For my analysis, I wanted to investigate my hypotheses about the influx of quality, must-see TV in recent years. I attempted to answer the following questions:

  1. Are there really more “good” shows than ever before?
  2. Are subscription/online networks displacing traditional channels?
  3. Should we trust the critics – do critics and users usually agree?
  4. What should I actually watch next?

Data Insights

Critics Vs. User Ratings

First, I wanted to explore the relationship between Critic and User Ratings. Metacritic provides a Critic Rating for TV shows with at least seven critic reviews, as well as an average User Rating for shows with at least four user reviews. I graphed both and colored the points based on the Rating Description, which is a qualitative variable based on the quantitative Critic Rating.Scraping TV Data From MetacriticBased on the plot, the relationship between Critic Rating and User Rating seemed fairly linear, and sentiments seem to converge as the Critic Rating improves. For shows that critics rated "Average" or "Unfavorable", user opinions vary widely. Further analysis revealed a correlation of 53%, indicating a significant, moderate linear relationship between Critic and User Rating.

Distribution

I also tested my theory of consensus among higher rated shows by conducting several two-sample t-tests on Critic and User Ratings for each Rating Description group, but I found that the population means were different across all four groups. We can look at the distribution of Critic and User Ratings as well:Scraping TV Data From Metacritic

Critics are "harsher" than users - the median rating given by critics is 65, while the median rating submitted by users is 74. The distribution of User Ratings is also narrower and is concentrated towards the higher end of the scale.

Number of Seasons

Critic and User Ratings may be correlated, but I wanted to know whether they matter for a show's success. By graphing the total number of seasons for each show, I found that almost half of the shows in the data set lasted for only one season. I also checked to see if there was any correlation between Critic and User Ratings and a show's total seasons, but there was a fairly weak linear relationship in both cases. Most shows only last for one or two seasons, regardless of how reviled or beloved they are.

Scraping TV Data From Metacritic

The Era of Peak TV?

Next, I wanted to know whether there really has been an increase in the quantity and quality of TV over the last few years. I graphed the number of shows on TV by year, as well as the number of highly rated shows (Critic Rating > 60).Looking specifically at the years between 2005 (when Metacritic launched its TV section) and 2016 (since 2017 data is incomplete), it is clear that the number of shows, as well as the number of "good" shows, has increased. However, looking at the average Critic and User Rating by year does not show such a clear trend.Critic and User Ratings have fluctuated in recent years, even as the total number of highly rated shows has increased. We can conclude that shows are not necessarily getting better - we may just have more shows (and therefore more "good" shows)!

Data Trends in Television Networks

The increase in show volume may be related to the many networks that have entered the TV ring over the last few years (e.g., Netflix, Hulu). We can see this increase by looking at the number of networks with at least one show on the air by year.

The number of TV networks essentially doubled between 2005 and 2015 - no wonder we feel overwhelmed by our options! Access to many of these networks may require additional cable packages or online subscriptions beyond the basic channels, so how do we decide where to spend our time and money? To explore this, I looked at the top 10 and bottom 10 networks based on average Critic Rating. Of all networks with at least 15 existing shows, PBS and HBO have the most critical acclaim. A&E and NBC have the least.

Bubble Chart

To incorporate User Ratings and show volume into my analysis, I created a bubble chart. In the image below, the size of the bubbles represents the number of shows each network has produced. Only networks with at least 15 existing shows are included.


This chart shows that HBO is essentially the "model" network - it has produced a large number of shows that are beloved by both users and critics. Other subscription networks, such as Netflix and Showtime, also stack up well against the competition. On the other hand, traditional basic cable networks are lagging behind in terms of creating "prestige" TV. This reflects the growing industry trend of cable package cancellation in favor of streaming services and other online subscriptions.

What Should I Watch?

Finally, I used the data I collected to cut through all of the "buzz" and determine what I should actually watch next. I averaged Critic and User Ratings across all shows with at least three seasons, and selected the top 10 Dramas and Comedies. Hopefully this will help any of you out there looking for a new show to binge watch!

Conclusions

Based on the data from Metacritic, it is difficult to say whether we are really experiencing a "golden age of television". There are more shows being produced than ever before (and therefore more good shows), though the average quality of shows may not be improving. The influx of new networks has likely contributed to this increase, though many of the new "prestige" shows seem to be coming from platforms like HBO and Netflix, not traditional cable. Unless cable networks start producing "must-see TV" to improve demand, they will probably continue to see a drop in subscription rates among their customers.

 

Link to my Github.

About Author

Julia Goldstein

Julia has over five years of experience delivering business insight through data analysis and visualization. As an analytics and management consultant, she was responsible for managing projects, identifying solutions, and developing support among senior-level stakeholders. Moving forward, Julia...
View all posts by Julia Goldstein >

Related Articles

Leave a Comment

No comments found.

View Posts by Categories


Our Recent Popular Posts


View Posts by Tags

#python #trainwithnycdsa 2019 2020 Revenue 3-points agriculture air quality airbnb airline alcohol Alex Baransky algorithm alumni Alumni Interview Alumni Reviews Alumni Spotlight alumni story Alumnus ames dataset ames housing dataset apartment rent API Application artist aws bank loans beautiful soup Best Bootcamp Best Data Science 2019 Best Data Science Bootcamp Best Data Science Bootcamp 2020 Best Ranked Big Data Book Launch Book-Signing bootcamp Bootcamp Alumni Bootcamp Prep boston safety Bundles cake recipe California Cancer Research capstone car price Career Career Day citibike classic cars classpass clustering Coding Course Demo Course Report covid 19 credit credit card crime frequency crops D3.js data data analysis Data Analyst data analytics data for tripadvisor reviews data science Data Science Academy Data Science Bootcamp Data science jobs Data Science Reviews Data Scientist Data Scientist Jobs data visualization database Deep Learning Demo Day Discount disney dplyr drug data e-commerce economy employee employee burnout employer networking environment feature engineering Finance Financial Data Science fitness studio Flask flight delay gbm Get Hired ggplot2 googleVis H20 Hadoop hallmark holiday movie happiness healthcare frauds higgs boson Hiring hiring partner events Hiring Partners hotels housing housing data housing predictions housing price hy-vee Income Industry Experts Injuries Instructor Blog Instructor Interview insurance italki Job Job Placement Jobs Jon Krohn JP Morgan Chase Kaggle Kickstarter las vegas airport lasso regression Lead Data Scienctist Lead Data Scientist leaflet league linear regression Logistic Regression machine learning Maps market matplotlib Medical Research Meet the team meetup methal health miami beach movie music Napoli NBA netflix Networking neural network Neural networks New Courses NHL nlp NYC NYC Data Science nyc data science academy NYC Open Data nyc property NYCDSA NYCDSA Alumni Online Online Bootcamp Online Training Open Data painter pandas Part-time performance phoenix pollutants Portfolio Development precision measurement prediction Prework Programming public safety PwC python Python Data Analysis python machine learning python scrapy python web scraping python webscraping Python Workshop R R Data Analysis R language R Programming R Shiny r studio R Visualization R Workshop R-bloggers random forest Ranking recommendation recommendation system regression Remote remote data science bootcamp Scrapy scrapy visualization seaborn seafood type Selenium sentiment analysis sentiment classification Shiny Shiny Dashboard Spark Special Special Summer Sports statistics streaming Student Interview Student Showcase SVM Switchup Tableau teachers team team performance TensorFlow Testimonial tf-idf Top Data Science Bootcamp Top manufacturing companies Transfers tweets twitter videos visualization wallstreet wallstreetbets web scraping Weekend Course What to expect whiskey whiskeyadvocate wildfire word cloud word2vec XGBoost yelp youtube trending ZORI