Hollywood's Sequel Problem

Posted on Nov 11, 2015

Contributed by Jake Lehrhoff. Jake took NYC Data Science Academy 12 week full time Data Science Bootcamp program between Sept 23 to Dec 18, 2015. The post was based on his third class project(due at 6th week of the program).

Header artwork courtesy of Fortune.com

Lately, it feels like a shocking number of "new releases" are either sequels or reboots. This holiday season's biggest box office draws are certain to be the conclusion to The Hunger Games trilogy and the highly anticipated, J.J. Abrams-helmed Star Wars Episode VII. Christopher Nolan's Dark Knight trilogy is but a recent memory, and yet suddenly Ben Affleck (Ben Affleck!) is poised to debut as our rebooted caped crusader in early 2016. Marvel has as many as 70 superhero movies planned for release from 2015 through 2028. There were eight American Pie movies. Eight.

So what is it about sequels? Why do we flock to them in droves?

Maybe we are hedging our bets on the tried-and-true (James Bond). In some cases, we are certainly too attached to the characters to pass up on a new installment (The Muppets). But what if the tried-and-true or our attachments fail us? Audiences and critics alike loved Peter Jackson's Lord of the Rings trilogy, but even inserting our favorite characters couldn't save The Hobbit from itself. (That isn't to say it faired poorly at the box office.)

The psychology behind our moviegoing choices may require more digging, but the data to explain our behavior is readily available. For the current study, daily box office receipts from the-number.com, ratings from rottentomatoes.com, and lifetime revenues from boxofficemojo.com were compiled to paint a picture of sequels today.

For a more hands-on look at the data, please check out the Shiny web application I developed to investigate how various movies compare in daily box office receipts and rankings, lifetime gross, and rating. Also, check out the "Franchise Fights" tab to pit some of your favorite franchises or movie collections against each other.

Additionally, all the code and data are available on my public github repo.

Before we get ahead of ourselves: what exactly is a sequel?

The definition is murkier than it may seem. Of course, we could call sequels movies that have a single, direct predecessor. But what about Avengers? It's more of a super-sequel, with three current film franchises funneling into a single mega-blockbuster. For the purposes of this study, it's a sequel.

Avengers may be a fringe case, but what about reboots? They are everywhereBatman Begins was the first of the Christopher Nolan's Dark Knight trilogy, but it's certainly no an original film. Such classification quickly gets cloudy. There was a Transformers animated movie in the 1980's--does that make Shia Labeouf's 2007 film a reboot? Well, yes. Same with the various Teenage Mutant Ninja Turtle movies (while we're on the subject of Michael Bay) and those Alvin and the Chipmunk movies. Reboots are simply unavoidable.

The Code

Various scraping functions were used for each of the aforementioned websites, each tailored to the idiosyncrasies of the source HTML. Search functions allow the HTML text content to be filtered down to the relevant data and pandas recompiles the data into clean, mutable tables.

Daily Box Office Receipts: the-numbers.com

The below function scrapes daily box office data for specific movies using Beautiful Soup, regular expressions, and pandas. Columns are retained for titles, rank, gross, total gross, days in theater, among other variables.

from bs4 import BeautifulSoup
import requests
import re
import pandas as pd

def scraper(list_):
    '''Scrape the-numbers.com for daily box office data.'''
    movieDF = pd.DataFrame()

    for url in list_:
        date = re.findall('daily.*', url)
        date = date[0][6:]
        date = re.sub('/','-', date)
        data = requests.get(url).text
        data = BeautifulSoup(data)
        aTag = data.find_all('a')
        movies = [tag.string for tag in aTag]
        movies = movies[52:82]
        data_sublists = [movies[i:i+2] for i in range(0, len(movies), 2)]
        dataframe = pd.DataFrame(data_sublists [0:], 
                                 columns = ['Title','Studio'])
        moviedata = data.find_all('td', {'class': "data"})
        moviedata = [tag.string for tag in moviedata]
        data_sublists2 = [moviedata[i:i+8] for i in range(0, 
                          len(moviedata), 8)]
        dataframe2 = pd.DataFrame(data_sublists2 [0:], 
        dataframe2['Title'] = dataframe['Title']
        dataframe2['Studio'] = dataframe['Studio']
        dataframe2['date'] = date
        dataframe2 = dataframe2[:15]
        movieDF = movieDF.append(dataframe2)
    return movieDF

url = 'http://www.the-numbers.com/box-office-chart/daily/'

Example code for scraping The Bourne Identity daily receipts

The code below specifies the dates of interest for the given movie, concatenating the strings with the box office chart url. After calling the scraper function, the code filters the resulting data frame for the movie in question.

'''Scraping Bourne Identity daily data'''
urlBourneI = []
for i in range(14,31):
    urlBourneI.append(url + '2002/06/' + str(i))
for i in range(1,15):
    if i < 10:
        urlBourneI.append(url + '2002/07/' + '0' + str(i))
        urlBourneI.append(url + '2002/07/' + str(i))

BourneI = scraper(urlBourneI)
BourneI = BourneI[(BourneI.Title == 'The Bourne Identity')]

Cleaning Daily Data and Plotting

With individual data frames for each of the desired movies, the following code combines the data into a single data frame and cleans the columns. The cleaning removes punctuation, converts numeric strings into integers, and converts date strings into mutable datetime.

frames = [SPR, BourneI, BourneS, Batman, BourneU, DarkK, GreenZ, 
          Descendants, HungerG, Avengers, DarkKR, SilverLP, Elysium, CatchF,          AmHust, Divergent, Fault, Guard, Interstellar, Mockingjay, 
          Insurgent, AvengersU, JurassicW, Martian, LOTR1, LOTR2, LOTR3, 
          Hobbit1, Hobbit2, Hobbit3, HP1, HP2, HP3, HP4, HP5, HP6, HP7a, 
          HP7b, Twilight1, Twilight2, Twilight3, Twilight4a, Twilight4b, 
          Transf1, Transf2, Transf3, Transf4, MadMax]
test = pd.concat(frames)
from datetime import datetime
def removePunc(x):
    return re.sub( "[^\\d]", "", x)
test['date'] = test['date'].apply(lambda x: datetime.strptime(x, "%Y-%m-%d"))
test.index = test['date']
test = test.sort(ascending=True,columns='date')
test['Gross'] = test['Gross'].apply(removePunc).apply(int)
test['TotalGross'] = test['TotalGross'].apply(removePunc).apply(int)

Example Plot Code

The following code uses Seaborn and matplotlib to create the plot seen at the top of the next section. By adjusting the title filter in the 6th line and adding "or" statements, similar plots can be created to inspect the growth of disparate films within the dataset.

import seaborn as sns
import matplotlib.pyplot as plt
%matplotlib inline
plt.rcParams['figure.figsize'] = 14, 10
sns.pointplot(x="Days", y="TotalGross", hue="Title", 
plt.ylabel('Total Gross (Million of $)', fontsize=16)
plt.xlabel('Days in Theaters', fontsize=16)
plt.xticks(rotation = 0, fontsize = 14)
plt.yticks(fontsize = 14)

The Sequel Effect

The Bourne Series

download-7Why start with a 13-year-old action movie, you ask? The Bourne series is one in a laundry list of movie franchises that depicts the "sequel effect." The graph to the left quantifies just how much a sequel can be worth. Successive Bourne movies command greater and greater box office receipts: the first month of the second film grossed a full $50 million more than that of the first, and the third film took home another $50 million on top of that. download-6


Can you blame Universal Studios for trying for a fourth, even if they couldn't convince Matt Damon to reprise the now-iconic roll? When Jeremy Renner took the helm for Legacy, lifetime gross dropped below that of even the first film. What happened here? Did audiences see this as a bastardization of the series and boycott? Do movies now have to abide by sitcom rules: at the end of the day, nothing ever changes? Or, was it simply not very good? It turns out, the simplest explanation was the correct one. We also have our first evidence that retaining a franchise's star is worth a lot of money.


What about movie franchises with lots of installments? There are eight Harry Potter movies. Could all eight movies possibly show the sequel effect? Nearly.download

The Code Part II: The Code Strikes Back

Scraping Reviews: Rotten Tomatoes

A similar scraping function locates review data, using a find function to identify where in the HTML the relevant data begins. Rather than requiring specific dates, this function can take a list of movie keywords. The benefit here is that inputting "Bourne" and "Hunger Games" as the keywordList argument returns the review scores for all the movies in both series--much more efficient, requiring minimal hard-coding. The boxofficemojo.com scraper functions similarly, returning lifetime gross data. Code for the scatterplot in the next section is also included.

def find(lst, predicate):
    return (i for i, j in enumerate(lst) if predicate(j)).next()

def reviewscrape(keywordList):

reviews = pd.DataFrame()
    url = 'http://www.rottentomatoes.com/search/?search='
    for keyword in keywordList:
        search = re.sub(' ','+',keyword)
        data = requests.get(url + search).text
        data = BeautifulSoup(data)
        Tag2 = data.find_all('class' == ['tMeterScore', 
                                         'unstyled articleLink'])
        movies1 = [tag.string for tag in Tag2]
        movies1 = [x for x in movies1 if x is not None]
        movies1 = movies1[300:]
        index = find(movies1, lambda x: '%' in x)
        movies1 = movies1[index:]
        sublist = [movies1[i:i+6] for i in range(0, len(movies1), 6)]
        dataframe = pd.DataFrame(sublist [0:], 
                                 columns = ['Rating','Title','','','',''])
        dataframe = dataframe[['Rating','Title']][0:6]
        reviews = reviews.append(dataframe)
    return reviews

sns.lmplot(x="Rating", y="Lifetime Gross", hue = "Title", fit_reg = False, 
                      )].sort(ascending=False), scatter_kws={"s": 100}
                      legend = False) 
plt.ylabel('Total Gross (Millions of $)', fontsize=16)
plt.xlabel('Rotten Tomatoes Rating', fontsize=16)
plt.xticks(rotation = 0, fontsize = 14)
plt.yticks(fontsize = 14)
plt.legend(loc='upper right')


CUljWgAAAAASUVORK5CYII=As strong as the sequel effect is, we've learned from The Bourne Legacy that not all installments in a series are created equal, and audiences may not open their wallets if their hero gets replaced and the quality drops substantially. But what happens if a hero is replaced but the quality was never there in the first place? Look no further than Tranformers: Age of Extinction. Without a rise in quality or the stars of the original, Mark Whalberg's Transformers brought home over $60 million less than the 2007 reboot.   Af2RY8j1P01NAAAAAElFTkSuQmCC



But reviews alone can't necessarily slow the momentum of a film series. None of the Twilight films had a Rotten Tomatoes rating above 50 (and two were below 30), and yet after the original, the four successive movies all raked in between $280 and $300 million. The equation for longevity is coming into focus: as long as you have a fanbase, they'll keep coming as long as you don't replace their stars.

The Code Part III: Return of the Code

Manipulating Yearly Top Ten Lists

The following code takes a data frame of the top 10 films of each year for the last 20 years (scraped from boxofficemojo.com) and reorganizes the data into categories depicting the yearly percentages and average gross of original films and sequels. The code for the pointplot in the next section can also be found.

grp = totals.groupby(('Year', 'Sequel'))
yearly = pd.DataFrame(grp['Sequel'].count())
yearly['Percent'] = yearly['Sequel']/10
yearly = yearly.drop('Sequel',1)
yearly.reset_index(inplace = True)

yearlygross = pd.DataFrame(grp['Total Gross'].mean())

top10s = pd.merge(yearly, yearlygross)

sns.pointplot(x = 'Year', y = 'Percent', 
              data = top10merge[top10merge['Sequel']=='Y'], 
                                fit_reg=False, scatter_kws={"s": 100},
                                legend = False) 
plt.ylabel('Percent of top-ten films', fontsize=16)
plt.xlabel('Year', fontsize=16)
plt.title('Prevalence of sequels in top-ten grossing films, 1996-2015', fontsize = 18)
plt.xticks(rotation = 30, fontsize = 14)
plt.yticks(fontsize = 14)

Conclusions: A New Hope

Is the sequel trend here to stay?

Thdownload-2e evidence says "yes," emphatically so. 64% of the last decade's yearly top-ten grossing movies were sequels, and those sequels on average brought home $33 million more than the originals. At the top of the charts it's even worse: nine of the top ten grossing films of the last decade were sequels, and 19 of the top 25.

These numbers may not be startling, but consider them in light of the previous decade: from 1996-2005, sequels comprised 37% of yearly top-ten grossing films. But even when sequels comprised less of the market, they were worth, on average, $23 million more than their original counterparts. With such a modest boost, its not surprising that sequels and reboots took over the market.


download-3Original ideas can still dominate at the box office, even original ideas written by software engineers who double as casual astrophysicists. The adaptation of Andy Weir's The Martian stood at the top of the box office two day's longer than Joss Whedon's first Avengers movie, and nine days longer than the sequel. The takeaway here is that original movies can be a boon for studios, particularly when Matt Damon is the star.



Cutting Hollywood Some Slack

We've learned a few things:

  • Sequels make more money.
  • Sequels that willfully replace lead actors are both inferior and less lucrative.
  • Poorly reviewed sequels will even make a lot of money.
  • The sequel trend has momentum.

But before we excoriate Hollywood for undermining creativity to cushion their bottom line, there's another way to tell that story. With predictable revenue streams and tentpole marketing, studios have room for lower-margin and higher-risk projects. They can gamble on more fringe concepts, be it in search of the next great franchise or the unusually creative--the avant garde beauty of WALL-E, the fantastical imagery of Pan's Labyrinth, or the bizarre existentialism of Being John Malkovich. Unfortunately, this theory--though compelling--is unfounded, and I'm no the only one who's worried. The Atlantic, VarietyVogueHuffPo, E!, and Mashable have all weighed in. (Forbes has moved on. For them it's all about the battle between sequels and reboots.)

Future Directions

This study has been a relatively surface-level investigation into the movie industry. Predicting Hollywood's boons and flops is a big business best served with heaps of data and a fine toothed comb. Mining Twitter for chatter on new releases can accurately predict box office revenues, but at that point, the investment has long since been made. By combining genre, proposed budget, actors, directors, and writers, an algorithm may be formed to help understand expected revenue for any proposed release a la baseball's "Moneyball." An important metric, as we've learned here, would be the "novelty" status: is a film a sequel, a reboot, an adaptation, or original? Hopefully all this adds up to not just mega-blockbusters but the success of the new and innovative.

Beyond all the data science that may go into building the next great film, we can all rest assured that the sequel and reboot trend has brought us at least one true gift: in 2016 we all get to enjoy Paul Feig's all-women Ghostbusters reboot.

About Author

Jake Lehrhoff

Jake Lehrhoff is a man of many hats. Currently, he is an analyst at Spotify, helping to improve an already amazing product. Previously, he spent six years teaching middle school English and chairing the department at a school...
View all posts by Jake Lehrhoff >

Related Articles

Leave a Comment

precious metals investing April 3, 2017
If you are going for finest contents like me, simply pay a visit this web site daily as it presents feature contents, thanks
شركة تطوير January 14, 2016
If you have the tendency to horde things, you will have to learn let go. Many mothers today work outside the home in addition to raising children, and sometimes housework suffers as a result. These mops can be a bit pricey initially, but over time will add up to hundreds of dollars in savings on floor products, especially if you have many types of flooring.

View Posts by Categories

Our Recent Popular Posts

View Posts by Tags

#python #trainwithnycdsa 2019 2020 Revenue 3-points agriculture air quality airbnb airline alcohol Alex Baransky algorithm alumni Alumni Interview Alumni Reviews Alumni Spotlight alumni story Alumnus ames dataset ames housing dataset apartment rent API Application artist aws bank loans beautiful soup Best Bootcamp Best Data Science 2019 Best Data Science Bootcamp Best Data Science Bootcamp 2020 Best Ranked Big Data Book Launch Book-Signing bootcamp Bootcamp Alumni Bootcamp Prep boston safety Bundles cake recipe California Cancer Research capstone car price Career Career Day citibike classic cars classpass clustering Coding Course Demo Course Report covid 19 credit credit card crime frequency crops D3.js data data analysis Data Analyst data analytics data for tripadvisor reviews data science Data Science Academy Data Science Bootcamp Data science jobs Data Science Reviews Data Scientist Data Scientist Jobs data visualization database Deep Learning Demo Day Discount disney dplyr drug data e-commerce economy employee employee burnout employer networking environment feature engineering Finance Financial Data Science fitness studio Flask flight delay gbm Get Hired ggplot2 googleVis H20 Hadoop hallmark holiday movie happiness healthcare frauds higgs boson Hiring hiring partner events Hiring Partners hotels housing housing data housing predictions housing price hy-vee Income Industry Experts Injuries Instructor Blog Instructor Interview insurance italki Job Job Placement Jobs Jon Krohn JP Morgan Chase Kaggle Kickstarter las vegas airport lasso regression Lead Data Scienctist Lead Data Scientist leaflet league linear regression Logistic Regression machine learning Maps market matplotlib Medical Research Meet the team meetup methal health miami beach movie music Napoli NBA netflix Networking neural network Neural networks New Courses NHL nlp NYC NYC Data Science nyc data science academy NYC Open Data nyc property NYCDSA NYCDSA Alumni Online Online Bootcamp Online Training Open Data painter pandas Part-time performance phoenix pollutants Portfolio Development precision measurement prediction Prework Programming public safety PwC python Python Data Analysis python machine learning python scrapy python web scraping python webscraping Python Workshop R R Data Analysis R language R Programming R Shiny r studio R Visualization R Workshop R-bloggers random forest Ranking recommendation recommendation system regression Remote remote data science bootcamp Scrapy scrapy visualization seaborn seafood type Selenium sentiment analysis sentiment classification Shiny Shiny Dashboard Spark Special Special Summer Sports statistics streaming Student Interview Student Showcase SVM Switchup Tableau teachers team team performance TensorFlow Testimonial tf-idf Top Data Science Bootcamp Top manufacturing companies Transfers tweets twitter videos visualization wallstreet wallstreetbets web scraping Weekend Course What to expect whiskey whiskeyadvocate wildfire word cloud word2vec XGBoost yelp youtube trending ZORI