Hollywood's Sequel Problem
Contributed by Jake Lehrhoff. Jake took NYC Data Science Academy 12 week full time Data Science Bootcamp program between Sept 23 to Dec 18, 2015. The post was based on his third class project(due at 6th week of the program).
Header artwork courtesy of Fortune.com
Lately, it feels like a shocking number of "new releases" are either sequels or reboots. This holiday season's biggest box office draws are certain to be the conclusion to The Hunger Games trilogy and the highly anticipated, J.J. Abrams-helmed Star Wars Episode VII. Christopher Nolan's Dark Knight trilogy is but a recent memory, and yet suddenly Ben Affleck (Ben Affleck!) is poised to debut as our rebooted caped crusader in early 2016. Marvel has as many as 70 superhero movies planned for release from 2015 through 2028. There were eight American Pie movies. Eight.
So what is it about sequels? Why do we flock to them in droves?
Maybe we are hedging our bets on the tried-and-true (James Bond). In some cases, we are certainly too attached to the characters to pass up on a new installment (The Muppets). But what if the tried-and-true or our attachments fail us? Audiences and critics alike loved Peter Jackson's Lord of the Rings trilogy, but even inserting our favorite characters couldn't save The Hobbit from itself. (That isn't to say it faired poorly at the box office.)
The psychology behind our moviegoing choices may require more digging, but the data to explain our behavior is readily available. For the current study, daily box office receipts from the-number.com, ratings from rottentomatoes.com, and lifetime revenues from boxofficemojo.com were compiled to paint a picture of sequels today.
For a more hands-on look at the data, please check out the Shiny web application I developed to investigate how various movies compare in daily box office receipts and rankings, lifetime gross, and rating. Also, check out the "Franchise Fights" tab to pit some of your favorite franchises or movie collections against each other.
Additionally, all the code and data are available on my public github repo.
Before we get ahead of ourselves: what exactly is a sequel?
The definition is murkier than it may seem. Of course, we could call sequels movies that have a single, direct predecessor. But what about Avengers? It's more of a super-sequel, with three current film franchises funneling into a single mega-blockbuster. For the purposes of this study, it's a sequel.
Avengers may be a fringe case, but what about reboots? They are everywhere. Batman Begins was the first of the Christopher Nolan's Dark Knight trilogy, but it's certainly no an original film. Such classification quickly gets cloudy. There was a Transformers animated movie in the 1980's--does that make Shia Labeouf's 2007 film a reboot? Well, yes. Same with the various Teenage Mutant Ninja Turtle movies (while we're on the subject of Michael Bay) and those Alvin and the Chipmunk movies. Reboots are simply unavoidable.
The Code
Various scraping functions were used for each of the aforementioned websites, each tailored to the idiosyncrasies of the source HTML. Search functions allow the HTML text content to be filtered down to the relevant data and pandas recompiles the data into clean, mutable tables.
Daily Box Office Receipts: the-numbers.com
The below function scrapes daily box office data for specific movies using Beautiful Soup, regular expressions, and pandas. Columns are retained for titles, rank, gross, total gross, days in theater, among other variables.
from bs4 import BeautifulSoup import requests import re import pandas as pd def scraper(list_): '''Scrape the-numbers.com for daily box office data.''' movieDF = pd.DataFrame() for url in list_: date = re.findall('daily.*', url) date = date[0][6:] date = re.sub('/','-', date) data = requests.get(url).text data = BeautifulSoup(data) aTag = data.find_all('a') movies = [tag.string for tag in aTag] movies = movies[52:82] data_sublists = [movies[i:i+2] for i in range(0, len(movies), 2)] dataframe = pd.DataFrame(data_sublists [0:], columns = ['Title','Studio']) moviedata = data.find_all('td', {'class': "data"}) moviedata = [tag.string for tag in moviedata] data_sublists2 = [moviedata[i:i+8] for i in range(0, len(moviedata), 8)] dataframe2 = pd.DataFrame(data_sublists2 [0:], columns=['Rank','PreviousRank','Gross', 'Change','Theaters','PerTheater', 'TotalGross','Days']) dataframe2['Title'] = dataframe['Title'] dataframe2['Studio'] = dataframe['Studio'] dataframe2['date'] = date dataframe2 = dataframe2[:15] movieDF = movieDF.append(dataframe2) return movieDF url = 'http://www.the-numbers.com/box-office-chart/daily/'
Example code for scraping The Bourne Identity daily receipts
The code below specifies the dates of interest for the given movie, concatenating the strings with the box office chart url. After calling the scraper function, the code filters the resulting data frame for the movie in question.
'''Scraping Bourne Identity daily data''' urlBourneI = [] for i in range(14,31): urlBourneI.append(url + '2002/06/' + str(i)) for i in range(1,15): if i < 10: urlBourneI.append(url + '2002/07/' + '0' + str(i)) else: urlBourneI.append(url + '2002/07/' + str(i)) BourneI = scraper(urlBourneI) BourneI = BourneI[(BourneI.Title == 'The Bourne Identity')]
Cleaning Daily Data and Plotting
With individual data frames for each of the desired movies, the following code combines the data into a single data frame and cleans the columns. The cleaning removes punctuation, converts numeric strings into integers, and converts date strings into mutable datetime.
frames = [SPR, BourneI, BourneS, Batman, BourneU, DarkK, GreenZ, Descendants, HungerG, Avengers, DarkKR, SilverLP, Elysium, CatchF, AmHust, Divergent, Fault, Guard, Interstellar, Mockingjay, Insurgent, AvengersU, JurassicW, Martian, LOTR1, LOTR2, LOTR3, Hobbit1, Hobbit2, Hobbit3, HP1, HP2, HP3, HP4, HP5, HP6, HP7a, HP7b, Twilight1, Twilight2, Twilight3, Twilight4a, Twilight4b, Transf1, Transf2, Transf3, Transf4, MadMax] test = pd.concat(frames) from datetime import datetime def removePunc(x): return re.sub( "[^\\d]", "", x) test['date'] = test['date'].apply(lambda x: datetime.strptime(x, "%Y-%m-%d")) test.index = test['date'] test = test.sort(ascending=True,columns='date') test['Gross'] = test['Gross'].apply(removePunc).apply(int) test['TotalGross'] = test['TotalGross'].apply(removePunc).apply(int)
Example Plot Code
The following code uses Seaborn and matplotlib to create the plot seen at the top of the next section. By adjusting the title filter in the 6th line and adding "or" statements, similar plots can be created to inspect the growth of disparate films within the dataset.
import seaborn as sns import matplotlib.pyplot as plt %matplotlib inline plt.rcParams['figure.figsize'] = 14, 10 sns.pointplot(x="Days", y="TotalGross", hue="Title", data=test2[(test2.Title.str.contains('Bourne'))]) plt.ylabel('Total Gross (Million of $)', fontsize=16) plt.ylim(0,) plt.xlabel('Days in Theaters', fontsize=16) plt.legend(fontsize=16) plt.xticks(rotation = 0, fontsize = 14) plt.yticks(fontsize = 14)
The Sequel Effect
The Bourne Series
Why start with a 13-year-old action movie, you ask? The Bourne series is one in a laundry list of movie franchises that depicts the "sequel effect." The graph to the left quantifies just how much a sequel can be worth. Successive Bourne movies command greater and greater box office receipts: the first month of the second film grossed a full $50 million more than that of the first, and the third film took home another $50 million on top of that. 
Can you blame Universal Studios for trying for a fourth, even if they couldn't convince Matt Damon to reprise the now-iconic roll? When Jeremy Renner took the helm for Legacy, lifetime gross dropped below that of even the first film. What happened here? Did audiences see this as a bastardization of the series and boycott? Do movies now have to abide by sitcom rules: at the end of the day, nothing ever changes? Or, was it simply not very good? It turns out, the simplest explanation was the correct one. We also have our first evidence that retaining a franchise's star is worth a lot of money.
What about movie franchises with lots of installments? There are eight Harry Potter movies. Could all eight movies possibly show the sequel effect? Nearly.
The Code Part II: The Code Strikes Back
Scraping Reviews: Rotten Tomatoes
A similar scraping function locates review data, using a find function to identify where in the HTML the relevant data begins. Rather than requiring specific dates, this function can take a list of movie keywords. The benefit here is that inputting "Bourne" and "Hunger Games" as the keywordList argument returns the review scores for all the movies in both series--much more efficient, requiring minimal hard-coding. The boxofficemojo.com scraper functions similarly, returning lifetime gross data. Code for the scatterplot in the next section is also included.
def find(lst, predicate): return (i for i, j in enumerate(lst) if predicate(j)).next() def reviewscrape(keywordList): reviews = pd.DataFrame() url = 'http://www.rottentomatoes.com/search/?search=' for keyword in keywordList: search = re.sub(' ','+',keyword) data = requests.get(url + search).text data = BeautifulSoup(data) Tag2 = data.find_all('class' == ['tMeterScore', 'unstyled articleLink']) movies1 = [tag.string for tag in Tag2] movies1 = [x for x in movies1 if x is not None] movies1 = movies1[300:] index = find(movies1, lambda x: '%' in x) movies1 = movies1[index:] sublist = [movies1[i:i+6] for i in range(0, len(movies1), 6)] dataframe = pd.DataFrame(sublist [0:], columns = ['Rating','Title','','','','']) dataframe = dataframe[['Rating','Title']][0:6] reviews = reviews.append(dataframe) return reviews sns.lmplot(x="Rating", y="Lifetime Gross", hue = "Title", fit_reg = False, data=gross[((gross['Title'].str.contains('Transformers')) )].sort(ascending=False), scatter_kws={"s": 100} legend = False) plt.ylabel('Total Gross (Millions of $)', fontsize=16) plt.xlabel('Rotten Tomatoes Rating', fontsize=16) plt.xticks(rotation = 0, fontsize = 14) plt.yticks(fontsize = 14) plt.legend(loc='upper right') plt.xlim(0,100)
Transformers
As strong as the sequel effect is, we've learned from The Bourne Legacy that not all installments in a series are created equal, and audiences may not open their wallets if their hero gets replaced and the quality drops substantially. But what happens if a hero is replaced but the quality was never there in the first place? Look no further than Tranformers: Age of Extinction. Without a rise in quality or the stars of the original, Mark Whalberg's Transformers brought home over $60 million less than the 2007 reboot. 
But reviews alone can't necessarily slow the momentum of a film series. None of the Twilight films had a Rotten Tomatoes rating above 50 (and two were below 30), and yet after the original, the four successive movies all raked in between $280 and $300 million. The equation for longevity is coming into focus: as long as you have a fanbase, they'll keep coming as long as you don't replace their stars.
The Code Part III: Return of the Code
Manipulating Yearly Top Ten Lists
The following code takes a data frame of the top 10 films of each year for the last 20 years (scraped from boxofficemojo.com) and reorganizes the data into categories depicting the yearly percentages and average gross of original films and sequels. The code for the pointplot in the next section can also be found.
grp = totals.groupby(('Year', 'Sequel')) yearly = pd.DataFrame(grp['Sequel'].count()) yearly['Percent'] = yearly['Sequel']/10 yearly = yearly.drop('Sequel',1) yearly.reset_index(inplace = True) yearlygross = pd.DataFrame(grp['Total Gross'].mean()) yearlygross.reset_index(inplace=True) top10s = pd.merge(yearly, yearlygross) sns.pointplot(x = 'Year', y = 'Percent', data = top10merge[top10merge['Sequel']=='Y'], fit_reg=False, scatter_kws={"s": 100}, legend = False) plt.ylabel('Percent of top-ten films', fontsize=16) plt.xlabel('Year', fontsize=16) plt.title('Prevalence of sequels in top-ten grossing films, 1996-2015', fontsize = 18) plt.xticks(rotation = 30, fontsize = 14) plt.yticks(fontsize = 14)
Conclusions: A New Hope
Is the sequel trend here to stay?
Th
e evidence says "yes," emphatically so. 64% of the last decade's yearly top-ten grossing movies were sequels, and those sequels on average brought home $33 million more than the originals. At the top of the charts it's even worse: nine of the top ten grossing films of the last decade were sequels, and 19 of the top 25.
These numbers may not be startling, but consider them in light of the previous decade: from 1996-2005, sequels comprised 37% of yearly top-ten grossing films. But even when sequels comprised less of the market, they were worth, on average, $23 million more than their original counterparts. With such a modest boost, its not surprising that sequels and reboots took over the market.
However...
Original ideas can still dominate at the box office, even original ideas written by software engineers who double as casual astrophysicists. The adaptation of Andy Weir's The Martian stood at the top of the box office two day's longer than Joss Whedon's first Avengers movie, and nine days longer than the sequel. The takeaway here is that original movies can be a boon for studios, particularly when Matt Damon is the star.
Cutting Hollywood Some Slack
We've learned a few things:
- Sequels make more money.
- Sequels that willfully replace lead actors are both inferior and less lucrative.
- Poorly reviewed sequels will even make a lot of money.
- The sequel trend has momentum.
But before we excoriate Hollywood for undermining creativity to cushion their bottom line, there's another way to tell that story. With predictable revenue streams and tentpole marketing, studios have room for lower-margin and higher-risk projects. They can gamble on more fringe concepts, be it in search of the next great franchise or the unusually creative--the avant garde beauty of WALL-E, the fantastical imagery of Pan's Labyrinth, or the bizarre existentialism of Being John Malkovich. Unfortunately, this theory--though compelling--is unfounded, and I'm no the only one who's worried. The Atlantic, Variety, Vogue, HuffPo, E!, and Mashable have all weighed in. (Forbes has moved on. For them it's all about the battle between sequels and reboots.)
Future Directions
This study has been a relatively surface-level investigation into the movie industry. Predicting Hollywood's boons and flops is a big business best served with heaps of data and a fine toothed comb. Mining Twitter for chatter on new releases can accurately predict box office revenues, but at that point, the investment has long since been made. By combining genre, proposed budget, actors, directors, and writers, an algorithm may be formed to help understand expected revenue for any proposed release a la baseball's "Moneyball." An important metric, as we've learned here, would be the "novelty" status: is a film a sequel, a reboot, an adaptation, or original? Hopefully all this adds up to not just mega-blockbusters but the success of the new and innovative.
Beyond all the data science that may go into building the next great film, we can all rest assured that the sequel and reboot trend has brought us at least one true gift: in 2016 we all get to enjoy Paul Feig's all-women Ghostbusters reboot.
About Author
Jake Lehrhoff
View all posts by Jake Lehrhoff >Related Articles
Leave a Comment
