Data Web Scraping Blockbuster Budgets
The skills we demoed here can be learned through taking Data Science with Machine Learning bootcamp with NYC Data Science Academy.
Contributed by Michael Winfield. Michael is currently in the NYC Data Science Academy 12-week full time Data Science Bootcamp program taking place between April 11th to July 1st, 2016. This post is based on his third class project – Web Scraping.
Introduction
As an avid movie fan, I decided to use Scrapy to web scrape data from the-numbers.com, a website dedicated to chronicling film box office performance.
I was ultimately interested in developing a model to classify the box office performance of films on the basis of their production budgets and release dates. I would then apply this model to unreleased films.
For that reason, I scraped a page of 5125 movies, arranged by budget, release date, and domestic and worldwide grosses.
Data Selection
I then selected 340 of these films. These 340 films range from $94 million to $425 million in production budget. They were released in comparable time periods, from 1995 until 2016. Included in these 340 are the lowest budgeted film to reach a billion dollars in worldwide grosses, as well as every film to reach a billion dollars in worldwide grosses (as of the time of scraping). Although it is not included in the data that I scraped, most of these films opened in at least 3000 theaters domestically, played in greater than or equal to 70 territories, and are either sci-fi/fantasy/action films for adults or animated comedies for children.
Data Visualization in Python using the Seaborn package
Before I applied any machine learning techniques to this data, i.e., Support Vector Machines and Boosted Trees, I wanted to see what the basic relationships were between time, foreign grosses and domestic grosses.
In this first graph we can see that foreign grosses are higher than domestic grosses over time, and this appears to be true as both foreign and domestic grosses increase.
From the second graph, we can see that increases in foreign grosses track with relatively stable production budgets, and foreign grosses routinely exceed production budgets.
In the third graph, we can see it is sometimes the case that domestic grosses are lower than production budgets.
Based on these graphs, I decided to investigate timing and production budgets using machine learning techniques.
Data Munging in R
After loading the scraped dataset and adding a Foreign Grosses column, I added timing factors and a box office performance classifier for use in applying Support Vector Machines and Boosted Trees. The timing factors correspond to dump months (1), Awards season (2), summer (3), and ‘Aggressive’(4). The term ‘Aggressive’ is intended to capture releasing a film in March or April to beat out the summer competition early.
Support Vector Machines
I loaded the e1071 library for SVM and created a training and test set.
The training set consisted of 334 released movies. The test set consisted of the 6 unreleased films on the-numbers.com movie budgets page with budgets above $94 million:
- 2020-12-31 Singularity ($175 million)
- 2016-12-21 Passengers ($150 million)
- 2016-11-04 Trolls ($120 million)
- 2017-07-21 Valerian and the City of a Thousand Planets ($180 million)
- 2016-10-21 Jack Reacher: Never Go Back ($96 million)
- 2016-06-24 Independence Day: Resurgence ($200 million)
Here is a screenshot of a 3D scatterplot of my cost, gamma, and error, resulting from the cross-validation I used to fine-tune my model.
The classification predictions are as follows:
37 159 55 82 122 149
3 3 3 3 3 3
Levels: -1 1 2 3
As all of these films are in classification 3, the inference, according to my model, is that they will all make less than $531 million in worldwide grosses. Then again, SVM is very sensitive, and an additional data point can dramatically shift the number of support vectors and the hyperplane that divides the classes.
Boosted Trees Data
I next wanted to look deeper into the effect of timing on foreign and domestic grosses. I used the GBM package to do so, using 1000 trees with an interaction depth of 4.
One can see that production budgets have more of an influence on both domestic and foreign grosses than the timing of release.
We can see that foreign grosses are higher during Awards season (2) and Summer (3) than other times, respectively.
One can see that domestic grosses are higher during summer (3) and Awards season (2), respectively. By comparing these graphs, one can also see that production budgets have a much greater impact on foreign grosses than domestic grosses.
Conclusions
- Production budgets and timing of release affect foreign and domestic box office performance differently.
- Production budgets have a greater influence on box office performance than timing of release.
- Foreign markets and domestic markets are more responsive at different times.
- Given their budgets and the rule of thumb for blockbuster success (3x the production budget in worldwide grosses), Independence Day: Resurgence and Valerian and the City of a Thousand Planets might be in trouble.
- For further reading on this topic, here is a useful link.
About Author
