Web Scraping Blockbuster Budgets

Michael Winfield
Posted on May 30, 2016

Contributed by Michael Winfield. Michael is currently in the NYC Data Science Academy 12-week full time Data Science Bootcamp program taking place between April 11th to July 1st, 2016. This post is based on his third class project – Web Scraping.


As an avid movie fan, I decided to use Scrapy to web scrape the-numbers.com, a website dedicated to chronicling film box office performance.


I was ultimately interested in developing a model to classify the box office performance of films on the basis of their production budgets and release dates. I would then apply this model to unreleased films.

For that reason, I scraped a page of 5125 movies, arranged by budget, release date, and domestic and worldwide grosses.



I then selected 340 of these films. These 340 films range from $94 million to $425 million in production budget. They were released in comparable time periods, from 1995 until 2016. Included in these 340 are the lowest budgeted film to reach a billion dollars in worldwide grosses, as well as every film to reach a billion dollars in worldwide grosses (as of the time of scraping). Although it is not included in the data that I scraped, most of these films opened in at least 3000 theaters domestically, played in greater than or equal to 70 territories, and are either sci-fi/fantasy/action films for adults or animated comedies for children.

Data Visualization in Python using the Seaborn package

Before I applied any machine learning techniques to this data, i.e., Support Vector Machines and Boosted Trees, I wanted to see what the basic relationships were between time, foreign grosses and domestic grosses.


In this first graph we can see that foreign grosses are higher than domestic grosses over time, and this appears to be true as both foreign and domestic grosses increase.


In the second graph, we can see that increases in foreign grosses track with relatively stable production budgets, and foreign grosses routinely exceed production budgets.


In the third graph, we can see it is sometimes the case that domestic grosses are lower than production budgets.

Based on these graphs, I decided to investigate timing and production budgets using machine learning techniques.

Data Munging in R

After loading the scraped dataset and adding a Foreign Grosses column, I added timing factors and a box office performance classifier for use in applying Support Vector Machines and Boosted Trees. The timing factors correspond to dump months (1), Awards season (2), summer (3), and ‘Aggressive’(4). The term ‘Aggressive’ is intended to capture releasing a film in March or April to beat out the summer competition early.


Support Vector Machines

I loaded the e1071 library for SVM and created a training and test set.


The training set consisted of 334 released movies. The test set consisted of the 6 unreleased films on the-numbers.com movie budgets page with budgets above $94 million:

  • 2020-12-31     Singularity ($175 million)
  • 2016-12-21     Passengers ($150 million)
  • 2016-11-04     Trolls ($120 million)
  • 2016-10-21     Jack Reacher: Never Go Back ($96 million)
  • 2016-06-24     Independence Day: Resurgence ($200 million)
  • 2017-07-21     Valerian and the City of a Thousand Planets ($180 million)

Here is a screenshot of a 3D scatterplot of my cost, gamma, and error, resulting from the cross-validation I used to fine-tune my model.


The classification predictions are as follows:

37 159 55 82 122 149
3 3 3 3 3 3
Levels: -1 1 2 3

As all of these films are in classification 3, the inference, according to my model, is that they will all make less than $531 million in worldwide grosses. Then again, SVM is very sensitive, and an additional data point can dramatically shift the number of support vectors and the hyperplane that divides the classes.

Boosted Trees

I next wanted to look deeper into the effect of timing on foreign and domestic grosses. I used the GBM package to do so, using 1000 trees with an interaction depth of 4.


One can see that production budgets have more of an influence on both domestic and foreign grosses than the timing of release.


One can see that foreign grosses are higher during Awards season (2) and Summer (3) than other times, respectively.


One can see that domestic grosses are higher during summer (3) and Awards season (2), respectively. By comparing these graphs, one can also see that production budgets have a much greater impact on foreign grosses than domestic grosses.



  • Production budgets and timing of release affect foreign and domestic box office performance differently.
  • Production budgets have a greater influence on box office performance than timing of release.
  • Foreign markets and domestic markets are more responsive at different times.
  • Given their budgets and the rule of thumb for blockbuster success (3x the production budget in worldwide grosses), Independence Day: Resurgence and Valerian and the City of a Thousand Planets might be in trouble.
  • For further reading on this topic, here is a useful link.

About Author

Michael Winfield

Michael Winfield

Michael has a passion for finding strategic insights for businesses, managers, and organizations engaged in competitive dynamics. With a background in corporate litigation and white collar criminal defense, as well as graduate-level education in strategic management, Michael is...
View all posts by Michael Winfield >

Related Articles

Leave a Comment

No comments found.

View Posts by Categories

Our Recent Popular Posts

View Posts by Tags

#python #trainwithnycdsa 2019 airbnb Alex Baransky alumni Alumni Interview Alumni Reviews Alumni Spotlight alumni story Alumnus API Application artist aws beautiful soup Best Bootcamp Best Data Science 2019 Best Data Science Bootcamp Best Data Science Bootcamp 2020 Best Ranked Big Data Book Launch Book-Signing bootcamp Bootcamp Alumni Bootcamp Prep Bundles California Cancer Research capstone Career Career Day citibike clustering Coding Course Demo Course Report D3.js data Data Analyst data science Data Science Academy Data Science Bootcamp Data science jobs Data Science Reviews Data Scientist Data Scientist Jobs data visualization Deep Learning Demo Day Discount dplyr employer networking feature engineering Finance Financial Data Science Flask gbm Get Hired ggplot2 googleVis Hadoop higgs boson Hiring hiring partner events Hiring Partners Industry Experts Instructor Blog Instructor Interview Job Job Placement Jobs Jon Krohn JP Morgan Chase Kaggle Kickstarter lasso regression Lead Data Scienctist Lead Data Scientist leaflet linear regression Logistic Regression machine learning Maps matplotlib Medical Research Meet the team meetup Networking neural network Neural networks New Courses nlp NYC NYC Data Science nyc data science academy NYC Open Data NYCDSA NYCDSA Alumni Online Online Bootcamp Online Training Open Data painter pandas Part-time Portfolio Development prediction Prework Programming PwC python Python Data Analysis python machine learning python scrapy python web scraping python webscraping Python Workshop R R language R Programming R Shiny r studio R Visualization R Workshop R-bloggers random forest Ranking recommendation recommendation system regression Remote remote data science bootcamp Scrapy scrapy visualization seaborn Selenium sentiment analysis Shiny Shiny Dashboard Spark Special Special Summer Sports statistics streaming Student Interview Student Showcase SVM Switchup Tableau team TensorFlow Testimonial tf-idf Top Data Science Bootcamp twitter visualization web scraping Weekend Course What to expect word cloud word2vec XGBoost yelp