Data Web Scraping Blockbuster Budgets

Posted on May 30, 2016
The skills we demoed here can be learned through taking Data Science with Machine Learning bootcamp with NYC Data Science Academy.

Contributed by Michael Winfield. Michael is currently in the NYC Data Science Academy 12-week full time Data Science Bootcamp program taking place between April 11th to July 1st, 2016. This post is based on his third class project – Web Scraping.


As an avid movie fan, I decided to use Scrapy to web scrape data from, a website dedicated to chronicling film box office performance.

Data Web Scraping Blockbuster Budgets

I was ultimately interested in developing a model to classify the box office performance of films on the basis of their production budgets and release dates. I would then apply this model to unreleased films.

For that reason, I scraped a page of 5125 movies, arranged by budget, release date, and domestic and worldwide grosses.

Data Web Scraping Blockbuster Budgets

Data Selection

I then selected 340 of these films. These 340 films range from $94 million to $425 million in production budget. They were released in comparable time periods, from 1995 until 2016. Included in these 340 are the lowest budgeted film to reach a billion dollars in worldwide grosses, as well as every film to reach a billion dollars in worldwide grosses (as of the time of scraping). Although it is not included in the data that I scraped, most of these films opened in at least 3000 theaters domestically, played in greater than or equal to 70 territories, and are either sci-fi/fantasy/action films for adults or animated comedies for children.

Data Visualization in Python using the Seaborn package

Before I applied any machine learning techniques to this data, i.e., Support Vector Machines and Boosted Trees, I wanted to see what the basic relationships were between time, foreign grosses and domestic grosses.

Data Web Scraping Blockbuster Budgets

In this first graph we can see that foreign grosses are higher than domestic grosses over time, and this appears to be true as both foreign and domestic grosses increase.

Data Web Scraping Blockbuster Budgets

From the second graph, we can see that increases in foreign grosses track with relatively stable production budgets, and foreign grosses routinely exceed production budgets.


In the third graph, we can see it is sometimes the case that domestic grosses are lower than production budgets.

Based on these graphs, I decided to investigate timing and production budgets using machine learning techniques.

Data Munging in R

After loading the scraped dataset and adding a Foreign Grosses column, I added timing factors and a box office performance classifier for use in applying Support Vector Machines and Boosted Trees. The timing factors correspond to dump months (1), Awards season (2), summer (3), and ‘Aggressive’(4). The term ‘Aggressive’ is intended to capture releasing a film in March or April to beat out the summer competition early.

Support Vector Machines

I loaded the e1071 library for SVM and created a training and test set.

The training set consisted of 334 released movies. The test set consisted of the 6 unreleased films on movie budgets page with budgets above $94 million:

  • 2020-12-31     Singularity ($175 million)
  • 2016-12-21     Passengers ($150 million)
  • 2016-11-04     Trolls ($120 million)
  • 2017-07-21     Valerian and the City of a Thousand Planets ($180 million)
  • 2016-10-21     Jack Reacher: Never Go Back ($96 million)
  • 2016-06-24     Independence Day: Resurgence ($200 million)

Here is a screenshot of a 3D scatterplot of my cost, gamma, and error, resulting from the cross-validation I used to fine-tune my model.


The classification predictions are as follows:

37 159 55 82 122 149
3 3 3 3 3 3
Levels: -1 1 2 3

As all of these films are in classification 3, the inference, according to my model, is that they will all make less than $531 million in worldwide grosses. Then again, SVM is very sensitive, and an additional data point can dramatically shift the number of support vectors and the hyperplane that divides the classes.

Boosted Trees Data

I next wanted to look deeper into the effect of timing on foreign and domestic grosses. I used the GBM package to do so, using 1000 trees with an interaction depth of 4.

One can see that production budgets have more of an influence on both domestic and foreign grosses than the timing of release.


We can see that foreign grosses are higher during Awards season (2) and Summer (3) than other times, respectively.


One can see that domestic grosses are higher during summer (3) and Awards season (2), respectively. By comparing these graphs, one can also see that production budgets have a much greater impact on foreign grosses than domestic grosses.



  • Production budgets and timing of release affect foreign and domestic box office performance differently.
  • Production budgets have a greater influence on box office performance than timing of release.
  • Foreign markets and domestic markets are more responsive at different times.
  • Given their budgets and the rule of thumb for blockbuster success (3x the production budget in worldwide grosses), Independence Day: Resurgence and Valerian and the City of a Thousand Planets might be in trouble.
  • For further reading on this topic, here is a useful link.

About Author

Michael Winfield

Michael has a passion for finding strategic insights for businesses, managers, and organizations engaged in competitive dynamics. With a background in corporate litigation and white collar criminal defense, as well as graduate-level education in strategic management, Michael is...
View all posts by Michael Winfield >

Related Articles

Leave a Comment

No comments found.

View Posts by Categories

Our Recent Popular Posts

View Posts by Tags

#python #trainwithnycdsa 2019 2020 Revenue 3-points agriculture air quality airbnb airline alcohol Alex Baransky algorithm alumni Alumni Interview Alumni Reviews Alumni Spotlight alumni story Alumnus ames dataset ames housing dataset apartment rent API Application artist aws bank loans beautiful soup Best Bootcamp Best Data Science 2019 Best Data Science Bootcamp Best Data Science Bootcamp 2020 Best Ranked Big Data Book Launch Book-Signing bootcamp Bootcamp Alumni Bootcamp Prep boston safety Bundles cake recipe California Cancer Research capstone car price Career Career Day citibike classic cars classpass clustering Coding Course Demo Course Report covid 19 credit credit card crime frequency crops D3.js data data analysis Data Analyst data analytics data for tripadvisor reviews data science Data Science Academy Data Science Bootcamp Data science jobs Data Science Reviews Data Scientist Data Scientist Jobs data visualization database Deep Learning Demo Day Discount disney dplyr drug data e-commerce economy employee employee burnout employer networking environment feature engineering Finance Financial Data Science fitness studio Flask flight delay gbm Get Hired ggplot2 googleVis H20 Hadoop hallmark holiday movie happiness healthcare frauds higgs boson Hiring hiring partner events Hiring Partners hotels housing housing data housing predictions housing price hy-vee Income Industry Experts Injuries Instructor Blog Instructor Interview insurance italki Job Job Placement Jobs Jon Krohn JP Morgan Chase Kaggle Kickstarter las vegas airport lasso regression Lead Data Scienctist Lead Data Scientist leaflet league linear regression Logistic Regression machine learning Maps market matplotlib Medical Research Meet the team meetup methal health miami beach movie music Napoli NBA netflix Networking neural network Neural networks New Courses NHL nlp NYC NYC Data Science nyc data science academy NYC Open Data nyc property NYCDSA NYCDSA Alumni Online Online Bootcamp Online Training Open Data painter pandas Part-time performance phoenix pollutants Portfolio Development precision measurement prediction Prework Programming public safety PwC python Python Data Analysis python machine learning python scrapy python web scraping python webscraping Python Workshop R R Data Analysis R language R Programming R Shiny r studio R Visualization R Workshop R-bloggers random forest Ranking recommendation recommendation system regression Remote remote data science bootcamp Scrapy scrapy visualization seaborn seafood type Selenium sentiment analysis sentiment classification Shiny Shiny Dashboard Spark Special Special Summer Sports statistics streaming Student Interview Student Showcase SVM Switchup Tableau teachers team team performance TensorFlow Testimonial tf-idf Top Data Science Bootcamp Top manufacturing companies Transfers tweets twitter videos visualization wallstreet wallstreetbets web scraping Weekend Course What to expect whiskey whiskeyadvocate wildfire word cloud word2vec XGBoost yelp youtube trending ZORI