Using Data to Web Scrape - Record Sales

Posted on Jun 10, 2021


Over the past fifty years, audio technology has changed dramatically, and recently, the music production industry has seen a massive transition in user preferences.  With the advent of streaming services in the early 2010s, more users now prefer to stream music as opposed to owning physical copies of media. In this text will use data to web scrape.

Through these changes, the once dominant vinyl record industry remains intact, and many users prefer vinyl because of the many benefits of vinyl formatting. Vinyl records lack the compression loss of digital audio files, often have greater sound quality, and can serve a collectible or novelty for the user. 


In evaluating the current state of the vinyl record industry, a few questions become apparent.

  • Are modern artists releasing their music in vinyl format, and if so, how do the number of modern releases compare to that of older artists?
  • How does the cost modern vinyl releases compare to that of older releases>
  • What can the vinyl industry do to keep up with digital formats and streaming platforms?

The code used in the following analysis can be found on Github.

Popular Music Trends

To investigate this problem. I referred to the Billboard 200 album of the week archive available on Wikipedia. This is a comprehensive list of the highest rated album for each week from 1953 to today according to the culture magazine, Billboard. The dataset stored on Wikipedia contains the year of the release, the week the release was most popular, the name of the release, and the name of the artist. I used the data from 1975 to 2020 to evaluate the trends in vinyl releases over time from when vinyl was the most popular media format to the age of streaming.

Using Data to Web Scrape - Record Sales
Figure 1: Billboard 200 Album of the Week 2020 (January - April)

I used the Python library Scrapy to scrape the tabular data from wikipedia. Though the data from Billboard gives us insight into artist and release popularity over time, it provides no indication as to how vinyl record sales performed in the market. Discogs is a digital marketplace where users can sell audio media in all formats, and it has become a popular platform for vinyl sales, and I scraped this site to better understand the vinyl marketplace.

Vinyl Releases Over the Years

To begin investigating this problem, I began by observing the total number of vinyl releases by artist per year. In a given year, an artist may have many releases even though they have only released or recorded a single album. The reason that album releases appear this way is a result of the same recording being releases in a different format.

For example, two vinyl releases that contain identical audio recordings but have different record sleeves are considered separate vinyl releases. The total number of vinyl releases per artist per year will give us a good understanding of how many different vinyl formats were created for a particular album accounting for international releases, alternative album art, collaboration albums, and limited edition copies.

First, I chose to simply compare two artists, one artist that had a Billboard album of the week listed in 2019 (DaBaby) and another artist that had a Billboard album of the week listed in 1984 (Huey Lewis and the News). Presumably, Huey Lewis and the News would have a higher number of releases per year than DaBaby because of the popularity of vinyl at the time, and this is reflected by Figure 2.

The total number of releases per artist, particularly the number of releases by Huey Lewis and the News in 1986 were much higher than I expected. One has to remember that these are only the total number of releases that have been listed for sale on Discogs, and many more releases could have been put into production.

Using Data to Web Scrape - Record Sales
Figure 2: Vinyl releases per year for Huey Lewis and the News (1984) versus DaBaby (2019)

With the findings from these two artists in mind, I decided to pick two years and plot all releases from all Billboard 200 number one album producing artists from each year in the same format. The years picked were 1976 and 2010. Vinyl was more popular in 1976 than in 2010, and it can be assumed that the same trend present in the number of releases from Huey Lewis and the News versus Dababy will be observed in the difference between the number of releases in 1976 versus 2010.

2010 was chosen instead of a more recent year, because it is clear based on Figure 2 that album releases can be produced many years after the initial release of the audio recording to account for delayed international popularity and other factors such as collaboration and greatest hits albums. Additionally, an artist with a popular release in 2010 may continue to release other albums in subsequent years. Observing the trends from 2010 instead of a more recent year will give us enough time to see how many releases are present in subsequent years.

Using Data to Web Scrape - Record Sales
Figure 3: International releases per year for Billboard Album of the Week artists from 1976 versus 2010.

Surprisingly, the number of releases per year observed from Discogs does not reflect the initial hypothesis. There are many factors that contribute to this.

  1. A greater number of artists in 2010 were listed on the Billboard Album of the Week charts than in 1976
  2. Discogs has only existed since the year 2000, and it was originally intended to serve as a database for online releases of electronic music
  3. International popularity of (mostly) United States based music has greatly increased between the years of 1976 and 2010, and this has supported both a greater number of international releases and shorter delays in releases of international record formats.

Even when considering the above factors, the data does suggest that, at the very least, modern artists do appear to be releasing vinyl formats and individuals are listing modern vinyl records for sale on Discogs in greater quantities than records produced by older artists.

Album Prices

In addition to the total number of releases from popular artists listed on Discord, I am interested in the price of albums for sale and how release year affects the price of an album. In order to collect this data, I scraped each entry from the first 200 pages of recent listings on the discord marketplace which contained a total of 5000 album listings.

From each listing, I recorded the price and year of each album released in order to track the trend of album prices based on release data. The data was smoothed using a running average filter to capture the trend of album prices per year, and the results show a clear trend where album price increases as the release date approaches the modern day.

Using Data to Web Scrape - Record Sales
Figure 4: Album prices based on release date as of June 2020.

There are several shortcomings of the data used to generate this analysis of album price trends.

Sample Size per Year

Because the data was retrieved from the first pages of the entire Discogs marketplace, there are not an equal number of samples from each year represented in the price data collected. For example, there were very few albums listed in the marketplace from 1966, and a single, expensive anthology album skewed the average price that year so that it appears much higher than it might have been if more data were collected.

In order to account for these inconsistencies, the running average filter was used to capture a general trend of the price data, however, analyzing a larger dataset with equal samples per year could have better solved this problem.

Time Dependence

Similarly, individuals post listings to the Discogs marketplace continuously, and the data scraped from the Discogs marketplace only represents the price at a momentary state. If this data were scraped a day later than it were, as many as hundreds of new entries could have been added to the marketplace home page, resulting in a near complete turnover of the data represented in Figure 4. On a more macroscale, if the data were to be collected years ago or years in the future, music trends and vinyl record popularity could have completely changed, resulting in different data trends.

Discogs Resale Philosophy

Discogs is strictly a resale platform, and it does not allow record labels to directly sell original products to customers. Because of this philosophy, it is unclear how skewed the price of recently released records are by the reality that they were also recently sold to a user by a record label or vendor at an invoice price.

For example, an released in the mid 20th century, though potentially popular and valuable, could be sold on discogs at a lower price than its true value because its price depends more heavily on the seller's attachment to it than it does on the initial invoice value. Contrarily, it is reasonable to think that someone selling a record released only a few years prior remembers the invoice price that they have paid and intends to sell the record at a similar price.

Given these limitations of the data collected, it is clear that record sales on Discogs are trending upwards per year with a few caveats.

  1. The trends are measured as of June 2021
  2. Sales represent resale value only
  3. The data analyzed is subject to the listings available on Discogs


Modern vinyl sales appear to be performing very well in resale markets in comparison to what I expected to find. It appears that artists are in fact producing vinyl formats of their new releases, and users appear to be buying and selling them on resale platforms at higher rates and higher prices than older releases.

It still remains somewhat unclear what the vinyl record industry can do to compete with streaming platforms, because the data used in this analysis is limited to vinyl resale. To further investigate this problem, it would be important to also analyze the streaming user habits and the vinyl record sales trends of record labels selling media at invoice prices.

The skills I demoed here can be learned through taking Data Science with Machine Learning bootcamp with NYC Data Science Academy.

About Author

Matthew Boubin

Matt Boubin is an electrical engineer with three years of digital signal processing experience in commercial aviation.
View all posts by Matthew Boubin >

Leave a Comment

No comments found.

View Posts by Categories

Our Recent Popular Posts

View Posts by Tags

#python #trainwithnycdsa 2019 2020 Revenue 3-points agriculture air quality airbnb airline alcohol Alex Baransky algorithm alumni Alumni Interview Alumni Reviews Alumni Spotlight alumni story Alumnus ames dataset ames housing dataset apartment rent API Application artist aws bank loans beautiful soup Best Bootcamp Best Data Science 2019 Best Data Science Bootcamp Best Data Science Bootcamp 2020 Best Ranked Big Data Book Launch Book-Signing bootcamp Bootcamp Alumni Bootcamp Prep boston safety Bundles cake recipe California Cancer Research capstone car price Career Career Day citibike classic cars classpass clustering Coding Course Demo Course Report covid 19 credit credit card crime frequency crops D3.js data data analysis Data Analyst data analytics data for tripadvisor reviews data science Data Science Academy Data Science Bootcamp Data science jobs Data Science Reviews Data Scientist Data Scientist Jobs data visualization database Deep Learning Demo Day Discount disney dplyr drug data e-commerce economy employee employee burnout employer networking environment feature engineering Finance Financial Data Science fitness studio Flask flight delay gbm Get Hired ggplot2 googleVis H20 Hadoop hallmark holiday movie happiness healthcare frauds higgs boson Hiring hiring partner events Hiring Partners hotels housing housing data housing predictions housing price hy-vee Income Industry Experts Injuries Instructor Blog Instructor Interview insurance italki Job Job Placement Jobs Jon Krohn JP Morgan Chase Kaggle Kickstarter las vegas airport lasso regression Lead Data Scienctist Lead Data Scientist leaflet league linear regression Logistic Regression machine learning Maps market matplotlib Medical Research Meet the team meetup methal health miami beach movie music Napoli NBA netflix Networking neural network Neural networks New Courses NHL nlp NYC NYC Data Science nyc data science academy NYC Open Data nyc property NYCDSA NYCDSA Alumni Online Online Bootcamp Online Training Open Data painter pandas Part-time performance phoenix pollutants Portfolio Development precision measurement prediction Prework Programming public safety PwC python Python Data Analysis python machine learning python scrapy python web scraping python webscraping Python Workshop R R Data Analysis R language R Programming R Shiny r studio R Visualization R Workshop R-bloggers random forest Ranking recommendation recommendation system regression Remote remote data science bootcamp Scrapy scrapy visualization seaborn seafood type Selenium sentiment analysis sentiment classification Shiny Shiny Dashboard Spark Special Special Summer Sports statistics streaming Student Interview Student Showcase SVM Switchup Tableau teachers team team performance TensorFlow Testimonial tf-idf Top Data Science Bootcamp Top manufacturing companies Transfers tweets twitter videos visualization wallstreet wallstreetbets web scraping Weekend Course What to expect whiskey whiskeyadvocate wildfire word cloud word2vec XGBoost yelp youtube trending ZORI