Using Data to Web Scrape - Record Sales
Overview
Over the past fifty years, audio technology has changed dramatically, and recently, the music production industry has seen a massive transition in user preferences. With the advent of streaming services in the early 2010s, more users now prefer to stream music as opposed to owning physical copies of media. In this text will use data to web scrape.
Through these changes, the once dominant vinyl record industry remains intact, and many users prefer vinyl because of the many benefits of vinyl formatting. Vinyl records lack the compression loss of digital audio files, often have greater sound quality, and can serve a collectible or novelty for the user.
In evaluating the current state of the vinyl record industry, a few questions become apparent.
- Are modern artists releasing their music in vinyl format, and if so, how do the number of modern releases compare to that of older artists?
- How does the cost modern vinyl releases compare to that of older releases>
- What can the vinyl industry do to keep up with digital formats and streaming platforms?
The code used in the following analysis can be found on Github.
Popular Music Trends
To investigate this problem. I referred to the Billboard 200 album of the week archive available on Wikipedia. This is a comprehensive list of the highest rated album for each week from 1953 to today according to the culture magazine, Billboard. The dataset stored on Wikipedia contains the year of the release, the week the release was most popular, the name of the release, and the name of the artist. I used the data from 1975 to 2020 to evaluate the trends in vinyl releases over time from when vinyl was the most popular media format to the age of streaming.
I used the Python library Scrapy to scrape the tabular data from wikipedia. Though the data from Billboard gives us insight into artist and release popularity over time, it provides no indication as to how vinyl record sales performed in the market. Discogs is a digital marketplace where users can sell audio media in all formats, and it has become a popular platform for vinyl sales, and I scraped this site to better understand the vinyl marketplace.
Vinyl Releases Over the Years
To begin investigating this problem, I began by observing the total number of vinyl releases by artist per year. In a given year, an artist may have many releases even though they have only released or recorded a single album. The reason that album releases appear this way is a result of the same recording being releases in a different format.
For example, two vinyl releases that contain identical audio recordings but have different record sleeves are considered separate vinyl releases. The total number of vinyl releases per artist per year will give us a good understanding of how many different vinyl formats were created for a particular album accounting for international releases, alternative album art, collaboration albums, and limited edition copies.
First, I chose to simply compare two artists, one artist that had a Billboard album of the week listed in 2019 (DaBaby) and another artist that had a Billboard album of the week listed in 1984 (Huey Lewis and the News). Presumably, Huey Lewis and the News would have a higher number of releases per year than DaBaby because of the popularity of vinyl at the time, and this is reflected by Figure 2.
The total number of releases per artist, particularly the number of releases by Huey Lewis and the News in 1986 were much higher than I expected. One has to remember that these are only the total number of releases that have been listed for sale on Discogs, and many more releases could have been put into production.
With the findings from these two artists in mind, I decided to pick two years and plot all releases from all Billboard 200 number one album producing artists from each year in the same format. The years picked were 1976 and 2010. Vinyl was more popular in 1976 than in 2010, and it can be assumed that the same trend present in the number of releases from Huey Lewis and the News versus Dababy will be observed in the difference between the number of releases in 1976 versus 2010.
2010 was chosen instead of a more recent year, because it is clear based on Figure 2 that album releases can be produced many years after the initial release of the audio recording to account for delayed international popularity and other factors such as collaboration and greatest hits albums. Additionally, an artist with a popular release in 2010 may continue to release other albums in subsequent years. Observing the trends from 2010 instead of a more recent year will give us enough time to see how many releases are present in subsequent years.
Surprisingly, the number of releases per year observed from Discogs does not reflect the initial hypothesis. There are many factors that contribute to this.
- A greater number of artists in 2010 were listed on the Billboard Album of the Week charts than in 1976
- Discogs has only existed since the year 2000, and it was originally intended to serve as a database for online releases of electronic music
- International popularity of (mostly) United States based music has greatly increased between the years of 1976 and 2010, and this has supported both a greater number of international releases and shorter delays in releases of international record formats.
Even when considering the above factors, the data does suggest that, at the very least, modern artists do appear to be releasing vinyl formats and individuals are listing modern vinyl records for sale on Discogs in greater quantities than records produced by older artists.
Album Prices
In addition to the total number of releases from popular artists listed on Discord, I am interested in the price of albums for sale and how release year affects the price of an album. In order to collect this data, I scraped each entry from the first 200 pages of recent listings on the discord marketplace which contained a total of 5000 album listings.
From each listing, I recorded the price and year of each album released in order to track the trend of album prices based on release data. The data was smoothed using a running average filter to capture the trend of album prices per year, and the results show a clear trend where album price increases as the release date approaches the modern day.
There are several shortcomings of the data used to generate this analysis of album price trends.
Sample Size per Year
Because the data was retrieved from the first pages of the entire Discogs marketplace, there are not an equal number of samples from each year represented in the price data collected. For example, there were very few albums listed in the marketplace from 1966, and a single, expensive anthology album skewed the average price that year so that it appears much higher than it might have been if more data were collected.
In order to account for these inconsistencies, the running average filter was used to capture a general trend of the price data, however, analyzing a larger dataset with equal samples per year could have better solved this problem.
Time Dependence
Similarly, individuals post listings to the Discogs marketplace continuously, and the data scraped from the Discogs marketplace only represents the price at a momentary state. If this data were scraped a day later than it were, as many as hundreds of new entries could have been added to the marketplace home page, resulting in a near complete turnover of the data represented in Figure 4. On a more macroscale, if the data were to be collected years ago or years in the future, music trends and vinyl record popularity could have completely changed, resulting in different data trends.
Discogs Resale Philosophy
Discogs is strictly a resale platform, and it does not allow record labels to directly sell original products to customers. Because of this philosophy, it is unclear how skewed the price of recently released records are by the reality that they were also recently sold to a user by a record label or vendor at an invoice price.
For example, an released in the mid 20th century, though potentially popular and valuable, could be sold on discogs at a lower price than its true value because its price depends more heavily on the seller's attachment to it than it does on the initial invoice value. Contrarily, it is reasonable to think that someone selling a record released only a few years prior remembers the invoice price that they have paid and intends to sell the record at a similar price.
Given these limitations of the data collected, it is clear that record sales on Discogs are trending upwards per year with a few caveats.
- The trends are measured as of June 2021
- Sales represent resale value only
- The data analyzed is subject to the listings available on Discogs
Conclusions
Modern vinyl sales appear to be performing very well in resale markets in comparison to what I expected to find. It appears that artists are in fact producing vinyl formats of their new releases, and users appear to be buying and selling them on resale platforms at higher rates and higher prices than older releases.
It still remains somewhat unclear what the vinyl record industry can do to compete with streaming platforms, because the data used in this analysis is limited to vinyl resale. To further investigate this problem, it would be important to also analyze the streaming user habits and the vinyl record sales trends of record labels selling media at invoice prices.
The skills I demoed here can be learned through taking Data Science with Machine Learning bootcamp with NYC Data Science Academy.
About Author
