Scraping Data in Ulta with Scrapy
The skills I demoed here can be learned through taking Data Science with Machine Learning bootcamp with NYC Data Science Academy.
GitHub | LinkedIn
Background
Skincare is more popular than ever. While beauty sales have fallen in 2020, skincare products, including face washes and moisturizers, have only increased in popularity. From February to March 2020, data shows some skincare product sales increased by over 600% (Statista). Within these products, moisturizers alone generate $700 million annually, in the US.
For my web scraping project, I chose to investigate trends in facial moisturizers sold on Ulta.com. Given the popularity of this industry (and my personal love for skincare), I wished to uncover and visualize to better understand information about these products.
Ulta.com is an all-in-one beauty retailer which prides itself on its immense product varieties. Ulta offers products at a wider range of price points than just about anyone else -- from drugstore to high-end. This massive variety made it preferable to similar retailers.
I had three main research questions to guide my analyses:
- Does the product price impact its rating? Specifically, I was curious if less expensive products had poorer ratings. I hypothesized that relatively cheaper products may be made with ingredients of slightly lower quality.
- What are the best-rated brands?
- What are the best-rated moisturizers at each price point?
The Web Scraping Process
I used scrapy to scrape the website. All code was written in Python. On each product result page, I parsed the following information from each item: product name, brand name, number of reviews, and average rating (out of 5 stars).
Data Cleaning and Analysis
Using a Jupyter notebook, I imported, cleaned, and analyzed the data.
Some of the products did not have any ratings. These were removed from the analyses. I also removed products that had less than 10 ratings, as these ratings were deemed to be not particularly reliable (discussed in depth below). Data cleaning was completed using numpy and pandas. Plots were produced using matplotlib and seaborn.
Does Product Price Impact Rating?
The short answer: most likely, not.
One initial question I had was whether a relationship existed between a product's price and its average rating. I produced a scatterplot fit with a simple linear regression line, and came to the conclusion that there does not appear to be a substantial association between the two measures.
I was less certain that no linear relationship exists, and more certain that this dataset was not optimal for answering the question. One possible explanation for this is the unbalanced aspect of the dataset. There are far too few observations of higher-priced products (and their reviews) to assess if such a pattern exists. This lead me to investigate the distribution of prices in the dataset.
Data Distribution of Prices
I produced a scatterplot in order to analyze the distribution of moisturizer prices. The median price (indicated with a blue line) is around $30. As suspected, the data was highly skewed -- the majority of products were between $20 and $40, with a few products costing as much as $100 or more.
This is to be expected, to some degree. There are simply more moisturizers that cost less. However, I suspected that this imbalance in the data may mask patterns that would otherwise appear. In order to produce further insights of value, I concluded I should group the data by price. This would allow for me to analyze products within a certain range, and between these ranges. I grouped the products into one of three groups by price range: Under $25, $25 to $50, and Over $50.
Review Data Distribution by Price Group
Now that the products were grouped by cost, I produced a boxplot to analyze the distribution of ratings within the three price ranges. The entire distribution rises ever so slightly, as one moves through the price ranges in ascending order. Products under $25 have the lowest median (approximately 4.4); the $25-$50 range has a median that is ever so slightly greater (approximately 4.5). The distribution of the products over $50 has a median of about 4.5 as well. However, this distribution almost appears to be entirely shifted up from the others. Put another way, the group of products that cost more than $50 possess (relatively) more products with ratings above 4.5.
Overall, the difference in mean product review is slight. I would not claim that more expensive products yield higher ratings.
Identifying Bestselling Products Data
Take two hypothetical products:
- Product A - Rating: 4.9 stars, number of ratings: 2
- Product B - Rating: 4.7 stars, number of ratings: 3,000
We can see that product B has a lower mean rating than Product A; however, Product B's rating is much more trustworthy.
My next goal was to determine the bestselling products (and by extension, bestselling brands). For the reason given above, I decided not to rely on solely a product's rating. To better identify bestselling products, I developed a "bestseller rating" metric, derived from a product's rating as well as the number of reviews in each product rating.
First, I identified the top 10 bestselling products in each of the three price ranges.
Bestselling Products at Each Price Point
The Top 10 products under $25 are charted above. The highest-rated of this group is Neutrogena's Hydro Boost Gel-Cream. Notably, the brands Neutrogena and L'Oréal each have multiple products in this ranking. This suggests they are two of the most popular brands at this price point.
Moving on to products costing $25-$50, the market appears to narrow. The highest-rated of this group is Olay's Regenerist Micro Sculpting Cream. Only 4 brands are represented in this list; with Clinique and Olay dominating. I suspect many products at this point are "tried and true" -- purchasers may look to a few trusted brands to buy moisturizers at this price point.
Above $50, there are a variety of brands represented here. Only two brands have more than one product in this ranking, suggesting the market is more varied. The highest-rated of this group is Dermalogica's Dynamic Skin Recovery Broad Spectrum SPF 50. Interestingly, there is more variety in ratings for this price point's bestseller list than in the other two lists. All products display ratings over 4 stars; however, it seems certain that higher prices do not indicate higher ratings.
Bestselling Brands
The charts below detail information on the Top 10 bestselling brands. By aggregating the bestseller ratings of each product by brand, I determined the top 10 bestseller-rated brands.
Unsurprisingly, all of the bestselling brands average above 4 stars for their listed products. Neutrogena has the relatively lowest rating, and Olay has the relatively greatest rating.
The majority of the brands above sell products that average under $50. Only two brands -- Dermalogica and Lancôme -- average more expensive, around $70.
A noteworthy feature of this chart is that the bars almost group themselves into the three price ranges I grouped the brands into. From left to right, the first four brands fall into the under $25 range; the next four from $30-50; and, the last two at $70. This indicates to me that my previous groupings may be reflective of price points in the industry, and represent a good way to group skincare products of this nature.
Conclusion
To return to the three original research questions:
- Does the product price impact its rating? Not substantially. More expensive products are not likely to be higher rated.
- What are the best-rated brands? As discussed above, I developed a metric to indicate the best-rated brands (accounting for the unreliability of products with few reviews). From each price point, the best brands appear to be Garnier, Olay, and Lancôme.
- What are the best-rated moisturizers at each price point? All around, anti-aging products seem to be the highest rated. At lower price points, there appears to be more competition: there are many brands, each with one or two highly-rated products. The highly-rated brands are all represented in the best-selling products lists, suggesting they are a solid choice for any consumers seeking to test a new skincare product.
About Author
