Scraping Ulta with Scrapy

Casey Hoffman
Posted on Feb 22, 2021

GitHub | LinkedIn 

Background

Skincare is more popular than ever. While beauty sales have fallen in 2020, skincare products, including face washes and moisturizers, have only increased in popularity. From February to March 2020, some skincare product sales increased by over 600% (Statista). Within these products, moisturizers alone generate $700 million annually, in the US. 

For my web scraping project, I chose to investigate trends in facial moisturizers sold on Ulta.com. Given the popularity of this industry (and my personal love for skincare), I wished to uncover and visualize to better understand information about these products.

Ulta.com is an all-in-one beauty retailer which prides itself on its immense product varieties. Ulta offers products at a wider range of price points than just about anyone else -- from drugstore to high-end. This massive variety made it preferable to similar retailers.

I had three main research questions to guide my analyses:

  1. Does the product price impact its rating? Specifically, I was curious if less expensive products had poorer ratings. I hypothesized that relatively cheaper products may be made with ingredients of slightly lower quality.
  2. What are the best-rated brands?
  3. What are the best-rated moisturizers at each price point?

The Web Scraping Process

I used scrapy to scrape the website. All code was written in Python. On each product result page, I parsed the following information from each item: product name, brand name, number of reviews, and average rating (out of 5 stars).

Data Cleaning and Analysis

Using a Jupyter notebook, I imported, cleaned, and analyzed the data.

Some of the products did not have any ratings. These were removed from the analyses. I also removed products that had less than 10 ratings, as these ratings were deemed to be not particularly reliable (discussed in depth below). Data cleaning was completed using numpy and pandas. Plots were produced using matplotlib and seaborn.

Does Product Price Impact Rating?

A scatterplot with a linear regression line, plotting the association between price in USD and mean rating. The data does not fit a linear pattern.

The short answer: most likely, not.

One initial question I had was whether a relationship existed between a product's price and its average rating. I produced a scatterplot fit with a simple linear regression line, and came to the conclusion that there does not appear to be a substantial association between the two measures.

I was less certain that no linear relationship exists, and more certain that this dataset was not optimal for answering the question. One possible explanation for this is the unbalanced aspect of the dataset. There are far too few observations of higher-priced products (and their reviews) to assess if such a pattern exists. This lead me to investigate the distribution of prices in the dataset.

Distribution of Prices

A histogram of the scraped products' prices, in USD. The blue vertical line represents the median.

I produced a scatterplot in order to analyze the distribution of moisturizer prices. The median price (indicated with a blue line) is around $30. As suspected, the data was highly skewed -- the majority of products were between $20 and $40, with a few products costing as much as $100 or more.

This is to be expected, to some degree. There are simply more moisturizers that cost less. However, I suspected that this imbalance in the data may mask patterns that would otherwise appear. In order to produce further insights of value, I concluded I should group the data by price. This would allow for me to analyze products within a certain range, and between these ranges. I grouped the products into one of three groups by price range: Under $25, $25 to $50, and Over $50. 

Review Distribution by Price Group

A boxplot displaying the distribution of ratings for the three price ranges. Left to right: $25-$50, Under $25, Over $50. The points outside boxes represent outliers.

Now that the products were grouped by cost, I produced a boxplot to analyze the distribution of ratings within the three price ranges. The entire distribution rises ever so slightly, as one moves through the price ranges in ascending order. Products under $25 have the lowest median (approximately 4.4); the $25-$50 range has a median that is ever so slightly greater (approximately 4.5). The distribution of the products over $50 has a median of about 4.5 as well. However, this distribution almost appears to be entirely shifted up from the others. Put another way, the group of products that cost more than $50 possess (relatively) more products with ratings above 4.5.

Overall, the difference in mean product review is slight. I would not claim that more expensive products yield higher ratings.

Identifying Bestselling Products

Take two hypothetical products:

  • Product A - Rating: 4.9 stars, number of ratings: 2
  • Product B - Rating: 4.7 stars, number of ratings: 3,000

Product B has a lower mean rating than Product A; however, Product B's rating is much more trustworthy.

My next goal was to determine the bestselling products (and by extension, bestselling brands). For the reason given above, I decided not to rely on solely a product's rating. To better identify bestselling products, I developed a "bestseller rating" metric, derived from a product's rating as well as the number of reviews in each product rating. 

First, I identified the top 10 bestselling products in each of the three price ranges. 

Bestselling Products at Each Price Point

This barchart displays the Top 10 bestselling moisturizers, costing less than $25, with their mean rating.

The Top 10 products under $25 are charted above. The highest-rated of this group is Neutrogena's Hydro Boost Gel-Cream. Notably, the brands Neutrogena and L'Oréal each have multiple products in this ranking. This suggests they are two of the most popular brands at this price point.

This barchart displays the Top 10 bestselling moisturizers, priced between $25 and $50, with their mean rating.

Moving on to products costing $25-$50, the market appears to narrow. The highest-rated of this group is Olay's Regenerist Micro Sculpting Cream. Only 4 brands are represented in this list; with Clinique and Olay dominating. I suspect many products at this point are "tried and true" -- purchasers may look to a few trusted brands to buy moisturizers at this price point.

This barchart displays the Top 10 bestselling moisturizers over $50, with their mean rating.

Above $50, there are a variety of brands represented here. Only two brands have more than one product in this ranking, suggesting the market is more varied. The highest-rated of this group is Dermalogica's Dynamic Skin Recovery Broad Spectrum SPF 50. Interestingly, there is more variety in ratings for this price point's bestseller list than in the other two lists. All products display ratings over 4 stars; however, it seems certain that higher prices do not indicate higher ratings.

Bestselling Brands

The charts below detail information on the Top 10 bestselling brands. By aggregating the bestseller ratings of each product by brand, I determined the top 10 bestseller-rated brands.

Bar chart of mean product ratings for the top 10 bestselling brands.

Unsurprisingly, all of the bestselling brands average above 4 stars for their listed products. Neutrogena has the relatively lowest rating, and Olay has the relatively greatest rating.

Bar chart of mean product cost for the top 10 bestselling brands.

The majority of the brands above sell products that average under $50. Only two brands -- Dermalogica and Lancôme -- average more expensive, around $70. 

A noteworthy feature of this chart is that the bars almost group themselves into the three price ranges I grouped the brands into. From left to right, the first four brands fall into the under $25 range; the next four from $30-50; and, the last two at $70. This indicates to me that my previous groupings may be reflective of price points in the industry, and represent a good way to group skincare products of this nature.

Conclusion

To return to the three original research questions:

  1. Does the product price impact its rating? Not substantially. More expensive products are not likely to be higher rated.
  2. What are the best-rated brands? As discussed above, I developed a metric to indicate the best-rated brands (accounting for the unreliability of products with few reviews). From each price point, the best brands appear to be Garnier, Olay, and Lancôme.
  3. What are the best-rated moisturizers at each price point? All around, anti-aging products seem to be the highest rated. At lower price points, there appears to be more competition: there are many brands, each with one or two highly-rated products. The highly-rated brands are all represented in the best-selling products lists, suggesting they are a solid choice for any consumers seeking to test a new skincare product.

About Author

Casey Hoffman

Casey Hoffman

Casey Hoffman is an experienced data professional with a background in academic research and higher education. She holds an M.A. and B.A. in Experimental Psychology from New York University. Casey is passionate about solving real-world problems through the...
View all posts by Casey Hoffman >

Leave a Comment

No comments found.

View Posts by Categories


Our Recent Popular Posts


View Posts by Tags

#python #trainwithnycdsa 2019 airbnb Alex Baransky alumni Alumni Interview Alumni Reviews Alumni Spotlight alumni story Alumnus API Application artist aws beautiful soup Best Bootcamp Best Data Science 2019 Best Data Science Bootcamp Best Data Science Bootcamp 2020 Best Ranked Big Data Book Launch Book-Signing bootcamp Bootcamp Alumni Bootcamp Prep Bundles California Cancer Research capstone Career Career Day citibike clustering Coding Course Demo Course Report D3.js data Data Analyst data science Data Science Academy Data Science Bootcamp Data science jobs Data Science Reviews Data Scientist Data Scientist Jobs data visualization Deep Learning Demo Day Discount dplyr employer networking feature engineering Finance Financial Data Science Flask gbm Get Hired ggplot2 googleVis Hadoop higgs boson Hiring hiring partner events Hiring Partners Industry Experts Instructor Blog Instructor Interview Job Job Placement Jobs Jon Krohn JP Morgan Chase Kaggle Kickstarter lasso regression Lead Data Scienctist Lead Data Scientist leaflet linear regression Logistic Regression machine learning Maps matplotlib Medical Research Meet the team meetup Networking neural network Neural networks New Courses nlp NYC NYC Data Science nyc data science academy NYC Open Data NYCDSA NYCDSA Alumni Online Online Bootcamp Online Training Open Data painter pandas Part-time Portfolio Development prediction Prework Programming PwC python python machine learning python scrapy python web scraping python webscraping Python Workshop R R language R Programming R Shiny r studio R Visualization R Workshop R-bloggers random forest Ranking recommendation recommendation system regression Remote remote data science bootcamp Scrapy scrapy visualization seaborn Selenium sentiment analysis Shiny Shiny Dashboard Spark Special Special Summer Sports statistics streaming Student Interview Student Showcase SVM Switchup Tableau team TensorFlow Testimonial tf-idf Top Data Science Bootcamp twitter visualization web scraping Weekend Course What to expect word cloud word2vec XGBoost yelp