Scraping Sephora Dataset: An Ingredients Analysis
The skills I demoed here can be learned through taking Data Science with Machine Learning bootcamp with NYC Data Science Academy.
In 2017, Senator Chuck Schumer (NY) began campaigning for the Food and Drug Administration (FDA) to remove 1,4-dioxane from consumer products goods, citing concerns over the levels of the known carcinogen in Long Island's water supply. 1,4-dioxane is never listed as an ingredient in a product, instead it is often the byproduct of the synthesis of the ethylation of other, desired, substances.
While this effort is admirable, it's important to note that there are actually a lot of chemicals that the US allows in our cosmetics and consumer products goods that many other countries have either restricted or outright prohibited. This not only affects cosmetics, but other consumer products goods from household cleaners to dry-erase markers. The use of "fragrance" as an ingredient allows companies to hide all sorts of chemicals in our products, without listing any ingredients, under the guise of intellectual property.
As consumer awareness grows, so does the desire for "clean beauty" products: products free from some of the more commonly known "bad" chemicals like sodium lauryl sulfate, sodium laureth sulfate, phthalates, parabens, and fragrance. However, how do these products actually stand up to a list of prohibited ingredients from the European Union (EU)? How do these products stand up to common "clean" beauty standards? And does paying more actually result in a "cleaner" product? Do fewer ingredients mean "better" ingredients? I scraped Sephora to find out.
Methodology
Scraping: Sephora's website was scraped using Selenium WebDriver with some optimization with Scrapy. The full code and details can be found here. An example product page screen shot is shown above. For each product, the name, price, size (in oz, mL, or both), average rating, number of "loves", and number of reviews were scraped. In addition, the categorization was also scraped from the upper left corner: whether it was fragrance, makeup, skincare, men's, or bath & body, and any subcategorization thereof.
In the above example, the categorization would be "Hair", then "Hair Styling & Treatments" then "Hair Oil". Also, in the tabbed product information section, I scraped the "details" tab, and the "ingredients" tab. The ingredients tab would occasionally list certain "highlighted" ingredients, which would be listed above the main ingredients. In the above screen shot, the hair oil contains Bis-aminopropyl Diglycol Dimaleate, Vitis Vinifera (Grape) Seed Oil, and Fermented Green Tea Oil.
The EU's list of prohibited cosmetics ingredients can be found here.* I did my best to parse it into substance names, including alternative names, to get an ingredients list to compare to the Sephora products. However, some substances were listed as classes of compounds (for example it would list an ingredient "and its salts"), and there are often many alternative names for ingredients, so by no means did I create a comprehensive list of prohibited substances.
Prohibited Ingredients
Another caveat to consider is that a lot of the ingredients are prohibited because of where they come from: petroleum and coal by-products are often contaminated with carcinogens like benzene and 1,3-butadiene, so alkanes from petroleum are prohibited, but coconut alkanes are fine.
The list of "common" bad ingredients is from Sephora's own "Clean Sephora" claims on the ingredients lists of their products.*
Ingredients Analysis
A total of 6008 products were scraped from Sephora, of those, 5427 contained ingredients lists. Of the products with ingredients lists, 4474 (85.3%) contained at least one ingredient on one of the lists, and 3973 (75.8%) contained at least one ethylated ingredient that may contain 1,4-dioxane. Above is plotted the counts of the top ten most prevalent ingredients** found in the products from the EU's list of prohibited ingredients (left) and the list of "bad" ingredients commonly found in products (right).
(Here, H3CC stands for hydroxyisohexyl 3-cyclohexene carboxaldehyde.) Of the 5427 products, 1123 (21.4%) of them contained some ingredient from the EU's list, with coumarin having the highest prevalence. Coumarin is used as a fragrance ingredient because it smells like vanilla, and is also toxic to the kidneys and liver. H3CC is also a fragrance compound and irritant. Paraffin, petrolatum, and mineral oil, butane, and isobutane are all petroleum by-products. Styrene, acrylonitrile, and acrylamide are all carcinogens.
From the right plot, we see that 4456 (84.9%) products contain some ingredient from the "Clean Sephora" list. The most common is phenoxyethanol, which has limited data and conflicting reports of it's health effects, but is generally considered an irritant and nervous system toxicant.
EU restricts
The EU restricts its use to less than 1%. The second most common is fragrance, which as mentioned above is not an actual ingredient but a bunch of ingredients, and may contain carcinogenic aromatic compounds or endocrine disruptors like phthalates. The third most common are PEGs, which are contaminated with 1,4-dioxane. There's BHT, some other acrylated compounds, and acrylamide, which are all carcinogens or potential carcinogens. Talc is often contaminated with asbestos, and parabens are endocrine disruptors.
What if you avoid all explicitly fragranced items, like perfume and cologne? Would there be less fragrance than otherwise? Below are plotted the counts for these non-fragrance products. We see that the rate of coumarin is halved, and while the rate of fragrance is significantly lower, it is still at 40%.
One question I had was whether or not spending more on a product, or more specifically, opting for a generally more expensive brand, would increase your likelihood of getting a "clean" product. Sephora has 355 brands, and so I limited this part of the analysis to brands that had at least 10 products with ingredients and prices and sizes listed.
Price/oz
Therefore a direct comparison could be made between the median price/oz for each brand, and the percentage of products containing "bad" ingredients from that brand. I also split the data into different categories: hair, makeup, and skincare. However, as seen below, there is not a correlation between price and the likelihood of getting a "clean" product. However, Edible Beauty might actually be edible!
Product Trends
When examining the ingredients lists, it's important not to forget the "highlighted" ingredients listed at the top of the page. A lot of these were trademarked ingredients or specific to a brand, but here is a list of the counts for the top ten most popular:
It's interesting to note that hyaluronic acid and vitamin E are both considered anti-aging ingredients, and they are the two leading most common ingredients. This got me wondering how the products are being marketed, so I decided to do an N-grams analysis of the "details" tabs of the products. Below are the results for the most common three word chunks in the Sephora details tabs.
Of the 1956 skincare products, by far the most common is "fine lines and wrinkles", with half of products, followed by "dullness and uneven texture". This clearly delineates a desire to have younger, brighter looking skin. After that, there are skin type descriptions, and an emphasis on ingredient quality. This is interesting given that 85% of the products appear to have some questionable ingredient quality.
Below are the trigrams for men's skincare products. There were only 68 products. While "fine lines and wrinkles" does come up, it's only in about 25% of products. Much more popular is emphasis on ingredient quality and skin types.
Conclusions
We can see that the US has a bit of an ingredients problem, with more carcinogens in our products than I am personally comfortable with. It's important to realize that most of these ingredients are not biodegradable, and thus they do not simply go away when washed down the drain. Even if you do not have skin sensitivities, common irritants can effect those who do.
We also see the effects of societal pressure on people to stay young looking, especially on women. This is already a well-documented phenomenon, but here we can also see that this pressure results in people exposing themselves to chemicals that are not entirely safe.
*Chemical lists:
The EU's list is quite comprehensive, including many we would probably never consider finding in our cosmetics, like narcotics. It includes common allergens, irritants, carcinogens, mutagens, teratogens, radioactive substances, endocrine disruptors, and human cells of any kind. It also includes substances, like petroleum by-products, that could be contaminated with carcinogens.
For each substance, there was listed the "chemical name" which often contained classes of compounds, like "Chromium (IV) compounds" which could be unfortunately vague. There were also lists of "known ingredients" to help with finding some of these product names. I parsed both columns into one long list. (Chromium compounds are pretty common in green pigments at Sephora, but those happen to be chromium (III) compounds, and are thus not prohibited by the EU.)
Ingredients
The "Clean Sephora" list includes phthalates, parabens, formaldehyde, formaldehyde-producing-substances, mineral oil, retinyl palmitate, oxybenzone, coal tar, hydroquinone, triclosan, triclocarbon, ethyl methacrylate, butyl methacrylate, hydroxypropyl methacrylate, tetrahydrofurfuryl trimethacrylate, aluminum salt, musk (from animals), benzophenone, butoxyethanol, carbon black, lead, lead acetate, methyl cellosolve, methoxyethanol, methylchloroisothiazolinone,
methylisothiazolinone, mercury, thimerisol, resorcinol, talc, toluene, butylated hydroxyanisole, BHA, butylated hydroxytoluene, BHT, ethanolamine, ethanolamine DEA, ethanolamine TEA, ethanolamine MEA, ethanolamine ETA, petrolatum, paraffin, phenoxyethanol, polyacrylamide, acrylamide, bromostyrene, deastyrene, acrylates, divinylbenzene (DVB) copolymer, sodium styrene, styrene oxide, and styrene. To this list I also added polyethylene glycols (PEGs), because a common by-product of their synthesis is 1,4-dioxane.
** The other ingredients prohibited by the EU found in Sephora products were phytonadione, diethylene glycol, phosphorus, hydrocarbons, galactaric acid, sodium borate, ficus carica extract, citrus reticulata peel extract, tagetes erecta flower extract, citrus reticulata leaf oil, citrus grandis peel oil, vinylidene chloride, DEA-oleth-3 phosphate, avobenzone, phenol, and formic acid.
Other ingredients found from the Sephora list in Sephora products were oxybenzone, retinyl palmitate, petrolatum, BHA, styrene, mineral oil, benzophenone, polyacrylamide, phthalate, methylisothiazolinonem, resorcinol, methylchloroisothiazolinone, ethanolamine, triclosan, carbon black, hydroquinone, musk, toluene, formaldehyde, sodium styrene, hydroxypropyl methacrylate, and butyl methacrylate.
About Author
Jennifer Ruddock
View all posts by Jennifer Ruddock >Leave a Comment
