Uncovering What Makes a Great Running Shoe? Scraping Zappos

Posted on Oct 19, 2019


In the shoe market there are many categories; athletic, running, casual, boots, hiking, etc. One thing is certain, people love shoes. Additionally, I find people are most critical of shoes that require performance more than just style. For that reason I wanted to find out, what makes a great men's running shoe?

Zappos has a plethora of running shoes and many different ratings systems. They offer written customer reviews, overall shoe rating, total number of customer reviews, favorite likes, true-to-size feeling, true-to-width feeling and arch support. With all of these different ratings I was curious to find out which were the top brands and models. What do customers have to say about the highest rated shoes? What makes a running shoe stand out from the rest? Is it style, traction, light weight, price, fitment, or just plain comfort? Before we get to the results, let’s take a look a my method for scraping.



For scraping Zappos I used the automation testing framework Selenium. I was able to scrape the entire men's running shoe inventory, totaling over 500 models. The data was cleaned then loaded into a python environment to gain useful insights. Figure 1(a) shoes a web page sample. The info gathered here is brand, model, price, overall rating, and favorite likes. Figure 1(b) shows the details page where the info for total number of reviews, true-to-size, true-to-feel, and arch support were collected. Additionally customer reviews were collected for the top 4 models was collected. 


Figure 1(a): Web page sample. Figure 1(b): Model detail page sample




Figure 2: Overall ratings density plot for the top 5 brands


Above in figure 2 is a density plot of the overall ratings for the top 5 brands. This was determined by the summing the overall ratings for each model of a given brand. However this did not factor in the number of reviews per model. That will be addressed shortly.

Immediately we see Asics stands out with a large peak in the 4/5 category however, falls short in the 5/5 star category.  Weighing in both 4 and 5 star categories all 5 brands are close, but we can see the winner is Nike.

I wanted to find out how these top 5 brands ranked with other brands in price, number of reviews, and likes. Below we can see the results on horizontal bar graphs, in figures 3(a), 3(b), and 3(c).



 Figure 3(a): Horizontal Bar chart of price


We see from figure 3(a) that all top 5 brands price point fall in the middle to lower range. We can conclude that the average runner cares about quality but only within a certain price range.



 Figure 3(b): Horizontal Bar chart of number of reviews


We see from figure 3(b) that all top 5 brands rank in the top for number of customer reviews. This makes sense as we imagine people have the most to say about something they really like. And this same logic goes for things that people really dislike too.


Figure 4(a): Scatter plot of likes vs price. Figure 4(b): Scatter plot of likes vs price


Figure 4(a) is a scatter plot of likes vs price. We notice the highest number of likes, in the 1000+ range, are the in the price range of  50$ to 125$. Figure 4(b) is a scatter plot of overall rating vs likes. We see this plot verifies our intuition that likes more are associated with higher ratings.

It is worth mentioning there was a small surprise. That the most liked shoe has a 4/5 star rating. Upon looking into this matter, I found that this was a new model shoe that fell short of its beloved predecessor.



Top 7 models:

Figure 5: Top 5 running shoe models


Figure 5  is a bar chart of the top 5 running shoe models and displays the 4 categories of ranking; 5/5 star rating, true-to-size, true-to-width, and arch support. These results are a weighted average of the number of customer reviews. We can see there are really only 2 top running models. Top ranked is Brooks Ghost 11, and Second is the New Balance M990V4.

I wanted to gain insight into what customers were saying about these two top ranked shoes along with the next two following, Salomon XA Pro 3D and the Hoka One One Bondi 6. I did this by generating word clouds for each model from all of the associated customer reviews. I made sure to filter out general words, and additionally for each model I filtered out brand name and model. Figures 6(a,b,c,d) show the results respectively.


Figure 6: Top 4 models word clouds


From the word clouds we see a few words stand out. Comfortable, fit and great. This also agrees with what qualities we imagine a performance shoe should have. That is it should be comfortable and fit perfectly. However I was surprised to find that no words associated with style or aesthetic bubbled to the top. 



From these results it's easy to verify that our general intuition about what qualities we seek in a running shoe are true. We seek comfort and fitment most. Although, it was surprisingly to find that style doesn't play a bigger role it the top models.  

Further these results agree with our intuition in that higher rated shoes are associated with more likes and number of customer reviews.


Future Work:

There is a whole other side to explore, women's running shoes. I’m curious how these results would compare to men’s running shoes. Would they simply be the same or would other factors drive a specific shoe to the top?Additionally I’d like to explore the subcategories of running shoes; trail running and regular running shoes. Also, how does the color pallet weigh in on customer’s preferences? And lastly I'd like to analyze bad reviews. What customer's don't like  



Thanks for reading!



About Author

Leave a Comment

No comments found.

View Posts by Categories

Our Recent Popular Posts

View Posts by Tags

#python #trainwithnycdsa 2019 airbnb Alex Baransky alumni Alumni Interview Alumni Reviews Alumni Spotlight alumni story Alumnus API Application artist aws beautiful soup Best Bootcamp Best Data Science 2019 Best Data Science Bootcamp Best Data Science Bootcamp 2020 Best Ranked Big Data Book Launch Book-Signing bootcamp Bootcamp Alumni Bootcamp Prep Bundles California Cancer Research capstone Career Career Day citibike clustering Coding Course Demo Course Report D3.js data Data Analyst data science Data Science Academy Data Science Bootcamp Data science jobs Data Science Reviews Data Scientist Data Scientist Jobs data visualization Deep Learning Demo Day Discount dplyr employer networking feature engineering Finance Financial Data Science Flask gbm Get Hired ggplot2 googleVis Hadoop higgs boson Hiring hiring partner events Hiring Partners Industry Experts Instructor Blog Instructor Interview Job Job Placement Jobs Jon Krohn JP Morgan Chase Kaggle Kickstarter lasso regression Lead Data Scienctist Lead Data Scientist leaflet linear regression Logistic Regression machine learning Maps matplotlib Medical Research Meet the team meetup Networking neural network Neural networks New Courses nlp NYC NYC Data Science nyc data science academy NYC Open Data NYCDSA NYCDSA Alumni Online Online Bootcamp Online Training Open Data painter pandas Part-time Portfolio Development prediction Prework Programming PwC python python machine learning python scrapy python web scraping python webscraping Python Workshop R R language R Programming R Shiny r studio R Visualization R Workshop R-bloggers random forest Ranking recommendation recommendation system regression Remote remote data science bootcamp Scrapy scrapy visualization seaborn Selenium sentiment analysis Shiny Shiny Dashboard Spark Special Special Summer Sports statistics streaming Student Interview Student Showcase SVM Switchup Tableau team TensorFlow Testimonial tf-idf Top Data Science Bootcamp twitter visualization web scraping Weekend Course What to expect word cloud word2vec XGBoost yelp