Scrape Best Buy website to track phone rating changes

Posted on Jul 25, 2019

Github link to the project


Smart phones have become an essential part of modern life. The huge size of the smart phone market results in fierce competition between cell phone manufacturers. Cell phone companies try to roll out new phones with new features periodically to attract more buyers and win more market shares. The customer ratings are indication of the popularity of the phones, which will impact the willingness of buying for potential buyers.  Therefore, it's important to track the rating changes of phones.

How does the popular phones perform overtime? Does the release of new phones from other companies affect the ratings of the old phones? What types of phones are being sold in each carrier? To address these questions, I did web-scraping using Scrapy on the Best Buy website to get phone models, ratings, the date that rating was posted, and the carrier it is bounded. I scraped around 274 phones and got 257,870 phone ratings. 

What types of phones are popular?

To answer this question, I analyzed the features of phones on the list. The number of phones with certain feature on the list is a good indicator of the popularity for the feature.

As we can see from the above plots, Apple has the most phones on the list, followed by Android phone manufacturer Samsung and LG. The most popular memory size is 32GB, followed by 64GB and 128GB. Silver is the most favored color, followed by Gold, Rose Gold, Space Gray and the Black. Of the three main carriers, Sprint has the most phones on the list, followed by Verizon and AT&T. Sprint also has the most diverse phone listings. Apple phones are the main items each carrier is trying to sell. Google phones are currently exclusively sold by Verizon.

What's the average number of ratings each type of phone can get?

Apple phones have accumulated the highest number of customer ratings as expected. However, if consider the number of ratings per phone, Samsung has the most ratings, which is 1261 in average, followed by Apple phones, which is around 1054.

How does ratings change over time?

Iphones have generally higher ratings compared to other brands. They maintained good ratings between 4.75 and 5. Interestingly, between April 2016 and July 2016, when Samsung phones appeared on the listing, the ratings for iphones dropped and bounced back later. 

I selected the main phones released by Apple and Samsung in recent years and tracked the rating change over time. When iphone 6s came out, it had very high ratings. Then the ratings fluctuated between 4.7 and 5. Interestingly, when Galaxy S7 came out in April 2016, it had average ratings around 4.6 and as its rating went up, the rating for iphone 6s dropped. The ratings for iphone 6s reached a low point around June 2016, but it bounced back, which is accompanied by the continued declining of Galaxy S7 ratings.


Apple dominates the smart phone market by having the most phones on the list for each carrier. However, in terms of customer feedback, Samsung customers tend to have more feedback than Apple's. The competition between Apple and Samsung may affect ratings of phones that released in closed window. Rising of the ratings by one phone is accompanied by the declining of the competitor's phone.  But this effect only occurred in a short period. Eventually, iphones maintain a higher ratings than their competitors. 

About Author

Jun Kui Chen

Jun obtained Ph.D. from Columbian University in Immunology. He is currently working in a Fintech start up as a Data Analyst.
View all posts by Jun Kui Chen >

Leave a Comment

No comments found.

View Posts by Categories

Our Recent Popular Posts

View Posts by Tags

#python #trainwithnycdsa 2019 airbnb Alex Baransky alumni Alumni Interview Alumni Reviews Alumni Spotlight alumni story Alumnus API Application artist aws beautiful soup Best Bootcamp Best Data Science 2019 Best Data Science Bootcamp Best Data Science Bootcamp 2020 Best Ranked Big Data Book Launch Book-Signing bootcamp Bootcamp Alumni Bootcamp Prep Bundles California Cancer Research capstone Career Career Day citibike clustering Coding Course Demo Course Report D3.js data Data Analyst data science Data Science Academy Data Science Bootcamp Data science jobs Data Science Reviews Data Scientist Data Scientist Jobs data visualization Deep Learning Demo Day Discount dplyr employer networking feature engineering Finance Financial Data Science Flask gbm Get Hired ggplot2 googleVis Hadoop higgs boson Hiring hiring partner events Hiring Partners Industry Experts Instructor Blog Instructor Interview Job Job Placement Jobs Jon Krohn JP Morgan Chase Kaggle Kickstarter lasso regression Lead Data Scienctist Lead Data Scientist leaflet linear regression Logistic Regression machine learning Maps matplotlib Medical Research Meet the team meetup Networking neural network Neural networks New Courses nlp NYC NYC Data Science nyc data science academy NYC Open Data NYCDSA NYCDSA Alumni Online Online Bootcamp Online Training Open Data painter pandas Part-time Portfolio Development prediction Prework Programming PwC python Python Data Analysis python machine learning python scrapy python web scraping python webscraping Python Workshop R R Data Analysis R language R Programming R Shiny r studio R Visualization R Workshop R-bloggers random forest Ranking recommendation recommendation system regression Remote remote data science bootcamp Scrapy scrapy visualization seaborn Selenium sentiment analysis Shiny Shiny Dashboard Spark Special Special Summer Sports statistics streaming Student Interview Student Showcase SVM Switchup Tableau team TensorFlow Testimonial tf-idf Top Data Science Bootcamp twitter visualization web scraping Weekend Course What to expect word cloud word2vec XGBoost yelp