Data Scraping Best Buy website to track phone rating changes
The skills the author demoed here can be learned through taking Data Science with Machine Learning bootcamp with NYC Data Science Academy.
Github link to the project
Introduction
Smart phones have become an essential part of modern life. The huge size of the smart phone market results in fierce competition between cell phone manufacturers. Cell phone companies try to roll out new phones with new features periodically to attract more buyers and win more market shares. The customer ratings are indication of the popularity of the phones, which will impact the willingness of buying for potential buyers. Therefore, it's important to track the data on rating changes of phones.
How does the popular phones perform overtime? Does the release of new phones from other companies affect the ratings of the old phones? What types of phones are being sold in each carrier? To address these questions, I did web-scraping using Scrapy on the Best Buy website to get phone models, ratings, the date that rating was posted, and the carrier it is bounded. I scraped around 274 phones and got 257,870 phone ratings.
Data on Popular Phones
To answer this question, I analyzed the features of phones on the list. The number of phones with certain feature on the list is a good indicator of the popularity for the feature.
As we can see from the above plots, Apple has the most phones on the list, followed by Android phone manufacturer Samsung and LG. The most popular memory size is 32GB, followed by 64GB and 128GB. Silver is the most favored color, followed by Gold, Rose Gold, Space Gray and the Black. Of the three main carriers, Sprint has the most phones on the list, followed by Verizon and AT&T. Sprint also has the most diverse phone listings. Apple phones are the main items each carrier is trying to sell. Google phones are currently exclusively sold by Verizon.
Data on Average Rating
Apple phones have accumulated the highest number of customer ratings as expected. However, if consider the number of ratings per phone, Samsung has the most ratings, which is 1261 in average, followed by Apple phones, which is around 1054.
Data on Rating Changes Over Time
Iphones have generally higher ratings compared to other brands. They maintained good ratings between 4.75 and 5. Interestingly, between April 2016 and July 2016, when Samsung phones appeared on the listing, the ratings for iphones dropped and bounced back later.
I selected the main phones released by Apple and Samsung in recent years and tracked the rating change over time. When iphone 6s came out, it had very high ratings. Then the ratings fluctuated between 4.7 and 5. Interestingly, when Galaxy S7 came out in April 2016, it had average ratings around 4.6 and as its rating went up, the rating for iphone 6s dropped. The ratings for iphone 6s reached a low point around June 2016, but it bounced back, which is accompanied by the continued declining of Galaxy S7 ratings.
Conclusion
Apple dominates the smart phone market by having the most phones on the list for each carrier. However, in terms of customer feedback, Samsung customers tend to have more feedback than Apple's. The competition between Apple and Samsung may affect ratings of phones that released in closed window. Rising of the ratings by one phone is accompanied by the declining of the competitor's phone. But this effect only occurred in a short period. Eventually, iphones maintain a higher ratings than their competitors.
About Author
