Web Scraping and Analysis of CPU and Graphic Cards on Newegg.com

Posted on Nov 21, 2016

As 2016 Black Friday is just around the corner, this is the right time for people to upgrade their desktop. As a computer fan for years, I am aiming to explore the hardware that are selling trend on Newegg.com for my web scraping project. The questions that I am trying to answer are:

Which brand of CPU and GPU that currently sells the best, and how does the parameters of the hardware affect the price?

1. Introduction

When someone wants to check out a newly released hardware, Newegg.com is always the right place to go. The broad product line, great deals and good service makes this online retailer very popular among computer fans. The highly organized web structure also makes it well-suited for getting and comparing hardware parameters. Therefore I decided to scrape this website for all the data needed for analysis.


Figure 1. Left: AMD and Intel are competing for CPU market. Right: Nvidia's flagship graphic card---Titan X Pascal

CPU and graphic card, in my opinion, are top-2 hardware in a desktop that directly determine its computing power. Hundreds of channels can be found on Youtube that benchmarking the gaming performance for each CPU and graphic card product everyday. Besides video gamers, data scientists need good computers too. A good CPU and save great amount of computation time when working on a relatively large dataset. More interestingly, GPU computing via graphic cards is going mainstream at this moment. The reason behind this is that a modern GPU will have far more cores than a CPU, thus make GPU well-suited for parallel computing. Today more and more people use high-end graphic cards to perform heavy computational work, like training models for neural network.

2. Web Scraping


Figure 2. Introduction to Newegg pages and the web structure

As shown above, when searching for a type of desktop components, a page consisting of 36 products will be displayed. The URL contains the information like page number, number of products in a page, and the criteria for sorting the products. The main panel consists of 36 icons. When clicking the "View Detail" bottom, you will be directed to single product page. In this page, most of the parameters can be found under "Specification" tab.


Figure 3. Web scraping plan for extracting CPU and graphic cards information

With the web structure in mind, the strategy for scraping is clear. Python Scrapy package was selected for web scraping job. The workflow of web spider can be briefly described as:

  • Generate a list of URLs ranging from the first page to the last page.
  • At each page, grab the customer rating data, generate a ranking number for each product, filter the undesired (Refurbished and Open box) item, and extract the URL for each product.
  • Go to each product page, grab the hardware parameters under "Specification" Tab, and yield the item info.

It should be noted that the product information becomes less organized in the latter pages. Such case are handled by multiple "try/except" statements. The complete code for web scraping can be found here.

3. Data

After web scraping from Newegg.com, the data were cleaned using this R script.

For CPU data, the key variables that are included in analysis are:

name brand series # cores freq (GHz) power (W) price ($) URL rating rank
chr chr factor int num num num chr factor int

For graphic cards data, the key variables are:

name brand chip gpu coreclock (MHz) memory size price ($) URL rating rank
chr chr factor chr num num num chr factor int

Other variables like L3/L2 cache for CPU, and memory clock/interface for graphic cards are not included in data analysis.

4. Visualization

In this part, exploratory graphs are drawn to visualize the selling trend of CPUs and graphic cards. In terms of the overall market distribution, AMD and Intel are competing for dominance in CPU area, while AMD and Nvidia are two biggest GPU chip manufacturers.


Figure 4. Distribution of CPU and graphic cards rating

First let's look at the customer rating distribution of CPU and graphic card that are currently selling on Newegg. For CPU, the customer ratings is ranging from 4 points to 5 points, indicating that CPU is overall reliable for nearly all products. On the other hand, some graphic card get rated only 1 points and 3 points, meaning the quality for graphic cards varies from product to product. When looking at the overall markets distribution, the number of Intel products is greater than AMD CPU. As for graphic card market, AMD fall behind Nvidia.


Figure 5. Barplot of graphic cards distribution against brands

From the barplot above, there are 10 companies sharing the graphic card markets. Some of them, like Powercolor and Sapphire, stand on the AMD side, while others like EVGA and PNY adopt Nvidia GPUs only. It is interesting to note that for companies that use both GPUs, they tends to have more Nvidia chip products over AMD chip.


Figure 6. Count and average price of CPUs against series

From the figure above, we can see that in top-100 best selling CPUs, although the average price of Intel product is higher than it opponent AMD, it is still leading in popularity.


Figure 7. Price of CPUs versus selling rank

Above is the product price plotting against selling rank for top-50 CPU products. The average price increases from $200 to $500 as the rank increases to around 25, and drop below 200 when reaching rank 50. There are three Intel products ranking between 20 and 30 that are selling at above $1000 dollars, thus pulling up the average price in this region.


Figure 8. Boxplot showing price of CPUs against number of cores

As shown from the boxplot above, a positive correlation can be observed between price and number of cores. We can also clearly observe the price gap between two CPU manufactures, and such gap becomes more clear at higher number of cores.

5. Statistical Analysis

In this section, the relationship between the price of CPU products and their parameters is explored. First let's look at the correlation plot including all numeric variables in the CPU dataset.


Figure 9. Correlation plot of CPU dataset.

It is interesting to note that the price of CPU almost has no correlation with the operating frequency, meaning that the operating frequency no longer reflects the computing power. Today CPU manufactures direct their attention on cores, threads and inner architecture to increase the performance, instead of simply adding clock speed.

Now let's see if we can use linear regression to model the price of CPUs against other variables. First let's check the normality of the price distribution.


Figure 10. Q-Q plot of price of CPUs

From the Q-Q plot above, the prices are not quite normally distributed. Therefore we do a box-cox transformation.


Figure 11. Log-likelihood vs. lambda indicating 95% confidence interval about the maximum observed value for lambda

From figure 11, we take the natural log of response value to increase the normality. For variable selection, CPU brands, series, number of core, power consumption and operating frequency are selected as the full model. Then AIC was applied to select the best linear model via step function in R. The variables for final model are series, power and number of cores.


Figure 12. Results of linear regression

Increasing of the power consumption will slightly increase the price, and CPU will cost a lot more when adding more cores to it. According to the p-value of the F-test, the overall regression model is significant, where 87% of the variability can be explained.

6. Conclusion and Future work

In summary, all CPU and graphic card product information was scraped from Newegg.com using Python Scrapy module. the data showed that the price of the CPU have a negligible correlation with the operating frequency. When looking at both CPU and graphic card data, although AMD has a wide range of product line, it is dominated by Intel and Nvidia in CPU and GPU market, respectively. A linear regression model is selected to model the natural log of price using CPU series, core and power consumption as predictors, which can explain 87% percent of variability.

In the future, this work can be expanded as follows:

  • Build an interactive app to allow others to explore and subset the dataset.
  • Scrape more features from the web, like product release date and customer reviews.
  • Apply machine learning algorithms to predict the user rating of a product based on the hardware parameters and reviews.

About Author

Xinyuan Wu

Xinyuan recently obtained his Ph.D. from North Carolina State University. He gained quantitative analysis, statistical knowledge and critical thinking from years of research on magnetic and photophysical chemistry. His belief in the trend of predictive analysis, along with...
View all posts by Xinyuan Wu >

Related Articles

Leave a Comment

Edna July 14, 2017
Thanks, it's quite informative

View Posts by Categories

Our Recent Popular Posts

View Posts by Tags

#python #trainwithnycdsa 2019 2020 Revenue 3-points agriculture air quality airbnb airline alcohol Alex Baransky algorithm alumni Alumni Interview Alumni Reviews Alumni Spotlight alumni story Alumnus ames dataset ames housing dataset apartment rent API Application artist aws bank loans beautiful soup Best Bootcamp Best Data Science 2019 Best Data Science Bootcamp Best Data Science Bootcamp 2020 Best Ranked Big Data Book Launch Book-Signing bootcamp Bootcamp Alumni Bootcamp Prep boston safety Bundles cake recipe California Cancer Research capstone car price Career Career Day citibike classic cars classpass clustering Coding Course Demo Course Report covid 19 credit credit card crime frequency crops D3.js data data analysis Data Analyst data analytics data for tripadvisor reviews data science Data Science Academy Data Science Bootcamp Data science jobs Data Science Reviews Data Scientist Data Scientist Jobs data visualization database Deep Learning Demo Day Discount disney dplyr drug data e-commerce economy employee employee burnout employer networking environment feature engineering Finance Financial Data Science fitness studio Flask flight delay gbm Get Hired ggplot2 googleVis H20 Hadoop hallmark holiday movie happiness healthcare frauds higgs boson Hiring hiring partner events Hiring Partners hotels housing housing data housing predictions housing price hy-vee Income Industry Experts Injuries Instructor Blog Instructor Interview insurance italki Job Job Placement Jobs Jon Krohn JP Morgan Chase Kaggle Kickstarter las vegas airport lasso regression Lead Data Scienctist Lead Data Scientist leaflet league linear regression Logistic Regression machine learning Maps market matplotlib Medical Research Meet the team meetup methal health miami beach movie music Napoli NBA netflix Networking neural network Neural networks New Courses NHL nlp NYC NYC Data Science nyc data science academy NYC Open Data nyc property NYCDSA NYCDSA Alumni Online Online Bootcamp Online Training Open Data painter pandas Part-time performance phoenix pollutants Portfolio Development precision measurement prediction Prework Programming public safety PwC python Python Data Analysis python machine learning python scrapy python web scraping python webscraping Python Workshop R R Data Analysis R language R Programming R Shiny r studio R Visualization R Workshop R-bloggers random forest Ranking recommendation recommendation system regression Remote remote data science bootcamp Scrapy scrapy visualization seaborn seafood type Selenium sentiment analysis sentiment classification Shiny Shiny Dashboard Spark Special Special Summer Sports statistics streaming Student Interview Student Showcase SVM Switchup Tableau teachers team team performance TensorFlow Testimonial tf-idf Top Data Science Bootcamp Top manufacturing companies Transfers tweets twitter videos visualization wallstreet wallstreetbets web scraping Weekend Course What to expect whiskey whiskeyadvocate wildfire word cloud word2vec XGBoost yelp youtube trending ZORI