eBay Scraping using BeautifulSoup in Python
The skills I demoed here can be learned through taking Data Science with Machine Learning bootcamp with NYC Data Science Academy.
Even though Amazon is the leader in e-commerce marketplaces – eBay still has its fair share in the online retail industry. Brands selling online should be monitoring prices on eBay as well to gain a competitive advantage.
Extracting data from eBay at a huge scale regularly is a challenging problem for data scientists. Here is an example of scraping eBay using python to identify prices of mobile phones.
Lets us imagine a use case where you need to monitor pricing of a product, say a mobile phone from eBay. Also, you want to visualise the range of price offering available on the mobile phone which you want to monitor. Moreover, you have other mobile phones under consideration so you may also want to compare their prices as well. In this blog, we will be scraping eBay to collect the prices of phones and find out the difference between their offerings on the eBay website.
Web scraping is an efficient mechanism for collecting data from the internet. You can also read about different myths about web scraping here.
Scraping eBay step by step
In this section, we will walk you through the step by step process of scraping eBay for products and their prices.
1.Selecting the required information
The very first task in web scraping is to identify the target web page. It is the web page from which you need to extract all the required information. We will be scraping eBay for the product listings so we can just open the eBay website and type our product in their search bar and hit enter. Once page loads with all the product listing of that product, all you need to do is pull that URL out from the browser.
This URL will be our target URL. In our case, the URL will be “https://www.ebay.com/sch/i.html?_from=R40&_nkw=galaxy+note+8&_sacat=0&_pgn=1“. Notice the two parameters in this URL i.e. “nkw” (new keyword) and “pgn” (page number) parameter. These parameters in the URL defines the search query. If we change “pgn” parameter to 2, then it will open the second page of the product listings for galaxy note 8 phone and if we were to change “nkw” to iPhone X then eBay will search for iPhone X and will show you the corresponding results.
2. Finalising the tags for extraction
Once we have finalised the target web page, we need to understand its HTML layout to scrape the results out. This is the most important and critical part of the web scraping and basic HTML knowledge is a pre-requisite for this step. When on the target web page, do “inspect element” and open the developer tools window or just do CTRL+SHIFT+I. In the new window, you will find the source code of the target web page.
In our case, all the products are mentioned as list elements so we have to grab all these lists. In order to grab an HTML element, we need to have an identifier associated with it. It can be an id of that element or any class name or any other HTML attribute of the particular element. We are using the class name as the identifier. All the lists have the same class name i.e. s-item. On further inspection, we got the class names for the product name and product price which are “s-item__title” and “s-item__price” respectively. With this information, we have successfully completed step 2!
3. Putting the scraped data in a structured format
After having our extractors/identifiers we only need to extract specific portions out from the HTML content. Once this is done, we need to organise this data into a suitable structured format. We will be creating a table where we will have all the product names in one column and their prices in the other.
4. Visualising the results (optional)
Since we are to compare the price offerings on two different mobile phones, we will visualise the results too. This is not a mandatory step for web scraping but is more of a process to turn your collected data into some actionable insights. We will be plotting boxplots to understand the distribution of the price offerings on both galaxy note 8 and iPhone 8 mobile phones.
Required libraries and Installation
To implement web scraping for this use case, you will need python, pip (package installer for python) and BeautifulSoup library in python for web scraping. You will also need pandas and numpy library to organise the collected data into a structured format.
- Installing Python and PIP
Depending upon your operating system, you can follow this blog link to setup python and Pip in your system. - Installing Beautiful soup library
apt-getinstallpython-bs4pipinstallbeautifulsoup4 - Installing pandas and numpy
pipinstallpandaspipinstallnumpyWe are done setting up our environment and now can begin with the scraping implementation using python. Implementation consists of the steps discussed in the earlier section.
-
Python implementation for scraping eBay
In this section, we will perform two scraping operations i.e. one for the iPhone 8 and other for the galaxy note 8 mobile phones. Implementation has been repeated for the two mobile phones for easier comprehension. A more optimised version can have two separate scrapping activity combined into one which is not required right now though.
Scrapping eBay for Galaxy Note 8 products
item_name=[]prices=[]foriinrange(1,10):ebayUrl="https://www.ebay.com/sch/i.html?_from=R40&_nkw=note+8&_sacat=0&_pgn="+str(i)r=requests.get(ebayUrl)data=r.textsoup=BeautifulSoup(data)listings=soup.find_all('li', attrs={'class':'s-item'})forlistinginlistings:prod_name=" "prod_price=" "fornameinlisting.find_all('h3', attrs={'class':"s-item__title"}):if(str(name.find(text=True, recursive=False))!="None"):prod_name=str(name.find(text=True, recursive=False))item_name.append(prod_name)if(prod_name!=" "):price=listing.find('span', attrs={'class':"s-item__price"})prod_price=str(price.find(text=True, recursive=False))prod_price=int(sub(",","",prod_price.split("INR")[1].split(".")[0]))prices.append(prod_price)fromscipyimportstatsimportnumpy as npdata_note_8=pd.DataFrame({"Name":item_name,"Prices": prices})data_note_8=data_note_8.iloc[np.abs(stats.zscore(data_note_8["Prices"]))<3,]Collected Data for Galaxy note 8
Scrapping eBay for iPhone 8
item_name = []prices = []for i in range(1,10): r= requests.get(ebayUrl) data=r.text soup=BeautifulSoup(data) listings = soup.find_all('li', attrs={'class': 's-item'}) for listing in listings: prod_name=" " prod_price = " " for name in listing.find_all('h3', attrs={'class':"s-item__title"}): if(str(name.find(text=True, recursive=False))!="None"): prod_name=str(name.find(text=True, recursive=False)) item_name.append(prod_name) if(prod_name!=" "): price = listing.find('span', attrs={'class':"s-item__price"}) prod_price = str(price.find(text=True, recursive=False)) prod_price = int(sub(",","",prod_price.split("INR")[1].split(".")[0])) prices.append(prod_price)from scipy import statsimport numpy as npdata_note_8 = pd.DataFrame({"Name":item_name, "Prices": prices})data_note_8 = data_note_8.iloc[np.abs(stats.zscore(data_note_8["Prices"])) < 3,] |
Collected data for iPhone 8
scraping eBay | Iphone data
Visualising the price of products
Now is the time to visualise the scraped results. We will be using the boxplots to visualise the distribution of prices of mobile phones. Box plot helps us in visualising a trend in numerical values. The green line is the median of the collected price data. The box extends from the Q1 to Q3 quartile values of the data, with a line at the median (Q2). The whiskers extend from the edges of the box to show the range of the data.
scraping eBay | Price Comparison
For iPhone 8, most of the prices lie between INR 25k-35k whereas most of the galaxy note 8 phones are available in the price range of 25k-30k. However, variation in prices of iPhone 8 is much more than galaxy note 8. iPhone 8 is available for minimum INR 15k on eBay whereas the minimum cost of galaxy note 8 on eBay is around 22-23K INR!
In this blog, we successfully used python for scraping eBay for two different products and their pricing. We also compared the available prices for galaxy note 8 and iPhone 8 to make a better purchase decision. Web scraping coupled with data science can be leveraged for smart decision making be it in the fortune 500 companies or in your day to day life.
About Author
