eBay Scraping using BeautifulSoup in Python

Posted on Sep 25, 2019
The skills I demoed here can be learned through taking Data Science with Machine Learning bootcamp with NYC Data Science Academy.

Even thoughΒ AmazonΒ is the leader in e-commerce marketplaces – eBay still has its fair share in the online retail industry. Brands selling online should be monitoring prices on eBay as well to gain a competitive advantage.Β 

Extracting data from eBay at a huge scale regularly is a challenging problem for data scientists. Here is an example of scraping eBay using python to identify prices of mobile phones.Β 

Lets us imagine a use case where you need to monitor pricing of a product, say a mobile phone from eBay. Also, you want to visualise the range of price offering available on the mobile phone which you want to monitor. Moreover, you have other mobile phones under consideration so you may also want to compare their prices as well. In this blog, we will be scraping eBay toΒ collect the pricesΒ of phones and find out the difference between their offerings on the eBay website.

Web scrapingΒ is an efficient mechanism for collecting data from the internet.Β You can also read about different myths about web scrapingΒ here.

Scraping eBay step by step

In this section, we will walk you through the step by step process ofΒ scraping eBayΒ for products and their prices.Β 

1.Selecting the required information

The very first task in web scraping is toΒ identify the target web page. It is the web page from which you need to extract all the required information. We will beΒ scraping eBay for the product listingsΒ so we can just open the eBay website and type our product in their search bar and hit enter. Once page loads with all the product listing of that product, all you need to do is pull that URL out from the browser.

This URL will be our target URL. In our case, the URL will be β€œhttps://www.ebay.com/sch/i.html?_from=R40&_nkw=galaxy+note+8&_sacat=0&_pgn=1β€œ. Notice the two parameters in this URL i.e. β€œnkw” (new keyword) and β€œpgn” (page number) parameter. These parameters in the URL defines the search query. If we change β€œpgn” parameter to 2, then it will open the second page of the product listings for galaxy note 8 phone and if we were to change β€œnkw” to iPhone X then eBay will search for iPhone X and will show you the corresponding results.

2. Finalising the tags for extraction

Once we have finalised the target web page, we need to understand its HTML layout to scrape the results out. This is the most important and critical part of the web scraping and basic HTML knowledge is a pre-requisite for this step. When on the target web page, do β€œinspect element” and open the developer tools window or just do CTRL+SHIFT+I. In the new window, you will find the source code of the target web page.

In our case,Β  all the products are mentioned as list elements so we have to grab all these lists. In order to grab an HTML element,Β we need to have an identifier associated with it. It can be an id of that element or any class name or any other HTML attribute of the particular element. We are using the class name as the identifier. All the lists have the same class name i.e.Β s-item. On further inspection, we got the class names for the product name and product price which are β€œs-item__title” and β€œs-item__price” respectively. With this information, we have successfully completed step 2!

3. Putting the scraped data in a structured format
After having our extractors/identifiers we only need to extract specific portions out from the HTML content. Once this is done, we need to organise this data into a suitable structured format. We will be creating a table where we will have all the product names in one column and their prices in the other.

4. Visualising the results (optional)
Since we are to compare the price offerings on two different mobile phones, we will visualise the results too. This is not a mandatory step for web scraping but is more of a process to turn your collected data into some actionable insights. We will be plotting boxplots to understand the distribution of the price offerings on both galaxy note 8 and iPhone 8 mobile phones.

Required libraries and Installation

To implement web scraping for this use case, you will needΒ python, pipΒ (package installer for python) andΒ BeautifulSoup libraryΒ in python for web scraping. You will also need pandas and numpy library to organise the collected data into a structured format.Β 

  1. Installing Python and PIP
    Depending upon your operating system, you can follow this blog link to setup python and Pip in your system.
  2. Installing Beautiful soup library
    apt-get install python-bs4
    pip install beautifulsoup4
  3. Installing pandas and numpy
    pip install pandas
    pip install numpy

    We are done setting up our environment and now can begin with the scraping implementation using python. Implementation consists of the steps discussed in the earlier section.

  4. Python implementation for scraping eBay

    In this section, we will perform two scraping operations i.e. one for the iPhone 8 and other for the galaxy note 8 mobile phones. Implementation has been repeated for the two mobile phones for easier comprehension. A more optimised version can have two separate scrapping activity combined into one which is not required right now though.

    Scrapping eBay for Galaxy Note 8 products

    item_name = []
    prices = []
    Β 
    for i in range(1,10):
    Β 
    Β Β Β Β ebayUrl = "https://www.ebay.com/sch/i.html?_from=R40&_nkw=note+8&_sacat=0&_pgn="+str(i)
    Β Β Β Β r= requests.get(ebayUrl)
    Β Β Β Β data=r.text
    Β Β Β Β soup=BeautifulSoup(data)
    Β 
    Β Β Β Β listings = soup.find_all('li', attrs={'class': 's-item'})
    Β 
    Β Β Β Β for listing in listings:
    Β Β Β Β Β Β Β Β prod_name=" "
    Β Β Β Β Β Β Β Β prod_price = " "
    Β Β Β Β Β Β Β Β for name in listing.find_all('h3', attrs={'class':"s-item__title"}):
    Β Β Β Β Β Β Β Β Β Β Β Β if(str(name.find(text=True, recursive=False))!="None"):
    Β Β Β Β Β Β Β Β Β Β Β Β Β Β Β Β prod_name=str(name.find(text=True, recursive=False))
    Β Β Β Β Β Β Β Β Β Β Β Β Β Β Β Β item_name.append(prod_name)
    Β 
    Β Β Β Β Β Β Β Β if(prod_name!=" "):
    Β Β Β Β Β Β Β Β Β Β Β Β price = listing.find('span', attrs={'class':"s-item__price"})
    Β Β Β Β Β Β Β Β Β Β Β Β prod_price = str(price.find(text=True, recursive=False))
    Β Β Β Β Β Β Β Β Β Β Β Β prod_price = int(sub(",","",prod_price.split("INR")[1].split(".")[0]))
    Β Β Β Β Β Β Β Β Β Β Β Β prices.append(prod_price)
    Β 
    from scipy import stats
    import numpy as np
    Β 
    data_note_8 = pd.DataFrame({"Name":item_name, "Prices": prices})
    data_note_8 = data_note_8.iloc[np.abs(stats.zscore(data_note_8["Prices"]))< 3,]

    Collected Data for Galaxy note 8

  5. Scraping eBay using BeautifulSoup in Python

Scrapping eBay for iPhone 8Β 

item_name = []
prices = []
Β 
for i in range(1,10):
Β 
Β Β Β Β ebayUrl = "https://www.ebay.com/sch/i.html?_from=R40&_nkw=iphone+8_sacat=0_pgn="+str(i)
Β Β Β Β r= requests.get(ebayUrl)
Β Β Β Β data=r.text
Β Β Β Β soup=BeautifulSoup(data)
Β 
Β Β Β Β listings = soup.find_all('li', attrs={'class': 's-item'})
Β 
Β Β Β Β for listing in listings:
Β Β Β Β Β Β Β Β prod_name=" "
Β Β Β Β Β Β Β Β prod_price = " "
Β Β Β Β Β Β Β Β for name in listing.find_all('h3', attrs={'class':"s-item__title"}):
Β Β Β Β Β Β Β Β Β Β Β Β if(str(name.find(text=True, recursive=False))!="None"):
Β Β Β Β Β Β Β Β Β Β Β Β Β Β Β Β prod_name=str(name.find(text=True, recursive=False))
Β Β Β Β Β Β Β Β Β Β Β Β Β Β Β Β item_name.append(prod_name)
Β 
Β Β Β Β Β Β Β Β if(prod_name!=" "):
Β Β Β Β Β Β Β Β Β Β Β Β price = listing.find('span', attrs={'class':"s-item__price"})
Β Β Β Β Β Β Β Β Β Β Β Β prod_price = str(price.find(text=True, recursive=False))
Β Β Β Β Β Β Β Β Β Β Β Β prod_price = int(sub(",","",prod_price.split("INR")[1].split(".")[0]))
Β Β Β Β Β Β Β Β Β Β Β Β prices.append(prod_price)
Β 
from scipy import stats
import numpy as np
Β 
data_note_8 = pd.DataFrame({"Name":item_name, "Prices": prices})
data_note_8 = data_note_8.iloc[np.abs(stats.zscore(data_note_8["Prices"])) < 3,]

Collected data for iPhone 8

Scraping eBay using BeautifulSoup in Python

scraping eBay | Iphone data

Visualising the price of products

Now is the time to visualise the scraped results. We will be using the boxplots to visualise the distribution of prices of mobile phones. Box plot helps us in visualising a trend in numerical values. The green line is the median of the collected price data. The box extends from the Q1 to Q3 quartile values of the data, with a line at the median (Q2). The whiskers extend from the edges of the box to show the range of the data.

Scraping eBay using BeautifulSoup in Python

scraping eBay | Price Comparison

For iPhone 8, most of the prices lie between INR 25k-35k whereas most of the galaxy note 8 phones are available in the price range of 25k-30k. However, variation in prices of iPhone 8 is much more than galaxy note 8. iPhone 8 is available for minimum INR 15k on eBay whereas the minimum cost of galaxy note 8 on eBay is around 22-23K INR!

In this blog, we successfully usedΒ python for scraping eBay for two different products and their pricing. We also compared the available prices for galaxy note 8 and iPhone 8 to make a better purchase decision. Web scraping coupled with data science can be leveraged for smart decision making be it in the fortune 500 companies or in your day to day life.

 

 

About Author

Related Articles

Leave a Comment

No comments found.

View Posts by Categories


Our Recent Popular Posts


View Posts by Tags

#python #trainwithnycdsa 2019 2020 Revenue 3-points agriculture air quality airbnb airline alcohol Alex Baransky algorithm alumni Alumni Interview Alumni Reviews Alumni Spotlight alumni story Alumnus ames dataset ames housing dataset apartment rent API Application artist aws bank loans beautiful soup Best Bootcamp Best Data Science 2019 Best Data Science Bootcamp Best Data Science Bootcamp 2020 Best Ranked Big Data Book Launch Book-Signing bootcamp Bootcamp Alumni Bootcamp Prep boston safety Bundles cake recipe California Cancer Research capstone car price Career Career Day citibike classic cars classpass clustering Coding Course Demo Course Report covid 19 credit credit card crime frequency crops D3.js data data analysis Data Analyst data analytics data for tripadvisor reviews data science Data Science Academy Data Science Bootcamp Data science jobs Data Science Reviews Data Scientist Data Scientist Jobs data visualization database Deep Learning Demo Day Discount disney dplyr drug data e-commerce economy employee employee burnout employer networking environment feature engineering Finance Financial Data Science fitness studio Flask flight delay gbm Get Hired ggplot2 googleVis H20 Hadoop hallmark holiday movie happiness healthcare frauds higgs boson Hiring hiring partner events Hiring Partners hotels housing housing data housing predictions housing price hy-vee Income Industry Experts Injuries Instructor Blog Instructor Interview insurance italki Job Job Placement Jobs Jon Krohn JP Morgan Chase Kaggle Kickstarter las vegas airport lasso regression Lead Data Scienctist Lead Data Scientist leaflet league linear regression Logistic Regression machine learning Maps market matplotlib Medical Research Meet the team meetup methal health miami beach movie music Napoli NBA netflix Networking neural network Neural networks New Courses NHL nlp NYC NYC Data Science nyc data science academy NYC Open Data nyc property NYCDSA NYCDSA Alumni Online Online Bootcamp Online Training Open Data painter pandas Part-time performance phoenix pollutants Portfolio Development precision measurement prediction Prework Programming public safety PwC python Python Data Analysis python machine learning python scrapy python web scraping python webscraping Python Workshop R R Data Analysis R language R Programming R Shiny r studio R Visualization R Workshop R-bloggers random forest Ranking recommendation recommendation system regression Remote remote data science bootcamp Scrapy scrapy visualization seaborn seafood type Selenium sentiment analysis sentiment classification Shiny Shiny Dashboard Spark Special Special Summer Sports statistics streaming Student Interview Student Showcase SVM Switchup Tableau teachers team team performance TensorFlow Testimonial tf-idf Top Data Science Bootcamp Top manufacturing companies Transfers tweets twitter videos visualization wallstreet wallstreetbets web scraping Weekend Course What to expect whiskey whiskeyadvocate wildfire word cloud word2vec XGBoost yelp youtube trending ZORI