Web Scraping Project: Experience Difference between Airbnb and TripAdvisor

Ziqiao Liu
Posted on Nov 21, 2016

Python Scrapy is  a powerful application framework for extracting structured data from web pages, you could extract data and generate a data file from websites based on your interests and needs, and also it offers the capability to easily post-process and persist your data. This technology resolves a big problem for me especially when I could not find an ideal dataset for a data science project. The problem i am interested in is what is the difference between Airbnb and TripAdvisor? Could I use hotel price to help Airbnb hosts setting their listing price? Let’s do it.


Airbnb, as a popular peer-to-peer online homestay network, enabling people to list or rent short-term lodging in residential properties, with the cost of such accommodation set by the property owner. So setting a correct rent price for Airbnb hosts is one of the keys to maximizing revenue. While hotels industry has already used dynamic pricing for decades for controlling prices and maximizing revenue, how would hosts go about setting their prices? Could we get some hints from hotel price trends for helping Airbnb hosts adjusting their listing price? Let’s Scrapy some data. In this project, the trends of the hotel price could be web scraped from TripAdvisor.com. I would like to compare the price differences between Airbnb and hotel prices and I also would like to measure the differences between services. If it is possible, I would try to develop an Airbnb price suggestion model based on the scraped information.

Web Scraping

Let’s assume we would like to book a 2 people travel to Orlando, FL during Dec 23--27, 2016.

The information I would like to scraped from those websites are listed below.

Airbnb: listing URLs in the search results, locations of listings, prices (night price, cleaning fee, tax, service fee), room type ( private room, entire home, shared room), people capacity, bed type, amenities, cancellation rules, and review score.

TripAdvisor: hotel names, hotel addresses, hotel reviews, review tags, amenities, hotel stars, and hotel price ranges.

Due to the limitation that current hotel prices are updated from other hotel booking websites through TripAdvisor, this information could not be acquired. Fortunately, there is a hotel price range information in the TripAdvisor hotel listing, which could be adopted as a good hint for price suggestions.



Figure 1


Figure 1  shows the scrapy process of getting data from a website. In the “Create Spider” part, a Scrapy project can be created with a working directory which stores a number of project files (scrapy.cfg, items.py, pipelines.py, settings.py, spiders).  Within the Scrapy project folder, I need to write code in those files to realize different functions as below.

Items.py : containers that used to collect scraped data. 

Spiders : create a spider.py, define Python classes (start_urls, Xpath) according to a certain site (or group of sites ) you would like to scrapy; construct appropriate XPath expression to extract the data from the websites, here I use Chrome which provides a developer’s tool for inspecting the structure of web pages (right click and then inspect)

pipelines.py : define Python classes that sequentially process the data

After I finish and validating all of the coding files, I could deploy the spider by using “scrapy crawl” command. In the “Scrapy Engine” part, once an item has been scraped, it is sent to the item pipeline, here the pipeline is used to validating scraped data, storing the scraped items in a file or a database in the “Data Downloader” part. Finally, a csv/txt file with data could be generated by using Scrapy. Please check my web Scrapy code below  and the whole Python code here.



Data Analysis

Price differences

Based on review scores of Airbnb listings, the price range is as shown in Figure, not big differences among listings with different review scores. The median prices with different review scores are around $50, the price range is wide among Airbnb with 4.5 and 5 in reviews. The review score is a comprehensive evaluation based on Accuracy, Communication, Cleanings, Location, Check-in and Value.

Figure 3 &4 show price differences among different hotel stars. Hotel stars are often used to classify hotels according to their quality, price differences is obvious among hotels with different starts and generally higher than Airbnb rent prices.


Figure 2


Figure 3


Figure 4


Figure 5

Services differences

Statistics visualization of amenities is shown in Figure, from which we could get a general idea that the top 10 amenities Airbnb hosts provide include Air Conditioning, Wireless Internet, Free Parking on Premises, Heating, Internet, Shampoo, TV, Dryer, Washer, Family Kid Friendly. Whereas in the hotel amenities, the top 10 ones are No Smoking Rooms, Wheelchair Accessories, High Speed Internet(Wifi), Free Internet, Pool, Self-Service Laundry, Family Rooms, Multilingual Staff. The big difference in Amenities between Hotel and Airbnb is that Hotel has more advanced  amenities service such as Wheelchair, Meeting Rooms, International Drive, Hot Tub, Bar Lounge,  Airport Transportation.


Figure 6


Figure 7


Figure 8

Factors drive the price

I assumed the differences in price is caused by the amenities difference between Airbnb and hotel service, an average Airbnb listing is just a less functioned hotel room. Based on this assumption, I build a random forest model, let’s check what features affects hotel price and Airbnb price most.


Figure 9


Figure 10

TripAdvisor Hotel Airbnb Listing
lon,lat lon,lat
Review score review_score_a
Free Parking Free Parking on Premises
Laundry Service Washer, Dryer
Kitchenette Kitchen
Hotel Star, Babysitting, Airport Transportation, Multilingual Staff, Suites, Golf course, ConferenceFacilities,

TennisCourt, ShuttleBusService



  • The Airbnb price could not be advised based on the hotel price at the same time, more information needed.
  • Factors affect hotel price most depend on the location, reviews and transportation.
  • Factors affect Airbnb price most depend on the tax, cleaning fee, person capacity and location.


About Author

Related Articles

Leave a Comment

Arturo February 15, 2017
Hi, Very nice post! Do you know if there is anyway to ScraPy/access information regarding if a listing is actually available or taken for a period of time? Or for example get a list of dates available for the listing? Thanks a lot!

View Posts by Categories

Our Recent Popular Posts

View Posts by Tags

#python #trainwithnycdsa 2019 airbnb Alex Baransky alumni Alumni Interview Alumni Reviews Alumni Spotlight alumni story Alumnus API Application artist aws beautiful soup Best Bootcamp Best Data Science 2019 Best Data Science Bootcamp Best Data Science Bootcamp 2020 Best Ranked Big Data Book Launch Book-Signing bootcamp Bootcamp Alumni Bootcamp Prep Bundles California Cancer Research capstone Career Career Day citibike clustering Coding Course Demo Course Report D3.js data Data Analyst data science Data Science Academy Data Science Bootcamp Data science jobs Data Science Reviews Data Scientist Data Scientist Jobs data visualization Deep Learning Demo Day Discount dplyr employer networking feature engineering Finance Financial Data Science Flask gbm Get Hired ggplot2 googleVis Hadoop higgs boson Hiring hiring partner events Hiring Partners Industry Experts Instructor Blog Instructor Interview Job Job Placement Jobs Jon Krohn JP Morgan Chase Kaggle Kickstarter lasso regression Lead Data Scienctist Lead Data Scientist leaflet linear regression Logistic Regression machine learning Maps matplotlib Medical Research Meet the team meetup Networking neural network Neural networks New Courses nlp NYC NYC Data Science nyc data science academy NYC Open Data NYCDSA NYCDSA Alumni Online Online Bootcamp Online Training Open Data painter pandas Part-time Portfolio Development prediction Prework Programming PwC python Python Data Analysis python machine learning python scrapy python web scraping python webscraping Python Workshop R R language R Programming R Shiny r studio R Visualization R Workshop R-bloggers random forest Ranking recommendation recommendation system regression Remote remote data science bootcamp Scrapy scrapy visualization seaborn Selenium sentiment analysis Shiny Shiny Dashboard Spark Special Special Summer Sports statistics streaming Student Interview Student Showcase SVM Switchup Tableau team TensorFlow Testimonial tf-idf Top Data Science Bootcamp twitter visualization web scraping Weekend Course What to expect word cloud word2vec XGBoost yelp