Data Web Scraping: Difference between Airbnb and TripAdvisor

Posted on Nov 21, 2016
The skills the author demoed here can be learned through taking Data Science with Machine Learning bootcamp with NYC Data Science Academy.

Python Scrapy is  a powerful application framework for extracting structured data from web pages, you could extract data and generate a data file from websites based on your interests and needs, and also it offers the capability to easily post-process and persist your data. This technology resolves a big problem for me especially when I could not find an ideal dataset for a data science project. The problem i am interested in is what is the difference between Airbnb and TripAdvisor? Could I use hotel price to help Airbnb hosts setting their listing price? Let’s do it.

Introduction

Airbnb, as a popular peer-to-peer online homestay network, enabling people to list or rent short-term lodging in residential properties, with the cost of such accommodation set by the property owner. So setting a correct rent price for Airbnb hosts is one of the keys to maximizing revenue. While hotels industry has already used dynamic pricing for decades for controlling prices and maximizing revenue, how would hosts go about setting their prices? Could we get some hints from hotel price trends for helping Airbnb hosts adjusting their listing price?

Let’s Scrapy some data. In this project, the trends of the hotel price could be web scraped from TripAdvisor.com. I would like to compare the price differences between Airbnb and hotel prices and I also would like to measure the differences between services. If it is possible, I would try to develop an Airbnb price suggestion model based on the scraped information.

Data Web Scraping

Let’s assume we would like to book a 2 people travel to Orlando, FL during Dec 23--27, 2016.

The information I would like to scraped from those websites are listed below.

Airbnb: listing URLs in the search results, locations of listings, prices (night price, cleaning fee, tax, service fee), room type ( private room, entire home, shared room), people capacity, bed type, amenities, cancellation rules, and review score.

TripAdvisor: hotel names, hotel addresses, hotel reviews, review tags, amenities, hotel stars, and hotel price ranges.

Due to the limitation that current hotel prices are updated from other hotel booking websites through TripAdvisor, this information could not be acquired. Fortunately, there is a hotel price range information in the TripAdvisor hotel listing, which could be adopted as a good hint for price suggestions.

 

Data Web Scraping: Difference between Airbnb and TripAdvisor

Figure 1

 

Figure 1  shows the scrapy process of getting data from a website. In the “Create Spider” part, a Scrapy project can be created with a working directory which stores a number of project files (scrapy.cfg, items.py, pipelines.py, settings.py, spiders).  Within the Scrapy project folder, I need to write code in those files to realize different functions as below.

Items.py : containers that used to collect scraped data. 

Spiders : create a spider.py, define Python classes (start_urls, Xpath) according to a certain site (or group of sites ) you would like to scrapy; construct appropriate XPath expression to extract the data from the websites, here I use Chrome which provides a developer’s tool for inspecting the structure of web pages (right click and then inspect)

pipelines.py : define Python classes that sequentially process the data

After I finish and validating all of the coding files, I could deploy the spider by using “scrapy crawl” command. In the “Scrapy Engine” part, once an item has been scraped, it is sent to the item pipeline, here the pipeline is used to validating scraped data, storing the scraped items in a file or a database in the “Data Downloader” part. Finally, a csv/txt file with data could be generated by using Scrapy. Please check my web Scrapy code below  and the whole Python code here.

Data Analysis

Data on Price differences

Based on review scores of Airbnb listings, the price range is as shown in Figure, not big differences among listings with different review scores. The median prices with different review scores are around $50, the price range is wide among Airbnb with 4.5 and 5 in reviews. The review score is a comprehensive evaluation based on Accuracy, Communication, Cleanings, Location, Check-in and Value.

Figure 3 &4 show price differences among different hotel stars. Hotel stars are often used to classify hotels according to their quality, price differences is obvious among hotels with different starts and generally higher than Airbnb rent prices.

Data Web Scraping: Difference between Airbnb and TripAdvisor

Figure 2

tripminhotelstar

Source 3

Data Web Scraping: Difference between Airbnb and TripAdvisor

Figure 4

distributionat

Figure 5

Services differences

Statistics visualization of amenities is shown in Figure, from which we could get a general idea that the top 10 amenities Airbnb hosts provide include Air Conditioning, Wireless Internet, Free Parking on Premises, Heating, Internet, Shampoo, TV, Dryer, Washer, Family Kid Friendly. Whereas in the hotel amenities, the top 10 ones are No Smoking Rooms, Wheelchair Accessories, High Speed Internet(Wifi), Free Internet, Pool, Self-Service Laundry, Family Rooms, Multilingual Staff. The big difference in Amenities between Hotel and Airbnb is that Hotel has more advanced  amenities service such as Wheelchair, Meeting Rooms, International Drive, Hot Tub, Bar Lounge,  Airport Transportation.

airbnbam

Figure 6

tripadvisoram

Source 7

wordcloud

Figure 8

Factors drive the price

I assumed the differences in price is caused by the amenities difference between Airbnb and hotel service, an average Airbnb listing is just a less functioned hotel room. Based on this assumption, I build a random forest model, let’s check what features affects hotel price and Airbnb price most.

triplh

Figure 9

varimportanceairbnb

Figure 10

TripAdvisor Hotel Airbnb Listing
lon,lat lon,lat
Review score review_score_a
Free Parking Free Parking on Premises
Laundry Service Washer, Dryer
Kitchenette Kitchen
Hotel Star, Babysitting, Airport Transportation, Multilingual Staff, Suites, Golf course, ConferenceFacilities,

TennisCourt, ShuttleBusService

N/A

Conclusion

  • The Airbnb price could not be advised based on the hotel price at the same time, more information needed.
  • Factors affect hotel price most depend on the location, reviews and transportation.
  • Factors affect Airbnb price most depend on the tax, cleaning fee, person capacity and location.

 

About Author

Related Articles

Leave a Comment

Arturo February 15, 2017
Hi, Very nice post! Do you know if there is anyway to ScraPy/access information regarding if a listing is actually available or taken for a period of time? Or for example get a list of dates available for the listing? Thanks a lot!

View Posts by Categories


Our Recent Popular Posts


View Posts by Tags

#python #trainwithnycdsa 2019 2020 Revenue 3-points agriculture air quality airbnb airline alcohol Alex Baransky algorithm alumni Alumni Interview Alumni Reviews Alumni Spotlight alumni story Alumnus ames dataset ames housing dataset apartment rent API Application artist aws bank loans beautiful soup Best Bootcamp Best Data Science 2019 Best Data Science Bootcamp Best Data Science Bootcamp 2020 Best Ranked Big Data Book Launch Book-Signing bootcamp Bootcamp Alumni Bootcamp Prep boston safety Bundles cake recipe California Cancer Research capstone car price Career Career Day citibike classic cars classpass clustering Coding Course Demo Course Report covid 19 credit credit card crime frequency crops D3.js data data analysis Data Analyst data analytics data for tripadvisor reviews data science Data Science Academy Data Science Bootcamp Data science jobs Data Science Reviews Data Scientist Data Scientist Jobs data visualization database Deep Learning Demo Day Discount disney dplyr drug data e-commerce economy employee employee burnout employer networking environment feature engineering Finance Financial Data Science fitness studio Flask flight delay gbm Get Hired ggplot2 googleVis H20 Hadoop hallmark holiday movie happiness healthcare frauds higgs boson Hiring hiring partner events Hiring Partners hotels housing housing data housing predictions housing price hy-vee Income Industry Experts Injuries Instructor Blog Instructor Interview insurance italki Job Job Placement Jobs Jon Krohn JP Morgan Chase Kaggle Kickstarter las vegas airport lasso regression Lead Data Scienctist Lead Data Scientist leaflet league linear regression Logistic Regression machine learning Maps market matplotlib Medical Research Meet the team meetup methal health miami beach movie music Napoli NBA netflix Networking neural network Neural networks New Courses NHL nlp NYC NYC Data Science nyc data science academy NYC Open Data nyc property NYCDSA NYCDSA Alumni Online Online Bootcamp Online Training Open Data painter pandas Part-time performance phoenix pollutants Portfolio Development precision measurement prediction Prework Programming public safety PwC python Python Data Analysis python machine learning python scrapy python web scraping python webscraping Python Workshop R R Data Analysis R language R Programming R Shiny r studio R Visualization R Workshop R-bloggers random forest Ranking recommendation recommendation system regression Remote remote data science bootcamp Scrapy scrapy visualization seaborn seafood type Selenium sentiment analysis sentiment classification Shiny Shiny Dashboard Spark Special Special Summer Sports statistics streaming Student Interview Student Showcase SVM Switchup Tableau teachers team team performance TensorFlow Testimonial tf-idf Top Data Science Bootcamp Top manufacturing companies Transfers tweets twitter videos visualization wallstreet wallstreetbets web scraping Weekend Course What to expect whiskey whiskeyadvocate wildfire word cloud word2vec XGBoost yelp youtube trending ZORI