Data Web Scraping: Difference between Airbnb and TripAdvisor
The skills the author demoed here can be learned through taking Data Science with Machine Learning bootcamp with NYC Data Science Academy.
Python Scrapy is a powerful application framework for extracting structured data from web pages, you could extract data and generate a data file from websites based on your interests and needs, and also it offers the capability to easily post-process and persist your data. This technology resolves a big problem for me especially when I could not find an ideal dataset for a data science project. The problem i am interested in is what is the difference between Airbnb and TripAdvisor? Could I use hotel price to help Airbnb hosts setting their listing price? Let’s do it.
Introduction
Airbnb, as a popular peer-to-peer online homestay network, enabling people to list or rent short-term lodging in residential properties, with the cost of such accommodation set by the property owner. So setting a correct rent price for Airbnb hosts is one of the keys to maximizing revenue. While hotels industry has already used dynamic pricing for decades for controlling prices and maximizing revenue, how would hosts go about setting their prices? Could we get some hints from hotel price trends for helping Airbnb hosts adjusting their listing price?
Let’s Scrapy some data. In this project, the trends of the hotel price could be web scraped from TripAdvisor.com. I would like to compare the price differences between Airbnb and hotel prices and I also would like to measure the differences between services. If it is possible, I would try to develop an Airbnb price suggestion model based on the scraped information.
Data Web Scraping
Let’s assume we would like to book a 2 people travel to Orlando, FL during Dec 23--27, 2016.
The information I would like to scraped from those websites are listed below.
Airbnb: listing URLs in the search results, locations of listings, prices (night price, cleaning fee, tax, service fee), room type ( private room, entire home, shared room), people capacity, bed type, amenities, cancellation rules, and review score.
TripAdvisor: hotel names, hotel addresses, hotel reviews, review tags, amenities, hotel stars, and hotel price ranges.
Due to the limitation that current hotel prices are updated from other hotel booking websites through TripAdvisor, this information could not be acquired. Fortunately, there is a hotel price range information in the TripAdvisor hotel listing, which could be adopted as a good hint for price suggestions.
Figure 1
Figure 1 shows the scrapy process of getting data from a website. In the “Create Spider” part, a Scrapy project can be created with a working directory which stores a number of project files (scrapy.cfg, items.py, pipelines.py, settings.py, spiders). Within the Scrapy project folder, I need to write code in those files to realize different functions as below.
Items.py : containers that used to collect scraped data.
Spiders : create a spider.py, define Python classes (start_urls, Xpath) according to a certain site (or group of sites ) you would like to scrapy; construct appropriate XPath expression to extract the data from the websites, here I use Chrome which provides a developer’s tool for inspecting the structure of web pages (right click and then inspect)
pipelines.py : define Python classes that sequentially process the data
After I finish and validating all of the coding files, I could deploy the spider by using “scrapy crawl” command. In the “Scrapy Engine” part, once an item has been scraped, it is sent to the item pipeline, here the pipeline is used to validating scraped data, storing the scraped items in a file or a database in the “Data Downloader” part. Finally, a csv/txt file with data could be generated by using Scrapy. Please check my web Scrapy code below and the whole Python code here.
Data Analysis
Data on Price differences
Based on review scores of Airbnb listings, the price range is as shown in Figure, not big differences among listings with different review scores. The median prices with different review scores are around $50, the price range is wide among Airbnb with 4.5 and 5 in reviews. The review score is a comprehensive evaluation based on Accuracy, Communication, Cleanings, Location, Check-in and Value.
Figure 3 &4 show price differences among different hotel stars. Hotel stars are often used to classify hotels according to their quality, price differences is obvious among hotels with different starts and generally higher than Airbnb rent prices.
Figure 2
Source 3
Figure 4
Figure 5
Services differences
Statistics visualization of amenities is shown in Figure, from which we could get a general idea that the top 10 amenities Airbnb hosts provide include Air Conditioning, Wireless Internet, Free Parking on Premises, Heating, Internet, Shampoo, TV, Dryer, Washer, Family Kid Friendly. Whereas in the hotel amenities, the top 10 ones are No Smoking Rooms, Wheelchair Accessories, High Speed Internet(Wifi), Free Internet, Pool, Self-Service Laundry, Family Rooms, Multilingual Staff. The big difference in Amenities between Hotel and Airbnb is that Hotel has more advanced amenities service such as Wheelchair, Meeting Rooms, International Drive, Hot Tub, Bar Lounge, Airport Transportation.
Figure 6
Source 7
Figure 8
Factors drive the price
I assumed the differences in price is caused by the amenities difference between Airbnb and hotel service, an average Airbnb listing is just a less functioned hotel room. Based on this assumption, I build a random forest model, let’s check what features affects hotel price and Airbnb price most.
Figure 9
Figure 10
| TripAdvisor Hotel | Airbnb Listing |
| lon,lat | lon,lat |
| Review score | review_score_a |
| Free Parking | Free Parking on Premises |
| Laundry Service | Washer, Dryer |
| Kitchenette | Kitchen |
| Hotel Star, Babysitting, Airport Transportation, Multilingual Staff, Suites, Golf course, ConferenceFacilities,
TennisCourt, ShuttleBusService |
N/A |
Conclusion
- The Airbnb price could not be advised based on the hotel price at the same time, more information needed.
- Factors affect hotel price most depend on the location, reviews and transportation.
- Factors affect Airbnb price most depend on the tax, cleaning fee, person capacity and location.
About Author
Ziqiao Liu
View all posts by Ziqiao Liu >Related Articles
Leave a Comment
