Scraping on TripAdvisor to decide important factors for hotel rating

Shuye Han
Posted on Aug 24, 2016

In this project, I am going to do some scraping works on the famous travel website TripAdvisor to see if I can get some useful information to conduct both descriptive and predictive analysis on which factor could influence the rating and ranking of a hotel most and how we can make corresponding strategies to improve the rating score of a specific hotel.

1.Web Scraping

In order to initiate the analysis process, we should first get the data from online. The structure of TripAdvisor website is comparatively simple and straightforward. But there are still some tricky works need to be done to get the clear information we need. So, let's first have a quick view of the multiple sections we are gonna to scrape inside the website and what kind of informations we can get.

1.1 Web Page Structure

In this project, I use the hotels in Krabi, a small town in southern part of Thailand, as an example to collect all the informations of nearly 300 hotels in this tour site. When we type the destination "Krabi" in the main page, the website will guide us into another page with a long list of all the hotels in Krabi:

屏幕快照 2016-08-17 上午10.03.52


When we click inside one of the hotel in list, we can get into the main page of the single hotel which contains lots of information for this particular one:

屏幕快照 2016-08-17 上午10.04.07

屏幕快照 2016-08-17 上午10.04.16


屏幕快照 2016-08-17 上午10.04.35


1.2 The Scraped Information

Inside each section in each web page, we can get different pieces of information as shown below. Those features are the factors we are gonna to take into consideration for further analysis:

屏幕快照 2016-08-24 上午10.48.48

屏幕快照 2016-08-24 上午10.49.47

屏幕快照 2016-08-24 上午10.50.27

Those are the features that we can easily get from simple browsing the page and iterating to the specific HTML locations. However, there is some part of the page in which contents show up dynamically and we need to interact with the Javascript to retrieve the very information we want. The reviews part in every hotel page is just the section that would change accordingly as users interact with the page by clicking the checkbox and radios:

屏幕快照 2016-08-24 上午10.58.15

In this section, the information we want to retrieve lies in the left part, the ratings from "Terrible" to "Excellent". But this rating is only meaningful after we selecting the specific groups of people on the right side. Thus, when scraping, we need to choose every combination of "Traveler type", "Time of year" and "Language" one by one and waiting for the HTML contents reloaded and show up on the left. This would be a long and laborious process and should be well organized to correctly collect all the group data without missing while at the same time get as efficient as possible.

1.3 Scraped data

After running algorithms to search through the whole website, we can finally download the data we want and save them inside the python data frame. There are totally three big frames according to the different sections the data resides. Their structures are as follows:

屏幕快照 2016-08-24 上午11.25.48


屏幕快照 2016-08-24 上午11.26.35


屏幕快照 2016-08-24 上午11.27.21

As we can see that the raw data frames have multiple drawbacks that we need to further modify to get the clear data. First, there are a bunch of missingness in some columns. Secondly, there are some ambiguity in most of the string values that need to be further clarified.


2. Imputation and Modification

In this step, I will focus on the imputation and modification to get the exact clear data to for descriptive and predictive analysis. For the missingness in the "area" column, I would just simply revisit the "street" column and retrieve it from inside, since there is not a strict format for the address content and the area label would always be missing and combined into the "street" label.

屏幕快照 2016-08-24 上午11.47.31

After the whole process of imputation, I would then focus on the modification to cut off the prefix and surfix of some string columns:

屏幕快照 2016-08-24 上午11.55.10

And last but not least, I need to add more columns derived from existing column for better convenience of the analysis afterwards. For an example, in the original highlight column there are multiple highlights in each cell of the column. I would like to split them out and set one additional column for one highlight inside:

屏幕快照 2016-08-24 下午12.00.19

We can see there are still missingness in the price and postcode columns. For the price column and the postcode, I used the KNN method for imputation:

屏幕快照 2016-08-24 下午12.01.45


3. Descriptive and Predictive Analysis

After all the pre-works are done, we can now use the clean data to do some analysis. We would first want to now if there is any single value that can influence the ranking most. The first assumption we would make is that " the higher the rating, the higher the ranking", which is probably true after being illustrated by the following violin plot:

屏幕快照 2016-08-24 下午12.10.17

But this plot doesn't tell us any useful information. We need to get more plots to see the relation between ranking/rating and other variables:

屏幕快照 2016-08-24 下午12.13.53

屏幕快照 2016-08-24 下午12.14.01

屏幕快照 2016-08-24 下午12.14.11

屏幕快照 2016-08-24 下午12.14.20

As a result, there is not a very clear and convincing simple binary relationship between the ranking/rating and other features. Thus we need to set a linear model to see if there is a linear relationship among multiple features to predict the expected raking or rating. Then come the multi-linear, logistic regression and decision tree models:

屏幕快照 2016-08-24 下午12.18.39

屏幕快照 2016-08-24 下午12.18.54

屏幕快照 2016-08-24 下午12.19.04

After exploring the model of the overall analysis, we can switch the focus onto how a specific group of people would rate a particular hotel based on the information provided. As usual, we would also first conduct some binary or trinary relation plotting:

屏幕快照 2016-08-24 下午12.30.25

屏幕快照 2016-08-24 下午12.30.33

屏幕快照 2016-08-24 下午12.30.44

屏幕快照 2016-08-24 下午12.30.51

We can see there are some interesting discoveries of the distributions of features like seasons, travel purposes and travel types. But we still need predictive model to step forward for further analysis. As usual, we also conduct linear regression and decision trees for the rating prediction, the results are promising somehow:屏幕快照 2016-08-24 下午12.35.32

屏幕快照 2016-08-24 下午12.35.40:


About Author

Leave a Comment

No comments found.

View Posts by Categories

Our Recent Popular Posts

View Posts by Tags

#python #trainwithnycdsa 2019 airbnb Alex Baransky alumni Alumni Interview Alumni Reviews Alumni Spotlight alumni story Alumnus API Application artist aws beautiful soup Best Bootcamp Best Data Science 2019 Best Data Science Bootcamp Best Data Science Bootcamp 2020 Best Ranked Big Data Book Launch Book-Signing bootcamp Bootcamp Alumni Bootcamp Prep Bundles California Cancer Research capstone Career Career Day citibike clustering Coding Course Demo Course Report D3.js data Data Analyst data science Data Science Academy Data Science Bootcamp Data science jobs Data Science Reviews Data Scientist Data Scientist Jobs data visualization Deep Learning Demo Day Discount dplyr employer networking feature engineering Finance Financial Data Science Flask gbm Get Hired ggplot2 googleVis Hadoop higgs boson Hiring hiring partner events Hiring Partners Industry Experts Instructor Blog Instructor Interview Job Job Placement Jobs Jon Krohn JP Morgan Chase Kaggle Kickstarter lasso regression Lead Data Scienctist Lead Data Scientist leaflet linear regression Logistic Regression machine learning Maps matplotlib Medical Research Meet the team meetup Networking neural network Neural networks New Courses nlp NYC NYC Data Science nyc data science academy NYC Open Data NYCDSA NYCDSA Alumni Online Online Bootcamp Open Data painter pandas Part-time Portfolio Development prediction Prework Programming PwC python python machine learning python scrapy python web scraping python webscraping Python Workshop R R language R Programming R Shiny r studio R Visualization R Workshop R-bloggers random forest Ranking recommendation recommendation system regression Remote remote data science bootcamp Scrapy scrapy visualization seaborn Selenium sentiment analysis Shiny Shiny Dashboard Spark Special Special Summer Sports statistics streaming Student Interview Student Showcase SVM Switchup Tableau team TensorFlow Testimonial tf-idf Top Data Science Bootcamp twitter visualization web scraping Weekend Course What to expect word cloud word2vec XGBoost yelp