Scraping on TripAdvisor to decide important factors for hotel rating
In this project, I am going to do some scraping works on the famous travel website TripAdvisor to see if I can get some useful information to conduct both descriptive and predictive analysis on which factor could influence the rating and ranking of a hotel most and how we can make corresponding strategies to improve the rating score of a specific hotel.
1.Web Scraping
In order to initiate the analysis process, we should first get the data from online. The structure of TripAdvisor website is comparatively simple and straightforward. But there are still some tricky works need to be done to get the clear information we need. So, let's first have a quick view of the multiple sections we are gonna to scrape inside the website and what kind of informations we can get.
1.1 Web Page Structure
In this project, I use the hotels in Krabi, a small town in southern part of Thailand, as an example to collect all the informations of nearly 300 hotels in this tour site. When we type the destination "Krabi" in the main page, the website will guide us into another page with a long list of all the hotels in Krabi:
When we click inside one of the hotel in list, we can get into the main page of the single hotel which contains lots of information for this particular one:
1.2 The Scraped Information
Inside each section in each web page, we can get different pieces of information as shown below. Those features are the factors we are gonna to take into consideration for further analysis:
Those are the features that we can easily get from simple browsing the page and iterating to the specific HTML locations. However, there is some part of the page in which contents show up dynamically and we need to interact with the Javascript to retrieve the very information we want. The reviews part in every hotel page is just the section that would change accordingly as users interact with the page by clicking the checkbox and radios:
In this section, the information we want to retrieve lies in the left part, the ratings from "Terrible" to "Excellent". But this rating is only meaningful after we selecting the specific groups of people on the right side. Thus, when scraping, we need to choose every combination of "Traveler type", "Time of year" and "Language" one by one and waiting for the HTML contents reloaded and show up on the left. This would be a long and laborious process and should be well organized to correctly collect all the group data without missing while at the same time get as efficient as possible.
1.3 Scraped data
After running algorithms to search through the whole website, we can finally download the data we want and save them inside the python data frame. There are totally three big frames according to the different sections the data resides. Their structures are as follows:
As we can see that the raw data frames have multiple drawbacks that we need to further modify to get the clear data. First, there are a bunch of missingness in some columns. Secondly, there are some ambiguity in most of the string values that need to be further clarified.
2. Imputation and Modification
In this step, I will focus on the imputation and modification to get the exact clear data to for descriptive and predictive analysis. For the missingness in the "area" column, I would just simply revisit the "street" column and retrieve it from inside, since there is not a strict format for the address content and the area label would always be missing and combined into the "street" label.
After the whole process of imputation, I would then focus on the modification to cut off the prefix and surfix of some string columns:
And last but not least, I need to add more columns derived from existing column for better convenience of the analysis afterwards. For an example, in the original highlight column there are multiple highlights in each cell of the column. I would like to split them out and set one additional column for one highlight inside:
We can see there are still missingness in the price and postcode columns. For the price column and the postcode, I used the KNN method for imputation:
3. Descriptive and Predictive Analysis
After all the pre-works are done, we can now use the clean data to do some analysis. We would first want to now if there is any single value that can influence the ranking most. The first assumption we would make is that " the higher the rating, the higher the ranking", which is probably true after being illustrated by the following violin plot:
But this plot doesn't tell us any useful information. We need to get more plots to see the relation between ranking/rating and other variables:
As a result, there is not a very clear and convincing simple binary relationship between the ranking/rating and other features. Thus we need to set a linear model to see if there is a linear relationship among multiple features to predict the expected raking or rating. Then come the multi-linear, logistic regression and decision tree models:
After exploring the model of the overall analysis, we can switch the focus onto how a specific group of people would rate a particular hotel based on the information provided. As usual, we would also first conduct some binary or trinary relation plotting:
We can see there are some interesting discoveries of the distributions of features like seasons, travel purposes and travel types. But we still need predictive model to step forward for further analysis. As usual, we also conduct linear regression and decision trees for the rating prediction, the results are promising somehow:
:
About Author
