Data Scraping TripAdvisor
The skills the author demoed here can be learned through taking Data Science with Machine Learning bootcamp with NYC Data Science Academy.
Contributed by Adam Owen. Adam is currently in the NYC Data Science Academy 12 week full time Data Science Bootcamp program taking place between January 11th to April 1st, 2016. This post is based on his third class project - web scraping (due on the 6th week of the program).
Introduction
TripAdvisor's treasure trove of customer reviews on all things travel is just too temping for a marketing data scientist to ignore. The amount of information available and insights to be gleaned from the TripAdvisor can range from sentiment analysis to competitive pricing to seasonal prediction and correlated complaints. In this post, I'll pull a few details on hotels listing using a simple scraping program using BeautifulSoup in Python. I started with a script provided by Boston University for its Data Mining class; however, that script was over a year old and I made several modifications to make it run properly.
For starters, we'll aim to scrape Hotel Name, City, State, Star Rating (1-5 stars), Review Count, Feature Tags, and City Ranking:
Data SCRAPING PROGRAM
After setting up a user_agent, base_url, and an empty data frame, we'll define how to find the create a url for the specific city/state and then return the hypertext reference (href) or pointer using BeautifulSoup:
Great! We found the city_url. Now let's jump to that page and pull all the html.
HTML: check! We have a large pile of html code and now we need to parse through it and extract what we want. I added options just in case the page is wrong. Additional options could be added depending on the outcome of scraped hotels.
Let's parse....
The three components of the program are set. Now, let's link them all together to create a simple program. I left out some of the logging details which could be included if you want to take a specific look at errors. In this case, I left them out.
Finally! We completed a simple program where we can load those definitions into our environment and use the function scrape_hotels(city, state, datadir) to build a data frame of the results. I created a simple for loop to go through 100 major metropolitan cities and, unfortunately, it came back scarce! I received results for Salt Lake City, Los Angeles, New York, and Chicago.
Complete List of Cities
In order to get a complete list of all 100 cities I'll have to adjust the get_city_page definition since TripAdvisor changed its website. This program is for educational purposes and if we were serious about scraping all 100 cities then we could employ several scraping services or rebuild the first href/pointer definition to capture all city pages. Another option is to surpass the first definition by manually finding each city page and put it in a list for html retrieval and scraping.
This case reminds us that scraping has a short shelf life because of the zero control a programmer has over the web development. Several services scrape basic websites and many of those are free including the plug-in for Google Sheets. That's the first line of defense for a new project and then one could proceed to complicate your life.
Let's press on and do a quick analysis.....
WHAT DID WE JUST SCRAPE?
1,106 rows with 4 cities. Sample below:
Note: City_Ranking is defined by TripAdvisor as the best hotels in each city.
After uploading RStudio, I began some initial analysis and visualizations of the data. You can checkout the R-code github link here.
Let's first look at a bubble chart of Hotel Ratings and City Reviews.
Hotel Ratings & City Reviews
This chart displays the rankings of hotels correlated with city ranking. We should see a high correlation between City Ranking and Hotel Star Ratings and that is largely what we see. A couple of interesting notes is there are a handful of 5-star hotels that have low City Ranking particularly in NYC and LA. Those contrary ranked hotels also don't have many reviews which could mean those are errors or hotels never traveled. Another interesting observation is the lack of 4-star hotels and above in LA. Also, Chicago overcomes has a large abundance of 4-4.5 star hotels while LA has an abundance of 3-3.5 star hotels.
Features
Not, let's take a quick look at the features grouped by city:
SLC has a surprising number of hotels that advertise "Pets Allowed" while LA is the king of "Budget" hotels. Chicago's advertised features mirror NYC but in smaller quantities. They are not visible because NYC overcomes the Chicago hotel counts.
Review Count and Ratings
Another graph we can explore is the review count and ratings by feature:
"Green" and "Luxury" hit the top of the list for ratings and "Mid-Range" and "Budget" are most reviewed. It's interesting that "Pets Allowed" hit high on both count and ratings. It's also very interesting to see how often neighborhoods are advertised on the featured hotel add such as "Magnificent Mile" and "Westside".
CONCLUSION
We found a handful of interesting insights from our TripAdvisor program. We also created a program to scrape TripAdvisor hotel listings and we can add new functions to scrape additional data. It will need additional coding before it's ready for full production but it's a great start. The next step in our analysis, once the scraping program is in top shape, is to add data including full reviews, highlighted reviews, pricing, and any other info we can find.
Another interesting point would be a quick analysis of TripAdvisors ad placement on the page which may give a competitor a hint to TripAdvisor's recommendation algorithm. We have a great starter script to continue additional analysis or for another scraping project. As with all most data science projects, many questions and limited time to answer them all!
About Author
Adam Owens
View all posts by Adam Owens >Related Articles
Leave a Comment
