Data Analyzing NYC Hotels on TripAdvisor
The skills the author demoed here can be learned through taking Data Science with Machine Learning bootcamp with NYC Data Science Academy.
Introduction
New York City is often described as one of the centers of the world. Data shows the city attracts outsiders for its diverse culture and business opportunities. Hotel businesses strive to meet this demand, however; competition is fierce. It is important to keep up with the competition and reflect on the company's actions for business success.
This was what I was considering when I took on this project. It was a great way of improving my understanding of learning the Scrapy package on Python. The objectives for this project include discovering: business opportunities to grow, successful business strategies, common consumer interest. We are going to answer some of these questions by analyzing the location and listed services on the website.
Methodology
For this project, we are going to scrap the NYC hotel listings of TripAdvisor.com. The website displays a table of hotel listings, each listing provides a summary and a hyperlink. These links leads to a detailed page of it’s listing. By inspecting these elements, l built a scrapy spider to compile a dataset of the details of each of the listed hotels. Before building the scrapy spider, observe the html links. One indication that information can be scraped by a spider is html links changing when interacting with the website. Within minutes, I was able to scrap 714 hotel details on TripAdvisor.
Surface Data Analysis
Let’s start with identifying the density and location of these hotels. By using a folium package on Python, I’ve visually produced a map that points the location of these hotels. However, address locations are not recognized by folium and would need to be converted into geographic coordinates. Warmer colors represent higher concentration of hotels, and cooler colors display a lower concentration of hotels. Location with no color did not have a hotel listing. It’s clear most NYC hotels are located within the borough of Manhattan. The location by JFK airport (in the borough of Queens) is interesting because the absence of color indicates that these hotel locations were not posted on TripAdvisor.
The boxplot below compares the overall rating of hotels to a ratio of their excellent reviews to total reviews. This method will measure their success by evaluating the value of their overall rating. Maintaining a high customer satisfaction is key to a successful business.
However, this method only works with established companies. For this study, I classified established companies as a company with a location that lists at least 50 reviews. As the graph shows there is a direct relationship between overall rating and the ratio of excellent reviews to total reviews. This is important because we are going to use this information to create a classification class of hotels.
Based on this graph, we are going to define the following terms to describe hotels: excellent, good, and poor.
- Excellent: Overall ratings > 4.0
- Good: Overall ratings 3.0 - 4.0
- Poor: Overall ratings < 3.0
Success Variable: Data on Location
Based on our classification system we have 157 excellent hotels, 284 good hotels, and 78 poor hotels.
While the location may play a role in business opportunity, it doesn’t determine business success. Focusing on Manhattan, there is a higher density of hotels in Midtown and downtown Wall Street. However, the hotels in these locations are not differentiated by success. A business in Wall Street is just as likely to be rated excellent, good or poor compared to a business in any other location.
Success Variable: Services
The word cloud above is a compilation of hotel services from all hotel listings, but a higher frequency of service will increase the text size. The word cloud shows that all the hotels regardless of their success provide things like flat screen TVs, high speed internet wifi, air conditioning, and the choice between smoking and non-smoking. Therefore, an expectation is set by the consumers for businesses to provide these services. A company that doesn’t provide these services will hurt its business without them. We are going to differentiate the services provided by the three classes of success to identify what services separate a poor hotel from an excellent hotel.
average of 14.75 listed services per hotel.
As the quality of class increases, the number of unique services also increases. However, the average listed services per hotel between the three classes doesn’t have significant change. Therefore, we can conclude excellent hotels aren’t listing more services, but are listing more unique services to separate themselves from their competition. Something interesting to note here is the higher frequency of “free” in poor hotels. There is little evidence to prove causation between usage of “free” and lower ratings, but is interesting to look into for the future.
Conclusion and Next Steps
Based on the 714 hotel entries scraped from TripAdvisor.com, we can conclude that most of the hotels are located in Manhattan. However, there is no distinct relationship between location and business success. To promotd business growth, companies should list unique services and aim to win excellent reviews. This information is valuable for consulting companies, hotel businesses, and completing travel platform sites.
There are a lot of different directions in which it is possible to take this project. You can delve deeper into this analysis by pulling information from other competing sites and comparing their location and services to their success. Creation of an application that will allow users to find a hotel with the desired features is another avenue. Incorporating a method to efficiently extract price deals for analyzing price prediction is the most challenging approach because scrapy is unable to extract prices from their respective listing. In that case, I would recommend using selenium for extracting information.
Interested in my spider? You can view the details my Github.
About Author
