Analyzing NYC Hotels on TripAdvisor

Yasuhiro Shinohara
Posted on Feb 3, 2020

Introduction

New York City is often described as one of the centers of the world. The city attracts outsiders for its diverse culture and business opportunities. Hotel businesses strive to meet this demand, however; competition is fierce. It is important to keep up with the competition and reflect on the company's actions for business success. 

This was what I was considering when I took on this project. It was a great way of improving my understanding of learning the Scrapy package on Python. The objectives for this project include discovering: business opportunities to grow, successful business strategies, common consumer interest. We are going to answer some of these questions by analyzing the location and listed services on the website.

Methodology

For this project, we are going to scrap the NYC hotel listings of TripAdvisor.com. The website displays a table of hotel listings, each listing provides a summary and a hyperlink. These links leads to a detailed page of it’s listing. By inspecting these elements, l built a scrapy spider to compile a dataset of the details of each of the listed hotels. Before building the scrapy spider, observe the html links. One indication that information can be scraped by a spider is html links changing when interacting with the website. Within minutes, I was able to scrap 714 hotel details on TripAdvisor. 

Surface Analysis

Let’s start with identifying the density and location of these hotels. By using a folium package on Python, I’ve visually produced a map that points the location of these hotels. However, address locations are not recognized by folium and would need to be converted into geographic coordinates. Warmer colors represent higher concentration of hotels, and cooler colors display a lower concentration of hotels. Location with no color did not have a hotel listing. It’s clear most NYC hotels are located within the borough of Manhattan. The location by JFK airport (in the borough of Queens) is interesting because the absence of color indicates that these hotel locations were not posted on TripAdvisor.

Heatmap of all hotels listed as NYC

The boxplot below compares the overall rating of hotels to a ratio of their excellent reviews to total reviews. This method will measure their success by evaluating the value of their overall rating. Maintaining a high customer satisfaction is key to a successful business. However, this method only works with established companies. For this study, I classified established companies as a company with a location that lists at least 50 reviews. As the graph shows there is a direct relationship between overall rating and the ratio of excellent reviews to total reviews. This is important because we are going to use this information to create a classification class of hotels.

Based on this graph, we are going to define the following terms to describe hotels: excellent, good, and poor. 

  • Excellent: Overall ratings > 4.0
  • Good: Overall ratings 3.0 - 4.0 
  • Poor: Overall ratings < 3.0

Success Variable: Location

Based on our classification system we have 157 excellent hotels, 284 good hotels, and 78 poor hotels. 

While the location may play a role in business opportunity, it doesn’t determine business success. Focusing on Manhattan, there is a higher density of hotels in Midtown and downtown Wall Street. However, the hotels in these locations are not differentiated by success. A business in Wall Street is just as likely to be rated excellent, good or poor compared to a business in any other location.

Success Variable: Services

The word cloud above is a compilation of hotel services from all hotel listings, but a higher frequency of service will increase the text size. The word cloud shows that all the hotels regardless of their success provide things like flat screen TVs, high speed internet wifi, air conditioning, and the choice between smoking and non-smoking. Therefore, an expectation is set by the consumers for businesses to provide these services. A company that doesn’t provide these services will hurt its business without them. We are going to differentiate the services provided by the three classes of success to identify what services separate a poor hotel from an excellent hotel.

Word cloud of services from excellent hotels. 214 unique services found, an

average of 14.75 listed services per hotel.

Word cloud of services from good hotels. 168 unique services found, average of 14.80 listed services per hotel.
Word cloud of services from poor hotels. 39 unique services found, average of 14.75 listed services per hotel.

As the quality of class increases, the number of unique services also increases. However, the average listed services per hotel between the three classes doesn’t have significant change. Therefore, we can conclude excellent hotels aren’t listing more services, but are listing more unique services to separate themselves from their competition. Something interesting to note here is the higher frequency of “free” in poor hotels. There is little evidence to prove causation between usage of “free” and lower ratings, but is interesting to look into for the future.

Conclusion and Next Steps

Based on the 714 hotel entries scraped from TripAdvisor.com, we can conclude that most of the hotels are located in Manhattan. However, there is no distinct relationship between location and business success. To promotd business growth, companies should list unique services and aim to win excellent reviews. This information is valuable for consulting companies, hotel businesses, and completing travel platform sites.

There are a lot of different directions in which it is possible to take this project. You can delve deeper into this analysis by pulling information from other competing sites and comparing their location and services to their success. Creation of an application that will allow users to find a hotel with the desired features is another avenue. Incorporating a method to efficiently extract price deals for analyzing price prediction is the most challenging approach because scrapy is unable to extract prices from their respective listing. In that case, I would recommend using selenium for extracting information.

Interested in my spider? You can view the details my Github.

About Author

Yasuhiro Shinohara

Yasuhiro Shinohara

Recent Data Scientist Fellow at NYC Data Science Academy. A former teacher with a Masters in Education with a focus on Earth Sciences. Emphasizes clear communication of data analytics through visualization and public speaking to promote positive and...
View all posts by Yasuhiro Shinohara >

Related Articles

Leave a Comment

No comments found.

View Posts by Categories


Our Recent Popular Posts


View Posts by Tags

#python #trainwithnycdsa 2019 airbnb Alex Baransky alumni Alumni Interview Alumni Reviews Alumni Spotlight alumni story Alumnus API Application artist aws beautiful soup Best Bootcamp Best Data Science 2019 Best Data Science Bootcamp Best Data Science Bootcamp 2020 Best Ranked Big Data Book Launch Book-Signing bootcamp Bootcamp Alumni Bootcamp Prep Bundles California Cancer Research capstone Career Career Day citibike clustering Coding Course Demo Course Report D3.js data Data Analyst data science Data Science Academy Data Science Bootcamp Data science jobs Data Science Reviews Data Scientist Data Scientist Jobs data visualization Deep Learning Demo Day Discount dplyr employer networking feature engineering Finance Financial Data Science Flask gbm Get Hired ggplot2 googleVis Hadoop higgs boson Hiring hiring partner events Hiring Partners Industry Experts Instructor Blog Instructor Interview Job Job Placement Jobs Jon Krohn JP Morgan Chase Kaggle Kickstarter lasso regression Lead Data Scienctist Lead Data Scientist leaflet linear regression Logistic Regression machine learning Maps matplotlib Medical Research Meet the team meetup Networking neural network Neural networks New Courses nlp NYC NYC Data Science nyc data science academy NYC Open Data NYCDSA NYCDSA Alumni Online Online Bootcamp Online Training Open Data painter pandas Part-time Portfolio Development prediction Prework Programming PwC python python machine learning python scrapy python web scraping python webscraping Python Workshop R R language R Programming R Shiny r studio R Visualization R Workshop R-bloggers random forest Ranking recommendation recommendation system regression Remote remote data science bootcamp Scrapy scrapy visualization seaborn Selenium sentiment analysis Shiny Shiny Dashboard Spark Special Special Summer Sports statistics streaming Student Interview Student Showcase SVM Switchup Tableau team TensorFlow Testimonial tf-idf Top Data Science Bootcamp twitter visualization web scraping Weekend Course What to expect word cloud word2vec XGBoost yelp