Data Analyzing NYC Hotels on TripAdvisor

Posted on Feb 3, 2020
The skills the author demoed here can be learned through taking Data Science with Machine Learning bootcamp with NYC Data Science Academy.


New York City is often described as one of the centers of the world. Data shows the city attracts outsiders for its diverse culture and business opportunities. Hotel businesses strive to meet this demand, however; competition is fierce. It is important to keep up with the competition and reflect on the company's actions for business success. 

This was what I was considering when I took on this project. It was a great way of improving my understanding of learning the Scrapy package on Python. The objectives for this project include discovering: business opportunities to grow, successful business strategies, common consumer interest. We are going to answer some of these questions by analyzing the location and listed services on the website.


For this project, we are going to scrap the NYC hotel listings of The website displays a table of hotel listings, each listing provides a summary and a hyperlink. These links leads to a detailed page of it’s listing. By inspecting these elements, l built a scrapy spider to compile a dataset of the details of each of the listed hotels. Before building the scrapy spider, observe the html links. One indication that information can be scraped by a spider is html links changing when interacting with the website. Within minutes, I was able to scrap 714 hotel details on TripAdvisor. 

Surface Data Analysis

Let’s start with identifying the density and location of these hotels. By using a folium package on Python, I’ve visually produced a map that points the location of these hotels. However, address locations are not recognized by folium and would need to be converted into geographic coordinates. Warmer colors represent higher concentration of hotels, and cooler colors display a lower concentration of hotels. Location with no color did not have a hotel listing. It’s clear most NYC hotels are located within the borough of Manhattan. The location by JFK airport (in the borough of Queens) is interesting because the absence of color indicates that these hotel locations were not posted on TripAdvisor.

Data Analyzing NYC Hotels on TripAdvisor
Heatmap of all hotels listed as NYC

The boxplot below compares the overall rating of hotels to a ratio of their excellent reviews to total reviews. This method will measure their success by evaluating the value of their overall rating. Maintaining a high customer satisfaction is key to a successful business.

However, this method only works with established companies. For this study, I classified established companies as a company with a location that lists at least 50 reviews. As the graph shows there is a direct relationship between overall rating and the ratio of excellent reviews to total reviews. This is important because we are going to use this information to create a classification class of hotels.

Data Analyzing NYC Hotels on TripAdvisor

Based on this graph, we are going to define the following terms to describe hotels: excellent, good, and poor. 

  • Excellent: Overall ratings > 4.0
  • Good: Overall ratings 3.0 - 4.0 
  • Poor: Overall ratings < 3.0

Success Variable: Data on Location

Based on our classification system we have 157 excellent hotels, 284 good hotels, and 78 poor hotels. 

While the location may play a role in business opportunity, it doesn’t determine business success. Focusing on Manhattan, there is a higher density of hotels in Midtown and downtown Wall Street. However, the hotels in these locations are not differentiated by success. A business in Wall Street is just as likely to be rated excellent, good or poor compared to a business in any other location.

Success Variable: Services

The word cloud above is a compilation of hotel services from all hotel listings, but a higher frequency of service will increase the text size. The word cloud shows that all the hotels regardless of their success provide things like flat screen TVs, high speed internet wifi, air conditioning, and the choice between smoking and non-smoking. Therefore, an expectation is set by the consumers for businesses to provide these services. A company that doesn’t provide these services will hurt its business without them. We are going to differentiate the services provided by the three classes of success to identify what services separate a poor hotel from an excellent hotel.

Word cloud of services from excellent hotels. 214 unique services found, an

average of 14.75 listed services per hotel.

Good hotel services word cloud. 168 unique services found, average of 14.80 listed services per hotel.
Word cloud of services from poor hotels. 39 unique services found, average of 14.75 listed services per hotel.

As the quality of class increases, the number of unique services also increases. However, the average listed services per hotel between the three classes doesn’t have significant change. Therefore, we can conclude excellent hotels aren’t listing more services, but are listing more unique services to separate themselves from their competition. Something interesting to note here is the higher frequency of “free” in poor hotels. There is little evidence to prove causation between usage of “free” and lower ratings, but is interesting to look into for the future.

Conclusion and Next Steps

Based on the 714 hotel entries scraped from, we can conclude that most of the hotels are located in Manhattan. However, there is no distinct relationship between location and business success. To promotd business growth, companies should list unique services and aim to win excellent reviews. This information is valuable for consulting companies, hotel businesses, and completing travel platform sites.

There are a lot of different directions in which it is possible to take this project. You can delve deeper into this analysis by pulling information from other competing sites and comparing their location and services to their success. Creation of an application that will allow users to find a hotel with the desired features is another avenue. Incorporating a method to efficiently extract price deals for analyzing price prediction is the most challenging approach because scrapy is unable to extract prices from their respective listing. In that case, I would recommend using selenium for extracting information.

Interested in my spider? You can view the details my Github.

About Author

Yasuhiro Shinohara

Recent Data Scientist Fellow at NYC Data Science Academy. A former teacher with a Masters in Education with a focus on Earth Sciences. Emphasizes clear communication of data analytics through visualization and public speaking to promote positive and...
View all posts by Yasuhiro Shinohara >

Related Articles

Leave a Comment

No comments found.

View Posts by Categories

Our Recent Popular Posts

View Posts by Tags

#python #trainwithnycdsa 2019 2020 Revenue 3-points agriculture air quality airbnb airline alcohol Alex Baransky algorithm alumni Alumni Interview Alumni Reviews Alumni Spotlight alumni story Alumnus ames dataset ames housing dataset apartment rent API Application artist aws bank loans beautiful soup Best Bootcamp Best Data Science 2019 Best Data Science Bootcamp Best Data Science Bootcamp 2020 Best Ranked Big Data Book Launch Book-Signing bootcamp Bootcamp Alumni Bootcamp Prep boston safety Bundles cake recipe California Cancer Research capstone car price Career Career Day citibike classic cars classpass clustering Coding Course Demo Course Report covid 19 credit credit card crime frequency crops D3.js data data analysis Data Analyst data analytics data for tripadvisor reviews data science Data Science Academy Data Science Bootcamp Data science jobs Data Science Reviews Data Scientist Data Scientist Jobs data visualization database Deep Learning Demo Day Discount disney dplyr drug data e-commerce economy employee employee burnout employer networking environment feature engineering Finance Financial Data Science fitness studio Flask flight delay gbm Get Hired ggplot2 googleVis H20 Hadoop hallmark holiday movie happiness healthcare frauds higgs boson Hiring hiring partner events Hiring Partners hotels housing housing data housing predictions housing price hy-vee Income Industry Experts Injuries Instructor Blog Instructor Interview insurance italki Job Job Placement Jobs Jon Krohn JP Morgan Chase Kaggle Kickstarter las vegas airport lasso regression Lead Data Scienctist Lead Data Scientist leaflet league linear regression Logistic Regression machine learning Maps market matplotlib Medical Research Meet the team meetup methal health miami beach movie music Napoli NBA netflix Networking neural network Neural networks New Courses NHL nlp NYC NYC Data Science nyc data science academy NYC Open Data nyc property NYCDSA NYCDSA Alumni Online Online Bootcamp Online Training Open Data painter pandas Part-time performance phoenix pollutants Portfolio Development precision measurement prediction Prework Programming public safety PwC python Python Data Analysis python machine learning python scrapy python web scraping python webscraping Python Workshop R R Data Analysis R language R Programming R Shiny r studio R Visualization R Workshop R-bloggers random forest Ranking recommendation recommendation system regression Remote remote data science bootcamp Scrapy scrapy visualization seaborn seafood type Selenium sentiment analysis sentiment classification Shiny Shiny Dashboard Spark Special Special Summer Sports statistics streaming Student Interview Student Showcase SVM Switchup Tableau teachers team team performance TensorFlow Testimonial tf-idf Top Data Science Bootcamp Top manufacturing companies Transfers tweets twitter videos visualization wallstreet wallstreetbets web scraping Weekend Course What to expect whiskey whiskeyadvocate wildfire word cloud word2vec XGBoost yelp youtube trending ZORI