Using Data to Analyze: Where Should You Eat In NYC?

Posted on Apr 28, 2019

The skills the author demonstrated here can be learned through taking Data Science with Machine Learning bootcamp with NYC Data Science Academy.


Everybody loves a good meal. Food has always been a big part of society’s culture and is immersed in society's day to day life. Based on data, NYC is known as having some of the best restaurants in the world and the food scene there is a big reason why tourists are attracted to come and visit. So where can we find the best restaurants? A majority of the restaurants in NYC are local establishments and aren’t well known like other chain companies. So if a tourist has no experience being in the city, they will want to know the best location to go to find the best restaurants.

Web Scraping: Opentable

Obviously a person’s food preference plays a big factor in what kind of food they like, but we can find the best locations to satisfy food cravings by looking at data. Opentable is an online restaurant reservation service founded by Chuck Templeton in 1998. The service is available in over 20 countries with around 48,000 restaurants. The data on restaurants available on the site will help us find which location in NYC has the best food. In terms of web scraping, I utilized scrapy and configured my spider to collect the data points below for each restaurant.

  • Restaurant Name
  • Location
  • Cuisine
  • Price Range
  • Food Rating
  • Service Rating
  • Ambience Rating
  • Value Rating
  • Number of Reviews
  • Recommendation Percentage
  • Chef Name
  • Dining Style
  • Dress Code
  • Noise Level

Scraping Limitations

After the collection of the data points, a few limitations needed to be addressed before moving forward with the analysis. First off, not all restaurants are available on Opentable. As much as society has grown on a technological standpoint there are still a handful of very good restaurants that do not take reservations or only allow call-in reservations. Secondly, whether a restaurant shows up an Opentable search depends on its availability. Try to make a reservation too far in advanced and the restaurant may not allow that. Make a reservation too late and all the open spots may be booked up. For the purposes of this study I used a two week outlook to maximize restaurant results.

Scraping Challenges

My main challenges with scraping Opentable were the similar x-paths for different data points. Data points including location, dining style, dress code, and chef name had the exact same x-paths. The problem arose when a restaurant did not provide one of those data points or the order in which they appeared was not consistent between restaurants. To solve this problem I had to clarify html tags on a deeper level than the standard class. An example of the code is shown below in red:

dining_style = response.xpath('//div[@class="_199894c6" and ./div[@class="_252cc398 _40f1eb59"]/span/text()="Dining Style"]/div[@class="_16c8fd5e _1f1541e1"]/text()').extract_first()

Data Analysis

Data on 1,441 different restaurants in NYC were collected as part of the web scraping. The first thing I wanted to solidify was a scoring system. As stated earlier, each restaurant had a food rating, service rating, ambience rating, and value rating. As shown below, each rating has a good positive correlation with the food rating. Moving forward, I will be using the food rating as a representation of how good a restaurant is because of the correlation it has with the other ratings. 

Group by Location

The main question in concern is where should people go to eat good food. We first want to compare restaurant ratings by locations in NYC and see if one location stands out above the others. Charted below is a bar chart of average restaurant ratings grouped by different neighborhoods in NYC. 

As seen from the chart, there is not a big discrepancy between restaurant food ratings between neighborhoods in NYC. In further viewing of the data collected, there may be factors we can eliminate to try and get a better representation of good restaurant ratings by neighborhood. Below I have laid out a few we will try.

  1. Some neighborhoods do not have a lot of restaurants. This does not show a good representation of how good the food scene is in those neighborhoods. We will eliminate the neighborhoods that have less than 5 restaurants.
  2. For the purposes of this study we want to eliminate bar/pub type restaurants. The rowdy atmosphere in those places usually result in lower food ratings and can skew some of the location's average food rating. We will remove any restaurant with an "energetic" noise level.
  3. When a chef name is available on the restaurant page, it adds a level of credibility to the food rating that makes it reliable. We will remove any restaurants that do not have the chef name available.
  4. Lastly, food rating is positively correlated to the amount of money spent on the food. The restaurants with higher food ratings unfortunately have higher prices, so we will eliminate restaurants that are in the lowest price range.

Removing all of those factors from consideration we have a bar chart that looks different from the initial chart we made. In this case, Nolita, Union Square, and East Village are our top locations for the best restaurants while Columbus Circle, South Street Seaport, and Harlem have the lowest restaurant ratings. The chart is shown below.

Conclusion and Future Work

NYC has a vast and diverse food culture. From the data it seems as the location does not make a huge impact on where you can find good restaurants in NYC. Even after eliminating restaurants with lower food ratings based on noise level, price, and a credible chef, it's not viable to say that one location stands above the other in terms of restaurant quality. I will continue to wrangle the data incorporating the type of cuisine as well as the number of reviews a restaurant has to try and pinpoint the best food in NYC. Updates to come...

Follow Along

You can follow the link provided for my source code on Github

About Author

Quoc Nguyen

Quoc graduated from Cornell University in 2015 with a B.S. in Civil Engineering and continued to obtain his M.Eng in Engineering Management in 2016. As a professional, Quoc worked at Skanska, an international construction and development group, for...
View all posts by Quoc Nguyen >

Related Articles

Leave a Comment

No comments found.

View Posts by Categories

Our Recent Popular Posts

View Posts by Tags

#python #trainwithnycdsa 2019 2020 Revenue 3-points agriculture air quality airbnb airline alcohol Alex Baransky algorithm alumni Alumni Interview Alumni Reviews Alumni Spotlight alumni story Alumnus ames dataset ames housing dataset apartment rent API Application artist aws bank loans beautiful soup Best Bootcamp Best Data Science 2019 Best Data Science Bootcamp Best Data Science Bootcamp 2020 Best Ranked Big Data Book Launch Book-Signing bootcamp Bootcamp Alumni Bootcamp Prep boston safety Bundles cake recipe California Cancer Research capstone car price Career Career Day citibike classic cars classpass clustering Coding Course Demo Course Report covid 19 credit credit card crime frequency crops D3.js data data analysis Data Analyst data analytics data for tripadvisor reviews data science Data Science Academy Data Science Bootcamp Data science jobs Data Science Reviews Data Scientist Data Scientist Jobs data visualization database Deep Learning Demo Day Discount disney dplyr drug data e-commerce economy employee employee burnout employer networking environment feature engineering Finance Financial Data Science fitness studio Flask flight delay gbm Get Hired ggplot2 googleVis H20 Hadoop hallmark holiday movie happiness healthcare frauds higgs boson Hiring hiring partner events Hiring Partners hotels housing housing data housing predictions housing price hy-vee Income Industry Experts Injuries Instructor Blog Instructor Interview insurance italki Job Job Placement Jobs Jon Krohn JP Morgan Chase Kaggle Kickstarter las vegas airport lasso regression Lead Data Scienctist Lead Data Scientist leaflet league linear regression Logistic Regression machine learning Maps market matplotlib Medical Research Meet the team meetup methal health miami beach movie music Napoli NBA netflix Networking neural network Neural networks New Courses NHL nlp NYC NYC Data Science nyc data science academy NYC Open Data nyc property NYCDSA NYCDSA Alumni Online Online Bootcamp Online Training Open Data painter pandas Part-time performance phoenix pollutants Portfolio Development precision measurement prediction Prework Programming public safety PwC python Python Data Analysis python machine learning python scrapy python web scraping python webscraping Python Workshop R R Data Analysis R language R Programming R Shiny r studio R Visualization R Workshop R-bloggers random forest Ranking recommendation recommendation system regression Remote remote data science bootcamp Scrapy scrapy visualization seaborn seafood type Selenium sentiment analysis sentiment classification Shiny Shiny Dashboard Spark Special Special Summer Sports statistics streaming Student Interview Student Showcase SVM Switchup Tableau teachers team team performance TensorFlow Testimonial tf-idf Top Data Science Bootcamp Top manufacturing companies Transfers tweets twitter videos visualization wallstreet wallstreetbets web scraping Weekend Course What to expect whiskey whiskeyadvocate wildfire word cloud word2vec XGBoost yelp youtube trending ZORI