Where Should You Eat In NYC?

Quoc Nguyen
Posted on Apr 28, 2019

Motivation

Everybody loves a good meal. Food has always been a big part of society’s culture and is immersed in society's day to day life. NYC is known as having some of the best restaurants in the world and the food scene there is a big reason why tourists are attracted to come and visit. So where can we find the best restaurants? A majority of the restaurants in NYC are local establishments and aren’t well known like other chain companies. So if a tourist has no experience being in the city, they will want to know the best location to go to find the best restaurants.

Web Scraping: Opentable

Obviously a person’s food preference plays a big factor in what kind of food they like, but we can find the best locations to satisfy food cravings by looking at data. Opentable is an online restaurant reservation service founded by Chuck Templeton in 1998. The service is available in over 20 countries with around 48,000 restaurants. The data on restaurants available on the site will help us find which location in NYC has the best food. In terms of web scraping, I utilized scrapy and configured my spider to collect the data points below for each restaurant.

  • Restaurant Name
  • Location
  • Cuisine
  • Price Range
  • Food Rating
  • Service Rating
  • Ambience Rating
  • Value Rating
  • Number of Reviews
  • Recommendation Percentage
  • Chef Name
  • Dining Style
  • Dress Code
  • Noise Level

Scraping Limitations

After the collection of the data points, a few limitations needed to be addressed before moving forward with the analysis. First off, not all restaurants are available on Opentable. As much as society has grown on a technological standpoint there are still a handful of very good restaurants that do not take reservations or only allow call-in reservations. Secondly, whether a restaurant shows up an Opentable search depends on its availability. Try to make a reservation too far in advanced and the restaurant may not allow that. Make a reservation too late and all the open spots may be booked up. For the purposes of this study I used a two week outlook to maximize restaurant results.

Scraping Challenges

My main challenges with scraping Opentable were the similar x-paths for different data points. Data points including location, dining style, dress code, and chef name had the exact same x-paths. The problem arose when a restaurant did not provide one of those data points or the order in which they appeared was not consistent between restaurants. To solve this problem I had to clarify html tags on a deeper level than the standard class. An example of the code is shown below in red:

dining_style = response.xpath('//div[@class="_199894c6" and ./div[@class="_252cc398 _40f1eb59"]/span/text()="Dining Style"]/div[@class="_16c8fd5e _1f1541e1"]/text()').extract_first()

Data Analysis

Data on 1,441 different restaurants in NYC were collected as part of the web scraping. The first thing I wanted to solidify was a scoring system. As stated earlier, each restaurant had a food rating, service rating, ambience rating, and value rating. As shown below, each rating has a good positive correlation with the food rating. Moving forward, I will be using the food rating as a representation of how good a restaurant is because of the correlation it has with the other ratings. 

Group by Location

The main question in concern is where should people go to eat good food. We first want to compare restaurant ratings by locations in NYC and see if one location stands out above the others. Charted below is a bar chart of average restaurant ratings grouped by different neighborhoods in NYC. 

As seen from the chart, there is not a big discrepancy between restaurant food ratings between neighborhoods in NYC. In further viewing of the data collected, there may be factors we can eliminate to try and get a better representation of good restaurant ratings by neighborhood. Below I have laid out a few we will try.

  1. Some neighborhoods do not have a lot of restaurants. This does not show a good representation of how good the food scene is in those neighborhoods. We will eliminate the neighborhoods that have less than 5 restaurants.
  2. For the purposes of this study we want to eliminate bar/pub type restaurants. The rowdy atmosphere in those places usually result in lower food ratings and can skew some of the location's average food rating. We will remove any restaurant with an "energetic" noise level.
  3. When a chef name is available on the restaurant page, it adds a level of credibility to the food rating that makes it reliable. We will remove any restaurants that do not have the chef name available.
  4. Lastly, food rating is positively correlated to the amount of money spent on the food. The restaurants with higher food ratings unfortunately have higher prices, so we will eliminate restaurants that are in the lowest price range.

Removing all of those factors from consideration we have a bar chart that looks different from the initial chart we made. In this case, Nolita, Union Square, and East Village are our top locations for the best restaurants while Columbus Circle, South Street Seaport, and Harlem have the lowest restaurant ratings. The chart is shown below.

Conclusion and Future Work

NYC has a vast and diverse food culture. From the data it seems as the location does not make a huge impact on where you can find good restaurants in NYC. Even after eliminating restaurants with lower food ratings based on noise level, price, and a credible chef, it's not viable to say that one location stands above the other in terms of restaurant quality. I will continue to wrangle the data incorporating the type of cuisine as well as the number of reviews a restaurant has to try and pinpoint the best food in NYC. Updates to come...

Follow Along

You can follow the link provided for my source code on Github

About Author

Quoc Nguyen

Quoc Nguyen

Quoc graduated from Cornell University in 2015 with a B.S. in Civil Engineering and continued to obtain his M.Eng in Engineering Management in 2016. As a professional, Quoc worked at Skanska, an international construction and development group, for...
View all posts by Quoc Nguyen >

Related Articles

Leave a Comment

No comments found.

View Posts by Categories


Our Recent Popular Posts


View Posts by Tags

2019 airbnb alumni Alumni Interview Alumni Spotlight alumni story Alumnus API artist aws beautiful soup Best Bootcamp Best Data Science 2019 Best Data Science Bootcamp Big Data bootcamp Bootcamp Prep Bundles California Cancer Research capstone Career citibike clustering Coding Course Report D3.js data Data Analyst data science Data Science Academy Data Science Bootcamp Data Scientist Data Scientist Jobs data visualization Deep Learning Demo Day dplyr employer networking feature engineering Finance Financial Data Science Flask gbm Get Hired ggplot2 googleVis Hadoop higgs boson Hiring hiring partner events Industry Experts Job JP Morgan Chase Kaggle lasso regression Lead Data Scienctist Lead Data Scientist leaflet linear regression Logistic Regression machine learning Maps matplotlib Medical Research meetup Networking neural network Neural networks New Courses nlp NYC NYC Data Science nyc data science academy NYC Open Data NYCDSA NYCDSA Alumni Open Data painter pandas Portfolio Development prediction Programming PwC python python machine learning python scrapy python web scraping python webscraping Python Workshop R R language R Programming R Shiny r studio R Visualization R Workshop R-bloggers random forest recommendation recommendation system regression Scrapy scrapy visualization seaborn Selenium sentiment analysis Shiny Shiny Dashboard Spark Special Special Summer Sports statistics streaming Student Interview Student Showcase SVM Tableau Testimonial tf-idf Top Data Science Bootcamp twitter visualization web scraping What to expect word cloud word2vec XGBoost yelp