Using Data to Analyze: Where Should You Eat In NYC?
The skills the author demonstrated here can be learned through taking Data Science with Machine Learning bootcamp with NYC Data Science Academy.
Motivation
Everybody loves a good meal. Food has always been a big part of society’s culture and is immersed in society's day to day life. Based on data, NYC is known as having some of the best restaurants in the world and the food scene there is a big reason why tourists are attracted to come and visit. So where can we find the best restaurants? A majority of the restaurants in NYC are local establishments and aren’t well known like other chain companies. So if a tourist has no experience being in the city, they will want to know the best location to go to find the best restaurants.
Web Scraping: Opentable
Obviously a person’s food preference plays a big factor in what kind of food they like, but we can find the best locations to satisfy food cravings by looking at data. Opentable is an online restaurant reservation service founded by Chuck Templeton in 1998. The service is available in over 20 countries with around 48,000 restaurants. The data on restaurants available on the site will help us find which location in NYC has the best food. In terms of web scraping, I utilized scrapy and configured my spider to collect the data points below for each restaurant.
- Restaurant Name
- Location
- Cuisine
- Price Range
- Food Rating
- Service Rating
- Ambience Rating
- Value Rating
- Number of Reviews
- Recommendation Percentage
- Chef Name
- Dining Style
- Dress Code
- Noise Level
Scraping Limitations
After the collection of the data points, a few limitations needed to be addressed before moving forward with the analysis. First off, not all restaurants are available on Opentable. As much as society has grown on a technological standpoint there are still a handful of very good restaurants that do not take reservations or only allow call-in reservations. Secondly, whether a restaurant shows up an Opentable search depends on its availability. Try to make a reservation too far in advanced and the restaurant may not allow that. Make a reservation too late and all the open spots may be booked up. For the purposes of this study I used a two week outlook to maximize restaurant results.
Scraping Challenges
My main challenges with scraping Opentable were the similar x-paths for different data points. Data points including location, dining style, dress code, and chef name had the exact same x-paths. The problem arose when a restaurant did not provide one of those data points or the order in which they appeared was not consistent between restaurants. To solve this problem I had to clarify html tags on a deeper level than the standard class. An example of the code is shown below in red:
dining_style = response.xpath('//div[@class="_199894c6" and ./div[@class="_252cc398 _40f1eb59"]/span/text()="Dining Style"]/div[@class="_16c8fd5e _1f1541e1"]/text()').extract_first()
Data Analysis
Data on 1,441 different restaurants in NYC were collected as part of the web scraping. The first thing I wanted to solidify was a scoring system. As stated earlier, each restaurant had a food rating, service rating, ambience rating, and value rating. As shown below, each rating has a good positive correlation with the food rating. Moving forward, I will be using the food rating as a representation of how good a restaurant is because of the correlation it has with the other ratings.
Group by Location
The main question in concern is where should people go to eat good food. We first want to compare restaurant ratings by locations in NYC and see if one location stands out above the others. Charted below is a bar chart of average restaurant ratings grouped by different neighborhoods in NYC.
As seen from the chart, there is not a big discrepancy between restaurant food ratings between neighborhoods in NYC. In further viewing of the data collected, there may be factors we can eliminate to try and get a better representation of good restaurant ratings by neighborhood. Below I have laid out a few we will try.
- Some neighborhoods do not have a lot of restaurants. This does not show a good representation of how good the food scene is in those neighborhoods. We will eliminate the neighborhoods that have less than 5 restaurants.
- For the purposes of this study we want to eliminate bar/pub type restaurants. The rowdy atmosphere in those places usually result in lower food ratings and can skew some of the location's average food rating. We will remove any restaurant with an "energetic" noise level.
- When a chef name is available on the restaurant page, it adds a level of credibility to the food rating that makes it reliable. We will remove any restaurants that do not have the chef name available.
- Lastly, food rating is positively correlated to the amount of money spent on the food. The restaurants with higher food ratings unfortunately have higher prices, so we will eliminate restaurants that are in the lowest price range.
Removing all of those factors from consideration we have a bar chart that looks different from the initial chart we made. In this case, Nolita, Union Square, and East Village are our top locations for the best restaurants while Columbus Circle, South Street Seaport, and Harlem have the lowest restaurant ratings. The chart is shown below.
Conclusion and Future Work
NYC has a vast and diverse food culture. From the data it seems as the location does not make a huge impact on where you can find good restaurants in NYC. Even after eliminating restaurants with lower food ratings based on noise level, price, and a credible chef, it's not viable to say that one location stands above the other in terms of restaurant quality. I will continue to wrangle the data incorporating the type of cuisine as well as the number of reviews a restaurant has to try and pinpoint the best food in NYC. Updates to come...
Follow Along
You can follow the link provided for my source code on Github.
About Author
