Web Scraping Yelp to Explore Bars/Pubs in NY

Sal Lascano
Posted on Feb 18, 2018

Bars of NYC

Analyzing the Bars and Pubs of New York reviewed in Yelp to have a better understanding of the industry as a new investor.

Motivation

Like many, I enjoy a cold beer every now and then. Exploring new bars around the city is one of my hobbies, so this research project was of personal interest.

Procedures

To get the data I scraped Yelp using the Python tool Scrapy. The project is fully coded in Python as well. There are 100 initial pages, each has the links to 10 bars, and each one of those one thousand bars has its own page. To start I recognized the pattern of the 100 initial pages and I implemented that in my algorithm to set those as the starting URLs in a list comprehension. Then I went on to extract the 10 hrefs of each page, giving me the links for all the 1000 pages of the bars that I need to start extracting the information from. Each of the pages look like the one pictured below:

bars

I easily got all the standard information from the page through each respective xpath except for the attributes of each bar. The attributes presented a challenge because not all bars have the same attributes. Both the attributes and descriptions were embedded as text under the same class, which made targeting a desired singular attribute from all the bars (for a proper analysis) rather challenging. I was able to use my engineering skills to solve the problem. I downloaded on the fly all attributes and their respective descriptions, then zipped them in a dictionary. Then I used a conditional as the xpath for the desired items to be scraped, as shown below a snippet of my algorithm for bar reservations:

key=response.xpath('//div[contains(@class,"short-def-list")]//dt/text()').extract()
key = [x.strip() for x in key]

value = response.xpath('//div[contains(@class,"short-def-list")]//dd/text()').extract()
value = [x.strip() for x in value]

attributes = dict(zip(key,value))

barReservations = (attributes['Takes Reservations'] if 'Takes Reservations' in attributes.keys() else 'N/A')

Findings, Analysis and Percentage Distribution

Yelp lists 13 bars in NY with the qualification of five star bars. However, 77% of them have less than 35 reviews, and a couple of them only one or two reviews. In order to find the true five star bar of NY I used the number of reviews as a validation tool. Accordingly The Alcove in Woodside Queens, with Gastro pub as its category, price range of 2 out of 4, with the most reviews (386) of all the five star pubs is to be regarded as the true five star bar of NYC.

null

The top five Neighborhoods with the most bars in NY are(from most popular): Midtown West, Midtown East, Upper West Side, West Village and East Village. Midtown is where you’d find most of the priciest bars, They include: The Grill, The Modern, Grand Salon, Butcher and Banker, Gabriel Kreuther, Upstairs at 21, and Ager, all of which rank as four out of four in price and require Dressy/Formal(Jacket Required). These bars also all had ratings of four stars or higher with an average of 217 reviews.

null

However, popularity is correlated with more moderate prices. The most popular bars of NY: Angel's Share, Blend on the Water and the Smith, all with a price range of two out of four and no special attire required. They are all four stars and above with at least 1,810 reviews each, clearly great bars. Very interesting note: there are 18 bars in NY that do not accept credit cards, three of which are in only one neighborhood. East village has: Tile Bar, Big Bar and Fish Bar, which are all categorized as Dive Bars (not the best place to take someone on a date). Not surprising that all these 3 bars have a price range of one out of four.

null

The most common number of stars for the bars in NYC by far is four stars, followed by 3.5 and 4.5 stars. That means that the bars of NYC overall have a good reputation as compared to bars from other states in Yelp with only three stars as their mode. New Yorkers tend to review their bars in a reasonable manner in comparison with other states (most common number of reviews being between 75 and 150), with the most common number of reviews being between 100 and 200 reviews.

null
null

Conclusion

From a business point of view I am certain that the information gathered in this project (extracted from the website Yelp) can successfully be used to advise new investors in this industry. Let’s take the example of one of the biggest issues in NYC -- parking. One might think that a key element to a successful establishment would be one that offers this scare resource. Surprisingly, though, my study shows that there is absolute no relationship between offering parking and high rankings.

null

So the question of course, is, what are the key attributes of a successful NYC bar? The typical successful/profitable bar in NYC is casual, with a price range of two out of four (Too expensive will make it too exclusive and we want as much variety as possible). The bar should be presentable enough to be able to reach four stars and most certainly accept credit cards for payment.

You can find my code and slides here at Sal's GitHub scrapy project

Thank you so much for reading, looking forward to any feedback.

About Author

Sal Lascano

Sal Lascano

Sal received his B.S. from Saint Peter's University in Jersey City, NJ with a major in Mathematics and a minor in Secondary Education. He worked for four years as an account manager and sales manager in the Interior...
View all posts by Sal Lascano >

Related Articles

Leave a Comment

No comments found.

View Posts by Categories


Our Recent Popular Posts


View Posts by Tags

2019 airbnb alumni Alumni Interview Alumni Spotlight alumni story Alumnus API artist aws beautiful soup Best Bootcamp Best Data Science 2019 Best Data Science Bootcamp Big Data bootcamp Bootcamp Prep Bundles California Cancer Research capstone Career citibike clustering Coding Course Report D3.js data Data Analyst data science Data Science Academy Data Science Bootcamp Data Scientist Data Scientist Jobs data visualization Deep Learning Demo Day dplyr employer networking feature engineering Finance Financial Data Science Flask gbm Get Hired ggplot2 googleVis Hadoop higgs boson Hiring hiring partner events Industry Experts Job JP Morgan Chase Kaggle lasso regression Lead Data Scienctist Lead Data Scientist leaflet linear regression Logistic Regression machine learning Maps matplotlib Medical Research meetup Networking neural network Neural networks New Courses nlp NYC NYC Data Science nyc data science academy NYC Open Data NYCDSA NYCDSA Alumni Open Data painter pandas Portfolio Development prediction Programming PwC python python machine learning python scrapy python web scraping python webscraping Python Workshop R R language R Programming R Shiny r studio R Visualization R Workshop R-bloggers random forest recommendation recommendation system regression Scrapy scrapy visualization seaborn Selenium sentiment analysis Shiny Shiny Dashboard Spark Special Special Summer Sports statistics streaming Student Interview Student Showcase SVM Tableau Testimonial tf-idf Top Data Science Bootcamp twitter visualization web scraping What to expect word cloud word2vec XGBoost yelp