Data Web Scraping Yelp to Explore Bars/Pubs in NY

Posted on Feb 18, 2018
The skills the author demoed here can be learned through taking Data Science with Machine Learning bootcamp with NYC Data Science Academy.

Bars of NYC

Data analyzing the Bars and Pubs of New York reviewed in Yelp to have a better understanding of the industry as a new investor.

Motivation

Like many, I enjoy a cold beer every now and then. Exploring new bars around the city is one of my hobbies, so this research project was of personal interest.

Procedures

To get the data I scraped Yelp using the Python tool Scrapy. The project is fully coded in Python as well. There are 100 initial pages, each has the links to 10 bars, and each one of those one thousand bars has its own page. To start I recognized the pattern of the 100 initial pages and I implemented that in my algorithm to set those as the starting URLs in a list comprehension. Then I went on to extract the 10 hrefs of each page, giving me the links for all the 1000 pages of the bars that I need to start extracting the information from. Each of the pages look like the one pictured below:

bars

I easily got all the standard information from the page through each respective xpath except for the attributes of each bar. The attributes presented a challenge because not all bars have the same attributes. Both the attributes and descriptions were embedded as text under the same class, which made targeting a desired singular attribute from all the bars (for a proper analysis) rather challenging. I was able to use my engineering skills to solve the problem. I downloaded on the fly all attributes and their respective descriptions, then zipped them in a dictionary. Then I used a conditional as the xpath for the desired items to be scraped, as shown below a snippet of my algorithm for bar reservations:

Algorithm for Bar Reservations

key=response.xpath('//div[contains(@class,"short-def-list")]//dt/text()').extract()
key = [x.strip() for x in key]

value = response.xpath('//div[contains(@class,"short-def-list")]//dd/text()').extract()
value = [x.strip() for x in value]

attributes = dict(zip(key,value))

barReservations = (attributes['Takes Reservations'] if 'Takes Reservations' in attributes.keys() else 'N/A')

Findings, Data Analysis and Percentage Distribution

Yelp lists 13 bars in NY with the qualification of five star bars. However, 77% of them have less than 35 reviews, and a couple of them only one or two reviews. In order to find the true five star bar of NY I used the number of reviews as a validation tool. Accordingly The Alcove in Woodside Queens, with Gastro pub as its category, price range of 2 out of 4, with the most reviews (386) of all the five star pubs is to be regarded as the true five star bar of NYC.

null

Top 5 Neighborhoods

The top five Neighborhoods with the most bars in NY are(from most popular): Midtown West, Midtown East, Upper West Side, West Village and East Village. Midtown is where you’d find most of the priciest bars, They include: The Grill, The Modern, Grand Salon, Butcher and Banker, Gabriel Kreuther, Upstairs at 21, and Ager, all of which rank as four out of four in price and require Dressy/Formal(Jacket Required). These bars also all had ratings of four stars or higher with an average of 217 reviews.

Popular Bars

null

However, popularity is correlated with more moderate prices. The most popular bars of NY: Angel's Share, Blend on the Water and the Smith, all with a price range of two out of four and no special attire required. They are all four stars and above with at least 1,810 reviews each, clearly great bars. Very interesting note: there are 18 bars in NY that do not accept credit cards, three of which are in only one neighborhood. East village has: Tile Bar, Big Bar and Fish Bar, which are all categorized as Dive Bars (not the best place to take someone on a date). Not surprising that all these 3 bars have a price range of one out of four.

Common Number of Starts

null

The most common number of stars for the bars in NYC by far is four stars, followed by 3.5 and 4.5 stars. That means that the bars of NYC overall have a good reputation as compared to bars from other states in Yelp with only three stars as their mode. New Yorkers tend to review their bars in a reasonable manner in comparison with other states (most common number of reviews being between 75 and 150), with the most common number of reviews being between 100 and 200 reviews.

null
null

Conclusion

From a business point of view I am certain that the information gathered in this project (extracted from the website Yelp) can successfully be used to advise new investors in this industry. Let’s take the example of one of the biggest issues in NYC -- parking. One might think that a key element to a successful establishment would be one that offers this scare resource. Surprisingly, though, my study shows that there is absolute no relationship between offering parking and high rankings.

null

So the question of course, is, what are the key attributes of a successful NYC bar? The typical successful/profitable bar in NYC is casual, with a price range of two out of four (Too expensive will make it too exclusive and we want as much variety as possible). The bar should be presentable enough to be able to reach four stars and most certainly accept credit cards for payment.

You can find my code and slides here at Sal's GitHub scrapy project

Thank you so much for reading, looking forward to any feedback.

About Author

Sal Lascano

Sal received his B.S. from Saint Peter's University in Jersey City, NJ with a major in Mathematics and a minor in Secondary Education. He worked for four years as an account manager and sales manager in the Interior...
View all posts by Sal Lascano >

Related Articles

Leave a Comment

No comments found.

View Posts by Categories


Our Recent Popular Posts


View Posts by Tags

#python #trainwithnycdsa 2019 2020 Revenue 3-points agriculture air quality airbnb airline alcohol Alex Baransky algorithm alumni Alumni Interview Alumni Reviews Alumni Spotlight alumni story Alumnus ames dataset ames housing dataset apartment rent API Application artist aws bank loans beautiful soup Best Bootcamp Best Data Science 2019 Best Data Science Bootcamp Best Data Science Bootcamp 2020 Best Ranked Big Data Book Launch Book-Signing bootcamp Bootcamp Alumni Bootcamp Prep boston safety Bundles cake recipe California Cancer Research capstone car price Career Career Day citibike classic cars classpass clustering Coding Course Demo Course Report covid 19 credit credit card crime frequency crops D3.js data data analysis Data Analyst data analytics data for tripadvisor reviews data science Data Science Academy Data Science Bootcamp Data science jobs Data Science Reviews Data Scientist Data Scientist Jobs data visualization database Deep Learning Demo Day Discount disney dplyr drug data e-commerce economy employee employee burnout employer networking environment feature engineering Finance Financial Data Science fitness studio Flask flight delay gbm Get Hired ggplot2 googleVis H20 Hadoop hallmark holiday movie happiness healthcare frauds higgs boson Hiring hiring partner events Hiring Partners hotels housing housing data housing predictions housing price hy-vee Income Industry Experts Injuries Instructor Blog Instructor Interview insurance italki Job Job Placement Jobs Jon Krohn JP Morgan Chase Kaggle Kickstarter las vegas airport lasso regression Lead Data Scienctist Lead Data Scientist leaflet league linear regression Logistic Regression machine learning Maps market matplotlib Medical Research Meet the team meetup methal health miami beach movie music Napoli NBA netflix Networking neural network Neural networks New Courses NHL nlp NYC NYC Data Science nyc data science academy NYC Open Data nyc property NYCDSA NYCDSA Alumni Online Online Bootcamp Online Training Open Data painter pandas Part-time performance phoenix pollutants Portfolio Development precision measurement prediction Prework Programming public safety PwC python Python Data Analysis python machine learning python scrapy python web scraping python webscraping Python Workshop R R Data Analysis R language R Programming R Shiny r studio R Visualization R Workshop R-bloggers random forest Ranking recommendation recommendation system regression Remote remote data science bootcamp Scrapy scrapy visualization seaborn seafood type Selenium sentiment analysis sentiment classification Shiny Shiny Dashboard Spark Special Special Summer Sports statistics streaming Student Interview Student Showcase SVM Switchup Tableau teachers team team performance TensorFlow Testimonial tf-idf Top Data Science Bootcamp Top manufacturing companies Transfers tweets twitter videos visualization wallstreet wallstreetbets web scraping Weekend Course What to expect whiskey whiskeyadvocate wildfire word cloud word2vec XGBoost yelp youtube trending ZORI