Data Web Scraping Yelp to Explore Bars/Pubs in NY
The skills the author demoed here can be learned through taking Data Science with Machine Learning bootcamp with NYC Data Science Academy.
Bars of NYC
Data analyzing the Bars and Pubs of New York reviewed in Yelp to have a better understanding of the industry as a new investor.
Motivation
Like many, I enjoy a cold beer every now and then. Exploring new bars around the city is one of my hobbies, so this research project was of personal interest.
Procedures
To get the data I scraped Yelp using the Python tool Scrapy. The project is fully coded in Python as well. There are 100 initial pages, each has the links to 10 bars, and each one of those one thousand bars has its own page. To start I recognized the pattern of the 100 initial pages and I implemented that in my algorithm to set those as the starting URLs in a list comprehension. Then I went on to extract the 10 hrefs of each page, giving me the links for all the 1000 pages of the bars that I need to start extracting the information from. Each of the pages look like the one pictured below:
I easily got all the standard information from the page through each respective xpath except for the attributes of each bar. The attributes presented a challenge because not all bars have the same attributes. Both the attributes and descriptions were embedded as text under the same class, which made targeting a desired singular attribute from all the bars (for a proper analysis) rather challenging. I was able to use my engineering skills to solve the problem. I downloaded on the fly all attributes and their respective descriptions, then zipped them in a dictionary. Then I used a conditional as the xpath for the desired items to be scraped, as shown below a snippet of my algorithm for bar reservations:
Algorithm for Bar Reservations
key=response.xpath('//div[contains(@class,"short-def-list")]//dt/text()').extract()
key = [x.strip() for x in key]
value = response.xpath('//div[contains(@class,"short-def-list")]//dd/text()').extract()
value = [x.strip() for x in value]
attributes = dict(zip(key,value))
barReservations = (attributes['Takes Reservations'] if 'Takes Reservations' in attributes.keys() else 'N/A')
Findings, Data Analysis and Percentage Distribution
Yelp lists 13 bars in NY with the qualification of five star bars. However, 77% of them have less than 35 reviews, and a couple of them only one or two reviews. In order to find the true five star bar of NY I used the number of reviews as a validation tool. Accordingly The Alcove in Woodside Queens, with Gastro pub as its category, price range of 2 out of 4, with the most reviews (386) of all the five star pubs is to be regarded as the true five star bar of NYC.
Top 5 Neighborhoods
The top five Neighborhoods with the most bars in NY are(from most popular): Midtown West, Midtown East, Upper West Side, West Village and East Village. Midtown is where you’d find most of the priciest bars, They include: The Grill, The Modern, Grand Salon, Butcher and Banker, Gabriel Kreuther, Upstairs at 21, and Ager, all of which rank as four out of four in price and require Dressy/Formal(Jacket Required). These bars also all had ratings of four stars or higher with an average of 217 reviews.
Popular Bars
However, popularity is correlated with more moderate prices. The most popular bars of NY: Angel's Share, Blend on the Water and the Smith, all with a price range of two out of four and no special attire required. They are all four stars and above with at least 1,810 reviews each, clearly great bars. Very interesting note: there are 18 bars in NY that do not accept credit cards, three of which are in only one neighborhood. East village has: Tile Bar, Big Bar and Fish Bar, which are all categorized as Dive Bars (not the best place to take someone on a date). Not surprising that all these 3 bars have a price range of one out of four.
Common Number of Starts
The most common number of stars for the bars in NYC by far is four stars, followed by 3.5 and 4.5 stars. That means that the bars of NYC overall have a good reputation as compared to bars from other states in Yelp with only three stars as their mode. New Yorkers tend to review their bars in a reasonable manner in comparison with other states (most common number of reviews being between 75 and 150), with the most common number of reviews being between 100 and 200 reviews.
Conclusion
From a business point of view I am certain that the information gathered in this project (extracted from the website Yelp) can successfully be used to advise new investors in this industry. Let’s take the example of one of the biggest issues in NYC -- parking. One might think that a key element to a successful establishment would be one that offers this scare resource. Surprisingly, though, my study shows that there is absolute no relationship between offering parking and high rankings.
So the question of course, is, what are the key attributes of a successful NYC bar? The typical successful/profitable bar in NYC is casual, with a price range of two out of four (Too expensive will make it too exclusive and we want as much variety as possible). The bar should be presentable enough to be able to reach four stars and most certainly accept credit cards for payment.
You can find my code and slides here at Sal's GitHub scrapy project
Thank you so much for reading, looking forward to any feedback.
About Author
