Data Web Scraping Restaurant Logs
The skills the author demoed here can be learned through taking Data Science with Machine Learning bootcamp with NYC Data Science Academy.
Preface
This post is about using web scraped online reviews data to (1) create a record log that restaurants can utilize to improve business performance and (2) create a performance evaluation and comparison tool. The latter can be particularly valuable when comparing the performance of different locations of a franchise or chain restaurants of the same brand or how one brand compares to another within the same geographical area or location. The basic ideas are also applicable to regular restaurants or any type of businesses that are subject to online customer reviews. I use web scraped Yelp ratings of Shake Shack and In-N-Out Burger to illustrate my ideas.
Introduction
Online reviews are a mainstay of modern consumerism. In the past, a restaurant that wanted customer feedback to understand what could be improved would have to rely on getting note in its suggestion box. This was mainly for the benefit of the restaurant. Customers then would rely on articles written by food critics to help them decide where to eat. But times have changed, and we now have a slew of online customer reviews websites, like Yelp and TripAdvisor, that, in some sense, benefits both the customers and the restaurants.
A rating, represented by number of stars (Yelp) or bubbles (TripAdvisor), is the most important part of an online review. It is the epitome of a customer’s impression about her dining experience. An online review usually also includes some content that tells online readers what is good or bad about the restaurant. Sometimes the review tells readers explicitly whether they should patronize the restaurant or not.
This arrangement creates a special scenario whereby restaurants are “exposed” to the public based on what customers are saying about them. The basic idea behind this scenario is not new. Food critics in the past often had the power to dictate the success or failure of restaurants through media exposure. But now some of that power is extended to the public. These days Yelp or TripAdvisor play the role of food critic ambassador, collecting and presenting customer reviews online.
Motivation
It is not surprising then that restaurants nowadays are under additional pressure to work more on their media image to stay competitive or even stay in business. A few weeks prior to this blog post, it was reported that after a customer left a 3-star rating on Yelp, a restaurant manager went as far as going to the customer’s house in the evening to talk about the less-than-favorable review. While his concern is understandable, there are certain lines that just should not be crossed.
Instead of reaching out to people in intimidating ways, I propose that restaurants, in particular chain restaurants, cope with online customer reviews by using the ratings as an aid to help improve their businesses. I illustrate my proposal using web scraped Yelp reviews on New York’s pride burger chain, Shake Shack, and California’s beloved In-N-Out Burger. I present how you can (1) create a database log of web scraped Yelp reviews and (2) utilize Yelp ratings as a performance evaluation and comparison tool. You can find more details about web scraping and analyses codes on my GitHub page.
Creating Your Own Database and Systematic Data Analysis of Yelp Ratings
The most important measure in a Yelp review is the average star rating, a value between 1 to 5 stars with 5 being the best rating. The higher the average rating, the higher the odds of your restaurant being selected over others in the area with lower ratings.
Reviews are initially presented on Yelp based on a “Yelp sort.” However, it may not be clear to a user exactly how the reviews are sorted. It is not efficient to read all the reviews one at a time unless there are only a few of them. Instead, you can first sort the reviews by date or rating and then identify the low ratings and read the corresponding reviews, if low ratings are what you are concerned about. But I think there is a better and more systematic way to do this.
The first step is to web scrape Yelp data using Beautiful Soup (or any other free web scraping software). After you have obtained Yelp data from web scraping, then you can sort the reviews yourself, however way you want! You can also visualize these data using whatever software you prefer. I will illustrate using Plotly, which is free, but you can also use Excel or the free open source software, R.
Yelp Ratings on In-N-Out
It is nice and convenient to have a single average Yelp rating for each restaurant but this rating does not really tell everything about a restaurant’s performance. What if the restaurant used to have high ratings but during the last few weeks there were many low ratings? The average rating might still be somewhat high, but the shift in direction could mean that a customer’s visit in the near future might not be favorable compared to a few weeks earlier. The restaurant manager should note and take action! In my opinion, it is best to visualize Yelp rating data over the dates the reviews were posted like so:
Observations
The data for this graph comes from Yelp ratings of In-N-Out Burger in Plano, Texas. This type of data visualization allows a restaurant manager to do the following:
- Visually inspect rating trends over time. With this, you can tell whether you are getting many low ratings lately, or worse, consecutively! In the example graph above, there were a few low ratings around April through July 2017 but after that were all good.
- Filter ratings that you may be interested in investigating further. Instead of reading all reviews on the Yelp website, you can read only those in your Yelp reviews database at particular times that you consider important. For example, if you are mainly concerned about 1 or 2-star ratings, then you can identify those in the graph and check the review contents at those specific time points in your database. This approach can help you understand what could have happened on those dates: Did the low ratings correspond to the approximate time when the restaurant was understaffed?
Record Log
With your new Yelp ratings database, you can turn it into a records log of actions that you can take to address the low Yelp ratings. Concretely, you can create new additional column names in your Yelp database table like “Nature of Yelp Review,” “Addressed When,” and “Addressed How.” The “Nature of Yelp Review” can be any category like logistics (floor not clean, trash on tables), food (salty burger, limp fries, shakes melted), or service (long wait time, rude cashier, incorrect orders).
This way you can note which category (logistics, food, or service) tends to occur more often than the others and identify areas of improvement. Perhaps your restaurant has too many service complaints. That could indicate you need to invest in additional staff training. By noting in your database when and how the Yelp review complaint was addressed, you can have a record to look back on and avoid repeating the same mistakes in the future. This could be particularly useful when there is a transition in the business’ management because the new manager would not have to start from scratch. Just turn over the records that you have accumulated from the previous manager.
Yelp ratings as a Performance Evaluation Tool
If you are a manager concerned about Yelp, how would you know whether your restaurant is improving its Yelp rating? It’s actually quite simple when you have web scraped Yelp data! As we have done earlier, order the ratings by the dates they were posted. For each rating date, take the average of all ratings until that date. The value that we just computed is the same as Yelp’s average rating posted on that date (before the next review is posted). See details about how this computation can be encoded on my GitHub page.
Using the same Yelp ratings of In-N-Out Burger in Plano, Texas, you can create a graph that looks like this:
There are two insights that can be gained from this graph. First, In-N-Out Burger’s Yelp rating dipped from 5 to 3 stars soon after opening around 2012. Second, in a span of around 1.5 years, the ratings improved to 4 stars and remained somewhat level until today. Great job In-N-Out Burger manager!
Comparing with In-N-Out in the Area
You can also compare your performance with other In-N-Out Burger restaurants in the area:
It looks like all three chains suffered from low ratings in the beginning, with Dallas North taking the biggest hit. But Dallas North improved by one star after three years. The Plano outlet still has the best average Yelp rating over time.
In-N-Out vs Shake Shack
But wait, there’s more! What about your competitor’s performance? Let’s bring in Shake Shack:
Shake Shack came into the Texas scene in late 2016. It appears that Shake Shack did not suffer as badly “Yelp-wise” as In-N-Out in the beginning. Does this mean that Texans like Shake Shack more than In-N-Out or simply that they have already acclimated to out-of-state imported burger chains? Towards 2018, the average ratings are about the same for all outlets except for In-N-Out in Plano, Texas, which outperformed everyone else. Perhaps the manager at this outlet really deserves a pay raise or a promotion?!
When to Web Scrape Yelp
Once you have your Yelp data in place within your computer system, you will continually need to update your database with new Yelp reviews. How frequently it has to be done depends on how many reviews your restaurant gets. Let us graph the number of reviews per month for In-N-Out Burger in Plano, Texas:
Number of Yelp Reviews
The graph indicates that there are only 1-3 Yelp reviews per month lately. It may be better to update your database every quarter. For outlets with more customer volume and reviews, such as Shake Shack in Madison Square Park, New York, the pattern looks quite different:
Number of Ratings
This location was the first Shake Shack outlet that opened around 2004-2005. Coincidentally, this is around the time that Yelp started doing business. The graph indicates that lately, there are approximately 5-15 reviews per month, so perhaps database updates can be done once each month. Note, however, that there were times when the number of reviews per month were much higher, reaching over a hundred reviews per month. I am not sure what happened during these years. Were these the peak years of Shake Shack’s popularity? During those years, a database update may be warranted once per week.
Shake Shack at Madison Square Park, New York
Final Thoughts
To some, if not most, restaurant businesses, Yelp or other online customer ratings may appear dreadful. But they can be used as an opportunity to improve your business. You can turn them into your own database or record log to note when and how you addressed unfavorable reviews. By categorizing customer review complaints, you can note areas where you can make improvements. You can also set a goal of improving your business performance in a “Yelp-wise” fashion using the performance evaluation and comparison tool proposed here. With web scraping and other free tools available nowadays, you can help improve your business with little or no cost, and only minimal effort!
Blog Post Image Source: https://stocksnap.io/
About Author
