Project 1: Exploratory visualizations of Yelp academic dataset
Contributed by Sung Pil Moon. He is currently in the NYC Data Science Academy 12 week full time Data Science Bootcamp program taking place between January 11th to April 1st, 2016. This post is based on his first class project - R Visualization (due on the 4th week of the program).
NOTE: The dataset, visualizations, and result outputs in this post are NOT representative for any types of overall business, users, reviews in Yelp.
* Main purpose of this post is to utilize several visualization techniques with a large dataset.
1. Yelp Academic Dataset #1 (Small)
1.1. Initial Questions
- Initial questions before getting the Yelp data set
- What are properties of average Yelp users?
- Age, gender distribution, average number of checkin,…?
- What types of food restaurants are rated higher?
- vegetarian restaurant, pub, steak restaurant, family restaurant, etc?
- New Yorkers are more likely to give higher review scores than people in other states?
- Female users tends to give higher review scores (and / or) more review counts than male users?
- More restaurants with high review scores in New York than other states?
- What are properties of average Yelp users?
- Data Set
- Yelp Academic Dataset for Yelp Dataset Challenge
- Requires a permission to download and disclose the results publicly
1.2. Simple exploratory analysis about the Yelp dataset:
After I received the access to download the Yelp dataset, I skimmed through the set to get the basic ideas, including how many tables are, what kinds of information is included in each table, how the tables are inter-connected, and so on.
Basically, the dataset contains a table, Business, consisting of 24 variables, 474,434 observations (289.4 MB).
Variables include
- business_id: id for each business (primary key)
- categories: categories names of business, such as pub, …
- name: name of business
- stars: review scores in the number of stars (5 max)
- review_counts: the number of review counts by Yelp users
- schools: school name(s) near business
- neighborhood: neighbor businesses
- city, state, longitude, latitudes, full_address, … more
With variables in the dataset, I realized that I would not be able to get answers to my initial questions. So, I started an exploratory analysis first with these two simple questions:
- Average review ratings by state
- More detailed review ratings by state
1.3. Average review ratings by state.
The first exploratory analysis began with looking at average review ratings by each state. Total 348,256 reviews on 13,730 business in 16 states were included in the analysis. The code blocks show the steps how to manipulate the data and load them into a table and on a leaflet map.
a. Pulling the data
dataGroupByStateStar <- ylpDataSmall %>%
filter(state != '') %>% mutate(tsum = n()) %>%
group_by(state, stars)
dataForTableByStateStar <- dataGroupByStateStar %>% group_by(state) %>%
summarise(total_business = n(), total_reviews = sum(review_count), avg_rating = round(mean(stars), 2)) %>% arrange(desc(avg_rating))b. Loading the data on the table
library(pander)
panderOptions("digits", 3)
pander(dataForTableByStateStar)| state | total_business | total_reviews | avg_rating |
|---|---|---|---|
| NC | 500 | 8147 | 3.84 |
| TX | 1000 | 23935 | 3.72 |
| IN | 393 | 3041 | 3.71 |
| CA | 4000 | 141119 | 3.69 |
| ON | 228 | 1014 | 3.69 |
| MI | 500 | 11634 | 3.66 |
| RI | 500 | 11086 | 3.64 |
| WA | 500 | 17998 | 3.64 |
| MA | 1298 | 54477 | 3.60 |
| GA | 500 | 15455 | 3.58 |
| VA | 189 | 1503 | 3.56 |
| PA | 1000 | 20078 | 3.55 |
| IL | 500 | 7188 | 3.51 |
| NY | 1382 | 23675 | 3.49 |
| NJ | 500 | 7904 | 3.37 |
| MD | 500 | 5813 | 3.36 |
c. Loading the data on the Leaflet map
library(leaflet)
leaflet(dataTotalAvgStarByState) %>% addTiles() %>% setView(lng = -96.503906,
lat = 38.68551, zoom = 4) %>% addCircles(lng = ~city_lng, lat = ~city_lat,
weight = 0, radius = ~exp(totAvgRatingByState * 1.4) * 800, fillOpacity = 0.5,
color = ~myCol(totAvgRatingByState), popup
= ~totAvgRatingByState) %>% addLegend("bottomleft",
pal = myCol, values = ~sort(totAvgRatingByState), title = "Avg.Ratings",
labFormat = labelFormat(prefix = ""), opacity = 0.5)
These table and map indicate that average review ratings are between 3.36 and 3.84. The rows in the table are sorted in descending order by average review scores. The color of a circle in the leaflet map corresponds to average rating scores shown in the bottom left corner in the map.
In these two plots, North Carolina is the state with the highest rating score (3.84) while Maryland has the lowest rating scores (3.36). Very different from my initial expectation, New Jersey and New York were not ranked within the top 5, but worst 2 and 3. To get more details, I created another plot, a distribution grid of average review rating scores, at a deeper level.
1.4. A grid of detailed average ratings by state.
Basically, the average review rating scores in each state were reclassified from 1.0 to 5.0 by 0.5 increase. For visualization purpose, a percentage of rating score is weighted.
a. Pulling the data
dataWeightedGroupByStateStar <- dataGroupByStateStar %>%
summarise(totalByStar = n()) %>% arrange(desc(stars)) %>%
mutate(total = sum(totalByStar)) %>% mutate(percent = round((totalByStar / total)*100, 1)) %>%
mutate(percentWeight = ifelse(percent >= 20, percent * 2.5, # custom column to weight the percent for size on the plot
ifelse(percent < 20 & percent >= 15, percent * 1.2,
ifelse(percent < 15 & percent >= 10, percent,
ifelse(percent < 10 & percent >= 5, percent * 0.8, 1)))))b. Loading the data on the ggplot bubble plot
library(ggplot2)
ggplot(dataWeightedGroupByStateStar, aes(x = state, y = stars, label = percent)) +
geom_point(aes(size = percentWeight * 2, colour = stars, alpha = 0.05)) +
geom_text(hjust = 0.4, size = 4) + scale_size(range = c(1, 30), guide = "none") +
scale_color_gradient(low = "darkblue", high = "red") + labs(title = "A grid of detailed avg.ratings by state ",
x = "State", y = "Detailed Avg.Ratings") + scale_y_continuous(breaks = seq(1,
5, 0.5)) + theme(legend.title = element_blank())
In this distribution grid of average rating score, each column indicates a proportion of score by state, while each row indicates a proportion at a specific review score in each state. For example, in the first column, California state has 17.8% of average rating score 5.0, 14.1% of average rating score 4.5, and 19.2% of average rating score 4.0. In the top row, California marked 17.8% of average rating score, Georgia marked 10.6% and Illinois marked 8.8% of average rating score 5.0.
From this grid, we can see the pattern that most states have businesses whose average rating scores are between 3.5 and 4.0. Interestingly, although California has the biggest proportion (17.8%) of an average score of 5.0, North Carolina seems to have more businesses with highest average rating scores (11.8% of 5.0, 20.4% of 4.5, 24.8% of 4.0, and 22.6% of 3.5, respectively). Unfortunately, New Jersey does not seem to be different from in the previous plot. All proportions are rather evenly distributed.
2. Yelp Academic Dataset #2 (Big)
I found another request link to download the Yelp Academic dataset. (The link to request an access is currently missing. I'll update a link once I found). This dataset is quite big (around 1.9GB in csv format) and contains 5 tables:
- Business (98 variables, 77,445 observations, 30.4MB)
- User (23 variables, 552,339 observations, 135.9MB)
- Reviews (10 variables, 2,225,213 observations, 1.64GB)
- Checkin (170 variables, 55,569 observations, 13.2MB)
- Tip (6 variables, 591,864 observations, 74.5MB)
2.1. Questions
Before I took a look at the dataset in detail, my initial questions were:
- Will Restaurants in higher price ranges have higher review rating scores?
- Will female Yelp users provide more frequent review count?
- Will users with more followers at Yelp purchase more?
- Will users with more followers at Yelp check in the business more frequently?
However, Due to privacy issue and missingness within the academic dataset, I couldn't find enough information and description about each entity, especially users, and connections to other tables. So, I had to also change my questions to:
- Will having more followers at Yelp rate more frequently than other users?
- Will having more followers at Yelp rate higher than other users?
Before we proceed, we need to know a concept of a special group of users at Yelp, called the 'Elite' user group (reference). They are active Yelp reviewers selected by Yelp's National Elite Squad Council. Selection criteria are unknown. Some of criteria include well-written reviews, high quality tips, a detailed personal profile, an active voting and complimenting record, and a history of playing well with others. They are invited to private events where up-and-coming restaurants and bars provide food and drinks for free.
2.2. [Q1]: Users having more followers at Yelp will rate more frequently?
The first question is about whether having more followers at Yelp will show a tendency to rate more frequently? To see more direct effect, I show visualizations of the overall users, elite users, and normal users separately.
a. Pulling the data
ylpUserSmElite <- ylpUserSm3 %>% filter(elite != "[]")
ylpUserSmNormal <- ylpUserSm3 %>% filter(elite == "[]")b. Loading the data on the box plot
* All users:
library(ggthemes)
# Yelp users in the boxplot
qplot(fans, review_count, data = ylpUserSm3, geom = "boxplot", group = Fan_Size,
color = Fan_Size) + labs(title = "Total review counts by the number of fans") +
theme(legend.position = "none")
The first visualization, boxplots, shows slowly but gradually an increase pattern until some points around 1000, then decreases. My hasty assumption is that Yelp users will aggressively provide reviews proportionate to the number of fans. However, once they have enough number of fans (around 1000 in this case), they would less likely to show similar frequency of review activity.
* Elite users:
# Elite Yelp group users in the boxplot
qplot(fans, review_count, data = ylpUserSmElite, geom = "boxplot", group = Fan_Size,
color = Fan_Size) + labs(title = "Total review counts by the number of fans (Elite users)") +
theme(legend.position = "none")
When we see the elite group alone in this boxplot group, the pattern is similar to the group of overall users. Notably, the number of review counts by outliers in early stages (when they have fewer fans) is quite higher and bigger in frequency than the next stage. It seems that elite users at the early stages provide more reviews to get more followers while elite users in the later stages seem to participate in more activities such as Yelp events for elite group.
* normal users:
# Non-elite Yelp group users in the boxplot
qplot(fans, review_count, data = ylpUserSmNormal, geom = "boxplot", group = Fan_Size,
color = Fan_Size) + labs(title = "Total review counts by the number of fans (Non-elite Users)") +
theme(legend.position = "none")
* All users:
# Yelp users in combination plots
qplot(fans, review_count, data = ylpUserSm1, geom = c("point", "smooth"), colour = fans) +
labs(title = "Total review counts by the number of fans") + scale_color_gradient(low = "darkblue",
high = "darkred") + stat_smooth(fill = "green", colour = "cyan", size = 1,
alpha = 0.1)
As shown in the previous boxplots graph for overall users, this plot shows the same pattern.
* elite user group:
# Elite Yelp group users in combination plots
qplot(fans, review_count, data = ylpUserSmElite, geom = c("point", "smooth"),
colour = fans) + labs(title = "Total review counts by the number of fans (Elite users)") +
scale_color_gradient(low = "darkblue", high = "darkred") + stat_smooth(fill = "green",
colour = "cyan", size = 1, alpha = 0.1)
The elite group shows the same pattern in this plot.
* normal users:
# Non-elite Yelp group users in combination plots
qplot(fans, review_count, data = ylpUserSmNormal, geom = c("point", "smooth"),
colour = fans) + labs(title = "Total review counts by the number of fans (Non-elite users)") +
scale_color_gradient(low = "darkblue", high = "darkred") + stat_smooth(fill = "green",
colour = "cyan", size = 1, alpha = 0.1)
2.3. [Q2]: Users having more followers at Yelp will rate higher?
The second question is about whether having more followers at Yelp is related to higher rate scores? This analysis is done similarly to the previous question.
a. Loading the data on the box plot
* All users:
# Yelp users in the boxplot
qplot(fans, average_stars, data = ylpUserSm3, geom = "boxplot", group = Fan_Size,
color = Fan_Size) + labs(title = "Average ratings by the number of fans") +
theme(legend.position = "none")
In the boxplots of overall users, we can see a pattern that the average rating scores gets increased as the number of fans increases.
* elite user group:
# Elite Yelp group users in the boxplot
qplot(fans, average_stars, data = ylpUserSmElite, geom = "boxplot", group = Fan_Size,
color = Fan_Size) + labs(title = "Average ratings by the number of fans (Elite users)") +
theme(legend.position = "none")
The boxplot graph also shows the similar pattern as the previous one. Here, I'm a little bit curious about the elite group, especially having more than 1000 followers, whether they try to show themselves as nice reviewers providing higher rate scores, and whether they were invited better businesses more often as the number of fans increases.
* normal users:
# Non-elite Yelp group users in the boxplot
qplot(fans, average_stars, data = ylpUserSmNormal, geom = "boxplot", group = Fan_Size,
color = Fan_Size) + labs(title = "Average ratings by the number of fans (Non-elite users)") +
theme(legend.position = "none")
This boxplot for normal users does not tell much about the pattern, except for that 50% of average ratings are between 3.5 and 4.0.
c. Loading the data on the combination plots (point+smooth)
* All users:
# Yelp users in combination plots
qplot(fans, average_stars, data = ylpUserSm1, geom = c("point", "smooth"), colour = fans) +
labs(title = "Average ratings by the number of fans") + scale_color_gradient(low = "darkblue",
high = "darkred") + stat_smooth(fill = "green", colour = "cyan", size = 1,
alpha = 0.1)
This combination plot for overall users shows a similar gradual but slow increase pattern in the boxplot above for overall users. Interestingly, for users who do not have many fans, the range of their reviews are quite wider than the ones of users having more fans.
* elite user group:
# Elite Yelp group users in combination plots
qplot(fans, average_stars, data = ylpUserSmElite, geom = c("point", "smooth"),
colour = fans) + labs(title = "Average ratings by the number of fans (Elite users)") +
scale_color_gradient(low = "darkblue", high = "red") + stat_smooth(fill = "green",
colour = "cyan", size = 1, alpha = 0.1)
This plot shows the similar gradual but slow increase pattern as the number of fans increase.
* normal users:
# Non-elite Yelp group users in combination plots
qplot(fans, average_stars, data = ylpUserSmNormal, geom = c("point", "smooth"),
colour = fans) + labs(title = "Average ratings by the number of fans (Non-elite users)") +
scale_color_gradient(low = "darkblue", high = "red") + stat_smooth(fill = "green",
colour = "cyan", size = 1, alpha = 0.1)
Differently from previous plots above, normal user group do not show steady increase pattern as the number of fans increase. Possible reason include they are less biased by the number of fans or they rated negatively as they were not selected as elite squad.
3. Conclusions
3.1. Findings
- A relationship between the number of followers at Yelp and average rating frequency
- Yelp users tend to rate more frequently until some point as the number of followers increase
- A relationship between the number of followers at Yelp and average rating scores
- Yelp users, (specifically Elite users), tend to rate more higher as the number of followers increase.
- Interesting to investigate whether they try to show themselves as nice reviewers and whether they are more invited to better business as the number of fans increase?
- Other
- It seems that the existence of remote fake reviewer is possible.
3.2. Next steps
- Implement these datasets within an interactive Shiny app
- Incorporate statistical analyses for significance of the result
NOTE: The dataset, visualizations, and result outputs in this post are NOT representative for any types of overall business, users, reviews in Yelp.
About Author
Sung Pil Moon
View all posts by Sung Pil Moon >Related Articles
Leave a Comment
