Retailers: Here's What Your Employees Are Saying About You - A Project on Web Scraping

Posted on Jun 15, 2017

To view the original source code, you can visit my Github page here. If you are interested in learning more about my background, you can visit my LinkedIn page here.

About This Project

This is the second of four projects for the NYC Data Science Academy. In this project, students are required to collect their datasets using a method called webscraping. Essentially, webscraping is the process of collecting information (i.e. texts) from a series of websites into structured databases such as .csv files. Webscraping enables analyses to be done on data that is not already provided in a structured format.

In this analysis, we will scrape employee reviews for the 8 retail companies that made it to Indeed's 50 Best Corporate Employers in the US list. See below for additional details regarding each of these 8 retail firms in discussion. The rank and overall score corresponds to the employer's rank and overall score as they appear on Indeed's website.


Given a two-week timeframe, the scope of the analysis was limited to these three primary objectives:

  • To understand when employees are more likely to submit positive and negative reviews
  • To understand the sentiments and emotions¬†associated with positive and negative reviews (i.e. Sentiment Analysis)
  • To understand the topics of positive and negative reviews (i.e. Topic Modeling)

Web Scraping with Scrapy

Scrapy (a Python framework) was used for collecting employee reviews for each of the retailers on the 50 Best Corporate Employers List. Each review scraped contains the following elements:

  • Company
  • Industry
  • Job Title
  • Date
  • Month
  • Location
  • Rating
  • Header
  • Comment (Main Review)
  • Pro
  • Con

As seen in the illustration below, not every review contains a pro and a con, as those are optional fields for the reviewer. To see an example of a real review, you can visit Indeed's Walt Disney Parks and Resorts review page here.

Employee Reviews At a Glance

In total, we have scraped over 20,000 reviews. The number of reviews scraped for each firm varies significantly across the board. This is likely due to the difference in the number of employees hired by each firm across the nation.

Employee reviews for these eight retail firms tend to peak around the beginning of the year (January through April). This is expected as we know that retail is a cyclical industry in which the majority of sales happen during the holiday season (November through January). Because employees are oftentimes hired only for the duration of this period, it makes sense  that the majority of the reviews come in around the beginning of the year.

Among the eight retailers on the 50 Best Corporate Employers list, we see that the review ratings skew toward a score of 5, which is the highest score a review can get.

There is no significant differences in average rating across retailers.

There is no significant differences in average rating across months.

When Are Employees At Retail Firms More Likely to Submit Positive and Negative Reviews?

Average Employee Rating by Month

When we consider reviews of all ratings (i.e. reviews with ratings 1 to 5), retailers receive the highest average ratings during the months of October, November, December, and January. Note that the ratings have been scaled to 0 to 1 to allow for comparison across firms and months.

Most Negative Reviews By Month (Reviews with Ratings Below 3)

If we only consider the count of negative reviews (i.e. reviews with ratings 1 or 2), retailers on average receive the most negative reviews during the months of October, February, March, and April (i.e. these months scored higher across retailers).

Most Positive Reviews By Month (Reviews with Ratings Above 3)

If we only consider the count of positive reviews (i.e. reviews with ratings 4 or 5), retailers on average receive the most positive reviews during the months of January, February, and April (i.e. these months scored higher across retailers).

Summary Observations

In summary, while the highest average ratings concentrate toward the end of the year, the highest count of both positive and negative reviews are given during the beginning of the year. This aligns with our understanding that the retail industry is cyclical. In other words, employees are oftentimes hired specifically to aid sales around the holiday season (i.e. Nov to Jan). When their work concludes around the beginning of the year, that's when we can expect employees to write the majority of the reviews (i.e. both positive and negative).

Understanding Employees Reviews Using Sentiment Analysis

A sentiment analysis was done using R's Syuzhet library to visualize the emotions across employee reviews. In the illustration below, each graph represents the intensity of the an emotion from January to December on a positive scale (i.e. 0 and above). For example, for Build-A-Bear Workshop's 'positive' graph, we can see that there is an uptick in positive sentiment toward March and November, and each of those months scored approximately a 6. This is a high score relative to the scores recorded across all other emotions.

Generally, the sentiment observed align with our understanding that the majority of the employee reviews are positive, as we are narrowly focused on analyzing employee reviews among the best 8 corporate retail employers according to Indeed's ranking. As an aside, it is interesting to see that Build-A-Bear Workshop and Trader Joe's recorded more volatility in its scoring across the negative emotions (i.e. negative, sadness, fear, disgust, anger).

Using Topic Modeling to Identify Patterns Among Positive and Negative Employee Reviews

Topic modeling is a form of unsupervised machine learning, and it can help us identify patterns by clustering similar items into groups. In this analysis, pyLDAvis, a Python library, was used to analyze the groups among positive and negative reviews (using pros and cons of the employee reviews). Essentially, the library takes a list of documents as inputs (i.e. a list of reviews) and attempts to group the reviews based on common combinations of keywords appearing in each of these reviews.

The drawback of using topic modeling, as with other clustering techniques, is that the user has to specify the number of groups to cluster the documents into. This presents a catch-22 issue as the initial objective is to understand the number of topics or groups among your inputs. Given this hurdle, the user must use their business judgment as well as running multiple trials with different number of topics as inputs to determine results that are most interpretable. More sophisticated clustering libraries aid the users by automatically selecting the best number of topics, but this should only be used as a guide to begin the analysis.

In the illustration below, we can see that words like 'great' and 'benefit,' among many other words, are oftentimes seen together among pro reviews.

Observed Topics Among Pros and Cons in Employee Reviews

Two topic modeling analyses were conducted using pro reviews and con reviews. Below are the most common combinations of words appearing in each of the groups for each analysis. It is not a surprise that the groups overlap in the common words identified among them, as we have chosen a relatively high number of topics. Nevertheless, the results provide a useful start for understanding what employees are thinking.

Visualizing Employee Reviews Using Word Clouds

R's Wordcloud2 library was used to further aid the visualization of what employees are saying about these firms. Essentially, the library takes a list of words along with their frequencies, and produces a word cloud based on those inputs. The higher the frequency a word is associated with, the larger it appears in the word cloud.

Word Cloud - Positive Reviews

The positive reviews word cloud was created using only the pros from each of the reviews. (Note: Each review contains the main review, a pro, and a con.) Some of the common words we see among pros are 'great,''work,' 'job,’'company,' and 'customers.'

Word Cloud - Negative Reviews

Similarly, a word cloud was created using only the cons from each of the reviews. We can see that common words appearing include work, management, hours, employees, and pay.

Future Directions

Beyond the scope of this project, below are other interesting areas that are worth looking into if additional time allows.

  • Evaluate the credibility of Indeed‚Äôs Top 50 Ranking corporate employers by sampling comparable employers that did not make it to the list
  • Better¬†understand the rationale why positive and negative reviews are concentrated in the months they were observed to be in, and evaluate whether this behavior is prevalent across other industries
  • Perform statistical methods to evaluate the relationships among variables

About Author

Jack Yip

Jack is passionate about using state-of-the-art data analytic techniques to help companies get ahead of the curve in monetizing their data. He combines effective storytelling and simple visualizations to translate highly technical analyses into actionable insights. Jack has...
View all posts by Jack Yip >

Related Articles

Leave a Comment

No comments found.

View Posts by Categories

Our Recent Popular Posts

View Posts by Tags

#python #trainwithnycdsa 2019 2020 Revenue 3-points agriculture air quality airbnb airline alcohol Alex Baransky algorithm alumni Alumni Interview Alumni Reviews Alumni Spotlight alumni story Alumnus ames dataset ames housing dataset apartment rent API Application artist aws bank loans beautiful soup Best Bootcamp Best Data Science 2019 Best Data Science Bootcamp Best Data Science Bootcamp 2020 Best Ranked Big Data Book Launch Book-Signing bootcamp Bootcamp Alumni Bootcamp Prep boston safety Bundles cake recipe California Cancer Research capstone car price Career Career Day citibike classic cars classpass clustering Coding Course Demo Course Report covid 19 credit credit card crime frequency crops D3.js data data analysis Data Analyst data analytics data for tripadvisor reviews data science Data Science Academy Data Science Bootcamp Data science jobs Data Science Reviews Data Scientist Data Scientist Jobs data visualization database Deep Learning Demo Day Discount disney dplyr drug data e-commerce economy employee employee burnout employer networking environment feature engineering Finance Financial Data Science fitness studio Flask flight delay gbm Get Hired ggplot2 googleVis H20 Hadoop hallmark holiday movie happiness healthcare frauds higgs boson Hiring hiring partner events Hiring Partners hotels housing housing data housing predictions housing price hy-vee Income Industry Experts Injuries Instructor Blog Instructor Interview insurance italki Job Job Placement Jobs Jon Krohn JP Morgan Chase Kaggle Kickstarter las vegas airport lasso regression Lead Data Scienctist Lead Data Scientist leaflet league linear regression Logistic Regression machine learning Maps market matplotlib Medical Research Meet the team meetup methal health miami beach movie music Napoli NBA netflix Networking neural network Neural networks New Courses NHL nlp NYC NYC Data Science nyc data science academy NYC Open Data nyc property NYCDSA NYCDSA Alumni Online Online Bootcamp Online Training Open Data painter pandas Part-time performance phoenix pollutants Portfolio Development precision measurement prediction Prework Programming public safety PwC python Python Data Analysis python machine learning python scrapy python web scraping python webscraping Python Workshop R R Data Analysis R language R Programming R Shiny r studio R Visualization R Workshop R-bloggers random forest Ranking recommendation recommendation system regression Remote remote data science bootcamp Scrapy scrapy visualization seaborn seafood type Selenium sentiment analysis sentiment classification Shiny Shiny Dashboard Spark Special Special Summer Sports statistics streaming Student Interview Student Showcase SVM Switchup Tableau teachers team team performance TensorFlow Testimonial tf-idf Top Data Science Bootcamp Top manufacturing companies Transfers tweets twitter videos visualization wallstreet wallstreetbets web scraping Weekend Course What to expect whiskey whiskeyadvocate wildfire word cloud word2vec XGBoost yelp youtube trending ZORI