Data Visualization on Sales Trends in Retail

Posted on Feb 4, 2019
The skills the author demoed here can be learned through taking Data Science with Machine Learning bootcamp with NYC Data Science Academy.


Data shows retail is probably the most talked about industry when it comes to disruption these days. Empty malls are a common blog topic and unusually high number of bankruptcies span across all subsectors. Some of the familiar names that filed for bankruptcy in the last few years span from well know Sears, ToysRUs, Limited Brands to lesser known Aerosoles, Gander Mountain and the Walking Company.

Given all this hyper activity and my interest in Retail, when I had to pick a topic for my Visualization project I quickly jumped on to a readily available Retail dataset for exploration. Exploring this dataset has helped me understand the industry better and has also thrown some surprise insights at me.

Data Set Details

Federal Reserve of Saint Louis publishes various data series related to Retail Sales data every month. They also have this data available via API and excel add-in from their FRED system. Same data is also available from U.S. Bureau of the Census. I have downloaded the data available till Oct 2018 for my project and building Shiny Dashboard WebApp.

Exploring the data using the WebApp, we can observe that Inventory data is available only for very few Sub Sectors. Even though we can analyze the inventory levels for these Sub Sectors, we will not be able to draw overarching conclusions at Retail level. 

Dashboard Purpose

I quickly realized that one of the aspects of building a dashboard is to be clear about the purpose of the dashboard as that impacts the type of interactivity to be built. Some of the ways a dashboard can be used are:

  1. Constant data points like metrics but changing data: For example, a dashboard showing metrics and current work progress on a shop floor. The goal here is not to explore data using the dashboard, but to consume insights
  2. Constant data and constant metrics: This is more of a onetime dashboard.  Presenting research findings could be an example here. Here the researcher has done all the exploratory analysis and presenting a set of conclusions. Again, the goal here is to consume insights and not explore data in depth. Another example is this blog post where insights as of 2018 Q3 data are being posted.
  3. Explore Data that is refreshed regularly: Analyzing Retail Sales data (topic of this blog) is an example. As fresh data gets loaded every month, it is not entirely clear what new insights are waiting to be found unless someone explores the data. So, the dashboard should help explore the data by slicing and dicing the data and applying some statistical measures. Tools like Tableau are well suited for the task. The WebApp I have built using R and Shiny tries to emulate this to a limited extent.

Retail Level Data Observations 

The Retail Industry as a whole is growing at faster clip than the economy. It has grown from $3.5 trillion in 2007 (pre-2008 crisis) to about $4.5 trillion at end of 2017 as seen from the chart below. This is about 2.9% annualized growth, well above U.S. GDP growth over same time, for lack of better comparison.

So, the numbers do not support the notion of disruption as generally perceived and understood. Retail industry is not going to diminish in size substantially or vanish in a traditional sense of “disruption”, like tape, CD/DVD manufacturers being disrupted by streaming or potential of blockchain to disintermediate entire industries. Disruption here actually means the customer interaction models and tool used (like AI, newer delivery models etc) are changing and the is going to change the mix of dominant players. But Retail as industry is consumption based and the net retail dollars are not going anywhere.

Data on eCommerce vs Overall Retail

eCommerce accounts for about 9.8% of retail sales at end of 2018 third quarter. Note that eCommerce data is available only at Quarterly granularity. But “Non Store Retailers” sub sector loosely corresponds to eCommerce. So that sub sector data can be used for further analysis.

Even though eCommerce growth has moderated compared to early 2000’s, it is still growing 4 times faster than overall retail. What is surprising is that Retail excluding eCommerce is still growing at a 2.6% rate. This is not bad given the size of the industry. Another interesting insight is, that the eCommerce is not at all impacted in the Retail slowdown in 2015-2016.

Analyzing Sub Sectors Data

Barring for few changes, the Top 5 and Bottom 5 sub sectors remained fairly stable since 1992. We can see “Non Store Retailers” (read as eCommerce) emerging into Top 5.

However, positions with respect to relative size has shifted considerably for many sectors over a period. Tree Maps are useful to quickly compare relative size of large number of sub sectors. The snapshots for years 1992, 2005 and 2017 demonstrate the shift in relative positions.

It is always interesting and useful for industry analysts to know which Sub Sectors are both growth and revenue leaders and vice versa. The following chart shows the same. 

For this Growth and Revenue leader analysis above, it is better to look at limited time range (like the seasonality chart mentioned later) to still get the average numbers than point numbers and at the same time avoid skewing current trends by very old data. From the following examples, one can see how the leaders change based on the time period used.

Seasonality with Retail Data

In analyzing Retail sales and inventory data, one would generally like to understand Level, Trend and Seasonality. Level corresponds to base level on top of which Trend and Seasonality can be explained. Trend shows either upward movement or downward movement from the level over the selected period. This Trend is clearly seen in chart below. But decoding seasonality from a simple chart showing sales over period is bit difficult and error prone. 

Here I have used Linear Model to derive seasonality chart shown below. First step is to apply Linear Model (LM, LOESS etc) as applicable for the data. Next step is to collect the error terms from the model. Error terms are essentially what remains after removing the Level and Trend information from each month sales, and explains the deviations from perfectly fit model. The final step is to collect the terms by period (month of the year here) and use appropriate statistic (median in this case) to show the variation over that time intervals. 

When the data series spans a long duration, one caution while calculating current seasonality is to watch for any changes in trends of seasonality. Using all the available data all the time can mask shifting seasonality trend. 

Ideas for Future Development

Having the app pull the data at regular intervals direct from FRED would be a nice feature to add. Similarly integrating other retail data available like retail employment, eCommerce data by Subsector would make data more complete and increase the usefulness of this app. One advantage of using R (or something similar) is that all the data exploratory and statistical methods are available to play with. Hence a dashboard like this can be enhanced to create insights that can act as trigger points further actions as new data arrives. Though a lot more can be done with this WebApp, the current version showcases that a lot can be achieved quickly using simple toolset these days.


FRED Economic Data from St. Louis Fed

Shiny Dashboard WebApp

About Author

Related Articles

Leave a Comment

No comments found.

View Posts by Categories

Our Recent Popular Posts

View Posts by Tags

#python #trainwithnycdsa 2019 2020 Revenue 3-points agriculture air quality airbnb airline alcohol Alex Baransky algorithm alumni Alumni Interview Alumni Reviews Alumni Spotlight alumni story Alumnus ames dataset ames housing dataset apartment rent API Application artist aws bank loans beautiful soup Best Bootcamp Best Data Science 2019 Best Data Science Bootcamp Best Data Science Bootcamp 2020 Best Ranked Big Data Book Launch Book-Signing bootcamp Bootcamp Alumni Bootcamp Prep boston safety Bundles cake recipe California Cancer Research capstone car price Career Career Day citibike classic cars classpass clustering Coding Course Demo Course Report covid 19 credit credit card crime frequency crops D3.js data data analysis Data Analyst data analytics data for tripadvisor reviews data science Data Science Academy Data Science Bootcamp Data science jobs Data Science Reviews Data Scientist Data Scientist Jobs data visualization database Deep Learning Demo Day Discount disney dplyr drug data e-commerce economy employee employee burnout employer networking environment feature engineering Finance Financial Data Science fitness studio Flask flight delay gbm Get Hired ggplot2 googleVis H20 Hadoop hallmark holiday movie happiness healthcare frauds higgs boson Hiring hiring partner events Hiring Partners hotels housing housing data housing predictions housing price hy-vee Income Industry Experts Injuries Instructor Blog Instructor Interview insurance italki Job Job Placement Jobs Jon Krohn JP Morgan Chase Kaggle Kickstarter las vegas airport lasso regression Lead Data Scienctist Lead Data Scientist leaflet league linear regression Logistic Regression machine learning Maps market matplotlib Medical Research Meet the team meetup methal health miami beach movie music Napoli NBA netflix Networking neural network Neural networks New Courses NHL nlp NYC NYC Data Science nyc data science academy NYC Open Data nyc property NYCDSA NYCDSA Alumni Online Online Bootcamp Online Training Open Data painter pandas Part-time performance phoenix pollutants Portfolio Development precision measurement prediction Prework Programming public safety PwC python Python Data Analysis python machine learning python scrapy python web scraping python webscraping Python Workshop R R Data Analysis R language R Programming R Shiny r studio R Visualization R Workshop R-bloggers random forest Ranking recommendation recommendation system regression Remote remote data science bootcamp Scrapy scrapy visualization seaborn seafood type Selenium sentiment analysis sentiment classification Shiny Shiny Dashboard Spark Special Special Summer Sports statistics streaming Student Interview Student Showcase SVM Switchup Tableau teachers team team performance TensorFlow Testimonial tf-idf Top Data Science Bootcamp Top manufacturing companies Transfers tweets twitter videos visualization wallstreet wallstreetbets web scraping Weekend Course What to expect whiskey whiskeyadvocate wildfire word cloud word2vec XGBoost yelp youtube trending ZORI