Data Analysis of Annual Stock Return

Posted on Sep 11, 2017
The skills the author demoed here can be learned through taking Data Science with Machine Learning bootcamp with NYC Data Science Academy.

In this data web scraping project, I was focusing on the web scraping of the 10-K filings of all the listing companies on NASDAQ from the official website of US securities and exchange commission. The goal of this project is trying to find some clues between the financial performance of the companies and their stock return.

  • Areas:

All the listing companies in NASDAQ (4715 listings) in 2016

11 industry sectors

  • Question want to answer:

By evaluating what financial features can we get substantial information about the annual stock return?

Web scraping data on Yahoo finance, all listings

I scraped all the stock price history in 2016 of the 4715 listings from Yahoo Finance. They are separated to 11 sectors, according to companies by industries. The annual stock return can be calculated from the scraped figures, and they are box plotted in the figure below. It seems a majority of companies in the basic industries sector in 2016 had the positive annual stock return, while the healthcare sector had the lowest median annual stock return, which was negative. The rest of the sectors seems to have no big difference with each other if only judging from the box plot.

Data Analysis of Annual Stock Return 10-K filing scraping

For the investors, it’s crucial to know as much as possible about the companies before making the decisions of an investment. There are many ways for them to get knowledge of this information. The financial websites, twitter, Bloomberg, etc, everywhere. However, there’s one place that can provide all the needed information, and they are 100% authentic, kosher, and ready to use. It’s the official website of US securities and exchange commission. There are plenty of annual, quarterly report of different companies you would like to find. In this project, I scraped some needed information from the 10-K filings of all the listing companies.

 A. What is the 10-K filing?

A 10-K is a comprehensive summary report of a company's performance that must be submitted annually to the Securities and Exchange Commission. Typically, the 10-K contains much more detail than the annual report. It includes information such as company history, organizational structure, equity, holdings, earnings per share, subsidiaries, etc. The 10-K must be filed within 60 days (it used to be 90 days) after the end of the fiscal year.

B. What to scrape?

  • Return on assets: Net income / Total assets

Return on assets (ROA) is an indicator of how profitable a company is relative to its total assets. ROA gives an idea as to how efficient management is at using its assets to generate earnings.

  • Return on equity: Net Income / Shareholder's Equity

Return on equity (ROE) measures a corporation's profitability by revealing how much profit a company generates with the money shareholders have invested.

  • Current ratio:  Current assets / Current liabilities

The current ratio is mainly used to give an idea of the company's ability to pay back its liabilities with its assets,  it can be used to take a rough measurement of a company’s financial health.

So, all in all, five key figures to be scraped from the 10-K filings:

  • Net income (loss);
  • Total assets;
  • Shareholder’s equity;
  • Current assets;
  • Current liabilities.

Data Results

I scraped all the five figures from the 10-K filings on, they are plotted in the figures below. We can see that the ROE and ROA of finance sector are very centralized comparing with other sectors. And for the healthcare sector from the analysis of ROA and ROE, the median profitability of the companies in this sector seems to be lower than the companies in other sectors. However, the median current ratio of this sector is among the highest in all the industry sectors.

Data Analysis of Annual Stock Return Data Analysis of Annual Stock Return

In order to answer this question:

     By evaluating what financial features can we get substantial information about the annual stock return?

I did some simple linear regression to try to find the answer, here’s the result:


Annual stock return vs. Return on equity

Sectors Intercept Slope p-value for Intercept p-value for slope
Basic Industries 0.355231621 0.18247217 1.58E-15 0.028546516
Capital Goods 0.336702101 0.126801624 4.02E-28 0.106128202
Consumer non Durables 0.225510762 0.018302826 1.49E-12 0.466077375
Consumer services 0.165893504 0.00813999 4.33E-30 0.494754911
Energy 0.368675068 -0.002923266 8.07E-10 0.855902605
Finance 0.358726929 0.347776918 2.06E-60 0.006366251
Healthcare 0.05743789 0.200523609 0.055878211 7.04E-10
Miscellaneous 0.215358904 0.07464239 1.89E-05 0.231190295
Public Utilities 0.240638978 0.263049961 2.72E-12 0.000595729
Technology 0.261536342 0.069303433 1.04E-22 0.095897413
Transportation 0.373388909 -0.009137745 1.88E-06 0.79221854


Annual stock return vs. Return on assets

Sectors Intercept Slope p-value for Intercept p-value for slope
Basic Industries 0.330594989 0.596238709 1.66E-15 0.004437525
Capital Goods 0.303064458 0.361889481 3.36E-30 1.58E-05
Consumer non Durables 0.170056788 0.765273808 9.45E-07 0.012500255
Consumer services 0.142080656 0.69679516 7.88E-26 1.58E-07
Energy 0.366266572 0.232100176 2.70E-13 0.281701288
Finance 0.352035813 -0.125288529 1.03E-71 0.72050919
Healthcare 0.025691108 0.299685086 0.457359828 2.92E-08
Miscellaneous 0.198963377 0.210699384 2.89E-05 0.156874055
Public Utilities 0.208522685 2.405875682 1.58E-14 1.31E-06
Technology 0.252271318 0.281154548 3.06E-23 0.000146544
Transportation 0.36413647 -0.229605123 4.19E-06 0.743215475


Annual stock return vs. Current ratio

Sectors Intercept Slope p-value for Intercept p-value for slope
Basic Industries 0.370656669 -1.13E-12 6.87E-19 0.910464037
Capital Goods 0.305918824 -2.42E-12 7.05E-30 0.309531081
Consumer non Durables 0.215105954 -1.47E-12 2.43E-15 0.770846634
Consumer services 0.167368009 -2.18E-12 2.43E-21 0.512133029
Energy 0.267368348 7.92E-13 1.51E-10 0.940114503
Finance 0.132306628 2.75E-12 0.001364733 0.497072028
Healthcare -0.134979771 6.94E-12 2.72E-07 0.096389706
Miscellaneous 0.238054911 -6.02E-12 1.27E-07 0.506012428
Public Utilities 0.20567902 1.01E-11 5.27E-13 0.245790194
Technology 0.158163786 1.02E-12 4.46E-13 0.514683949
Transportation 0.277715154 3.14E-13 2.53E-06 0.985370257


The bolded ones in here indicate there is a relationship between a specific financial feature and the annual stock return for companies in a specific industry sector. It’s very interesting here that in no industry sector, the current ratio seems to have the relationship with the annual stock return, while for many of the sectors, the annual stock return can be reflected by their return on assets and return on equity.


Two financial features can be used to be evaluated before potential investments:

  • Return on assets:

Basic Industries / Capital Goods / Consumer on Durables / Consumer Services / Healthcare / Public Utilities / Technology;

  • Return on equity:

Basic Industries / Finance / Healthcare / Public Utilities.

Warren Buffett used to say: Return on assets is the best way to evaluate a bank. Well, I'm open to this. I may need to take out the banks from the finance sector, and analyze solely on this group, to get a better understanding and more interesting result.

About Author

Yiming Wu

Yiming is a data science fellow in NYC data science academy. He is always passionate about machine learning, data visualization, and coding reusable scripts. He received his Ph.D. degree in Electrical Engineering from the Université de Technologie Belfort-Montbéliard,...
View all posts by Yiming Wu >

Related Articles

Leave a Comment

john m September 3, 2019
Great article. What did you use to scrape 10-Ks? I have not had great success using various methods in python.

View Posts by Categories

Our Recent Popular Posts

View Posts by Tags

#python #trainwithnycdsa 2019 2020 Revenue 3-points agriculture air quality airbnb airline alcohol Alex Baransky algorithm alumni Alumni Interview Alumni Reviews Alumni Spotlight alumni story Alumnus ames dataset ames housing dataset apartment rent API Application artist aws bank loans beautiful soup Best Bootcamp Best Data Science 2019 Best Data Science Bootcamp Best Data Science Bootcamp 2020 Best Ranked Big Data Book Launch Book-Signing bootcamp Bootcamp Alumni Bootcamp Prep boston safety Bundles cake recipe California Cancer Research capstone car price Career Career Day citibike classic cars classpass clustering Coding Course Demo Course Report covid 19 credit credit card crime frequency crops D3.js data data analysis Data Analyst data analytics data for tripadvisor reviews data science Data Science Academy Data Science Bootcamp Data science jobs Data Science Reviews Data Scientist Data Scientist Jobs data visualization database Deep Learning Demo Day Discount disney dplyr drug data e-commerce economy employee employee burnout employer networking environment feature engineering Finance Financial Data Science fitness studio Flask flight delay gbm Get Hired ggplot2 googleVis H20 Hadoop hallmark holiday movie happiness healthcare frauds higgs boson Hiring hiring partner events Hiring Partners hotels housing housing data housing predictions housing price hy-vee Income Industry Experts Injuries Instructor Blog Instructor Interview insurance italki Job Job Placement Jobs Jon Krohn JP Morgan Chase Kaggle Kickstarter las vegas airport lasso regression Lead Data Scienctist Lead Data Scientist leaflet league linear regression Logistic Regression machine learning Maps market matplotlib Medical Research Meet the team meetup methal health miami beach movie music Napoli NBA netflix Networking neural network Neural networks New Courses NHL nlp NYC NYC Data Science nyc data science academy NYC Open Data nyc property NYCDSA NYCDSA Alumni Online Online Bootcamp Online Training Open Data painter pandas Part-time performance phoenix pollutants Portfolio Development precision measurement prediction Prework Programming public safety PwC python Python Data Analysis python machine learning python scrapy python web scraping python webscraping Python Workshop R R Data Analysis R language R Programming R Shiny r studio R Visualization R Workshop R-bloggers random forest Ranking recommendation recommendation system regression Remote remote data science bootcamp Scrapy scrapy visualization seaborn seafood type Selenium sentiment analysis sentiment classification Shiny Shiny Dashboard Spark Special Special Summer Sports statistics streaming Student Interview Student Showcase SVM Switchup Tableau teachers team team performance TensorFlow Testimonial tf-idf Top Data Science Bootcamp Top manufacturing companies Transfers tweets twitter videos visualization wallstreet wallstreetbets web scraping Weekend Course What to expect whiskey whiskeyadvocate wildfire word cloud word2vec XGBoost yelp youtube trending ZORI