MOOC Insights through Scraping edX

Posted on Aug 28, 2017

This analysis looks into one corner of the online education market - but first, let's talk about higher education more broadly. The cost of a traditional 4-year undergraduate degree has risen dramatically over recent decades. The College Board reports the total cost (including tuition, fees, room and board) of a 4-year private non-profit education rose from $16,760 in 1976 to $45,370 in 2016. The cost of a public 4-year education rose from $8,160 to $20,090 across the same years. Allan Cartter was the first (that I could find) to suggest the existence of a higher education bubble when he was Chancellor of NYU in 1964, predicting higher education in the U.S. was churning out 30-50% more grads than the number of jobs requiring degrees. Even more have echoed his calls in recent years.

Online courses, popularly known as Massive Open Online Courses, or MOOCs, are the latest means through which institutions are delivering higher educational services.

Some argue that MOOCs may one day replace traditional brick-and-mortar universities. Others see them as complements to university; or an inherently inefficient model as evidenced by low retention rates. Regardless, the popularity of MOOCs is growing, and it's important for educators, students, and financiers (shout out to Navient, my student loan servicer) to understand the market for online courses.

According to Class Central, a search engine for online courses, Coursera leads all MOOC providers with 23 million users as of December 2016, followed by edX (10 million), XuetangX (6 million), Future Learn (5.3 million), and Udemy (4 million).

The major MOOC providers offer some "original content", but their main business is providing the online venue through which their institutional partners can push their own courses out to students.

In this analysis, I lump the institutional partners into three categories:

  1. Public and private non-profit universities (e.g. Harvard, MIT, Delft University of Technology, University of Michigan)
  2. Private for-profit companies (e.g. Microsoft, New York Institute of Finance, Tenaris University)
  3. Non-profit companies and other organizations (e.g. Amnesty International, The International Monetary Fund, The World Bank Group)


This project aims to extract high-level supply-side insights into the online course market, including the supply of courses offered in different subjects, the diversity of course offerings, and the characteristics of a typical course.

Web Scraping Methods

I used Selenium to scrape the edX website for information on all courses offered (in English), then conducted exploratory data analysis in python, primarily using the numpy and pandas packages. Check out my Github for the Selenium code. I won't bore you with the details.


edX offers 1257 courses in English. For each course, I pulled the following information:

  • Title
  • Description
  • Subject (e.g. Math, Literature, Business and Management)
  • Recommended course length (typically measured in weeks)
  • Commitment (typically measured in hours per week)
  • Prerequisites
  • Price (a vast majority of courses are free)
  • Level (introductory, intermediate, advanced)


Just as Caltech specializes in math and sciences and College of the Atlantic solely offers degrees in human ecology (link: ), edX favors Computer Science and Business and Management, which account for an impressive 262 and 177, respectively, of edX's 1257 online courses. Engineering, Economics and Finance, and Biology and Life Sciences round out the top 5. On the other end, edX only offers 5 Food and Nutrition courses and 2 Ethics courses.

The below graph shows the number of courses offered in each subject by the type of sponsoring institution (green = public and private non-profit universities, red = private for-profit companies, blue = non-profit companies and other organizations).

"Traditional" public and private non-profit universities touch all areas (read: subjects) of the online education space. This makes sense. These schools already offer courses across an array of subjects to the students who pay to sit in their physical classrooms, so the barriers to entry are low. Set up a camera in the back of the classroom, upload the materials to edX, and you're in (I'm overgeneralizing).

Private companies and other non-profit organizations do not typically offer brick-and-mortar educational services, so entering the online course market requires more effort, more investment. These institutions need to find instructors, develop curriculums, and put together materials before deciding to offer a course in astronomy, or literature. Higher barriers to entry mean there's more on the line if the online course fails to attract students.

This is perhaps why we see private for-profit companies and other non-profit organizations only in the more established, proven online education market segments like computer science, business, finance, and engineering. This leaves the newer, smaller segments, primarily in the humanities, to the public and private non-profit universities.

Now on to some features of the courses themselves. The total length of a course (measured as the course length (in weeks) multiplied by the weekly commitment (hours per week)) can tell us how strenuous or difficult it is. We can also look at the distribution of course lengths as a proxy for product "diversity" across different subjects.

The below violin plot (Social Sciences subjects only) shows the distribution of courses in each subject by the number of total hours for each course. Subjects with more course offerings (toward the left) tend to have more elongated distributions of course lengths than subjects with fewer offerings (toward the right). One could argue that saturated markets like Business and Management encourage product diversity and specialization - institutions are willing to take risks and deviate from the typical course length seen in a traditional university setting. Subjects with fewer course offerings (e.g. Ethics) are still dominated by the traditional universities with little incentive to stray from their proven model. This course length distribution trend generally exists among math/science and humanities subjects too - email me if you'd like to see those violin plots.

The distribution of course lengths differs by subject - how about by institution type?

Statistical Analysis

A one-way ANOVA test can tell us whether average course length varies among the three groups of institutional offerors (public/private non-profit universities, private for-profit companies, and non-profit companies/other organizations).

The distributions of course lengths by institution type are skewed right. A log transformation gives us more "Gaussian-like" distributions, and helps us to better satisfy the equal variance assumption required in order to produce meaningful ANOVA test results.

Institution Type Number of Courses Offered Course Length Average (hrs) Course Length Variance log(Course Length) Average log(Course Length) Variance
Public or Non-Profit University 997 40.4 1176.7 3.38 .70
Private Company 126 15.9 128.8 2.53 .54
Non-Profit Company or Other Organization 70 33.6 5132.7 2.82 1.11

Distribution of course lengths - pre-log transformation

Distribution of course hours - post-log transformation

Null Hypothesis: edX courses offered by 1) Public/Non-Profit Universities, 2) Private Companies, and 3) Non-Profit Companies/Other Organizations have the same average length.

Alternative Hypothesis: one or more of these groups has a different mean course length than the others.

One-way ANOVA test results:
F-statistic: 66.11
P-value: 5.93 e-28

Conclusion: reject the null hypothesis in favor of the alternative. The three institution types do not offer the same average course length.

Course length is just one feature of a MOOC, yet I argue it is one of the most important proxies we can analyze to gauge product diversity. Traditional universities generally offer courses of similar lengths. Deviation is costly. Other providers can be more flexible in their approach.

The higher education bubble may soon burst. Educators and students should understand their options and the relevant stakeholders.

About Author


Patrick Masi-Phelps

Patrick graduated from Wesleyan University with a bachelor's degree in Mathematics and Economics. After graduation, he worked in regulatory consulting for two years, primarily with traditional buyout private equity and quantitative trading firms. Patrick enrolled in NYCDSA to...
View all posts by Patrick Masi-Phelps >

Related Articles

Leave a Comment

No comments found.

View Posts by Categories

Our Recent Popular Posts

View Posts by Tags

#python #trainwithnycdsa 2019 airbnb Alex Baransky alumni Alumni Interview Alumni Reviews Alumni Spotlight alumni story Alumnus API Application artist aws beautiful soup Best Bootcamp Best Data Science 2019 Best Data Science Bootcamp Best Data Science Bootcamp 2020 Best Ranked Big Data Book Launch Book-Signing bootcamp Bootcamp Alumni Bootcamp Prep Bundles California Cancer Research capstone Career Career Day citibike clustering Coding Course Demo Course Report D3.js data Data Analyst data science Data Science Academy Data Science Bootcamp Data science jobs Data Science Reviews Data Scientist Data Scientist Jobs data visualization Deep Learning Demo Day Discount dplyr employer networking feature engineering Finance Financial Data Science Flask gbm Get Hired ggplot2 googleVis Hadoop higgs boson Hiring hiring partner events Hiring Partners Industry Experts Instructor Blog Instructor Interview Job Job Placement Jobs Jon Krohn JP Morgan Chase Kaggle Kickstarter lasso regression Lead Data Scienctist Lead Data Scientist leaflet linear regression Logistic Regression machine learning Maps matplotlib Medical Research Meet the team meetup Networking neural network Neural networks New Courses nlp NYC NYC Data Science nyc data science academy NYC Open Data NYCDSA NYCDSA Alumni Online Online Bootcamp Online Training Open Data painter pandas Part-time Portfolio Development prediction Prework Programming PwC python python machine learning python scrapy python web scraping python webscraping Python Workshop R R language R Programming R Shiny r studio R Visualization R Workshop R-bloggers random forest Ranking recommendation recommendation system regression Remote remote data science bootcamp Scrapy scrapy visualization seaborn Selenium sentiment analysis Shiny Shiny Dashboard Spark Special Special Summer Sports statistics streaming Student Interview Student Showcase SVM Switchup Tableau team TensorFlow Testimonial tf-idf Top Data Science Bootcamp twitter visualization web scraping Weekend Course What to expect word cloud word2vec XGBoost yelp