Data Study on Education in America

Posted on Mar 3, 2016
The skills the author demoed here can be learned through taking Data Science with Machine Learning bootcamp with NYC Data Science Academy.
Contributed by Thomas Boulenger. He is currently in the NYC Data Science Academy 12 week full time Data Science Bootcamp program taking place between January 11th to April 1st, 2016. This post is based on his second class project - R Shiny (due on the 4th week of the program).


In the Big Data era, taking a look into large datasets can be a daunting task. The more accurate and specialised the data, the more difficult it might be to extract some sense out of the thousands or even millions of rows and variables. The College Scorecard Data, also featured by Kaggle for exploratory purposes might be a good first example of exploratory difficulties we may face when dealing with big data. Although not too deep (only 7,804 schools listed), it is very wide with 1,728 variables listed. Going through all variables manually requires to carefully read the Data Documentation.

In order to make the data easier to explore, we use here a Shiny application to look into 46 variables we have selected from the initial 1,728. We also trim a little the initial dataset rows by discarding the schools listed in the overseas US territories. Overall we are therefore left with a sample of the initial dataset that contains 7,019 rows and 46 columns. Note we choose here to limit ourselves to only 46 variables for the sake of making our Shiny application as fast as possible, but many more could be included.

Data Set

To start exploring the dataset, follow this link.

Here is a quick description of the 46 variables we consider here:

'CITY', "State abbreviation" = 'STABBR', 'ZIP', 'LONGITUDE', 'LATITUDE',
"Institution Name" = 'INSTNM',
"Net tuition revenue per full-time equivalent student" = 'TUITFTE',
"Instructional expenditures divided by the number of FTE students" = 'INEXPFTE',
"Average faculty salary per month" = 'AVGFACSAL',
"Average incomes for families" = 'faminc',
"Average annual cost of attendance for academic year institutions" = 'COSTT4_A',
"Average annual cost of attendance for prgram year institutions" = 'COSTT4_P',
"Tuition for in-state students" = 'TUITIONFEE_IN',
"Tuition for out-of-state students" = 'TUITIONFEE_OUT',
"Tuititon for program-year institutions" = 'TUITIONFEE_PROG',
"Net total price for the public" = 'NPT4_PUB',
"Net total price for the private" = 'NPT4_PRIV',
"Highest award level offered at the school" = 'HIGHDEG',
"Type of degree that the school primarily awards" = 'PREDDEG',
"Predominant degree (0=N/A, 1=certificate..4=Graduate)" = 'sch_deg',
"Institution’s governance structure (public, private nonprofit, or private for-profit)" = 'CONTROL',
"Religious Affiliation" = 'RELAFFIL',
"Historically Black Colleges and Universities" = 'HBCU',
"Predominantly Black Institutions" = 'PBI',
"Alaska Native-/Native Hawaiian-serving Institutions" = 'ANNHI',
"Tribal Colleges and Universities" = 'TRIBAL',
"Asian American-/Native American-Pacific Islander-serving Institutions" = 'AANAPII',
"Hispanic-serving Institutions" = 'HSI',
"Native American Non-Tribal Institutions" = 'NANTI',
"Admissions over all campuses" = 'ADM_RATE_ALL',
"Admissions for each campuses for institutions with branches" = 'ADM_RATE',
"75th percentile for Math SAT" = 'SATMT75',
"75th percentile for Math ACT" = 'ACTMT75',
"Number of students enrolled in Fall" = 'UGDS',
"Ratio of female students" = 'female',
"Ratio of married students" = 'married',
"Percentage of White students" = 'pct_white',
"Percentage of Black students" = 'pct_black',
"Percentage of Asian students" = 'pct_asian',
"Percentage of Hispanic students" = 'pct_hispanic',
"Percentage of US born students" = 'pct_born_us',
"Poverty rate" = 'poverty_rate',
"Median loan debt" = 'DEBT_MDN',
"Median loan debt (completed students)" = 'GRAD_DEBT_MDN',
"Median loan debt (withdrawn students)" = "WDRAW_DEBT_MDN",
"3-year cohort default rate" = 'CDR3')

Of course students' future incomes and loan debts are likely to depend on which school they attend to. Most prestigious schools are for instance more likely to see their students be the most indebted as the tuition skyrockets, but they are also more likely to get the highest salaries. Can we find variables which influence by how much people are indebted? Are schools with higher average Math SAT scores less likely to see their students default within 3 years upon graduation? These are the kind of questions we may here address and answer by exploring the answers.

Data on States averages

The first tab consists in mapping averages for a slew of variables, namely: Average of the students' family incomes, Average students' tuition fee, Proportion of students who default within 3 years upon graduation, Median students' loan debt, Predominant degree types awarded in the school. To ease the exploration, a summary is provided with the value of the selected variable in the selected state. A scroll menu allows to select a sample of the dataset (either 200, 500, 1,000 or the Full 7,019 rows of the Dataset).

This feature is to prove very handy when the dataset becomes very large with many variables. Here the 46 variables still allow us to map the selected variables for the Full dataset quite quickly.

Data Study on Education in America

The last component is a Checkbox that allows the user to zoom in on the east coast to see more details

Screen Shot 2016-03-03 at 14.39.15


The second simply provides a view on the data we are using, as well as a search engine that allows the user to look for a specific variable.

Screen Shot 2016-03-03 at 14.44.30


To map the data in more details, we use a plot from the leaflet package that allows the user to select some variables and to compare it over different school in a given area after zooming in.

Screen Shot 2016-03-03 at 14.44.57

For instance should we be interested in investigating the admission rate of the listed schools in Manhattan, we may selected the appropriate variable, select the Full dataset and select the New-York city in the input selection, or simply zoom in the New-York area

Screen Shot 2016-03-03 at 14.48.00

Data Explorer

This tab is an interactive plotting device that are designed to help the interested user in finding correlations between the numerical variables of our dataset. For instance, should there be a link between a school's average Math SAT scores and the 3 years default rate of their students? The answer seems to be yes, as we observe a downward trend which indicates that the higher scores students get at their Math SAT tests, the less likely they are to default. Note this is only an average over the all school

Screen Shot 2016-03-03 at 14.50.34

Note there is a summary to explain what the selected variables are, along with the percentage of missing values for that variable. Moreover, a last scroll menu allows the user to group and color the observations by the categorical variables contained in the dataset. Here for instance we see that 81% of the data are missing for the Math SAT variable and that most observations available are among schools that predominantly award advanced degree (4 = Master degree, 3 = Bachelor, etc..).


The last tab is a prolongation of the explorer and more specifically aims at looking into the correlation of the Tuition fee to some variables within schools which are either public, private non-profit or private for-profit. For instance, looking again at the 3 years default-rate against the Tuition, we see most peak observations are concentrated around a Tuition of $10,000 p.a., while within the public dataset most observations are below $5,000.

Screen Shot 2016-03-03 at 15.42.32

About Author

Thomas Boulenger

Thomas holds a PhD in pure Mathematics from Paris, and started to pick up an interest for Machine Learning while still a Postdoc at Basel University. After spending a few years in academia, he decided to join NYC...
View all posts by Thomas Boulenger >

Leave a Comment

Mortal kombat X Hack November 4, 2016
Great post. I am facing a couple of these difficulties.
mortal kombat x hack October 25, 2016
Great post. I am confronting a couple of these difficulties.
piep.Net August 1, 2016
Biig brabd examples include JetBlue and CarMax Luigi A cold andd sterile office is likely the days seem a lot longer one individual get exactly the research applied traditional) printing typically more efficient web sitee There are certain trademarks to my a jjob Luigi What's more, most when a like to always install extra softsare just to make web page "work"

View Posts by Categories

Our Recent Popular Posts

View Posts by Tags

#python #trainwithnycdsa 2019 2020 Revenue 3-points agriculture air quality airbnb airline alcohol Alex Baransky algorithm alumni Alumni Interview Alumni Reviews Alumni Spotlight alumni story Alumnus ames dataset ames housing dataset apartment rent API Application artist aws bank loans beautiful soup Best Bootcamp Best Data Science 2019 Best Data Science Bootcamp Best Data Science Bootcamp 2020 Best Ranked Big Data Book Launch Book-Signing bootcamp Bootcamp Alumni Bootcamp Prep boston safety Bundles cake recipe California Cancer Research capstone car price Career Career Day citibike classic cars classpass clustering Coding Course Demo Course Report covid 19 credit credit card crime frequency crops D3.js data data analysis Data Analyst data analytics data for tripadvisor reviews data science Data Science Academy Data Science Bootcamp Data science jobs Data Science Reviews Data Scientist Data Scientist Jobs data visualization database Deep Learning Demo Day Discount disney dplyr drug data e-commerce economy employee employee burnout employer networking environment feature engineering Finance Financial Data Science fitness studio Flask flight delay gbm Get Hired ggplot2 googleVis H20 Hadoop hallmark holiday movie happiness healthcare frauds higgs boson Hiring hiring partner events Hiring Partners hotels housing housing data housing predictions housing price hy-vee Income Industry Experts Injuries Instructor Blog Instructor Interview insurance italki Job Job Placement Jobs Jon Krohn JP Morgan Chase Kaggle Kickstarter las vegas airport lasso regression Lead Data Scienctist Lead Data Scientist leaflet league linear regression Logistic Regression machine learning Maps market matplotlib Medical Research Meet the team meetup methal health miami beach movie music Napoli NBA netflix Networking neural network Neural networks New Courses NHL nlp NYC NYC Data Science nyc data science academy NYC Open Data nyc property NYCDSA NYCDSA Alumni Online Online Bootcamp Online Training Open Data painter pandas Part-time performance phoenix pollutants Portfolio Development precision measurement prediction Prework Programming public safety PwC python Python Data Analysis python machine learning python scrapy python web scraping python webscraping Python Workshop R R Data Analysis R language R Programming R Shiny r studio R Visualization R Workshop R-bloggers random forest Ranking recommendation recommendation system regression Remote remote data science bootcamp Scrapy scrapy visualization seaborn seafood type Selenium sentiment analysis sentiment classification Shiny Shiny Dashboard Spark Special Special Summer Sports statistics streaming Student Interview Student Showcase SVM Switchup Tableau teachers team team performance TensorFlow Testimonial tf-idf Top Data Science Bootcamp Top manufacturing companies Transfers tweets twitter videos visualization wallstreet wallstreetbets web scraping Weekend Course What to expect whiskey whiskeyadvocate wildfire word cloud word2vec XGBoost yelp youtube trending ZORI