A Glimpse into US Education

Thomas Boulenger
Posted on Mar 3, 2016

Contributed by Thomas Boulenger. He is currently in the NYC Data Science Academy 12 week full time Data Science Bootcamp program taking place between January 11th to April 1st, 2016. This post is based on his second class project - R Shiny (due on the 4th week of the program).

In the Big Data era, taking a look into large datasets can be a daunting task. The more accurate and specialised the data, the more difficult it might be to extract some sense out of the thousands or even millions of rows and variables. The College Scorecard Data, also featured by Kaggle for exploratory purposes might be a good first example of exploratory difficulties we may face when dealing with big data. Although not too deep (only 7,804 schools listed), it is very wide with 1,728 variables listed. Going through all variables manually requires to carefully read the Data Documentation. In order to make the data easier to explore, we use here a Shiny application to look into 46 variables we have selected from the initial 1,728. We also trim a little the initial dataset rows by discarding the schools listed in the overseas US territories. Overall we are therefore left with a sample of the initial dataset that contains 7,019 rows and 46 columns. Note we choose here to limit ourselves to only 46 variables for the sake of making our Shiny application as fast as possible, but many more could be included.

To start exploring the dataset, follow this link.

Here is a quick description of the 46 variables we consider here:

'CITY', "State abbreviation" = 'STABBR', 'ZIP', 'LONGITUDE', 'LATITUDE',
"Institution Name" = 'INSTNM',
"Net tuition revenue per full-time equivalent student" = 'TUITFTE',
"Instructional expenditures divided by the number of FTE students" = 'INEXPFTE',
"Average faculty salary per month" = 'AVGFACSAL',
"Average incomes for families" = 'faminc',
"Average annual cost of attendance for academic year institutions" = 'COSTT4_A',
"Average annual cost of attendance for prgram year institutions" = 'COSTT4_P',
"Tuition for in-state students" = 'TUITIONFEE_IN',
"Tuition for out-of-state students" = 'TUITIONFEE_OUT',
"Tuititon for program-year institutions" = 'TUITIONFEE_PROG',
"Net total price for the public" = 'NPT4_PUB',
"Net total price for the private" = 'NPT4_PRIV',
"Highest award level offered at the school" = 'HIGHDEG',
"Type of degree that the school primarily awards" = 'PREDDEG',
"Predominant degree (0=N/A, 1=certificate..4=Graduate)" = 'sch_deg',
"Institution’s governance structure (public, private nonprofit, or private for-profit)" = 'CONTROL',
"Religious Affiliation" = 'RELAFFIL',
"Historically Black Colleges and Universities" = 'HBCU',
"Predominantly Black Institutions" = 'PBI',
"Alaska Native-/Native Hawaiian-serving Institutions" = 'ANNHI',
"Tribal Colleges and Universities" = 'TRIBAL',
"Asian American-/Native American-Pacific Islander-serving Institutions" = 'AANAPII',
"Hispanic-serving Institutions" = 'HSI',
"Native American Non-Tribal Institutions" = 'NANTI',
"Admissions over all campuses" = 'ADM_RATE_ALL',
"Admissions for each campuses for institutions with branches" = 'ADM_RATE',
"75th percentile for Math SAT" = 'SATMT75',
"75th percentile for Math ACT" = 'ACTMT75',
"Number of students enrolled in Fall" = 'UGDS',
"Ratio of female students" = 'female',
"Ratio of married students" = 'married',
"Percentage of White students" = 'pct_white',
"Percentage of Black students" = 'pct_black',
"Percentage of Asian students" = 'pct_asian',
"Percentage of Hispanic students" = 'pct_hispanic',
"Percentage of US born students" = 'pct_born_us',
"Poverty rate" = 'poverty_rate',
"Median loan debt" = 'DEBT_MDN',
"Median loan debt (completed students)" = 'GRAD_DEBT_MDN',
"Median loan debt (withdrawn students)" = "WDRAW_DEBT_MDN",
"3-year cohort default rate" = 'CDR3')

Of course students' future incomes and loan debts are likely to depend on which school they attend to. Most prestigious schools are for instance more likely to see their students be the most indebted as the tuition skyrockets, but they are also more likely to get the highest salaries. Can we find variables which influence by how much people are indebted? Are schools with higher average Math SAT scores less likely to see their students default within 3 years upon graduation? These are the kind of questions we may here address and answer by exploring the answers.

States averages

The first tab consists in mapping averages for a slew of variables, namely: Average of the students' family incomes, Average students' tuition fee, Proportion of students who default within 3 years upon graduation, Median students' loan debt, Predominant degree types awarded in the school. To ease the exploration, a summary is provided with the value of the selected variable in the selected state. A scroll menu allows to select a sample of the dataset (either 200, 500, 1,000 or the Full 7,019 rows of the Dataset). This feature is to prove very handy when the dataset becomes very large with many variables. Here the 46 variables still allow us to map the selected variables for the Full dataset quite quickly.

Screen Shot 2016-03-03 at 14.36.45

The last component is a Checkbox that allows the user to zoom in on the east coast to see more details

Screen Shot 2016-03-03 at 14.39.15


The second simply provides a view on the data we are using, as well as a search engine that allows the user to look for a specific variable.

Screen Shot 2016-03-03 at 14.44.30


To map the data in more details, we use a plot from the leaflet package that allows the user to select some variables and to compare it over different school in a given area after zooming in.

Screen Shot 2016-03-03 at 14.44.57

For instance should we be interested in investigating the admission rate of the listed schools in Manhattan, we may selected the appropriate variable, select the Full dataset and select the New-York city in the input selection, or simply zoom in the New-York area

Screen Shot 2016-03-03 at 14.48.00


This tab is an interactive plotting device that are designed to help the interested user in finding correlations between the numerical variables of our dataset. For instance, should there be a link between a school's average Math SAT scores and the 3 years default rate of their students? The answer seems to be yes, as we observe a downward trend which indicates that the higher scores students get at their Math SAT tests, the less likely they are to default. Note this is only an average over the all school

Screen Shot 2016-03-03 at 14.50.34

Note there is a summary to explain what the selected variables are, along with the percentage of missing values for that variable. Moreover, a last scroll menu allows the user to group and color the observations by the categorical variables contained in the dataset. Here for instance we see that 81% of the data are missing for the Math SAT variable and that most observations available are among schools that predominantly award advanced degree (4 = Master degree, 3 = Bachelor, etc..).


The last tab is a prolongation of the explorer and more specifically aims at looking into the correlation of the Tuition fee to some variables within schools which are either public, private non-profit or private for-profit. For instance, looking again at the 3 years default-rate against the Tuition, we see most peak observations are concentrated around a Tuition of $10,000 p.a., while within the public dataset most observations are below $5,000.

Screen Shot 2016-03-03 at 15.42.32

About Author

Thomas Boulenger

Thomas Boulenger

Thomas holds a PhD in pure Mathematics from Paris, and started to pick up an interest for Machine Learning while still a Postdoc at Basel University. After spending a few years in academia, he decided to join NYC...
View all posts by Thomas Boulenger >

Leave a Comment

Mortal kombat X Hack November 4, 2016
Great post. I am facing a couple of these difficulties.
mortal kombat x hack October 25, 2016
Great post. I am confronting a couple of these difficulties.
piep.Net August 1, 2016
Biig brabd examples include JetBlue and CarMax Luigi A cold andd sterile office is likely the days seem a lot longer korot.uk one individual get exactly the research applied st.grogol.us traditional) printing typically more efficient web sitee There are certain trademarks to my a jjob Luigi What's more, most when a like to always install extra softsare just to make web page "work" tapeciarnia.waw.pl

View Posts by Categories

Our Recent Popular Posts

View Posts by Tags

#python #trainwithnycdsa 2019 airbnb Alex Baransky alumni Alumni Interview Alumni Reviews Alumni Spotlight alumni story Alumnus API Application artist aws beautiful soup Best Bootcamp Best Data Science 2019 Best Data Science Bootcamp Best Data Science Bootcamp 2020 Best Ranked Big Data Book Launch Book-Signing bootcamp Bootcamp Alumni Bootcamp Prep Bundles California Cancer Research capstone Career Career Day citibike clustering Coding Course Demo Course Report D3.js data Data Analyst data science Data Science Academy Data Science Bootcamp Data science jobs Data Science Reviews Data Scientist Data Scientist Jobs data visualization Deep Learning Demo Day Discount dplyr employer networking feature engineering Finance Financial Data Science Flask gbm Get Hired ggplot2 googleVis Hadoop higgs boson Hiring hiring partner events Hiring Partners Industry Experts Instructor Blog Instructor Interview Job Job Placement Jobs Jon Krohn JP Morgan Chase Kaggle Kickstarter lasso regression Lead Data Scienctist Lead Data Scientist leaflet linear regression Logistic Regression machine learning Maps matplotlib Medical Research Meet the team meetup Networking neural network Neural networks New Courses nlp NYC NYC Data Science nyc data science academy NYC Open Data NYCDSA NYCDSA Alumni Online Online Bootcamp Online Training Open Data painter pandas Part-time Portfolio Development prediction Prework Programming PwC python Python Data Analysis python machine learning python scrapy python web scraping python webscraping Python Workshop R R language R Programming R Shiny r studio R Visualization R Workshop R-bloggers random forest Ranking recommendation recommendation system regression Remote remote data science bootcamp Scrapy scrapy visualization seaborn Selenium sentiment analysis Shiny Shiny Dashboard Spark Special Special Summer Sports statistics streaming Student Interview Student Showcase SVM Switchup Tableau team TensorFlow Testimonial tf-idf Top Data Science Bootcamp twitter visualization web scraping Weekend Course What to expect word cloud word2vec XGBoost yelp