Data Study on Education in America
The skills the author demoed here can be learned through taking Data Science with Machine Learning bootcamp with NYC Data Science Academy.
Contributed by Thomas Boulenger. He is currently in the NYC Data Science Academy 12 week full time Data Science Bootcamp program taking place between January 11th to April 1st, 2016. This post is based on his second class project - R Shiny (due on the 4th week of the program).
Introduction
In the Big Data era, taking a look into large datasets can be a daunting task. The more accurate and specialised the data, the more difficult it might be to extract some sense out of the thousands or even millions of rows and variables. The College Scorecard Data, also featured by Kaggle for exploratory purposes might be a good first example of exploratory difficulties we may face when dealing with big data. Although not too deep (only 7,804 schools listed), it is very wide with 1,728 variables listed. Going through all variables manually requires to carefully read the Data Documentation.
In order to make the data easier to explore, we use here a Shiny application to look into 46 variables we have selected from the initial 1,728. We also trim a little the initial dataset rows by discarding the schools listed in the overseas US territories. Overall we are therefore left with a sample of the initial dataset that contains 7,019 rows and 46 columns. Note we choose here to limit ourselves to only 46 variables for the sake of making our Shiny application as fast as possible, but many more could be included.
Data Set
To start exploring the dataset, follow this link.
Here is a quick description of the 46 variables we consider here:
'CITY', "State abbreviation" = 'STABBR', 'ZIP', 'LONGITUDE', 'LATITUDE',
"Institution Name" = 'INSTNM',
"Net tuition revenue per full-time equivalent student" = 'TUITFTE',
"Instructional expenditures divided by the number of FTE students" = 'INEXPFTE',
"Average faculty salary per month" = 'AVGFACSAL',
"Average incomes
for families" = 'faminc',
"Average annual cost of attendance for academic year institutions" = 'COSTT4_A',
"Average annual cost of attendance for prgram year institutions" = 'COSTT4_P',
"Tuition for in-state students" = 'TUITIONFEE_IN',
"Tuition for out-of-state students" = 'TUITIONFEE_OUT',
"Tuititon for program-year institutions" = 'TUITIONFEE_PROG',
"Net total price for the public" = 'NPT4_PUB',
"Net total price for the private" = 'NPT4_PRIV',
"Highest award level offered at the school" = 'HIGHDEG',
"Type of degree that the school primarily awards" = 'PREDDEG',
"Predominant degree (0=N/A, 1=certificate..4=Graduate)" = 'sch_deg',
"Institution’s governance structure (public, private nonprofit, or private for-profit)" = 'CONTROL',
"Religious Affiliation" = 'RELAFFIL',
"Historically Black Colleges and Universities" = 'HBCU',
"Predominantly Black Institutions" = 'PBI',
"Alaska Native-/Native Hawaiian-serving Institutions" = 'ANNHI',
"Tribal Colleges and Universities" = 'TRIBAL',
"Asian American-/Native American-Pacific Islander-serving Institutions" = 'AANAPII',
"Hispanic-serving Institutions" = 'HSI',
"Native American Non-Tribal Institutions" = 'NANTI',
"Admissions over all campuses" = 'ADM_RATE_ALL',
"Admissions for each campuses for institutions with branches" = 'ADM_RATE',
"75th percentile for Math SAT" = 'SATMT75',
"75th percentile for Math ACT" = 'ACTMT75',
"Number of students enrolled in Fall" = 'UGDS',
"Ratio of female students" = 'female',
"Ratio of married students" = 'married',
"Percentage of White students" = 'pct_white',
"Percentage of Black students" = 'pct_black',
"Percentage of Asian students" = 'pct_asian',
"Percentage of Hispanic students" = 'pct_hispanic',
"Percentage of US born students" = 'pct_born_us',
"Poverty rate" = 'poverty_rate',
"Median loan debt" = 'DEBT_MDN',
"Median loan debt (completed students)" = 'GRAD_DEBT_MDN',
"Median loan debt (withdrawn students)" = "WDRAW_DEBT_MDN",
"3-year cohort default rate" = 'CDR3')
Of course students' future incomes and loan debts are likely to depend on which school they attend to. Most prestigious schools are for instance more likely to see their students be the most indebted as the tuition skyrockets, but they are also more likely to get the highest salaries. Can we find variables which influence by how much people are indebted? Are schools with higher average Math SAT scores less likely to see their students default within 3 years upon graduation? These are the kind of questions we may here address and answer by exploring the answers.
Data on States averages
The first tab consists in mapping averages for a slew of variables, namely: Average of the students' family incomes, Average students' tuition fee, Proportion of students who default within 3 years upon graduation, Median students' loan debt, Predominant degree types awarded in the school. To ease the exploration, a summary is provided with the value of the selected variable in the selected state. A scroll menu allows to select a sample of the dataset (either 200, 500, 1,000 or the Full 7,019 rows of the Dataset).
This feature is to prove very handy when the dataset becomes very large with many variables. Here the 46 variables still allow us to map the selected variables for the Full dataset quite quickly.
The last component is a Checkbox that allows the user to zoom in on the east coast to see more details
Data
The second simply provides a view on the data we are using, as well as a search engine that allows the user to look for a specific variable.
Overview
To map the data in more details, we use a plot from the leaflet package that allows the user to select some variables and to compare it over different school in a given area after zooming in.
For instance should we be interested in investigating the admission rate of the listed schools in Manhattan, we may selected the appropriate variable, select the Full dataset and select the New-York city in the input selection, or simply zoom in the New-York area
Data Explorer
This tab is an interactive plotting device that are designed to help the interested user in finding correlations between the numerical variables of our dataset. For instance, should there be a link between a school's average Math SAT scores and the 3 years default rate of their students? The answer seems to be yes, as we observe a downward trend which indicates that the higher scores students get at their Math SAT tests, the less likely they are to default. Note this is only an average over the all school
Note there is a summary to explain what the selected variables are, along with the percentage of missing values for that variable. Moreover, a last scroll menu allows the user to group and color the observations by the categorical variables contained in the dataset. Here for instance we see that 81% of the data are missing for the Math SAT variable and that most observations available are among schools that predominantly award advanced degree (4 = Master degree, 3 = Bachelor, etc..).
Tuition
The last tab is a prolongation of the explorer and more specifically aims at looking into the correlation of the Tuition fee to some variables within schools which are either public, private non-profit or private for-profit. For instance, looking again at the 3 years default-rate against the Tuition, we see most peak observations are concentrated around a Tuition of $10,000 p.a., while within the public dataset most observations are below $5,000.
About Author
Thomas Boulenger
View all posts by Thomas Boulenger >Leave a Comment
