A Performance Data Analysis on The History of Content
The skills I demoed here can be learned through taking Data Science with Machine Learning bootcamp with NYC Data Science Academy.
INTRO
Throughout history, there have been periods of growth and periods of stagnation. The same is true of the words we use. In the 1400s, for example, data showed that interestingly, humans spoke in different ways, with inflections and tones that simply don't match how we would use them today.
In the English Renaissance, words changed further, bringing in additions from brilliant minds like William Shakespeare and Edmund Spencer.
Then, in the 19th century, language changed. The establishment of a new outline - specifically called the First Consonant Shift - created a relationship in words that changed the way we speak, shifting our tones for a more complete interpretation of language.
Based on research by ThoughtCo, this was "among the earliest Germanic sound changes" and is now considered to be a well-established theory in the field of linguistics.
This study, using a subsample of Google Ngram, set out to examine how language, and specifically the words we use, have changed over time. Along the way, we saw that content – publication of content in the written form – has increased so dramatically over the years that it became a center for how we look at our data. With more time, this work would go continue with data engineering, including the extrapolation of larger sub samples (more on this below). It would also go into more depth with the analysis of letters and their combinations.
In a nutshell, this work examines the shift of clusters of letters over the last several decades.
Fig 0: Famous universities and their performance
THE DATA
This internal project used Google Ngram, a corpus of data that records word usage over time, to perform its analysis. Google Ngram started as a prototype called “Bookworm”, and was initially released to the public in December of 2010. At its core, it is a database of word usage, showing how trends change in a given language over the course of decades.
The subsample consisted of 28.9 million rows, 4 columns, and several additional features which were created during analysis. This subsample pulled from the 86+ million rows where our analysis was focusing on, so while the data was indicative of its larger sample of “A” grams, it was not representative of the entire English language, which was closer to 2.3 billion. One thing to note, the full corpus of American English language data is so large that the Google project itself had to limit its sampling so that it could hold all of the necessary information in aggregate.
Business challenge: How might we understand the history of the English language? How could this allow us to determine where language might be trending? How might this be relevant as we look at trends in content creation?
Technical challenge: The data itself, with sampling, is close to 30 million rows. The technical challenge was to download, store, and manipulate the data, given the time constraints at hand. Python, Koalas, PySpark, HDFS, SQL, and ultimately pandas were used to interpret and understand this data, with varying levels of success.
EXPLORATORY DATA ANALYSIS
The first challenge was to download and format the data.
The original set of data was formatted word by word, year by year, with counts for each word-year pair. For example, the word “Aang” had more than 500 different entries in a multi-million row dataset. In comparison, “Astronaut” had fewer years associated with it, making anlaysis more complex.
To clean up this data, I used regex / string cleaning functions followed by groupbys for organization. The simplest way to organize the data was by year. However, doing so would create a groupby dataframe organized by word – not by letter, which would allow us to aggregate information across categorical variables.
To do this, we had to turn the count of each letter into a continuous metric. This involved creating a new matrix, organizing by year, and with the count of each letter multiplied and summed it was easier to reference the split count of the word's performance and length as denoted by year. The final result was a set of 26 groupby dataframes, which could be aggregated together to determine performance of all segments from 1900 - 2010.
Fig 1: the aggregated data
This analysis looked at three dimensions: D1, the count of the letter; D2, the given year of that record, and D3, the average length of word.
TIME SERIES
As is to be expected, we see a curve shaped pattern in length and year. When we see what the relationship is, we have a more clear interpretation of the data. Let’s investigate this further by breaking down each dimension into a 2-D relationship, and go from there.
As a side note, as this data increases in time, the logarithmic scale does not apply. However, we can take the average count of how often the letter appears in a year, calling this frequency. For each letter, we measure one aggregate frequency, and then expand this to a noted frequency over the course of any given of years.
Fig 2: aggregated frequency by year
This analysis was conducted with the understanding that, as more years pass, letters will likely appear more often (as more books are published, more data is created). We did not create a measure of language-based inflation which could take into account the scale to which words are published, and in turn compare how different letters perform over the years.
Fig 3: aggregated word length and frequency by year
Next, we looked at how length of word performed over time. Here, we extracted a key insight, which could be attributed to multiple factors, including the advent of the internet and social media. As the years increase between 1900 to about 1980, word length increases in a linear relationship. Then, in the late 70s, we begin to see a decline in the length of words processed each year – a trend that shows how important these counts were to the analysis that we did.
Interestingly, this aggregated data offers only hints as to why the length of words has decreased since then. Here, we offer a few possible solutions.
- This data could indicate a surge in popularity among books with shortened words – fiction books, comic books, Sunday morning cartoons, etc.
- It could suggest that as more complex topics became less popular, like psychology, hard sciences, or information theory, the usage of terminology in these words has decreased in tune.
- Moreover, it could show the result a new technology, like the internet, or pagers, causing an impact on how we communicate.
- It could be cultural trend that draws from another sub-cultural arena.
One interesting point here is that this data does come from the full Google Ngram database - so while one might assume that the results are not indicative of the whole population, the corpus quite literally is the whole population. This scale is relatively unheard of in modern day data analysis.
Data Analysis of Variance (ANOVA)
Based on the current analysis, there appeared to be a positive correlation between frequency and year, and another interesting relationship between length and year. How does this data look across the different letters? Which letter segment performs the best?
Breaking away from total counts, an A/B Test showed the performance of the letters summed. This data was uploaded into a small interactive module with Dash.
Fig 5, frequency and word length by segment, by year
This chart shows the performance of each letter over the years, with the length shown as the color dimension. The representation of shade as a continuous variable allows us to examine up to 26 different segments, and their performance, at any given time.
As we might expect, the count of each letter changes as the years progress. Looking at segments, one notable point is that it is not possible to tell what the unadjusted increase would be over time - this calculation only shows degrees of increase, which vary by letter. It would be useful here to determine whether some sort of significant relative increase exists.
Fig 6, frequency and word length by category, by year
For the majority of the letter-segment combinations, there was no useful takeaway or insight. The segments were, for the most part, statistically significant, analyzed at an alpha of 0.05. So we can say that it is likely each segment had its own unique trend.
CONCLUSION
All things considered, this project allowed for a more comprehensive understanding of language.
Looking at the trends of the segments, we pair the changes over time to see where there might be a trend.
On the whole, it appeared that letter trends were unique. Statistical significance was achieved by confirming the letter frequency pair by length of word. This led to an analysis of clusters which, aside from being interesting, showed few notable insights.
We also see that, on the whole, the amount of content created has absolutely exploded since the 1900s. This should come as no surprise, but seeing it aggregated on a dataset was quite impactful.
On a more granular level, it was interesting to note the differences between letters. Again, an inflation calculation would have been useful here, as well as the ability to confirm that the variables were increased in a way that was statistically significant. This sample was 1/3 of 1/26 of the English language, and so cannot be considered to be representative of the data at large. Further analysis is needed to confirm that this data is representative of all data from the “A” frame, and this analysis would need to be multiplied across all other letter frames to understand trends in the English language on the whole.
Finally, as we consider the changing of language in the future, it is important to note that the future of language is, and its core, tied in with content. Therefore, on a macro-scale, solidifying these trends, and being able to question why the words and patterns we see impact the future of langauge, is a useful feature. It can provide us with a means of understanding, an easier way to work, and a glimpse into how we might prepare for language decades into the future.
About Author
