CoverSpy: Adventures Scraping Twitter and Tumblr
Introduction
As an avid reader and a nosy onlooker, my goal for this project was to answer the following questions: What are Americans reading across different cities? How greatly do reading choices differ by city and even by local environment? Can demographic and personal information on the readers be identified?
Although libraries, bookstores, and online retailers release general information on their most popular books, this project utilized a new and unique data source: Coverspy. Coverspy is a twitter page that encourages average citizens to become subway voyeurs and report what those around them are reading. A typical post includes the title of a book, author, and description of the reader (including location). Coverspy is an interesting source of information not only because it displays a snapshot of the popular authors and books at a period of time, but it is highly location-specific and descriptive. Because of the popularity of the New York City Coverspy page, twitter accounts have opened for other cities, including Boston, Chicago, Washington DC, and Barcelona.
Methodology
In order to obtain the desired information from Coverspy, I used the Selenium package in Python to obtain the following information from the Coverspy pages for New York City, Boston, and Chicago: Book title, Author, Timestamp (later discarded) and Description. The github link below contains further code used for the project:
https://github.com/KCritelli/Web_Scraping/tree/master/Scraping_Project_Files
The greatest limitation with the Coverspy data was that there were not very many observations to work with. For each city, there were roughly 800 entries, some of which were incomplete (for example, the viewer did not see the book title, only the author). Nevertheless, the results obtained were interesting and provided preliminary answers to the questions posed at the beginning of the project. As more entries are added to the Coverspy page, it will become a more interesting page to scrape.
Results
My first goal was to compare the most popular book titles and authors on the subways/metros of different cities. Acknowledging relatively few entries for each city, the results suggest that Americans in New York, Boston, and Chicago read quite differently. For example, the most popular author in identified by Coverspy New York was Zadie Smith- yet, she did not rank in the top 12 in either Boston or Chicago.
| Top Authors in New York | Top Authors in Boston | Top Authors in Chicago | ||||||||
| Rank | Author | Views | Rank | Author | Views | Rank | Author | Views | ||
| 1 | Zadie Smith | 12 | 1 | Gillian Flynn | 8 | 1 | Erik Larson | 10 | ||
| 2 | J.K. Rowling | 11 | 2 | J. K. Rowling | 7 | 2 | Anthony Doerr | 6 | ||
| 3 | Neil Gaiman | 11 | 3 | Cheryl Strayed | 6 | 3 | George R. R. Martin | 6 | ||
| 4 | Haruki Murakami | 10 | 4 | Chimamanda Ngozi Adichie | 5 | 4 | Gillian Flynn | 6 | ||
| 5 | Margaret Atwood | 10 | 5 | George R. R. Martin | 5 | 5 | Michael Chabon | 6 | ||
| 6 | Stephen King | 10 | 6 | Jojo Moyes | 5 | 6 | Stephen King | 6 | ||
| 7 | Chimamanda Ngozi Adichie | 9 | 7 | Diana Gabaldon | 4 | 7 | Chimamanda Ngozi Adichie | 4 | ||
| 8 | Paul Beatty | 9 | 8 | Donna Tartt | 4 | 8 | David Foster Wallace | 4 | ||
| 9 | George Orwell | 7 | 9 | Haruki Murakami | 4 | 9 | Haruki Murakami | 4 | ||
| 10 | Hanya Yanagihara | 6 | 10 | Lee Child | 4 | 10 | Liane Moriarty | 4 | ||
| 11 | Elena Ferrante | 5 | 11 | Malcolm Gladwell | 4 | 11 | Michael Crichton | 4 | ||
| 12 | Liane Moriarty | 5 | 12 | Stephen King | 4 | 12 | Michael Lewis | 4 | ||
After examining title and authors breakdowns by city, I pooled the results for different cities and compared reading patterns for men and women. Although there were far more data entries for women than men (1054 observations of women versus 688 observations of men), the most interesting result turned up in the list of top authors read by men:
| Men: Top Authors | ||
| Author | Views | |
| 1 | Stephen King | 12 |
| 2 | Gillian Flynn | 7 |
| 3 | Neil Gaiman | 7 |
| 4 | George Orwell | 6 |
| 5 | George R. R. Martin | 6 |
| 6 | David Foster Wallace | 5 |
| 7 | Haruki Murakami | 5 |
| 8 | Malcolm Gladwell | 5 |
| 9 | Cormac McCarthy | 4 |
| 10 | Erik Larson | 4 |
| 11 | Hanya Yanagihara | 4 |
| 12 | Michael Lewis | 4 |
| 13 | Andy Weir | 3 |
| 14 | Ayn Rand | 3 |
| 15 | Bill Bryson | 3 |
| 16 | Dan Simmons | 3 |
| 17 | Daniel Kahneman | 3 |
| 18 | David Mitchell | 3 |
| 19 | Frank Herbert | 3 |
| 20 | Fyodor Dostoevsky | 3 |
Interestingly, while the top authors read by women were spread evenly across genres and gender, male readers tended to heavily favor male authors (17 of the top 20 authors for male readers are also male). Although further research would need to be done, this data heavily suggests that men tend to read very differently from women, perhaps more narrowly.
To further investigate reader demographics, I produced a histogram of observations by age on the New York City subway. I found that the vast majority of observed readers are in their twenties, with counts decreasing steadily by age.
Finally, I decided to use the reader descriptions to create word clouds based on location. Word clouds are plots based on word frequency: in this case, the most frequent words in the reader descriptions appear as the largest, most centrally-located words. I sorted the descriptions by subway train to see if the readers of New York displayed identifiable "personalities" based on where they were reading. Though this last visualization is highly subjective and mostly for entertainment, I would argue that the readers on the F train (which goes back and forth to Brooklyn) stand out in their description.
Conclusions
On a broad level, this project taught me the importance of working with a large dataset. Although Coverspy is a novel and fun website and it allows viewers to look at many variables in each entry (title, author, reader gender, reader age, reader description), there are not enough total entries to do a highly comprehensive analysis on the results. As a result, I focused on displaying the information of interest in creative ways and highlighting surprising conclusions about readers patterns across cities, within New York, and across demographics. In the future, I hope to continue exploring data on how individuals read, what titles and authors are popular where, and eventually...what predictions can be made with this information?
About Author
