CoverSpy: Adventures Scraping Twitter and Tumblr

Posted on Aug 7, 2017

As an avid reader and a nosy onlooker, my goal for this project was to answer the following questions: What are Americans reading across different cities? How greatly do reading choices differ by city and even by local environment? Can demographic and personal information on the readers be identified?
Although libraries, bookstores, and online retailers release general information on their most popular books, this project utilized a new and unique data source: Coverspy. Coverspy is a twitter page that encourages average citizens to become subway voyeurs and report what those around them are reading. A typical post includes the title of a book, author, and description of the reader (including location). Coverspy is an interesting source of information not only because it displays a snapshot of the popular authors and books at a period of time, but it is highly location-specific and descriptive. Because of the popularity of the New York City Coverspy page, twitter accounts have opened for other cities, including Boston, Chicago, Washington DC, and Barcelona.

In order to obtain the desired information from Coverspy, I used the Selenium package in Python to obtain the following information from the Coverspy pages for New York City, Boston, and Chicago: Book title, Author, Timestamp (later discarded) and Description. The github link below contains further code used for the project:
The greatest limitation with the Coverspy data was that there were not very many observations to work with. For each city, there were roughly 800 entries, some of which were incomplete (for example, the viewer did not see the book title, only the author). Nevertheless, the results obtained were interesting and provided preliminary answers to the questions posed at the beginning of the project. As more entries are added to the Coverspy page, it will become a more interesting page to scrape.


My first goal was to compare the most popular book titles and authors on the subways/metros of different cities. Acknowledging relatively few entries for each city, the results suggest that Americans in New York, Boston, and Chicago read quite differently. For example, the most popular author in identified by Coverspy New York was Zadie Smith- yet, she did not rank in the top 12 in either Boston or Chicago.

Top Authors in New York Β  Top Authors in Boston Β  Top Authors in Chicago
Rank Author Views Β  Β Rank Author Views Β  Β Rank Author Views
1 Zadie Smith 12 1 Gillian Flynn 8 1 Erik Larson 10
2 J.K. Rowling 11 2 J. K. Rowling 7 2 Anthony Doerr 6
3 Neil Gaiman 11 3 Cheryl Strayed 6 3 George R. R. Martin 6
4 Haruki Murakami 10 4 Chimamanda Ngozi Adichie 5 4 Gillian Flynn 6
5 Margaret Atwood 10 5 George R. R. Martin 5 5 Michael Chabon 6
6 Stephen King 10 6 Jojo Moyes 5 6 Stephen King 6
7 Chimamanda Ngozi Adichie 9 7 Diana Gabaldon 4 7 Chimamanda Ngozi Adichie 4
8 Paul Beatty 9 8 Donna Tartt 4 8 David Foster Wallace 4
9 George Orwell 7 9 Haruki Murakami 4 9 Haruki Murakami 4
10 Hanya Yanagihara 6 10 Lee Child 4 10 Liane Moriarty 4
11 Elena Ferrante 5 11 Malcolm Gladwell 4 11 Michael Crichton 4
12 Liane Moriarty 5 12 Stephen King 4 12 Michael Lewis 4

After examining title and authors breakdowns by city, I pooled the results for different cities and compared reading patterns for men and women. Although there were far more data entries for women than men (1054 observations of women versus 688 observations of men), the most interesting result turned up in the list of top authors read by men:


Men: Top Authors
Author Views
1 Stephen King 12
2 Gillian Flynn 7
3 Neil Gaiman 7
4 George Orwell 6
5 George R. R. Martin 6
6 David Foster Wallace 5
7 Haruki Murakami 5
8 Malcolm Gladwell 5
9 Cormac McCarthy 4
10 Erik Larson 4
11 Hanya Yanagihara 4
12 Michael Lewis 4
13 Andy Weir 3
14 Ayn Rand 3
15 Bill Bryson 3
16 Dan Simmons 3
17 Daniel Kahneman 3
18 David Mitchell 3
19 Frank Herbert 3
20 Fyodor Dostoevsky 3


Interestingly, while the top authors read by women were spread evenly across genres and gender, male readers tended to heavily favor male authors (17 of the top 20 authors for male readers are also male). Although further research would need to be done, this data heavily suggests that men tend to read very differently from women, perhaps more narrowly.

To further investigate reader demographics, I produced a histogram of observations by age on the New York City subway. I found that the vast majority of observed readers are in their twenties, with counts decreasing steadily by age.

Finally, I decided to use the reader descriptions to create word clouds based on location. Word clouds are plots based on word frequency: in this case, the most frequent words in the reader descriptions appear as the largest, most centrally-located words. I sorted the descriptions by subway train to see if the readers of New York displayed identifiable "personalities" based on where they were reading. Though this last visualization is highly subjective and mostly for entertainment, I would argue that the readers on the F train (which goes back and forth to Brooklyn) stand out in their description.


On a broad level, this project taught me the importance of working with a large dataset. Although Coverspy is a novel and fun website and it allows viewers to look at many variables in each entry (title, author, reader gender, reader age, reader description), there are not enough total entries to do a highly comprehensive analysis on the results. As a result, I focused on displaying the information of interest in creative ways and highlighting surprising conclusions about readers patterns across cities, within New York, and across demographics. In the future, I hope to continue exploring data on how individuals read, what titles and authors are popular where, and eventually...what predictions can be made with this information?

About Author

Katie Critelli

Katie graduated from the University of Pennsylvania with a Bachelor's degree in Neuroscience and an Honor's thesis focused on protein-modeling in neurodegenerative diseases. She worked previously at Booz Allen Hamilton in the military healthcare division. Katie has joined...
View all posts by Katie Critelli >

Related Articles

Leave a Comment

No comments found.

View Posts by Categories

Our Recent Popular Posts

View Posts by Tags

#python #trainwithnycdsa 2019 2020 Revenue 3-points agriculture air quality airbnb airline alcohol Alex Baransky algorithm alumni Alumni Interview Alumni Reviews Alumni Spotlight alumni story Alumnus ames dataset ames housing dataset apartment rent API Application artist aws bank loans beautiful soup Best Bootcamp Best Data Science 2019 Best Data Science Bootcamp Best Data Science Bootcamp 2020 Best Ranked Big Data Book Launch Book-Signing bootcamp Bootcamp Alumni Bootcamp Prep boston safety Bundles cake recipe California Cancer Research capstone car price Career Career Day citibike classic cars classpass clustering Coding Course Demo Course Report covid 19 credit credit card crime frequency crops D3.js data data analysis Data Analyst data analytics data for tripadvisor reviews data science Data Science Academy Data Science Bootcamp Data science jobs Data Science Reviews Data Scientist Data Scientist Jobs data visualization database Deep Learning Demo Day Discount disney dplyr drug data e-commerce economy employee employee burnout employer networking environment feature engineering Finance Financial Data Science fitness studio Flask flight delay gbm Get Hired ggplot2 googleVis H20 Hadoop hallmark holiday movie happiness healthcare frauds higgs boson Hiring hiring partner events Hiring Partners hotels housing housing data housing predictions housing price hy-vee Income Industry Experts Injuries Instructor Blog Instructor Interview insurance italki Job Job Placement Jobs Jon Krohn JP Morgan Chase Kaggle Kickstarter las vegas airport lasso regression Lead Data Scienctist Lead Data Scientist leaflet league linear regression Logistic Regression machine learning Maps market matplotlib Medical Research Meet the team meetup methal health miami beach movie music Napoli NBA netflix Networking neural network Neural networks New Courses NHL nlp NYC NYC Data Science nyc data science academy NYC Open Data nyc property NYCDSA NYCDSA Alumni Online Online Bootcamp Online Training Open Data painter pandas Part-time performance phoenix pollutants Portfolio Development precision measurement prediction Prework Programming public safety PwC python Python Data Analysis python machine learning python scrapy python web scraping python webscraping Python Workshop R R Data Analysis R language R Programming R Shiny r studio R Visualization R Workshop R-bloggers random forest Ranking recommendation recommendation system regression Remote remote data science bootcamp Scrapy scrapy visualization seaborn seafood type Selenium sentiment analysis sentiment classification Shiny Shiny Dashboard Spark Special Special Summer Sports statistics streaming Student Interview Student Showcase SVM Switchup Tableau teachers team team performance TensorFlow Testimonial tf-idf Top Data Science Bootcamp Top manufacturing companies Transfers tweets twitter videos visualization wallstreet wallstreetbets web scraping Weekend Course What to expect whiskey whiskeyadvocate wildfire word cloud word2vec XGBoost yelp youtube trending ZORI