Using Data from Venmo to Map Social Networks

Posted on Aug 3, 2020
The skills the author demoed here can be learned through taking Data Science with Machine Learning bootcamp with NYC Data Science Academy.

Data shows Venmo has grown into a wildly popular P2P payment app.  It is one of those rare companies whose product dominates its space so absolutely that its name is used as a verb colloquially: splitting a check with a friend? "I'll venmo you later" 

A unique, sometimes controversial, feature of Venmo is its social media aspect. Every user has a feed showing off all of their public transactions. On top of being a source of FOMO, this is a great source of data! I wanted to see if I could use that data to map social networks. This sort of analysis can be extremely valuable for targeted advertising and marketing. 

Data Gathering:

To gather my Venmo dataset I used Python, taking advantage of this unofficial Venmo API, Pandas, and Pickle. 

I wrote a script thatgathers Venmo user and transaction data by following these steps: 

  1. Starts with a list of Venmo users
  2. Adds their user data to a dataframe
  3. Pulls their transactions (default to last 50, but can be set to a date to scrape after) and adds them to a dataframe
  4. Creates a new list of users from those transactions
  5. Repeats on that new list of users
  6. Code terminates after going through these steps N times

To avoid duplicate entries, my code checks the existing data frames frequently. Due to the  overlap between user transactions and the quickly growing sets of users (exponential! using the default 50 transactions per user the amount of users grows as much as 50n!) this method gets jammed up quickly.  If I were to do this again, I'd consider a different method, maybe growing a set of user and transaction IDs first and then adding them all to a dataframe later (this would avoid searching through data frames over and over). 

After running this script for a few hours I was able to gather 47,836 users and 13,7137 transactions.  Checkout the scripts I used on my github

Network Data Analysis

Now for the fun part! First the data needs to be organized as a graph. Each user is a node, and each transaction is an edge. I decided to define this data as an undirected multigraph. I consider a transaction between users a shared connection (therefore undirected), no matter which direction the money is going. It is a multigraph because if users pay each other more often (more edges between them) they should be more connected.  

Pagerank Algorithm

To map the network I used a personalized pagerank algorithm. Pagerank was developed by Google and was their original algorithm used for ranking web-pages in their search engine.  It works by calculating the probability a "random surfer" is at a given node at any time. The surfer randomly starts at any node and moves randomly along the edges. More connected nodes will therefore have a higher probability.

Personalized page-rank has the slight modification in that the surfer doesn't start at any node. Instead there is a given probability that they start at some subset of nodes. In this way it ranks the relevance of nodes given an initial set of nodes. In this case, it ranks Venmo users closely related to other Venmo users.  

My code is below. I used networkx to do this analysis: 

 

My Social Network

First, to test this idea, I used this method to rank nodes relative to me. Using this data and algorithm can I find my friends

Using Data from Venmo to Map Social Networks

Graph of me (in green), my 6 closest friends (red), and the next 400 closest ranked nodes

My 6 closest friends, output by running the personalized pagerank algorithm on the entire Venmo transaction dataset personalized on my node, are:

Venmo User Real World Relationship
Rachel Girlfriend
Eric Friend
Anna Roommate
Josh Brother
Ian Friend
Charles  Friend

Qualitatively this worked really well. These are indeed 6 of my closest friends. Obviously there is a lot of bias in this analysis. For example, these are only friends on Venmo who publicize transactions etc., but, especially in populations who use Venmo often (mine), it seems to work.

Case Study: Uber and Lyft

Each Venmo transaction has a note. The note is written by one of the users and can be any short string.  It is difficult to categorize the type of transaction using the note (I tried, check out my analysis in my github!). It is often just an irrelevant emoji. But occasionally there is a clear description. 

One quirk of this dataset is that users often write Uber or Lyft explicitly in their Venmo notes. In fact, 1% of all of the transactions I scraped mentioned Uber explicitly, and 0.3% mentioned Lyft.  So, using those notes, we have sets of users who we know use Uber and/or Lyft. 

The question I want to examine is: what is the overlap between these user groups? 

By just looking at the sets of users who explicitly use Uber and Lyft there is not a lot of overlap. 90% of users who have an Uber note do not have a Lyft note.  But by extending these networks using the personalized pagerank can we see if there is more of an overlap? 

The steps of the analysis are as follows: 

  1. Create sets of Uber users and Lyft users
  2. Run personalized pagerank on both sets
  3. Look at the overlap of the 5000 highest ranked users in each set

Findings

After this analysis there was a 33% overlap between sets. So, even after extending the networks, there seems to be distinct Uber and Lyft users.  If this knowledge were combined with more user data (i.e. demographic information of each user) it can be a powerful analysis showing the difference between the customer base of each ridesharing company. 

Note that this analysis does have flaws. Even with tens of thousands of users, this is a very incomplete network of the total Venmo user-base and an even more incomplete network of the total social network in the world. So there are a lot of missing connections and nodes, which will have a huge impact on the analysis. But as a proof of concept it works well. Venmo's public transaction data is a rich source of information. It can be used to group consumers and learn about how they spend their money. 

Conclusion:

Venmo's social media feeds are a rich source of data. They can be pretty easily scraped to gather large amounts of transaction data. That data can be used to map users’ networks and group consumers. With secondary sources, that analysis can be extended further to learn about different demographic's preferences. 

If you have any questions or want to get in touch with me feel free to email me at: [email protected] !

And checkout my code at: https://github.com/Msender98/venmo_scrape 

Thanks for reading!

About Author

Michael Sender

I have an MS in Applied Math from the University of Washington, and a BS in both Material Science and Math from Rutgers University. Before joining NYC Data Science Academy I worked for over 5 years as a...
View all posts by Michael Sender >

Related Articles

Leave a Comment

No comments found.

View Posts by Categories


Our Recent Popular Posts


View Posts by Tags

#python #trainwithnycdsa 2019 2020 Revenue 3-points agriculture air quality airbnb airline alcohol Alex Baransky algorithm alumni Alumni Interview Alumni Reviews Alumni Spotlight alumni story Alumnus ames dataset ames housing dataset apartment rent API Application artist aws bank loans beautiful soup Best Bootcamp Best Data Science 2019 Best Data Science Bootcamp Best Data Science Bootcamp 2020 Best Ranked Big Data Book Launch Book-Signing bootcamp Bootcamp Alumni Bootcamp Prep boston safety Bundles cake recipe California Cancer Research capstone car price Career Career Day citibike classic cars classpass clustering Coding Course Demo Course Report covid 19 credit credit card crime frequency crops D3.js data data analysis Data Analyst data analytics data for tripadvisor reviews data science Data Science Academy Data Science Bootcamp Data science jobs Data Science Reviews Data Scientist Data Scientist Jobs data visualization database Deep Learning Demo Day Discount disney dplyr drug data e-commerce economy employee employee burnout employer networking environment feature engineering Finance Financial Data Science fitness studio Flask flight delay gbm Get Hired ggplot2 googleVis H20 Hadoop hallmark holiday movie happiness healthcare frauds higgs boson Hiring hiring partner events Hiring Partners hotels housing housing data housing predictions housing price hy-vee Income Industry Experts Injuries Instructor Blog Instructor Interview insurance italki Job Job Placement Jobs Jon Krohn JP Morgan Chase Kaggle Kickstarter las vegas airport lasso regression Lead Data Scienctist Lead Data Scientist leaflet league linear regression Logistic Regression machine learning Maps market matplotlib Medical Research Meet the team meetup methal health miami beach movie music Napoli NBA netflix Networking neural network Neural networks New Courses NHL nlp NYC NYC Data Science nyc data science academy NYC Open Data nyc property NYCDSA NYCDSA Alumni Online Online Bootcamp Online Training Open Data painter pandas Part-time performance phoenix pollutants Portfolio Development precision measurement prediction Prework Programming public safety PwC python Python Data Analysis python machine learning python scrapy python web scraping python webscraping Python Workshop R R Data Analysis R language R Programming R Shiny r studio R Visualization R Workshop R-bloggers random forest Ranking recommendation recommendation system regression Remote remote data science bootcamp Scrapy scrapy visualization seaborn seafood type Selenium sentiment analysis sentiment classification Shiny Shiny Dashboard Spark Special Special Summer Sports statistics streaming Student Interview Student Showcase SVM Switchup Tableau teachers team team performance TensorFlow Testimonial tf-idf Top Data Science Bootcamp Top manufacturing companies Transfers tweets twitter videos visualization wallstreet wallstreetbets web scraping Weekend Course What to expect whiskey whiskeyadvocate wildfire word cloud word2vec XGBoost yelp youtube trending ZORI