Using Venmo to Map Social Networks

Michael Sender
Posted on Aug 3, 2020

Venmo has grown into a wildly popular P2P payment app.  It is one of those rare companies whose product dominates its space so absolutely that its name is used as a verb colloquially: splitting a check with a friend? "I'll venmo you later" 

A unique, sometimes controversial, feature of Venmo is its social media aspect. Every user has a feed showing off all of their public transactions. On top of being a source of FOMO, this is a great source of data! I wanted to see if I could use that data to map social networks. This sort of analysis can be extremely valuable for targeted advertising and marketing. 

Data Gathering:

To gather my Venmo dataset I used Python, taking advantage of this unofficial Venmo API, Pandas, and Pickle. 

I wrote a script thatgathers Venmo user and transaction data by following these steps: 

  1. Starts with a list of Venmo users
  2. Adds their user data to a dataframe
  3. Pulls their transactions (default to last 50, but can be set to a date to scrape after) and adds them to a dataframe
  4. Creates a new list of users from those transactions
  5. Repeats on that new list of users
  6. Code terminates after going through these steps N times

To avoid duplicate entries, my code checks the existing data frames frequently. Due to the  overlap between user transactions and the quickly growing sets of users (exponential! using the default 50 transactions per user the amount of users grows as much as 50n!) this method gets jammed up quickly.  If I were to do this again, I'd consider a different method, maybe growing a set of user and transaction IDs first and then adding them all to a dataframe later (this would avoid searching through data frames over and over). 

After running this script for a few hours I was able to gather 47,836 users and 13,7137 transactions.  Checkout the scripts I used on my github

Network Analysis

Now for the fun part! First the data needs to be organized as a graph. Each user is a node, and each transaction is an edge. I decided to define this data as an undirected multigraph. I consider a transaction between users a shared connection (therefore undirected), no matter which direction the money is going. It is a multigraph because if users pay each other more often (more edges between them) they should be more connected.  

To map the network I used a personalized pagerank algorithm. Pagerank was developed by Google and was their original algorithm used for ranking web-pages in their search engine.  It works by calculating the probability a "random surfer" is at a given node at any time. The surfer randomly starts at any node and moves randomly along the edges. More connected nodes will therefore have a higher probability. Personalized page-rank has the slight modification in that the surfer doesn't start at any node. Instead there is a given probability that they start at some subset of nodes. In this way it ranks the relevance of nodes given an initial set of nodes. In this case, it ranks Venmo users closely related to other Venmo users.  

My code is below. I used networkx to do this analysis: 

https://gist.github.com/Msender98/1a0b422d9fdb421a0800be99a3b9a045

 

My Social Network

First, to test this idea, I used this method to rank nodes relative to me. Using this data and algorithm can I find my friends

Graph of me (in green), my 6 closest friends (red), and the next 400 closest ranked nodes

My 6 closest friends, output by running the personalized pagerank algorithm on the entire Venmo transaction dataset personalized on my node, are:

Venmo User Real World Relationship
Rachel Girlfriend
Eric Friend
Anna Roommate
Josh Brother
Ian Friend
Charles  Friend

Qualitatively this worked really well. These are indeed 6 of my closest friends. Obviously there is a lot of bias in this analysis. For example, these are only friends on Venmo who publicize transactions etc., but, especially in populations who use Venmo often (mine), it seems to work.

Case Study: Uber and Lyft

Each Venmo transaction has a note. The note is written by one of the users and can be any short string.  It is difficult to categorize the type of transaction using the note (I tried, check out my analysis in my github!). It is often just an irrelevant emoji. But occasionally there is a clear description. 

One quirk of this dataset is that users often write Uber or Lyft explicitly in their Venmo notes. In fact, 1% of all of the transactions I scraped mentioned Uber explicitly, and 0.3% mentioned Lyft.  So, using those notes, we have sets of users who we know use Uber and/or Lyft. 

The question I want to examine is: what is the overlap between these user groups? 

By just looking at the sets of users who explicitly use Uber and Lyft there is not a lot of overlap. 90% of users who have an Uber note do not have a Lyft note.  But by extending these networks using the personalized pagerank can we see if there is more of an overlap? 

The steps of the analysis are as follows: 

  1. Create sets of Uber users and Lyft users
  2. Run personalized pagerank on both sets
  3. Look at the overlap of the 5000 highest ranked users in each set

After this analysis there was a 33% overlap between sets. So, even after extending the networks, there seems to be distinct Uber and Lyft users.  If this knowledge were combined with more user data (i.e. demographic information of each user) it can be a powerful analysis showing the difference between the customer base of each ridesharing company. 

Note that this analysis does have flaws. Even with tens of thousands of users, this is a very incomplete network of the total Venmo user-base and an even more incomplete network of the total social network in the world. So there are a lot of missing connections and nodes, which will have a huge impact on the analysis. But as a proof of concept it works well. Venmo's public transaction data is a rich source of information. It can be used to group consumers and learn about how they spend their money. 

Conclusion:

Venmo's social media feeds are a rich source of data. They can be pretty easily scraped to gather large amounts of transaction data. That data can be used to map users’ networks and group consumers. With secondary sources, that analysis can be extended further to learn about different demographic's preferences. 

If you have any questions or want to get in touch with me feel free to email me at: [email protected] !

And checkout my code at: https://github.com/Msender98/venmo_scrape 

Thanks for reading!

About Author

Michael Sender

Michael Sender

I have an MS in Applied Math from the University of Washington, and a BS in both Material Science and Math from Rutgers University. Before joining NYC Data Science Academy I worked for over 5 years as a...
View all posts by Michael Sender >

Related Articles

Leave a Comment

No comments found.

View Posts by Categories


Our Recent Popular Posts


View Posts by Tags

#python #trainwithnycdsa 2019 airbnb Alex Baransky alumni Alumni Interview Alumni Reviews Alumni Spotlight alumni story Alumnus API Application artist aws beautiful soup Best Bootcamp Best Data Science 2019 Best Data Science Bootcamp Best Data Science Bootcamp 2020 Best Ranked Big Data Book Launch Book-Signing bootcamp Bootcamp Alumni Bootcamp Prep Bundles California Cancer Research capstone Career Career Day citibike clustering Coding Course Demo Course Report D3.js data Data Analyst data science Data Science Academy Data Science Bootcamp Data science jobs Data Science Reviews Data Scientist Data Scientist Jobs data visualization Deep Learning Demo Day Discount dplyr employer networking feature engineering Finance Financial Data Science Flask gbm Get Hired ggplot2 googleVis Hadoop higgs boson Hiring hiring partner events Hiring Partners Industry Experts Instructor Blog Instructor Interview Job Job Placement Jobs Jon Krohn JP Morgan Chase Kaggle Kickstarter lasso regression Lead Data Scienctist Lead Data Scientist leaflet linear regression Logistic Regression machine learning Maps matplotlib Medical Research Meet the team meetup Networking neural network Neural networks New Courses nlp NYC NYC Data Science nyc data science academy NYC Open Data NYCDSA NYCDSA Alumni Online Online Bootcamp Open Data painter pandas Part-time Portfolio Development prediction Prework Programming PwC python python machine learning python scrapy python web scraping python webscraping Python Workshop R R language R Programming R Shiny r studio R Visualization R Workshop R-bloggers random forest Ranking recommendation recommendation system regression Remote remote data science bootcamp Scrapy scrapy visualization seaborn Selenium sentiment analysis Shiny Shiny Dashboard Spark Special Special Summer Sports statistics streaming Student Interview Student Showcase SVM Switchup Tableau team TensorFlow Testimonial tf-idf Top Data Science Bootcamp twitter visualization web scraping Weekend Course What to expect word cloud word2vec XGBoost yelp