Using Data from Venmo to Map Social Networks
The skills the author demoed here can be learned through taking Data Science with Machine Learning bootcamp with NYC Data Science Academy.
Data shows Venmo has grown into a wildly popular P2P payment app. It is one of those rare companies whose product dominates its space so absolutely that its name is used as a verb colloquially: splitting a check with a friend? "I'll venmo you later"
A unique, sometimes controversial, feature of Venmo is its social media aspect. Every user has a feed showing off all of their public transactions. On top of being a source of FOMO, this is a great source of data! I wanted to see if I could use that data to map social networks. This sort of analysis can be extremely valuable for targeted advertising and marketing.
Data Gathering:
To gather my Venmo dataset I used Python, taking advantage of this unofficial Venmo API, Pandas, and Pickle.
I wrote a script thatgathers Venmo user and transaction data by following these steps:
- Starts with a list of Venmo users
- Adds their user data to a dataframe
- Pulls their transactions (default to last 50, but can be set to a date to scrape after) and adds them to a dataframe
- Creates a new list of users from those transactions
- Repeats on that new list of users
- Code terminates after going through these steps N times
To avoid duplicate entries, my code checks the existing data frames frequently. Due to the overlap between user transactions and the quickly growing sets of users (exponential! using the default 50 transactions per user the amount of users grows as much as 50n!) this method gets jammed up quickly. If I were to do this again, I'd consider a different method, maybe growing a set of user and transaction IDs first and then adding them all to a dataframe later (this would avoid searching through data frames over and over).
After running this script for a few hours I was able to gather 47,836 users and 13,7137 transactions. Checkout the scripts I used on my github.
Network Data Analysis
Now for the fun part! First the data needs to be organized as a graph. Each user is a node, and each transaction is an edge. I decided to define this data as an undirected multigraph. I consider a transaction between users a shared connection (therefore undirected), no matter which direction the money is going. It is a multigraph because if users pay each other more often (more edges between them) they should be more connected.
Pagerank Algorithm
To map the network I used a personalized pagerank algorithm. Pagerank was developed by Google and was their original algorithm used for ranking web-pages in their search engine. It works by calculating the probability a "random surfer" is at a given node at any time. The surfer randomly starts at any node and moves randomly along the edges. More connected nodes will therefore have a higher probability.
Personalized page-rank has the slight modification in that the surfer doesn't start at any node. Instead there is a given probability that they start at some subset of nodes. In this way it ranks the relevance of nodes given an initial set of nodes. In this case, it ranks Venmo users closely related to other Venmo users.
My code is below. I used networkx to do this analysis:
My Social Network
First, to test this idea, I used this method to rank nodes relative to me. Using this data and algorithm can I find my friends?
Graph of me (in green), my 6 closest friends (red), and the next 400 closest ranked nodes
My 6 closest friends, output by running the personalized pagerank algorithm on the entire Venmo transaction dataset personalized on my node, are:
| Venmo User | Real World Relationship |
| Rachel | Girlfriend |
| Eric | Friend |
| Anna | Roommate |
| Josh | Brother |
| Ian | Friend |
| Charles | Friend |
Qualitatively this worked really well. These are indeed 6 of my closest friends. Obviously there is a lot of bias in this analysis. For example, these are only friends on Venmo who publicize transactions etc., but, especially in populations who use Venmo often (mine), it seems to work.
Case Study: Uber and Lyft
Each Venmo transaction has a note. The note is written by one of the users and can be any short string. It is difficult to categorize the type of transaction using the note (I tried, check out my analysis in my github!). It is often just an irrelevant emoji. But occasionally there is a clear description.
One quirk of this dataset is that users often write Uber or Lyft explicitly in their Venmo notes. In fact, 1% of all of the transactions I scraped mentioned Uber explicitly, and 0.3% mentioned Lyft. So, using those notes, we have sets of users who we know use Uber and/or Lyft.
The question I want to examine is: what is the overlap between these user groups?
By just looking at the sets of users who explicitly use Uber and Lyft there is not a lot of overlap. 90% of users who have an Uber note do not have a Lyft note. But by extending these networks using the personalized pagerank can we see if there is more of an overlap?
The steps of the analysis are as follows:
- Create sets of Uber users and Lyft users
- Run personalized pagerank on both sets
- Look at the overlap of the 5000 highest ranked users in each set
Findings
After this analysis there was a 33% overlap between sets. So, even after extending the networks, there seems to be distinct Uber and Lyft users. If this knowledge were combined with more user data (i.e. demographic information of each user) it can be a powerful analysis showing the difference between the customer base of each ridesharing company.
Note that this analysis does have flaws. Even with tens of thousands of users, this is a very incomplete network of the total Venmo user-base and an even more incomplete network of the total social network in the world. So there are a lot of missing connections and nodes, which will have a huge impact on the analysis. But as a proof of concept it works well. Venmo's public transaction data is a rich source of information. It can be used to group consumers and learn about how they spend their money.
Conclusion:
Venmo's social media feeds are a rich source of data. They can be pretty easily scraped to gather large amounts of transaction data. That data can be used to map users’ networks and group consumers. With secondary sources, that analysis can be extended further to learn about different demographic's preferences.
If you have any questions or want to get in touch with me feel free to email me at: msender98@gmail.com !
And checkout my code at: https://github.com/Msender98/venmo_scrape
Thanks for reading!
About Author
