Scraping Meetup.com To Build A Group Recommender App
A couple blogs ago, I mentioned my fascination over the site Meetup.com where users can find communities whose members all share a common interest. Its a great way to build a professional network, make a few friends or just learn from passionate and smart people. There are tons of groups to choose from and joining one may not be enough! This led me to build an app that recommends Meetup groups based on ones the app user has enjoyed before. Link to the app is here!
Since I am using scraped data to build this recommendation system, I've decided to focus on tech-related groups within the NYC area (so as to shorten the amount of scraping time). However I'd like to point out these concepts can be applied for all types of Meetup groups.
The project consisted of 3 phases:
- Scrape tech-group pages of Meetup.com using Selenium
- Apply apriori and association rule learning to group topics
- Build an interactive app in Shiny R
WEB SCRAPING WITH SELENIUM
My tool choice for scraping was Selenium. Using a python script, the code was simple, elegant and super intuitive. It uses a web driver to navigate to the pages we want to scrape. Then, by using XPath queries, I parsed out specific details about each tech-group that I wanted as my data. The starting page (https://www.meetup.com/find/tech) to scrape looks like this:
Each box is a link to a different group. The bottom of the page has a "Show more' button that when clicked, more groups are loaded. The strategy that was implemented in my scraping script was to keep clicking on the "Show more" button until all the groups have been loaded. I then parsed out the href link from each box that leads to their group page and stored that as a list. One by one, the web driver will go to each of these group pages and scrape the data I need. The most important data I needed for the recommendation system were the topics listed for that group. But in the end, I've scraped everything in the below image into a dictionary that was eventually written into a csv file:
APRIORI AND ASSOCIATION RULE LEARNING
Most people think of market basket analysis when they hear association rule learning. In the scenario of market basket analysis, each purchase at say a supermarket is a transaction, and each of these transactions will contain either one or more grocery items. Association rules mine what items will likely be bought given that a particular item was also bought.
I applied the same concept to my recommendation system. Only instead of purchases, EACH meetup group is a particular "transaction". And instead of grocery items, I am dealing with the topics a Meetup group can have. For example "Big Data","Cryptocurrency", "Machine Learning" etc. A Meetup group can have one or many topics associated with it. To put it simply, I am mining what topics are most likely to be associated with each other. I did this in R studio using the 'arules' package. Below is a graph of the actual rules that have been mined with the scraped data.
The general idea of the application can be explained with a simple example. Say I joined the group "Cryptocurrency Mondays NYC". There are 10 topics associated with this group: "Virtual Currency", "Decentralization", "Bitcoin" and etc ... My application will then search the LHS (a.k.a. the antecedents) in the rules that it has mined to find any matches with any of Cryptocurrency Monday's 10 topics. It will then find the topic on the RHS (a.k.a. the consequent) that is MOST likely to appear given the antecedents (the one with the highest lift).
Continuing with our example, we can see that if a group has the topics "Bitcoin", "Blockchain Development", "Digital Currency", and "Ethereum" (all are topics under Cryptocurrency Mondays), then the topic "Ethereum blockchain" is 34 times more likely to appear than expected. The assumption here is that if that member likes the group Cryptocurrency Mondays because of the topics "Bitcoin", "Ethereum" and etc., than this member would probably like a group that has the topic "Ethereum blockchain" as well. The app would then proceed to find a group that lists "Ethereum blockchain" as one of their topics.
Before association rules are mined, apriori rules must be set. I didn't mind setting the support threshold to 1% as the strongest recommendations will be provided first anyway so better to as much data as we can. I set the confidence threshold arbitrarily to 50% as I felt comfortable with the strength of the association.
The great thing about using these association rules is that it isn't looking for groups with exactly that same topics as another group. It is recommending groups with topics that a member is likely to like based on the topics we know they already like. This allows for greater flexibility.
INTERACTIVE R SHINY APP
The result of putting all this into an interactive app on R Shiny looks like this! :
The left panel contains a drop down menu allowing the user to choose a group that they've enjoyed in the past. Once chosen, the right panel will automatically populate with the name of the Meetup group recommended, a group description and topics listed under that group as well. Looks like if a user likes CryptoMondays NYC, this user might also like Blockchain NYC! Notice that "Ethereum blockchain" is listed as one of the topics for Blockchain NYC.
If the user would like another recommendation, he/she can click the button labeled "Recommend Me Another". The right side would then re-populate with another group!
CONCLUSION
I hope anyone who uses this app find themselves reaching out to more communities. The UI is not the prettiest as UI wasn't my top prioirity :/ but given some spare time in the future, I'll go ahead and make this more fun and entertaining to play with. And although the app focuses in on Tech groups, the concept can easily be scaled to reach many more categories.
About Author
