Scraping To Build A Group Recommender App

Posted on Apr 17, 2018

A couple blogs ago, I mentioned my fascination over the site where users can find communities whose members all share a common interest. Its a great way to build a professional network, make a few friends or just learn from passionate and smart people. There are tons of groups to choose from and joining one may not be enough! This led me to build an app that recommends Meetup groups based on ones the app user has enjoyed before. Link to the app is here!

Since I am using scraped data to build this recommendation system, I've decided to focus on tech-related groups within the NYC area (so as to shorten the amount of scraping time). However I'd like to point out these concepts can be applied for all types of Meetup groups.

The project consisted of 3 phases:

  1. Scrape tech-group pages of using Selenium
  2. Apply apriori and association rule learning to group topics
  3. Build an interactive app in Shiny R



My tool choice for scraping was Selenium. Using a python script, the code was simple, elegant and super intuitive. It uses a web driver to navigate to the pages we want to scrape. Then, by using XPath queries, I parsed out specific details about each tech-group that I wanted as my data. The starting page ( to scrape looks like this:

Each box is a link to a different group. The bottom of the page has a "Show more' button that when clicked, more groups are loaded. The strategy that was implemented in my scraping script was to keep clicking on the "Show more" button until all the groups have been loaded. I then parsed out the href link from each box that leads to their group page and stored that as a list. One by one, the web driver will go to each of these group pages and scrape the data I need. The most important data I needed for the recommendation system were the topics listed for that group. But in the end, I've scraped everything in the below image into a dictionary that was eventually written into a csv file:



Most people think of market basket analysis when they hear association rule learning. In the scenario of market basket analysis, each purchase at say a supermarket is a transaction, and each of these transactions will contain either one or more grocery items. Association rules mine what items will likely be bought given that a particular item was also bought.

I applied the same concept to my recommendation system. Only instead of purchases, EACH meetup group is a particular "transaction". And instead of grocery items, I am dealing withΒ the topicsΒ  a Meetup group can have. For example "Big Data","Cryptocurrency", "Machine Learning" etc. A Meetup group can have one or many topics associated with it.Β  To put it simply, I am mining what topics are most likely to be associated with each other. I did this in R studio using the 'arules' package. Below is a graph of the actual rules that have been mined with the scraped data.

The general idea of the application can be explained with a simple example. Say I joined the group "Cryptocurrency Mondays NYC". There are 10 topics associated with this group: "Virtual Currency", "Decentralization", "Bitcoin" and etc ... My application will then search the LHS (a.k.a. the antecedents) in the rules that it has mined to find any matches with any of Cryptocurrency Monday's 10 topics. It will then find the topic on the RHS (a.k.a. the consequent) that is MOST likely to appear given the antecedents (the one with the highest lift).

Continuing with our example, we can see that if a group has the topics "Bitcoin", "Blockchain Development", "Digital Currency", and "Ethereum" (all are topics under Cryptocurrency Mondays), then the topic "Ethereum blockchain" is 34 times more likely to appear than expected. The assumption here is that if that member likes the group Cryptocurrency Mondays because of the topics "Bitcoin", "Ethereum" and etc., than this member would probably like a group that has the topic "Ethereum blockchain" as well. The app would then proceed to find a group that lists "Ethereum blockchain" as one of their topics.

Before association rules are mined, apriori rules must be set. I didn't mind setting the support threshold to 1% as the strongest recommendations will be provided first anyway so better to as much data as we can. I set the confidence threshold arbitrarily to 50% as I felt comfortable with the strength of the association.

The great thing about using these association rules is that it isn't looking for groups with exactly that same topics as another group. It is recommending groups with topics that a member is likely to like based on the topics we know they already like. This allows for greater flexibility.



The result of putting all this into an interactive app on R Shiny looks like this! :

The left panel contains a drop down menu allowing the user to choose a group that they've enjoyed in the past. Once chosen, the right panel will automatically populate with the name of the Meetup group recommended, a group description and topics listed under that group as well. Looks like if a user likes CryptoMondays NYC, this user might also like Blockchain NYC! Notice that "Ethereum blockchain" is listed as one of the topics for Blockchain NYC.

If the user would like another recommendation, he/she can click the button labeled "Recommend Me Another". The right side would then re-populate with another group!



I hope anyone who uses this app find themselves reaching out to more communities. The UI is not the prettiest as UI wasn't my top prioirity :/ but given some spare time in the future, I'll go ahead and make this more fun and entertaining to play with. And although the app focuses in on Tech groups, the concept can easily be scaled to reach many more categories.

About Author

Kenny Moy

Kenny has years of experience providing data driven solutions in industries such as marketing, healthcare, real estate, and public service. In addition to machine learning, he loves the AHA! moments, storytelling, and the creativity aspects of data science.
View all posts by Kenny Moy >

Related Articles

Leave a Comment

No comments found.

View Posts by Categories

Our Recent Popular Posts

View Posts by Tags

#python #trainwithnycdsa 2019 2020 Revenue 3-points agriculture air quality airbnb airline alcohol Alex Baransky algorithm alumni Alumni Interview Alumni Reviews Alumni Spotlight alumni story Alumnus ames dataset ames housing dataset apartment rent API Application artist aws bank loans beautiful soup Best Bootcamp Best Data Science 2019 Best Data Science Bootcamp Best Data Science Bootcamp 2020 Best Ranked Big Data Book Launch Book-Signing bootcamp Bootcamp Alumni Bootcamp Prep boston safety Bundles cake recipe California Cancer Research capstone car price Career Career Day citibike classic cars classpass clustering Coding Course Demo Course Report covid 19 credit credit card crime frequency crops D3.js data data analysis Data Analyst data analytics data for tripadvisor reviews data science Data Science Academy Data Science Bootcamp Data science jobs Data Science Reviews Data Scientist Data Scientist Jobs data visualization database Deep Learning Demo Day Discount disney dplyr drug data e-commerce economy employee employee burnout employer networking environment feature engineering Finance Financial Data Science fitness studio Flask flight delay gbm Get Hired ggplot2 googleVis H20 Hadoop hallmark holiday movie happiness healthcare frauds higgs boson Hiring hiring partner events Hiring Partners hotels housing housing data housing predictions housing price hy-vee Income Industry Experts Injuries Instructor Blog Instructor Interview insurance italki Job Job Placement Jobs Jon Krohn JP Morgan Chase Kaggle Kickstarter las vegas airport lasso regression Lead Data Scienctist Lead Data Scientist leaflet league linear regression Logistic Regression machine learning Maps market matplotlib Medical Research Meet the team meetup methal health miami beach movie music Napoli NBA netflix Networking neural network Neural networks New Courses NHL nlp NYC NYC Data Science nyc data science academy NYC Open Data nyc property NYCDSA NYCDSA Alumni Online Online Bootcamp Online Training Open Data painter pandas Part-time performance phoenix pollutants Portfolio Development precision measurement prediction Prework Programming public safety PwC python Python Data Analysis python machine learning python scrapy python web scraping python webscraping Python Workshop R R Data Analysis R language R Programming R Shiny r studio R Visualization R Workshop R-bloggers random forest Ranking recommendation recommendation system regression Remote remote data science bootcamp Scrapy scrapy visualization seaborn seafood type Selenium sentiment analysis sentiment classification Shiny Shiny Dashboard Spark Special Special Summer Sports statistics streaming Student Interview Student Showcase SVM Switchup Tableau teachers team team performance TensorFlow Testimonial tf-idf Top Data Science Bootcamp Top manufacturing companies Transfers tweets twitter videos visualization wallstreet wallstreetbets web scraping Weekend Course What to expect whiskey whiskeyadvocate wildfire word cloud word2vec XGBoost yelp youtube trending ZORI