Web scraping- Insights of the Meetup.com groups in New York

Posted on Nov 21, 2016


Meetup.com is a social networking platform that facilitates offline groups in various locations, and members are brought together by common interests. Due to the success of the concept, there are large amount of the groups and events are held on the platform. However, how the new members choose which group or event they should join is the question to ask. How often does the group hosting events? What is the atmosphere of the group? Is it a large group that always hosts an event that has more than hundred people or is it a cozy small group of gathering? This project will provide an overview analysis about all the groups that based in New York and a groups sorting system.


Data Collecting 

The data was scraped from Meetup.com based on New York location, which includes all the groups’ information for all the categories. The total observation is 2327 groups. The variables that being collected are:

  • Group Name
  • Category name
  • Founded Date
  • Number of Members, Group Reviews
  • Location
  • Past Meetup Event Numbers
  • Upcoming Event Numbers
  • Most Recent Upcoming Event Name
  • Most Recent Upcoming Event Time
  • Most Recent Upcoming Event Date
  • Most Recent Upcoming Event’s Number of Participants
  • URL of the Groups



The tools that was using for scraping data were Selenium and Beautiful Soup in Python, Selenium provides the ability to scroll down to the bottom of each web page in order to collect all the groups’ URL. Beautiful Soup package is able to collect the data from each group page for the following analysis.

Before the Exploratory Data Analysis, all the data merging, cleaning was performed in R. The project was created in Shiny application. All the code can be found on GitHub.

Overview Statistics

The first tab gives a general overview statistics of the groups information by categories. Number of groups by location in each category graph indicates the ratio of groups’ number in different New York location. With no doubt New York, NY has the highest numbers of groups. By clicking off New York, NY in the legend, we can see that Brooklyn, NY is the second popular location.

Number of meetup in each category indicates that the total number of groups within each category. However, a lot of categories have 100 groups, and which is also the maximum number. There might be a possibility that Meetup.com is only showing 100 most activated groups in each category.

To have a better idea of the average group size of each category, Average member in each category provides the information. The graph shows that Social and Technology are the two category have the highest average number of group members.




Relationship between Number of members, Number of Past Events, and Number of Upcoming Event

The first scatter plot shows the relationship between number of members and numbers of past events of each category. The user can also select the categories that they want to compare by clicking the groups’ name in the legend. We can see that some of the categories have some positive relationship, but some of them do not.


The second scatter plot shows the relationship between numbers of members and numbers of upcoming events of each category. Similar as the first scatter plot, the graph shows that is there any positive relationship between members’ numbers and upcoming events numbers.


The third scatter plot is a 3D scatter plot, which allow the users to see the relationship between numbers of members, numbers of past events, and numbers of upcoming events at the same time.


The Popular Time

The Heat Map tab gives the information about the popular time slot of the weekdays by Number of members, Number of participants, and Average numbers of participant.

The heat map shows that the groups hosting events at Friday from 6pm – 9pm have most members in the groups. The upcoming events that hosting on Wednesday from 6pm-9pm have highest number people attending. However, the time slot that has highest average number of attending is Monday from 9pm – 12am.


Sorting System

Last but not least, the Select Your Group tab is a sorting system that allows users to pick the group that based their preference.

The users can select the groups based on location, weekday, range of members, range of the past event numbers, the range of upcoming event numbers.

The sorting system will provide the detail information of each group and the URL within the user’s parameters.



  1. New York, NY and Brooklyn, NY are the two most popular places in New York’s meetup groups.
  1. The categories that has highest average number of members are Social and Technology
  1. Some of the categories have positive relationship between numbers of past events and numbers of members, such as Social. However, some of them are not, such as Technology.
  1. Some of the categories have positive relationship between numbers of upcoming events and numbers of members, such as Outdoors & Adventure. However, some of them are not, such as Language & Culture.
  1. The groups hosting events at Friday from 6pm – 9pm have most members in the groups.
  1. The upcoming events that hosting on Wednesday from 6pm-9pm has highest number people attending.
  2. The upcoming events that hosting on Monday from 9pm – 12am has the highest average number of attending.

Future Steps

  • Predict the number of upcoming event by constructing a multiple linear regression.
  • Scrap the reviews of the groups, and turn the sorting system to a recommendation system.

About Author

Related Articles

Leave a Comment

TheWhiteningDepot UK December 18, 2017
There are five varieties of periodontitis, though a number of them tend to be common. One day, tooth will likely be eroded completely that only a few black part remains in the name of teeth. Now more adults are able to change their looks according changing fashion.
TheWhiteningDepot UK November 29, 2017
A good family dentist could also advise parents around the use of fluoride supplements, the most effective toothpaste to make use of, and the way to teach children to properly brush their teeth. A porcelain crown is reinforcement for any weakened tooth that's chipped, cracked or broken. Sometimes to leave dental surgeries or drilling of teeth they conceal pain or maybe avoid likely to dentist clinic or hospital.
copie bracelet love June 26, 2017
cartierlovejesduas Sounds like that Wells Fargo job is keeping you really happy, positive, open-minded, interested in fairness, valuing more than the almighty dollar, etc. copie bracelet love http://www.bestlevering.cn/fr/cheap-replica-cartier-love-white-gold-bracelet-paved-with-diamonds-p11/

View Posts by Categories

Our Recent Popular Posts

View Posts by Tags

#python #trainwithnycdsa 2019 airbnb Alex Baransky alumni Alumni Interview Alumni Reviews Alumni Spotlight alumni story Alumnus API Application artist aws beautiful soup Best Bootcamp Best Data Science 2019 Best Data Science Bootcamp Best Data Science Bootcamp 2020 Best Ranked Big Data Book Launch Book-Signing bootcamp Bootcamp Alumni Bootcamp Prep Bundles California Cancer Research capstone Career Career Day citibike clustering Coding Course Demo Course Report D3.js data Data Analyst data science Data Science Academy Data Science Bootcamp Data science jobs Data Science Reviews Data Scientist Data Scientist Jobs data visualization Deep Learning Demo Day Discount dplyr employer networking feature engineering Finance Financial Data Science Flask gbm Get Hired ggplot2 googleVis Hadoop higgs boson Hiring hiring partner events Hiring Partners Industry Experts Instructor Blog Instructor Interview Job Job Placement Jobs Jon Krohn JP Morgan Chase Kaggle Kickstarter lasso regression Lead Data Scienctist Lead Data Scientist leaflet linear regression Logistic Regression machine learning Maps matplotlib Medical Research Meet the team meetup Networking neural network Neural networks New Courses nlp NYC NYC Data Science nyc data science academy NYC Open Data NYCDSA NYCDSA Alumni Online Online Bootcamp Online Training Open Data painter pandas Part-time Portfolio Development prediction Prework Programming PwC python Python Data Analysis python machine learning python scrapy python web scraping python webscraping Python Workshop R R language R Programming R Shiny r studio R Visualization R Workshop R-bloggers random forest Ranking recommendation recommendation system regression Remote remote data science bootcamp Scrapy scrapy visualization seaborn Selenium sentiment analysis Shiny Shiny Dashboard Spark Special Special Summer Sports statistics streaming Student Interview Student Showcase SVM Switchup Tableau team TensorFlow Testimonial tf-idf Top Data Science Bootcamp twitter visualization web scraping Weekend Course What to expect word cloud word2vec XGBoost yelp