Data Web scraping- groups in New York

Posted on Nov 21, 2016
The skills we demoed here can be learned through taking Data Science with Machine Learning bootcamp with NYC Data Science Academy.

Motivation is a social networking platform that facilitates offline groups in various locations, and members are brought together by common interests. Due to the success of the concept, data shows there are large amount of the groups and events are held on the platform.

However, how the new members choose which group or event they should join is the question to ask. How often does the group hosting events? What is the atmosphere of the group? Is it a large group that always hosts an event that has more than hundred people or is it a cozy small group of gathering? This project will provide an overview data analysis about all the groups that based in New York and a groups sorting system.


Data Collecting 

The data was scraped from based on New York location, which includes all the groups’ information for all the categories. The total observation is 2327 groups. The variables that being collected are:

  • Group Name
  • Category name
  • Founded Date
  • Number of Members, Group Reviews
  • Location
  • Past Meetup Event Numbers
  • Upcoming Event Numbers
  • Most Recent Upcoming Event Name
  • Most Recent Upcoming Event Time
  • Highest Amount of Recent Upcoming Event Date
  • Most Recent Upcoming Event’s Number of Participants
  • URL of the Groups



The tools that was using for scraping data were Selenium and Beautiful Soup in Python, Selenium provides the ability to scroll down to the bottom of each web page in order to collect all the groups’ URL. Beautiful Soup package is able to collect the data from each group page for the following analysis.

Before the Exploratory Data Analysis, all the data merging, cleaning was performed in R. The project was created in Shiny application. All the code can be found on GitHub.

Overview Data Statistics

The first tab gives a general overview statistics of the groups information by categories. Number of groups by location in each category graph indicates the ratio of groups’ number in different New York location. With no doubt New York, NY has the highest numbers of groups. By clicking off New York, NY in the legend, we can see that Brooklyn, NY is the second popular location.

Number of meetup in each category indicates that the total number of groups within each category. However, a lot of categories have 100 groups, and which is also the maximum number. There might be a possibility that is only showing 100 most activated groups in each category.

To have a better idea of the average group size of each category, Average member in each category provides the information. The graph shows that Social and Technology are the two category have the highest average number of group members.

Data Web scraping- groups in New York

Data Web scraping- groups in New York


Relationship between Number of members, Number of Past Events, and Number of Upcoming Event

The first scatter plot shows the relationship between number of members and numbers of past events of each category. The user can also select the categories that they want to compare by clicking the groups’ name in the legend. We can see that some of the categories have some positive relationship, but some of them do not.

Data Web scraping- groups in New York

Relationship Between Number of Members and Upcoming Events

The second scatter plot shows the relationship between numbers of members and numbers of upcoming events of each category. Similar as the first scatter plot, the graph shows that is there any positive relationship between members’ numbers and upcoming events numbers.

Data Web scraping- groups in New York

Scatter Plot on Number of Members, Past Events, and Upcoming Events

The third scatter plot is a 3D scatter plot, which allow the users to see the relationship between numbers of members, numbers of past events, and numbers of upcoming events at the same time.


The Popular Time

The Heat Map tab gives the information about the popular time slot of the weekdays by Number of members, Number of participants, and Average numbers of participant.

The heat map shows that the groups hosting events at Friday from 6pm – 9pm have most members in the groups. The upcoming events that hosting on Wednesday from 6pm-9pm have highest number people attending. However, the time slot that has highest average number of attending is Monday from 9pm – 12am.


Sorting System

Last but not least, the Select Your Group tab is a sorting system that allows users to pick the group that based their preference.

The users can select the groups based on location, weekday, range of members, range of the past event numbers, the range of upcoming event numbers.

The sorting system will provide the detail information of each group and the URL within the user’s parameters.



  1. New York, NY and Brooklyn, NY are the two most popular places in New York’s meetup groups.
  1. The categories that has highest average number of members are Social and Technology
  1. Some of the categories have positive relationship between numbers of past events and numbers of members, such as Social. However, some of them are not, such as Technology.
  1. Some of the categories have positive relationship between numbers of upcoming events and numbers of members, such as Outdoors & Adventure. However, some of them are not, such as Language & Culture.
  1. The groups hosting events at Friday from 6pm – 9pm have most members in the groups.
  1. The upcoming events that hosting on Wednesday from 6pm-9pm has highest number people attending.
  2. The upcoming events that hosting on Monday from 9pm – 12am has the highest average number of attending.

Future Steps

  • Predict the number of upcoming event by constructing a multiple linear regression.
  • Scrap the reviews of the groups, and turn the sorting system to a recommendation system.

About Author

Related Articles

Leave a Comment

TheWhiteningDepot UK December 18, 2017
There are five varieties of periodontitis, though a number of them tend to be common. One day, tooth will likely be eroded completely that only a few black part remains in the name of teeth. Now more adults are able to change their looks according changing fashion.
TheWhiteningDepot UK November 29, 2017
A good family dentist could also advise parents around the use of fluoride supplements, the most effective toothpaste to make use of, and the way to teach children to properly brush their teeth. A porcelain crown is reinforcement for any weakened tooth that's chipped, cracked or broken. Sometimes to leave dental surgeries or drilling of teeth they conceal pain or maybe avoid likely to dentist clinic or hospital.
copie bracelet love June 26, 2017
cartierlovejesduas Sounds like that Wells Fargo job is keeping you really happy, positive, open-minded, interested in fairness, valuing more than the almighty dollar, etc. copie bracelet love

View Posts by Categories

Our Recent Popular Posts

View Posts by Tags

#python #trainwithnycdsa 2019 2020 Revenue 3-points agriculture air quality airbnb airline alcohol Alex Baransky algorithm alumni Alumni Interview Alumni Reviews Alumni Spotlight alumni story Alumnus ames dataset ames housing dataset apartment rent API Application artist aws bank loans beautiful soup Best Bootcamp Best Data Science 2019 Best Data Science Bootcamp Best Data Science Bootcamp 2020 Best Ranked Big Data Book Launch Book-Signing bootcamp Bootcamp Alumni Bootcamp Prep boston safety Bundles cake recipe California Cancer Research capstone car price Career Career Day citibike classic cars classpass clustering Coding Course Demo Course Report covid 19 credit credit card crime frequency crops D3.js data data analysis Data Analyst data analytics data for tripadvisor reviews data science Data Science Academy Data Science Bootcamp Data science jobs Data Science Reviews Data Scientist Data Scientist Jobs data visualization database Deep Learning Demo Day Discount disney dplyr drug data e-commerce economy employee employee burnout employer networking environment feature engineering Finance Financial Data Science fitness studio Flask flight delay gbm Get Hired ggplot2 googleVis H20 Hadoop hallmark holiday movie happiness healthcare frauds higgs boson Hiring hiring partner events Hiring Partners hotels housing housing data housing predictions housing price hy-vee Income Industry Experts Injuries Instructor Blog Instructor Interview insurance italki Job Job Placement Jobs Jon Krohn JP Morgan Chase Kaggle Kickstarter las vegas airport lasso regression Lead Data Scienctist Lead Data Scientist leaflet league linear regression Logistic Regression machine learning Maps market matplotlib Medical Research Meet the team meetup methal health miami beach movie music Napoli NBA netflix Networking neural network Neural networks New Courses NHL nlp NYC NYC Data Science nyc data science academy NYC Open Data nyc property NYCDSA NYCDSA Alumni Online Online Bootcamp Online Training Open Data painter pandas Part-time performance phoenix pollutants Portfolio Development precision measurement prediction Prework Programming public safety PwC python Python Data Analysis python machine learning python scrapy python web scraping python webscraping Python Workshop R R Data Analysis R language R Programming R Shiny r studio R Visualization R Workshop R-bloggers random forest Ranking recommendation recommendation system regression Remote remote data science bootcamp Scrapy scrapy visualization seaborn seafood type Selenium sentiment analysis sentiment classification Shiny Shiny Dashboard Spark Special Special Summer Sports statistics streaming Student Interview Student Showcase SVM Switchup Tableau teachers team team performance TensorFlow Testimonial tf-idf Top Data Science Bootcamp Top manufacturing companies Transfers tweets twitter videos visualization wallstreet wallstreetbets web scraping Weekend Course What to expect whiskey whiskeyadvocate wildfire word cloud word2vec XGBoost yelp youtube trending ZORI