Web scraping- Insights of the Meetup.com groups in New York

Avatar
Posted on Nov 21, 2016

Motivation 

Meetup.com is a social networking platform that facilitates offline groups in various locations, and members are brought together by common interests. Due to the success of the concept, there are large amount of the groups and events are held on the platform. However, how the new members choose which group or event they should join is the question to ask. How often does the group hosting events? What is the atmosphere of the group? Is it a large group that always hosts an event that has more than hundred people or is it a cozy small group of gathering? This project will provide an overview analysis about all the groups that based in New York and a groups sorting system.

 

Data Collecting 

The data was scraped from Meetup.com based on New York location, which includes all the groups’ information for all the categories. The total observation is 2327 groups. The variables that being collected are:

  • Group Name
  • Category name
  • Founded Date
  • Number of Members, Group Reviews
  • Location
  • Past Meetup Event Numbers
  • Upcoming Event Numbers
  • Most Recent Upcoming Event Name
  • Most Recent Upcoming Event Time
  • Most Recent Upcoming Event Date
  • Most Recent Upcoming Event’s Number of Participants
  • URL of the Groups

 

Process 

The tools that was using for scraping data were Selenium and Beautiful Soup in Python, Selenium provides the ability to scroll down to the bottom of each web page in order to collect all the groups’ URL. Beautiful Soup package is able to collect the data from each group page for the following analysis.

Before the Exploratory Data Analysis, all the data merging, cleaning was performed in R. The project was created in Shiny application. All the code can be found on GitHub.

Overview Statistics

The first tab gives a general overview statistics of the groups information by categories. Number of groups by location in each category graph indicates the ratio of groups’ number in different New York location. With no doubt New York, NY has the highest numbers of groups. By clicking off New York, NY in the legend, we can see that Brooklyn, NY is the second popular location.

Number of meetup in each category indicates that the total number of groups within each category. However, a lot of categories have 100 groups, and which is also the maximum number. There might be a possibility that Meetup.com is only showing 100 most activated groups in each category.

To have a better idea of the average group size of each category, Average member in each category provides the information. The graph shows that Social and Technology are the two category have the highest average number of group members.

m1

m2

 

Relationship between Number of members, Number of Past Events, and Number of Upcoming Event

The first scatter plot shows the relationship between number of members and numbers of past events of each category. The user can also select the categories that they want to compare by clicking the groups’ name in the legend. We can see that some of the categories have some positive relationship, but some of them do not.

m3

The second scatter plot shows the relationship between numbers of members and numbers of upcoming events of each category. Similar as the first scatter plot, the graph shows that is there any positive relationship between members’ numbers and upcoming events numbers.

m4

The third scatter plot is a 3D scatter plot, which allow the users to see the relationship between numbers of members, numbers of past events, and numbers of upcoming events at the same time.

m5

The Popular Time

The Heat Map tab gives the information about the popular time slot of the weekdays by Number of members, Number of participants, and Average numbers of participant.

The heat map shows that the groups hosting events at Friday from 6pm – 9pm have most members in the groups. The upcoming events that hosting on Wednesday from 6pm-9pm have highest number people attending. However, the time slot that has highest average number of attending is Monday from 9pm – 12am.

m6

Sorting System

Last but not least, the Select Your Group tab is a sorting system that allows users to pick the group that based their preference.

The users can select the groups based on location, weekday, range of members, range of the past event numbers, the range of upcoming event numbers.

The sorting system will provide the detail information of each group and the URL within the user’s parameters.

%e6%93%b7%e5%8f%96

Takeaways 

  1. New York, NY and Brooklyn, NY are the two most popular places in New York’s meetup groups.
  1. The categories that has highest average number of members are Social and Technology
  1. Some of the categories have positive relationship between numbers of past events and numbers of members, such as Social. However, some of them are not, such as Technology.
  1. Some of the categories have positive relationship between numbers of upcoming events and numbers of members, such as Outdoors & Adventure. However, some of them are not, such as Language & Culture.
  1. The groups hosting events at Friday from 6pm – 9pm have most members in the groups.
  1. The upcoming events that hosting on Wednesday from 6pm-9pm has highest number people attending.
  2. The upcoming events that hosting on Monday from 9pm – 12am has the highest average number of attending.

Future Steps

  • Predict the number of upcoming event by constructing a multiple linear regression.
  • Scrap the reviews of the groups, and turn the sorting system to a recommendation system.

About Author

Related Articles

Leave a Comment

Avatar
TheWhiteningDepot UK December 18, 2017
There are five varieties of periodontitis, though a number of them tend to be common. One day, tooth will likely be eroded completely that only a few black part remains in the name of teeth. Now more adults are able to change their looks according changing fashion.
Avatar
TheWhiteningDepot UK November 29, 2017
A good family dentist could also advise parents around the use of fluoride supplements, the most effective toothpaste to make use of, and the way to teach children to properly brush their teeth. A porcelain crown is reinforcement for any weakened tooth that's chipped, cracked or broken. Sometimes to leave dental surgeries or drilling of teeth they conceal pain or maybe avoid likely to dentist clinic or hospital.
Avatar
copie bracelet love June 26, 2017
cartierlovejesduas Sounds like that Wells Fargo job is keeping you really happy, positive, open-minded, interested in fairness, valuing more than the almighty dollar, etc. copie bracelet love http://www.bestlevering.cn/fr/cheap-replica-cartier-love-white-gold-bracelet-paved-with-diamonds-p11/

View Posts by Categories


Our Recent Popular Posts


View Posts by Tags

2019 airbnb alumni Alumni Interview Alumni Spotlight alumni story Alumnus API artist aws beautiful soup Best Bootcamp Best Data Science 2019 Best Data Science Bootcamp Big Data bootcamp Bootcamp Prep Bundles California Cancer Research capstone Career citibike clustering Coding Course Report D3.js data Data Analyst data science Data Science Academy Data Science Bootcamp Data Scientist Data Scientist Jobs data visualization Deep Learning Demo Day dplyr employer networking feature engineering Finance Financial Data Science Flask gbm Get Hired ggplot2 googleVis Hadoop higgs boson Hiring hiring partner events Industry Experts Job JP Morgan Chase Kaggle lasso regression Lead Data Scienctist Lead Data Scientist leaflet linear regression Logistic Regression machine learning Maps matplotlib Medical Research meetup Networking neural network Neural networks New Courses nlp NYC NYC Data Science nyc data science academy NYC Open Data NYCDSA NYCDSA Alumni Open Data painter pandas Portfolio Development prediction Programming PwC python python machine learning python scrapy python web scraping python webscraping Python Workshop R R language R Programming R Shiny r studio R Visualization R Workshop R-bloggers random forest recommendation recommendation system regression Scrapy scrapy visualization seaborn Selenium sentiment analysis Shiny Shiny Dashboard Spark Special Special Summer Sports statistics streaming Student Interview Student Showcase SVM Tableau Testimonial tf-idf Top Data Science Bootcamp twitter visualization web scraping What to expect word cloud word2vec XGBoost yelp