Data Web scraping- Meetup.com groups in New York
The skills we demoed here can be learned through taking Data Science with Machine Learning bootcamp with NYC Data Science Academy.
Motivation
Meetup.com is a social networking platform that facilitates offline groups in various locations, and members are brought together by common interests. Due to the success of the concept, data shows there are large amount of the groups and events are held on the platform.
However, how the new members choose which group or event they should join is the question to ask. How often does the group hosting events? What is the atmosphere of the group? Is it a large group that always hosts an event that has more than hundred people or is it a cozy small group of gathering? This project will provide an overview data analysis about all the groups that based in New York and a groups sorting system.
Data Collecting
The data was scraped from Meetup.com based on New York location, which includes all the groups’ information for all the categories. The total observation is 2327 groups. The variables that being collected are:
- Group Name
- Category name
- Founded Date
- Number of Members, Group Reviews
- Location
- Past Meetup Event Numbers
- Upcoming Event Numbers
- Most Recent Upcoming Event Name
- Most Recent Upcoming Event Time
- Highest Amount of Recent Upcoming Event Date
- Most Recent Upcoming Event’s Number of Participants
- URL of the Groups
Process
The tools that was using for scraping data were Selenium and Beautiful Soup in Python, Selenium provides the ability to scroll down to the bottom of each web page in order to collect all the groups’ URL. Beautiful Soup package is able to collect the data from each group page for the following analysis.
Before the Exploratory Data Analysis, all the data merging, cleaning was performed in R. The project was created in Shiny application. All the code can be found on GitHub.
Overview Data Statistics
The first tab gives a general overview statistics of the groups information by categories. Number of groups by location in each category graph indicates the ratio of groups’ number in different New York location. With no doubt New York, NY has the highest numbers of groups. By clicking off New York, NY in the legend, we can see that Brooklyn, NY is the second popular location.
Number of meetup in each category indicates that the total number of groups within each category. However, a lot of categories have 100 groups, and which is also the maximum number. There might be a possibility that Meetup.com is only showing 100 most activated groups in each category.
To have a better idea of the average group size of each category, Average member in each category provides the information. The graph shows that Social and Technology are the two category have the highest average number of group members.
Relationship between Number of members, Number of Past Events, and Number of Upcoming Event
The first scatter plot shows the relationship between number of members and numbers of past events of each category. The user can also select the categories that they want to compare by clicking the groups’ name in the legend. We can see that some of the categories have some positive relationship, but some of them do not.
Relationship Between Number of Members and Upcoming Events
The second scatter plot shows the relationship between numbers of members and numbers of upcoming events of each category. Similar as the first scatter plot, the graph shows that is there any positive relationship between members’ numbers and upcoming events numbers.
Scatter Plot on Number of Members, Past Events, and Upcoming Events
The third scatter plot is a 3D scatter plot, which allow the users to see the relationship between numbers of members, numbers of past events, and numbers of upcoming events at the same time.
The Popular Time
The Heat Map tab gives the information about the popular time slot of the weekdays by Number of members, Number of participants, and Average numbers of participant.
The heat map shows that the groups hosting events at Friday from 6pm – 9pm have most members in the groups. The upcoming events that hosting on Wednesday from 6pm-9pm have highest number people attending. However, the time slot that has highest average number of attending is Monday from 9pm – 12am.
Sorting System
Last but not least, the Select Your Group tab is a sorting system that allows users to pick the group that based their preference.
The users can select the groups based on location, weekday, range of members, range of the past event numbers, the range of upcoming event numbers.
The sorting system will provide the detail information of each group and the URL within the user’s parameters.
Takeaways
- New York, NY and Brooklyn, NY are the two most popular places in New York’s meetup groups.
- The categories that has highest average number of members are Social and Technology
- Some of the categories have positive relationship between numbers of past events and numbers of members, such as Social. However, some of them are not, such as Technology.
- Some of the categories have positive relationship between numbers of upcoming events and numbers of members, such as Outdoors & Adventure. However, some of them are not, such as Language & Culture.
- The groups hosting events at Friday from 6pm – 9pm have most members in the groups.
- The upcoming events that hosting on Wednesday from 6pm-9pm has highest number people attending.
- The upcoming events that hosting on Monday from 9pm – 12am has the highest average number of attending.
Future Steps
- Predict the number of upcoming event by constructing a multiple linear regression.
- Scrap the reviews of the groups, and turn the sorting system to a recommendation system.
About Author
Lydia Kan
View all posts by Lydia Kan >Related Articles
Leave a Comment
