Web scraping- Insights of the Meetup.com groups in New York
Meetup.com is a social networking platform that facilitates offline groups in various locations, and members are brought together by common interests. Due to the success of the concept, there are large amount of the groups and events are held on the platform. However, how the new members choose which group or event they should join is the question to ask. How often does the group hosting events? What is the atmosphere of the group? Is it a large group that always hosts an event that has more than hundred people or is it a cozy small group of gathering? This project will provide an overview analysis about all the groups that based in New York and a groups sorting system.
The data was scraped from Meetup.com based on New York location, which includes all the groups’ information for all the categories. The total observation is 2327 groups. The variables that being collected are:
- Group Name
- Category name
- Founded Date
- Number of Members, Group Reviews
- Past Meetup Event Numbers
- Upcoming Event Numbers
- Most Recent Upcoming Event Name
- Most Recent Upcoming Event Time
- Most Recent Upcoming Event Date
- Most Recent Upcoming Event’s Number of Participants
- URL of the Groups
The tools that was using for scraping data were Selenium and Beautiful Soup in Python, Selenium provides the ability to scroll down to the bottom of each web page in order to collect all the groups’ URL. Beautiful Soup package is able to collect the data from each group page for the following analysis.
The first tab gives a general overview statistics of the groups information by categories. Number of groups by location in each category graph indicates the ratio of groups’ number in different New York location. With no doubt New York, NY has the highest numbers of groups. By clicking off New York, NY in the legend, we can see that Brooklyn, NY is the second popular location.
Number of meetup in each category indicates that the total number of groups within each category. However, a lot of categories have 100 groups, and which is also the maximum number. There might be a possibility that Meetup.com is only showing 100 most activated groups in each category.
To have a better idea of the average group size of each category, Average member in each category provides the information. The graph shows that Social and Technology are the two category have the highest average number of group members.
Relationship between Number of members, Number of Past Events, and Number of Upcoming Event
The first scatter plot shows the relationship between number of members and numbers of past events of each category. The user can also select the categories that they want to compare by clicking the groups’ name in the legend. We can see that some of the categories have some positive relationship, but some of them do not.
The second scatter plot shows the relationship between numbers of members and numbers of upcoming events of each category. Similar as the first scatter plot, the graph shows that is there any positive relationship between members’ numbers and upcoming events numbers.
The third scatter plot is a 3D scatter plot, which allow the users to see the relationship between numbers of members, numbers of past events, and numbers of upcoming events at the same time.
The Popular Time
The Heat Map tab gives the information about the popular time slot of the weekdays by Number of members, Number of participants, and Average numbers of participant.
The heat map shows that the groups hosting events at Friday from 6pm – 9pm have most members in the groups. The upcoming events that hosting on Wednesday from 6pm-9pm have highest number people attending. However, the time slot that has highest average number of attending is Monday from 9pm – 12am.
Last but not least, the Select Your Group tab is a sorting system that allows users to pick the group that based their preference.
The users can select the groups based on location, weekday, range of members, range of the past event numbers, the range of upcoming event numbers.
The sorting system will provide the detail information of each group and the URL within the user’s parameters.
- New York, NY and Brooklyn, NY are the two most popular places in New York’s meetup groups.
- The categories that has highest average number of members are Social and Technology
- Some of the categories have positive relationship between numbers of past events and numbers of members, such as Social. However, some of them are not, such as Technology.
- Some of the categories have positive relationship between numbers of upcoming events and numbers of members, such as Outdoors & Adventure. However, some of them are not, such as Language & Culture.
- The groups hosting events at Friday from 6pm – 9pm have most members in the groups.
- The upcoming events that hosting on Wednesday from 6pm-9pm has highest number people attending.
- The upcoming events that hosting on Monday from 9pm – 12am has the highest average number of attending.
- Predict the number of upcoming event by constructing a multiple linear regression.
- Scrap the reviews of the groups, and turn the sorting system to a recommendation system.