NYC Data Science Academy| Blog
Bootcamps
Lifetime Job Support Available Financing Available
Bootcamps
Data Science with Machine Learning Flagship ๐Ÿ† Data Analytics Bootcamp Artificial Intelligence Bootcamp New Release ๐ŸŽ‰
Free Lesson
Intro to Data Science New Release ๐ŸŽ‰
Find Inspiration
Find Alumni with Similar Background
Job Outlook
Occupational Outlook Graduate Outcomes Must See ๐Ÿ”ฅ
Alumni
Success Stories Testimonials Alumni Directory Alumni Exclusive Study Program
Courses
View Bundled Courses
Financing Available
Bootcamp Prep Popular ๐Ÿ”ฅ Data Science Mastery Data Science Launchpad with Python View AI Courses Generative AI for Everyone New ๐ŸŽ‰ Generative AI for Finance New ๐ŸŽ‰ Generative AI for Marketing New ๐ŸŽ‰
Bundle Up
Learn More and Save More
Combination of data science courses.
View Data Science Courses
Beginner
Introductory Python
Intermediate
Data Science Python: Data Analysis and Visualization Popular ๐Ÿ”ฅ Data Science R: Data Analysis and Visualization
Advanced
Data Science Python: Machine Learning Popular ๐Ÿ”ฅ Data Science R: Machine Learning Designing and Implementing Production MLOps New ๐ŸŽ‰ Natural Language Processing for Production (NLP) New ๐ŸŽ‰
Find Inspiration
Get Course Recommendation Must Try ๐Ÿ’Ž An Ultimate Guide to Become a Data Scientist
For Companies
For Companies
Corporate Offerings Hiring Partners Candidate Portfolio Hire Our Graduates
Students Work
Students Work
All Posts Capstone Data Visualization Machine Learning Python Projects R Projects
Tutorials
About
About
About Us Accreditation Contact Us Join Us FAQ Webinars Subscription An Ultimate Guide to
Become a Data Scientist
    Login
NYC Data Science Acedemy
Bootcamps
Courses
Students Work
About
Bootcamps
Bootcamps
Data Science with Machine Learning Flagship
Data Analytics Bootcamp
Artificial Intelligence Bootcamp New Release ๐ŸŽ‰
Free Lessons
Intro to Data Science New Release ๐ŸŽ‰
Find Inspiration
Find Alumni with Similar Background
Job Outlook
Occupational Outlook
Graduate Outcomes Must See ๐Ÿ”ฅ
Alumni
Success Stories
Testimonials
Alumni Directory
Alumni Exclusive Study Program
Courses
Bundles
financing available
View All Bundles
Bootcamp Prep
Data Science Mastery
Data Science Launchpad with Python NEW!
View AI Courses
Generative AI for Everyone
Generative AI for Finance
Generative AI for Marketing
View Data Science Courses
View All Professional Development Courses
Beginner
Introductory Python
Intermediate
Python: Data Analysis and Visualization
R: Data Analysis and Visualization
Advanced
Python: Machine Learning
R: Machine Learning
Designing and Implementing Production MLOps
Natural Language Processing for Production (NLP)
For Companies
Corporate Offerings
Hiring Partners
Candidate Portfolio
Hire Our Graduates
Students Work
All Posts
Capstone
Data Visualization
Machine Learning
Python Projects
R Projects
About
Accreditation
About Us
Contact Us
Join Us
FAQ
Webinars
Subscription
An Ultimate Guide to Become a Data Scientist
Tutorials
Data Analytics
  • Learn Pandas
  • Learn NumPy
  • Learn SciPy
  • Learn Matplotlib
Machine Learning
  • Boosting
  • Random Forest
  • Linear Regression
  • Decision Tree
  • PCA
Interview by Companies
  • JPMC
  • Google
  • Facebook
Artificial Intelligence
  • Learn Generative AI
  • Learn ChatGPT-3.5
  • Learn ChatGPT-4
  • Learn Google Bard
Coding
  • Learn Python
  • Learn SQL
  • Learn MySQL
  • Learn NoSQL
  • Learn PySpark
  • Learn PyTorch
Interview Questions
  • Python Hard
  • R Easy
  • R Hard
  • SQL Easy
  • SQL Hard
  • Python Easy
Data Science Blog > Big Data > Cluster Analysis of Twitter: Understanding Human Interactions for Business Improvement

Cluster Analysis of Twitter: Understanding Human Interactions for Business Improvement

Mohammad Haseeb Durrani
Posted on Jun 22, 2017

Introduction

Most of the data readily available in the real world comes unlabeled. Getting the labels often entails manual classification, which can be a tedious and expensive process. While dealing with unlabeled data, we are limited to using unsupervised machine learning techniques since we do not have a definite response variable to work with. Unsupervised learning is generally used as an exploratory data analysis tool used to understand the data. I decided to work with unlabeled Twitter data in the  form of tweets to attempt  to understand  human thoughts and interactions. A business or organization can benefit a great deal from understanding the individuals in its  target audience and then leveraging this to connect with them on a more personal level. Thus, having an automated classification system of such unlabeled tweets can serve as a valuable tool for business improvement.

Workflow

The classification of tweets is a multi-step process, starting from the preprocessing of unlabeled tweets before applying unsupervised machine learning algorithms for clustering and topic modeling. Once the desired level of accuracy in tweet grouping is achieved, one can proceed to manually label them based on similarity in context. Now these labelled tweets can be fed into supervised machine learning models to train the model to understand the underlying similarities of tweets belonging to each category. A fully trained tweet classification model can then be applied to unseen data in an automated tweet labeling system. A visualization of the steps involved can be seen below.



The steps outlined in red form the focus of this blog. Only once reproducibility is achieved in the clustering and topic modeling of tweets, can one proceed to the supervised machine learning model.

The Data

The data for this project were obtained directly using Twitter's Streaming API which allows access to Twitter's global stream of Tweet data. The API gives the user control over downloading by specifying the language, as well as keywords being used in the tweets. For the purpose of the project, I was interested in English tweets.  For some of my data I used the keyword search option to narrow the focus of the tweets being downloaded. The data conveniently downloads in JSON format which can easily be parsed into a Python dictionary.

For analysis the data was then transformed from dictionary format into a Pandas dataframe. Preprocessing of the tweets was somewhat of a trivial process since they tend to be slightly different in structure from standard text documents. There tends to be more use of slang and shorthand to get the message across in limited 140 character working space. For that reason, one must be careful in removing unnecessary elements from each tweet while retaining features which would contribute to the meaning and context of the tweet.

I used regular expressions to remove the characters "RT" in any tweet because keeping it would lead to clustering retweets together which would be at odds with the purpose of my analysis. I also removed any user mentions from tweets which can be identified with the preceding "@" before the user's name. Similarly, I removed any links that were present as well. Finally, I used Python's NLTK package to remove stop words like is, the, are, etc. Stop words are words that are grammatically essential to structure, but contribute very little to the context of a sentence. Once the data was cleaned up it was now ready for machine learning.

Clustering/Topic Modeling

The unsupervised learning algorithms used for this analysis include Latent Dirichlet Allocation (LDA) and Non-negative Matrix Factorization (NMF) for topic modeling, and K-means for clustering of tweets. Conveniently, all three are available in Python's scikit-learn package.



K-means clustering algorithm essentially grouped individual tweets into only one of the specified number of clusters which could be problematic if a given tweet fell into more than one category. Despite this limitation, K-means did surprisingly well in grouping most tweets based on similarities. However, there would always be a "catch-all" cluster which would contain any tweets which the algorithm was unable to properly classify. In this regard the topic modeling algorithms allowed the flexibility of assigning multiple topics to tweets by weight of how likely it is to be a part of each of the topics. The following sections of this blog will show how these algorithms performed with different sets of tweets.

Keyword-search tweets

The first set of tweets I tested were obtained using different keyword searches related to health & fitness, travel, sports, news/politics, art, music, video games, lifestyle, dating/relationships, wedding/marriage, and other life changing events. This was a test to check if the unsupervised learning methods would group the tweets based on the keywords that were used to get them. A word cloud of the hashtags associated with this set of tweets can be seen below.



LDA: Below we have the top words used to determine topic using LDA.  Topic 5 is clearly related to US politics as it includes words as trump, comey, and president. Topic 12 generalizes to entertainment by including the words music, video, sports, and gameplay. Topic 8 seems to be a confused mixture between travel and politics that could partially be due to the mention of President Trump's travel ban.


NMF: For this set of tweets NMF performed substantially better than LDA as can be seen with the topic assignments below. The top words for each topic relate very well to the keywords used in my search. Topics 1, 3, 5, 7, 9, and 13 clearly show relevance to jobs/hiring, wedding, art, sports, health & fitness, and politics respectively. There were some unclear classifications with certain topics, but overall NMF performed very well in this case.


K-means: Clustering using the K-means algorithm also outperformed LDA in coherent grouping of tweets. With the exception of a few "not so good" clusters, grouping based on sports, politics, music, and health can be seen in clusters 5, 8, 11, and 13 respectively.

As mentioned earlier, cluster 9 was the catch-all cluster containing all tweets which did not fall into any of the other clusters.


Open-streaming tweets

The next set of tweets for cluster analysis were streamed with the only restriction of being in english. Given the absence of intentional underlying categories, it was interesting to see how the algorithms would perform in grouping randomly obtained tweets. Below is a  word cloud of the hashtags of these tweets.


LDA: For the open-streaming tweets LDA performed much better as compared to the previous set of tweets. Topic 2 is related to music entertainment, and topic 3 is specific to tweets about the elections in the United Kingdom. Topic 28 seems to talk about jobs, hiring, and careers. Topics 4 and 7 are somewhat unclear in determining the context of the tweets.


NMF: Once again NMF outperformed LDA in topic assignment as can be seen with the clear context of most of the topics determined using this algorithm. Topic 0, 2, 7, 8, and 10 clearly relate respectively to entertainment, UK elections, career, sports, and giveaway advertisements. Topic 11 refers to the unfortunate tragedy of the Arianna Grande concert in Manchester, UK, and topic 29 speaks of events related to FBI director Comey's testimony regarding President Trump and Russia.


K-means: K-means did a good job at clustering tweets by category as can be seen some of the clusters below. Cluster 5 grouped tweets mentioning different genres of music together in a music-related cluster. Cluster 14 and 25 group on the discussion of the UK and US politics respectively. 

The catch-all cluster in this case was cluster 22 containing a good portion of the tweets which were unclassified.


Focused tweets

The final set of tweets I obtained were from users following specific organizations and companies. The purpose of narrowing my focus to user tweets was to detect if multiple tweets from the same users can provide insights into topics of interest to the user. For this set I obtained about 100 tweets each from 2000 followers of organizations/companies such as Barclays, Fitbit, Tinder, Democratic/Republican National Party, and the Economist. For this blog I will discuss my analysis of tweets from followers of the Democrat or Republican Party.


                                                                                            Democrat Hashtags 

                                                                                           Republican Hashtags


Using LDA topic modeling, and visualizing some of the tweets associated with the assigned topics, we can see the opposing views from followers of two competing entities. Followers of the Democratic Party seem to be more critical of President Trump in their tweets, while followers of the Republican Party are more supportive of President Trump and more critical of Hillary Clinton and former President Obama.



Future directions

Future work on this project will involve cycling between cleaning the text of the tweets to obtain optimal topic modeling and clustering using the algorithms discussed in the blog. Afterwards, supervised learning techniques can be used to train a model to classify new tweets based on pre-existing or newly added categories.



This scope of this project and its methodology can be extended to include image analysis and video analysis to include posts from other forms of social media such as Facebook, Instagram, and Snapchat. The more data that can be added to this model, the better the model will serve as a valuable resource to organizations to understand their current and potential customers.

About Author

Mohammad Haseeb Durrani

Mohammad Haseeb Durrani is a highly motivated and insightful Data Scientist with a Master of Science in Biomedical Engineering from the NYU Tandon School of Engineering. With an extensive background in biomedical research including pre-clinical vaccine development, and...
View all posts by Mohammad Haseeb Durrani >

Related Articles

Capstone
Catching Fraud in the Healthcare System
Capstone
The Convenience Factor: How Grocery Stores Impact Property Values
Capstone
Acquisition Due Dilligence Automation for Smaller Firms
Machine Learning
Pandemic Effects on the Ames Housing Market and Lifestyle
Machine Learning
The Ames Data Set: Sales Price Tackled With Diverse Models

Leave a Comment

Cancel reply

You must be logged in to post a comment.

#30DaysDataScienceChallenge โ€“ ๐Ÿค– May 2, 2018
[โ€ฆ] Day 29: Twitter Analysis with Python (bonus: Cluster Analysis of Twitter) [โ€ฆ]

View Posts by Categories

All Posts 2399 posts
AI 7 posts
AI Agent 2 posts
AI-based hotel recommendation 1 posts
AIForGood 1 posts
Alumni 60 posts
Animated Maps 1 posts
APIs 41 posts
Artificial Intelligence 2 posts
Artificial Intelligence 2 posts
AWS 13 posts
Banking 1 posts
Big Data 50 posts
Branch Analysis 1 posts
Capstone 206 posts
Career Education 7 posts
CLIP 1 posts
Community 72 posts
Congestion Zone 1 posts
Content Recommendation 1 posts
Cosine SImilarity 1 posts
Data Analysis 5 posts
Data Engineering 1 posts
Data Engineering 3 posts
Data Science 7 posts
Data Science News and Sharing 73 posts
Data Visualization 324 posts
Events 5 posts
Featured 37 posts
Function calling 1 posts
FutureTech 1 posts
Generative AI 5 posts
Hadoop 13 posts
Image Classification 1 posts
Innovation 2 posts
Kmeans Cluster 1 posts
LLM 6 posts
Machine Learning 364 posts
Marketing 1 posts
Meetup 144 posts
MLOPs 1 posts
Model Deployment 1 posts
Nagamas69 1 posts
NLP 1 posts
OpenAI 5 posts
OpenNYC Data 1 posts
pySpark 1 posts
Python 16 posts
Python 458 posts
Python data analysis 4 posts
Python Shiny 2 posts
R 404 posts
R Data Analysis 1 posts
R Shiny 560 posts
R Visualization 445 posts
RAG 1 posts
RoBERTa 1 posts
semantic rearch 2 posts
Spark 17 posts
SQL 1 posts
Streamlit 2 posts
Student Works 1687 posts
Tableau 12 posts
TensorFlow 3 posts
Traffic 1 posts
User Preference Modeling 1 posts
Vector database 2 posts
Web Scraping 483 posts
wukong138 1 posts

Our Recent Popular Posts

AI 4 AI: ChatGPT Unifies My Blog Posts
by Vinod Chugani
Dec 18, 2022
Meet Your Machine Learning Mentors: Kyle Gallatin
by Vivian Zhang
Nov 4, 2020
NICU Admissions and CCHD: Predicting Based on Data Analysis
by Paul Lee, Aron Berke, Bee Kim, Bettina Meier and Ira Villar
Jan 7, 2020

View Posts by Tags

#python #trainwithnycdsa 2019 2020 Revenue 3-points agriculture air quality airbnb airline alcohol Alex Baransky algorithm alumni Alumni Interview Alumni Reviews Alumni Spotlight alumni story Alumnus ames dataset ames housing dataset apartment rent API Application artist aws bank loans beautiful soup Best Bootcamp Best Data Science 2019 Best Data Science Bootcamp Best Data Science Bootcamp 2020 Best Ranked Big Data Book Launch Book-Signing bootcamp Bootcamp Alumni Bootcamp Prep boston safety Bundles cake recipe California Cancer Research capstone car price Career Career Day ChatGPT citibike classic cars classpass clustering Coding Course Demo Course Report covid 19 credit credit card crime frequency crops D3.js data data analysis Data Analyst data analytics data for tripadvisor reviews data science Data Science Academy Data Science Bootcamp Data science jobs Data Science Reviews Data Scientist Data Scientist Jobs data visualization database Deep Learning Demo Day Discount disney dplyr drug data e-commerce economy employee employee burnout employer networking environment feature engineering Finance Financial Data Science fitness studio Flask flight delay football gbm Get Hired ggplot2 googleVis H20 Hadoop hallmark holiday movie happiness healthcare frauds higgs boson Hiring hiring partner events Hiring Partners hotels housing housing data housing predictions housing price hy-vee Income industry Industry Experts Injuries Instructor Blog Instructor Interview insurance italki Job Job Placement Jobs Jon Krohn JP Morgan Chase Kaggle Kickstarter las vegas airport lasso regression Lead Data Scienctist Lead Data Scientist leaflet league linear regression Logistic Regression machine learning Maps market matplotlib Medical Research Meet the team meetup methal health miami beach movie music Napoli NBA netflix Networking neural network Neural networks New Courses NHL nlp NYC NYC Data Science nyc data science academy NYC Open Data nyc property NYCDSA NYCDSA Alumni Online Online Bootcamp Online Training Open Data painter pandas Part-time performance phoenix pollutants Portfolio Development precision measurement prediction Prework Programming public safety PwC python Python Data Analysis python machine learning python scrapy python web scraping python webscraping Python Workshop R R Data Analysis R language R Programming R Shiny r studio R Visualization R Workshop R-bloggers random forest Ranking recommendation recommendation system regression Remote remote data science bootcamp Scrapy scrapy visualization seaborn seafood type Selenium sentiment analysis sentiment classification Shiny Shiny Dashboard Spark Special Special Summer Sports statistics streaming Student Interview Student Showcase SVM Switchup Tableau teachers team team performance TensorFlow Testimonial tf-idf Top Data Science Bootcamp Top manufacturing companies Transfers tweets twitter videos visualization wallstreet wallstreetbets web scraping Weekend Course What to expect whiskey whiskeyadvocate wildfire word cloud word2vec XGBoost yelp youtube trending ZORI

NYC Data Science Academy

NYC Data Science Academy teaches data science, trains companies and their employees to better profit from data, excels at big data project consulting, and connects trained Data Scientists to our industry.

NYC Data Science Academy is licensed by New York State Education Department.

Get detailed curriculum information about our
amazing bootcamp!

Please enter a valid email address
Sign up completed. Thank you!

Offerings

  • HOME
  • DATA SCIENCE BOOTCAMP
  • ONLINE DATA SCIENCE BOOTCAMP
  • Professional Development Courses
  • CORPORATE OFFERINGS
  • HIRING PARTNERS
  • About

  • About Us
  • Alumni
  • Blog
  • FAQ
  • Contact Us
  • Refund Policy
  • Join Us
  • SOCIAL MEDIA

    ยฉ 2025 NYC Data Science Academy
    All rights reserved. | Site Map
    Privacy Policy | Terms of Service
    Bootcamp Application