Data Analysis of cyber attacks between 2017 and 2018

Posted on Aug 13, 2018
The skills the author demoed here can be learned through taking Data Science with Machine Learning bootcamp with NYC Data Science Academy.


Project Type

Web Scraping and Data Visualization

Project Name

Analysis of cyber attacks between 2017 and 2018

Source and


Python numpy, pandas, seaborn, scrapy, matplotlib, wordcloud, NLP packages


  • Initiate spider to scrape for listed cyber attacks
  • Exploratory Data Analysis to identify trends
  • Initiate spider to scrape for information on affected organizations/institutions within healthcare sector
  • Exploratory Data Analysis and visualization

Key Findings

  • Number of cyber attacks in 2018 increases each month compared with 2017
  • Individuals are the main target for malware and account hijacking
    • hackers use campaigns to target millions simultaneously, mostly Android users
  • Top industry affected is healthcare sector
    • hackers target accounts of employees through phishing emails
    • hospitals are vulnerable due to their old IT infrastructure and large number of employees
    • small private organizations are vulnerable due to the lack of sophisticated cyber security (lack of funding)


Cyber attacks represent a serious and expensive threat for all industries. Deep Learning algorithms might be able to not only catch up with hackers but also be one step ahead of the next attack.


Every year, the increasing number of cyber attacks comes with evolved tactics and highly skilled hackers in searchĀ of rapid financial gain. According to the "Cost of a Data Breach Study" by the Ponemon Institute LLC, the average cost of a data breach is US$3.86 million with about 30% likelihood of a recurrent breach within the next 24 months. Across various industries, heavily regulated sectors such as healthcare and financial organizations suffer from the highest data breach costs per capita (US$408 and US$206, respectively compared to an average per capita cost of US$148).

Understanding the landscape of cyber threats and identifying predictive factors that contribute to increasing vulnerabilities is crucial to protect the sensitive informationĀ of individuals and enterprises. To link company information of organizations within the healthcare sector with reported cyber attack characteristics, I scraped hackmaggedonĀ to get a list of cyber attacks across multiple industries as well as available information from Glassdoor for selected organizations. Please visit my GitHub for the complete code and analysis.


The scrapyĀ package of Python was used to collect information on date, target, target_class, country, summary description, and type of attack from the siteĀ hackmageddon. Similarly, information on companies' revenue and employee size was scraped from glassdoorĀ using scrapy as the major tool. Data visualization was performed with python packages such as Pandas, Numpy, and Matplotlib. Text snippets were processed using filtering, tokenization, and lemmitizationĀ of NLP packages in python.


The landscape of cyber attacks

In comparison with last year, the number of cyber attacks increased for every month suggesting an overall increasing number of cyber attacks. Data Analysis of cyber attacks between 2017 and 2018

Based on the classification of, individuals are the main target of cyber attacks (2017: 61 attacks, 2018: 169 attacks)Ā  followed by organizations and companies of the healthcare and financial sectors. Other sectors such as science, transportation, or hospitality are less affected.

Data Analysis of cyber attacks between 2017 and 2018

Target: Individuals

The pie chart below displaysĀ the main types of cyber attacksĀ on individuals. More than 50% of cyber threats on individuals can be attributed to malware, followed by accountĀ hijacking, and targetedĀ attacks. Although within the top 5 categories and among the most feared, malware targeting point-of-sale interactions to steal credit card information and similar, are less dominant. Other categories include malvertising, vulnerabilities, and malicious scripts.

To gain more information on the nature of reported cyber attacks on individuals, the summary text wereĀ visualized in form of a WordCloud after processing and cleaning. Words appearing with high frequencies are "million", "campaign", "android", and "app". These words indicate that hacks often happen through downloads of apps by android users, affecting millions of individuals simultaneously. "Ransomware" is another word that gained more and more popularity, referring to blackmailing of affected individuals by locking down access to private and sensitive information.

Data Analysis of cyber attacks between 2017 and 2018

Target: Healthcare sector

The WordCloud based on the description of cyber attacks within the healthcare sector looked different. While apps and Android users described the main playfield of cyber attacks on individuals, it is "phishing", "email account", andĀ  "employee" that describe the main methodologies for the healthcare industry. Hackers gain access to the system through phishing emails, which leads to a breach of information and/or the distribution of ransomware.

The word "ransomware" also appeared as one of the most frequent words, emphasizing the danger of it as sensitive financial and health-related information of millions of people can be at stake. Based on the frequency WordCloud, the question is whether the number of employees correlates with the number of cyber attacks as more employees increase the likelihood of "successful" data breach.


Through, I found revenue and employee information on affected organizations operating within the healthcare sector. However, both variables were given as categorical data with different ranges. I applied the Chi-square test to confirm that the size of employees is dependent on revenue and vice versa (p-value < 0.001). Affected organizations and companies were further subdivided into hospitals, private companies (mostly associations of certain physicians), and non-profit organizations (e.g. health insurance companies and large hospital groups).

Non-profits showed a relative increase of cyber attacks with increasing size of the organization. About 30% of all cyber attacks within the nonprofits targeted organizations with more than 10,000 employees. Surprisingly, private companies showed an opposite trend. The relative number of cyber attacks decreased with increasing employees. Small private companies with 1-50 employees showed the highest frequency of cyber attacks, indicating the lack of sufficient IT security. Hospitals did not reveal a clear trend. More than 50% of all attacks were targeted towards hospitals with 1001-5000 employees.

It is possible, that this number reflects the standard size of most hospitals with little variance as larger hospital groups are included in the nonprofit category and smaller private practices in the private companies category. Together, the data suggests a correlation between size and number of attacks. Hospitals seems to be particularly vulnerable.

| Conclusion |

Cyber attacks represent a common and dangerous threat for every individual who is connected to the internet and all enterprises. As individual using an Android device, cautious downloads of apps can prevent malware and other malicious activities from occurring on those devices. Increased cyber security training for employees of hospitals is highly recommended to counteract the threat of phishing emails. Ransomware has become a new and lucrative trend among the hacker community, prompting increased investments in cyber security across all industries.

Unsupervised machine learning can be used to identify factors that cluster organizations into high-risk or low-risk groups, which in turn can provide helpful information to implement security measures that can not only detect cyber threats faster but also prevent recurring attacks. In the future, Deep Learning will be another powerful tool to detect unknown network intrusions.

About Author

Melanie Uhde

Iā€™m a Ph.D. in Immunology with hands-on experience in applying Machine Learning tools in Python and R. At Columbia University Medical Center, I led research projects on the identification of patient subsets to find new treatment options by...
View all posts by Melanie Uhde >

Leave a Comment

No comments found.

View Posts by Categories

Our Recent Popular Posts

View Posts by Tags

#python #trainwithnycdsa 2019 2020 Revenue 3-points agriculture air quality airbnb airline alcohol Alex Baransky algorithm alumni Alumni Interview Alumni Reviews Alumni Spotlight alumni story Alumnus ames dataset ames housing dataset apartment rent API Application artist aws bank loans beautiful soup Best Bootcamp Best Data Science 2019 Best Data Science Bootcamp Best Data Science Bootcamp 2020 Best Ranked Big Data Book Launch Book-Signing bootcamp Bootcamp Alumni Bootcamp Prep boston safety Bundles cake recipe California Cancer Research capstone car price Career Career Day citibike classic cars classpass clustering Coding Course Demo Course Report covid 19 credit credit card crime frequency crops D3.js data data analysis Data Analyst data analytics data for tripadvisor reviews data science Data Science Academy Data Science Bootcamp Data science jobs Data Science Reviews Data Scientist Data Scientist Jobs data visualization database Deep Learning Demo Day Discount disney dplyr drug data e-commerce economy employee employee burnout employer networking environment feature engineering Finance Financial Data Science fitness studio Flask flight delay gbm Get Hired ggplot2 googleVis H20 Hadoop hallmark holiday movie happiness healthcare frauds higgs boson Hiring hiring partner events Hiring Partners hotels housing housing data housing predictions housing price hy-vee Income Industry Experts Injuries Instructor Blog Instructor Interview insurance italki Job Job Placement Jobs Jon Krohn JP Morgan Chase Kaggle Kickstarter las vegas airport lasso regression Lead Data Scienctist Lead Data Scientist leaflet league linear regression Logistic Regression machine learning Maps market matplotlib Medical Research Meet the team meetup methal health miami beach movie music Napoli NBA netflix Networking neural network Neural networks New Courses NHL nlp NYC NYC Data Science nyc data science academy NYC Open Data nyc property NYCDSA NYCDSA Alumni Online Online Bootcamp Online Training Open Data painter pandas Part-time performance phoenix pollutants Portfolio Development precision measurement prediction Prework Programming public safety PwC python Python Data Analysis python machine learning python scrapy python web scraping python webscraping Python Workshop R R Data Analysis R language R Programming R Shiny r studio R Visualization R Workshop R-bloggers random forest Ranking recommendation recommendation system regression Remote remote data science bootcamp Scrapy scrapy visualization seaborn seafood type Selenium sentiment analysis sentiment classification Shiny Shiny Dashboard Spark Special Special Summer Sports statistics streaming Student Interview Student Showcase SVM Switchup Tableau teachers team team performance TensorFlow Testimonial tf-idf Top Data Science Bootcamp Top manufacturing companies Transfers tweets twitter videos visualization wallstreet wallstreetbets web scraping Weekend Course What to expect whiskey whiskeyadvocate wildfire word cloud word2vec XGBoost yelp youtube trending ZORI