Posted on Apr 9, 2021
The skills I demoed here can be learned through taking Data Science with Machine Learning bootcamp with NYC Data Science Academy.


During the Covid´s pandemic the world´s economy has suffered from a radical slowdown. This has been harshly felt in the labor market, affecting the opportunities of citizens to earn an income. This project aims not at providing an explanation of the general impact of the pandemic in the labor market, but instead to use data provide a snapshot to help identify possible niches of opportunity during one of the worst moments of the pandemic.

Thus, the aim is not to produce general claims about the labor market in the country, but to offer a picture of the available opportunities in the neighboring States of New York, New Jersey and Pennsylvania.

The Data snapshot 

The data has been obtained scraping Glassdoor. The popular job portal allows gathering information about vacancies within a geographical range and filtering by other categories such as industry and salary. For this project, any restriction was introduced in terms of salary or industry and the geographical range has been chosen to cover the States of New York, New Jersey and Pennsylvania.

From each vacancy, data has been collected for: the particular job function, the company offering the vacancy, the industry in which it operates, the number of employees (expressed in a range), the State in which the company is located and the salary offered for the particular job. Right before performing the analysis, the database accounts for 563 observations and 7 variables.

 Exploratory data visualization

The insights obtained from the exploratory data visualization are displayed below and offer an account of the labor market´ situation in the States of New York, New Jersey and Pennsylvania.

Firstly, we visualize the number of vacancies offered in each of the industries that conform our dataset.

Figure 1: Number of vacancies per industry


As we can see in Figure 1, IT is the industry offering a higher number of jobs. This is probably due to more demand for on-line services during the pandemic combined with the fact that telework is easier to implement in this industry. Health care is the second industry that offers a higher amount of vacancies. This is coherent with the greater need of health care professionals during the pandemic. As we can already begin to realize, the pandemic affects firms and workers in different industries differently.

Figure 2: Salaries per industry


When it comes to salaries, Figure 2 shows that Business Services is the industry that offers workers the highest remunerations. IT appears again as a good option in the second place, another signal that IT workers are doing better than most others during the pandemic. Health care is also an industry in which high salaries are offered. On the other hand, entertainment, non-profit and construction appear as the industries with the worst paying vacancies at the time. Especially in the case of the construction industry, this may be affected by the restrictions on the work opportunities, as most of the work cannot be done remotely.

Figure 3: Number of vacancies per State


As expected, figure 3 shows that New York is the state with a higher amount of offered vacancies. Pennsylvania´s low number is probably due to the geographical distance from New York, which makes jobs in New York and New Jersey to be more likely to be displayed by Glassdoor.

Figure 4: Average salaries per State

Figure 4 shows similar average salaries for New York and New Jersey, with Pennsylvania offering slightly lower ones. This may be a surprising result, as the prior might be that New York would offer higher salaries. In order to clarify what is driving this result, we compare average salaries across industries between the State of New York and New Jersey.

Figure 5: Average salaries across industries between the States of New York and New Jersey.

Figure 5 shows how the average salary per industry is actually higher in New York for every industry we have data on with only one exception. This is the Business Services industry, in which there is a big gap between both states. In the next Figure, we dig deeper into this industry to check what is behind this big difference.

Figure 6: Average salaries per firm in the business service industry

As we can see in Figure 6, there is one company in New Jersey, OptiBrand Rx, that is offering an average salary sharply above other companies in the industry. A close look at the data reveals that OptiBrand Rx is looking to hire for many executive positions, which imply higher wages. In addition, this company seems to be in a hurry to find suitable candidates, as it has published each of those vacancies several times, with slight salaries increases, thus augmenting the weight of the company in the State´s average. This appears to be a partial explanation for the difference in average salary between both States.

Figure 7: Proportion of vacancies per industry and State.

Figure 7 shows another possible explanation. It displays the proportion of vacancies offered by industry in each State. This allows to check whether the difference in average salaries may not only be due to differences across industries but also because a certain State offers a higher proportion of its vacancies in high paying industries.

As we can see in the graph, New Jersey offers a higher proportion of its vacancies in high paying industries like IT or insurance. Despite this, it is difficult to know for certain the net effect as it also offers a smaller proportion of vacancies in another high paying industry such as healthcare. Nevertheless, a look at the graph provides the insight that this could also help to explain why there are no differences in average salaries between New York and New Jersey although New York pays higher average salaries across almost every industry.

Figure 8: Salaries per number of employees in the firm.

In Figure 8 we turn to analyze how the number of employees in a firm is related to the salaries it offers.

Intuitively, one may expect firms with higher numbers of employees to offer higher salaries. However, we see that this is not the case. Actually, small firms (1 to 50 employees), seem to offer the highest ones while big companies offer lower wages. An explanation for this may be that firms hire a mix of employees with different sets of skills and wages. Following this reasoning, bigger firms may offer above average wages to certain employees but this may be offset by the wage offered to workers with a lower skill level. We try to confirm if this is the case in the next two figures.

Figure 9: Number of vacancies per salary level and number of employees in the firm.

In Figure 9, we break down the vacancies offered into two categories, low pay and high pay. In order to classify them, we calculate the average market salary and assign a vacancy as high pay if the offered salary is above the average market salary and low pay otherwise. Then, we compare the number of each of these types of vacancies across firms with different numbers of employees.

As we can see in the graph, smaller firms actually tend to offer a higher proportion of low paying vacancies that bigger companies do, with the exception of firms in the range of 1 to 50 employees. Now we can see how firms with more than 10.000 employees offer a higher proportion of high paying jobs. Why then do smaller firms offer higher salaries on average? In order to try to provide a more definite answer, we compare the average salaries of the low pay vacancies previously classified across firms with different numbers of employees.

Figure 10: Data on Average salaries of low pay vacancies per number of employees.


In Figure 10, the order in which the firms are displayed have been reversed to show the ones with the highest number of employees above. As we can see, companies with more employees offer lower average salaries in their low paying vacancies in comparison to smaller firms.


We provide evidence to support possible niches in the labor market that have been less affected during the pandemic. Specifically, the IT industry in a firm above 10.000 employees located in New York stands as one of the best options. The healthcare industry also provides good economic opportunities, although applicants should also weigh the health risk.

For high skilled workers, companies with a high number of employees stand as a better choice than smaller ones. However, this is not the case for job-seekers with less valuable skill sets. These are probably better off in a smaller firm. Finally, New York offers better economic opportunities than New Jersey or Pennsylvania. This may be due to the higher health risk associated with living and working in a huge metropolis in comparison to less densely populated areas in the neighboring States.

 The relevant code and data for this project, including web scraping and analysis, can be found here.

About Author

Guillermo Ruiz

Data Science Professional and Economist with a demonstrated history of data analysis and machine learning modeling with a focus on storytelling with data. Passionate about helping companies to gather and analyze data to make more informed decisions to...
View all posts by Guillermo Ruiz >

Related Articles

Leave a Comment

No comments found.

View Posts by Categories

Our Recent Popular Posts

View Posts by Tags

#python #trainwithnycdsa 2019 2020 Revenue 3-points agriculture air quality airbnb airline alcohol Alex Baransky algorithm alumni Alumni Interview Alumni Reviews Alumni Spotlight alumni story Alumnus ames dataset ames housing dataset apartment rent API Application artist aws bank loans beautiful soup Best Bootcamp Best Data Science 2019 Best Data Science Bootcamp Best Data Science Bootcamp 2020 Best Ranked Big Data Book Launch Book-Signing bootcamp Bootcamp Alumni Bootcamp Prep boston safety Bundles cake recipe California Cancer Research capstone car price Career Career Day citibike classic cars classpass clustering Coding Course Demo Course Report covid 19 credit credit card crime frequency crops D3.js data data analysis Data Analyst data analytics data for tripadvisor reviews data science Data Science Academy Data Science Bootcamp Data science jobs Data Science Reviews Data Scientist Data Scientist Jobs data visualization database Deep Learning Demo Day Discount disney dplyr drug data e-commerce economy employee employee burnout employer networking environment feature engineering Finance Financial Data Science fitness studio Flask flight delay gbm Get Hired ggplot2 googleVis H20 Hadoop hallmark holiday movie happiness healthcare frauds higgs boson Hiring hiring partner events Hiring Partners hotels housing housing data housing predictions housing price hy-vee Income Industry Experts Injuries Instructor Blog Instructor Interview insurance italki Job Job Placement Jobs Jon Krohn JP Morgan Chase Kaggle Kickstarter las vegas airport lasso regression Lead Data Scienctist Lead Data Scientist leaflet league linear regression Logistic Regression machine learning Maps market matplotlib Medical Research Meet the team meetup methal health miami beach movie music Napoli NBA netflix Networking neural network Neural networks New Courses NHL nlp NYC NYC Data Science nyc data science academy NYC Open Data nyc property NYCDSA NYCDSA Alumni Online Online Bootcamp Online Training Open Data painter pandas Part-time performance phoenix pollutants Portfolio Development precision measurement prediction Prework Programming public safety PwC python Python Data Analysis python machine learning python scrapy python web scraping python webscraping Python Workshop R R Data Analysis R language R Programming R Shiny r studio R Visualization R Workshop R-bloggers random forest Ranking recommendation recommendation system regression Remote remote data science bootcamp Scrapy scrapy visualization seaborn seafood type Selenium sentiment analysis sentiment classification Shiny Shiny Dashboard Spark Special Special Summer Sports statistics streaming Student Interview Student Showcase SVM Switchup Tableau teachers team team performance TensorFlow Testimonial tf-idf Top Data Science Bootcamp Top manufacturing companies Transfers tweets twitter videos visualization wallstreet wallstreetbets web scraping Weekend Course What to expect whiskey whiskeyadvocate wildfire word cloud word2vec XGBoost yelp youtube trending ZORI