Exploratory Data Analysis of Real Estate in Manhattan

Posted on Feb 16, 2021
The skills I demoed here can be learned through taking Data Science with Machine Learning bootcamp with NYC Data Science Academy.

Exploratory Data Analysis of Real Estate in Manhattan

Overview of Project

Using Python's Scrapy package, the goal of this project was to conduct an exploratory data analysis of on residential real estate listings in Manhattan. This analysis aims to answer three questions regarding the borough's real-estate listings:

  • What are some of the general trends in real-estate listings in Manhattan?
  • What factors can affect the price of a Manhattan listing?
  • How affordable and financially feasible is purchasing residential real-estate for the average New Yorker?

The data for this project was scrapped from Realtor.com and consists of 1663 individual listings containing information on the street address, price, number of bath & bedrooms, property type, and square footage.

General Dara Trends in Manhattan Listings

Exploratory Data Analysis of Real Estate in Manhattan
Figure 1: Bar plot of the total number of property listings. The plot counts the number of listings with the given property type.

The bar plot above shows the total number of property types in the data set. Overall, the two predominant property types in Manhattan appear to be Co-ops and Condos. Townhouses, Single Family Homes, Multi Family Homes, and hybrid properties collectively account for less than 10% of the total number of listings.

Exploratory Data Analysis of Real Estate in Manhattan
Figure 2: A set of boxplots showing the distribution of the log-scaled listing prices for each property type

The set of boxplots above depicts the listing prices for residencies across several property types. The prices were transformed on a logarithmic scale in order to better visualize the distributions. Townhouses have the largest overall listing price while condos and co-ops have the lowest overall prices.

Closer Examination of Co-Ops & Condos Data

Because Co-Ops and Condos comprised a large majority of the Manhattan real-estate listings, these two property types were further focused on for further analysis.

Exploratory Data Analysis of Real Estate in Manhattan
Figure 3: The following two histograms show the distribution of the total number of rooms in a listing between condos and co-ops

The following two histograms show the distribution of the total number of rooms in a listing between condos and co-ops. Overall the number of rooms clusters around 2-4 total rooms.

Figure 4: Boxplots of the distribution of square footage in Co-ops and Condos

The boxplots above show the distribution of square footages among the Co-Op and condo property types. Overall the distributions between the two groups is largely similar with most listings ranging between 900 to 2000 square feet.  

Evaluating Data on Factors Correlated with Listing Price 

In addition to evaluating some of the trends regarding Manhattan listings, this analysis also examined which housing features showed strong correlations with the listing price. A Pearson correlation test was used to evaluate the correlation among each of the variables. The two variables with the highest correlations were square footage (r = 0.807) and the total number of rooms (r = 0.744).

Figure 5: A scatter plot of square footage and log-scaled listing prices with a trend line. The shade region represents the error associated with the trend line.

The following graph is a plot of square footage against listing price modelled also with a trend line. The regression line shows a strong correlation near the start, but begins to show significant errors as the square footage increases

Figure 6: A scatter plot of total number of rooms and log-scaled listing prices with a trend line. The shade region represents the error associated with the trend line.

The following graph depicts the total number of rooms in a listing plotted against the listing price. A linear model was used to visualize the relationship between room number and price. From this graph, it appears that there is a positive correlation between the number of rooms and listing price.

Finances of Purchasing Manhattan Real-Estate

Figure 7: Boxplot of the distribution of the potential mortgage rates for Manhattan listings

The pair of boxplots above show the distributions of monthly mortgages. The mortgages were calculated by finding the difference between the listing price and down payment dividing three hundred and sixty, which is the number of months in thirty years. The red plot shows the monthly mortgage after a twenty percent down payment, and the blue plot shows the monthly mortgage after a five percent down payment.

Figure 8: Boxplot of the recommended salaries for Manhattan listings

The boxplots above show the distribution of recommended yearly salaries based on the potential monthly mortgage of a listing. The recommended salary was determined by calculating a salary in which 30 percent of the annual income goes towards the monthly mortgage payments. The red plot shows the distribution of salaries if a listing had a twenty percent down payment, and the blue plot shows the distribution of salaries if a listing had a five percent down payment. The horizontal line below the twenty-fifth percentile of each plot is the median household income of Manhattan residents in 2018, which was about $60,000.

Conclusions & Future Steps

From the scrapped real-estate data, the following patterns observed in Manhattan residential real-estate:

  • A large portion of listings consist mainly of Co-Op and Condos
  • Some overlaps in the prices and size of Co-Op and Condo listings
  • Very few large residential listings in Manhattan
  • Square footage and the number of rooms were positively correlated with the price of a listing
  • Most of the residential real-estate listings are potentially unaffordable to the average New York City resident

The data gathered for this analysis represents less than a third of the total number of Manhattan listings available on Realtor; however, due to time-constraints and technical issues, the analysis was conducted with just the data on hand, so the estimates do not offer a full & comprehensive overview on Manhattan residential real-estate. Furthermore, the data do not indicate how prices have shifted as a result of the COVID-19 pandemic, and the data also do not look towards prices in areas surrounding Manhattan like New Jersey, Upstate New York, and Queens. Therefore, a future iteration of the project would broaden the scope real-estate listings & make comparisons between historical pricing data.

About Author

Brian Perez Joseph

With a background in biomedical research and data science, Brian aims to utilize his quantitative background in the sciences and data programming skills to provide data-driven decision making strategies and key insights for real-world business problems.
View all posts by Brian Perez Joseph >

Leave a Comment

No comments found.

View Posts by Categories


Our Recent Popular Posts


View Posts by Tags

#python #trainwithnycdsa 2019 2020 Revenue 3-points agriculture air quality airbnb airline alcohol Alex Baransky algorithm alumni Alumni Interview Alumni Reviews Alumni Spotlight alumni story Alumnus ames dataset ames housing dataset apartment rent API Application artist aws bank loans beautiful soup Best Bootcamp Best Data Science 2019 Best Data Science Bootcamp Best Data Science Bootcamp 2020 Best Ranked Big Data Book Launch Book-Signing bootcamp Bootcamp Alumni Bootcamp Prep boston safety Bundles cake recipe California Cancer Research capstone car price Career Career Day citibike classic cars classpass clustering Coding Course Demo Course Report covid 19 credit credit card crime frequency crops D3.js data data analysis Data Analyst data analytics data for tripadvisor reviews data science Data Science Academy Data Science Bootcamp Data science jobs Data Science Reviews Data Scientist Data Scientist Jobs data visualization database Deep Learning Demo Day Discount disney dplyr drug data e-commerce economy employee employee burnout employer networking environment feature engineering Finance Financial Data Science fitness studio Flask flight delay gbm Get Hired ggplot2 googleVis H20 Hadoop hallmark holiday movie happiness healthcare frauds higgs boson Hiring hiring partner events Hiring Partners hotels housing housing data housing predictions housing price hy-vee Income Industry Experts Injuries Instructor Blog Instructor Interview insurance italki Job Job Placement Jobs Jon Krohn JP Morgan Chase Kaggle Kickstarter las vegas airport lasso regression Lead Data Scienctist Lead Data Scientist leaflet league linear regression Logistic Regression machine learning Maps market matplotlib Medical Research Meet the team meetup methal health miami beach movie music Napoli NBA netflix Networking neural network Neural networks New Courses NHL nlp NYC NYC Data Science nyc data science academy NYC Open Data nyc property NYCDSA NYCDSA Alumni Online Online Bootcamp Online Training Open Data painter pandas Part-time performance phoenix pollutants Portfolio Development precision measurement prediction Prework Programming public safety PwC python Python Data Analysis python machine learning python scrapy python web scraping python webscraping Python Workshop R R Data Analysis R language R Programming R Shiny r studio R Visualization R Workshop R-bloggers random forest Ranking recommendation recommendation system regression Remote remote data science bootcamp Scrapy scrapy visualization seaborn seafood type Selenium sentiment analysis sentiment classification Shiny Shiny Dashboard Spark Special Special Summer Sports statistics streaming Student Interview Student Showcase SVM Switchup Tableau teachers team team performance TensorFlow Testimonial tf-idf Top Data Science Bootcamp Top manufacturing companies Transfers tweets twitter videos visualization wallstreet wallstreetbets web scraping Weekend Course What to expect whiskey whiskeyadvocate wildfire word cloud word2vec XGBoost yelp youtube trending ZORI