Exploratory Data Analysis of Real Estate Listings in Manhattan

Brian Perez Joseph
Posted on Feb 16, 2021

Overview of Project

Using Python's Scrapy package, the goal of this project was to conduct an exploratory data analysis of on residential real estate listings in Manhattan. This analysis aims to answer three questions regarding the borough's real-estate listings:

  • What are some of the general trends in real-estate listings in Manhattan?
  • What factors can affect the price of a Manhattan listing?
  • How affordable and financially feasible is purchasing resdiential real-estate for the average New Yorker?

The data for this project was scrapped from Realtor.com and consists of 1663 individual listings containing information on the street address, price, number of bath & bedrooms, property type, and square footage.

General Trends in Manhattan Listings

Figure 1: Bar plot of the total number of property listings. The plot counts the number of listings with the given property type.

The bar plot above shows the total number of property types in the data set. Overall, the two predominant property types in Manhattan appear to be Co-ops and Condos. Townhouses, Single Family Homes, Multi Family Homes, and hybrid properties collectively account for less than 10% of the total number of listings.

Figure 2: A set of boxplots showing the distribution of the log-scaled listing prices for each property type

The set of boxplots above depicts the listing prices for residencies across several property types. The prices were transformed on a logarithmic scale in order to better visualize the distributions. Townhouses have the largest overall listing price while condos and co-ops have the lowest overall prices.

Closer Examination of Co-Ops & Condos

Because Co-Ops and Condos comprised a large majority of the Manhattan real-estate listings, these two property types were further focused on for further analysis.

Figure 3: The following two histograms show the distribution of the total number of rooms in a listing between condos and co-ops

The following two histograms show the distribution of the total number of rooms in a listing between condos and co-ops. Overall the number of rooms clusters around 2-4 total rooms.

Figure 4: Boxplots of the distribution of square footage in Co-ops and Condos

The boxplots above show the distribution of square footages among the Co-Op and condo property types. Overall the distributions between the two groups is largely similar with most listings ranging between 900 to 2000 square feet.  

Evaluating Factors Correlated with Listing Price

In addition to evaluating some of the trends regarding Manhattan listings, this analysis also examined which housing features showed strong correlations with the listing price. A Pearson correlation test was used to evaluate the correlation among each of the variables. The two variables with the highest correlations were square footage (r = 0.807) and the total number of rooms (r = 0.744).

Figure 5: A scatter plot of square footage and log-scaled listing prices with a trend line. The shade region represents the error associated with the trend line.

The following graph is a plot of square footage against listing price modelled also with a trend line. The regression line shows a strong correlation near the start, but begins to show significant errors as the square footage increases

Figure 6: A scatter plot of total number of rooms and log-scaled listing prices with a trend line. The shade region represents the error associated with the trend line.

The following graph depicts the total number of rooms in a listing plotted against the listing price. A linear model was used to visualize the relationship between room number and price. From this graph, it appears that there is a positive correlation between the number of rooms and listing price.

Finances of Purchasing Manhattan Real-Estate

Figure 7: Boxplot of the distribution of the potential mortgage rates for Manhattan listings

The pair of boxplots above show the distributions of monthly mortgages. The mortgages were calculated by finding the difference between the listing price and down payment dividing three hundred and sixty, which is the number of months in thirty years. The red plot shows the monthly mortgage after a twenty percent down payment, and the blue plot shows the monthly mortgage after a five percent down payment.

Figure 8: Boxplot of the recommended salaries for Manhattan listings

The boxplots above show the distribution of recommended yearly salaries based on the potential monthly mortgage of a listing. The recommended salary was determined by calculating a salary in which 30 percent of the annual income goes towards the monthly mortgage payments. The red plot shows the distribution of salaries if a listing had a twenty percent down payment, and the blue plot shows the distribution of salaries if a listing had a five percent down payment. The horizontal line below the twenty-fifth percentile of each plot is the median household income of Manhattan residents in 2018, which was about $60,000.

Conclusions & Future Steps

From the scrapped real-estate data, the following patterns observed in Manhattan residential real-estate:

  • A large portion of listings consist mainly of Co-Op and Condos
  • Some overlaps in the prices and size of Co-Op and Condo listings
  • Very few large residential listings in Manhattan
  • Square footage and the number of rooms were positively correlated with the price of a listing
  • Most of the residential real-estate listings are potentially unaffordable to the average New York City resident

The data gathered for this analysis represents less than a third of the total number of Manhattan listings available on Realtor; however, due to time-constraints and technical issues, the analysis was conducted with just the data on hand, so the estimates do not offer a full & comprehensive overview on Manhattan residential real-estate. Furthermore, the data do not indicate how prices have shifted as a result of the COVID-19 pandemic, and the data also do not look towards prices in areas surrounding Manhattan like New Jersey, Upstate New York, and Queens. Therefore, a future iteration of the project would broaden the scope real-estate listings & make comparisons between historical pricing data.

About Author

Brian Perez Joseph

Brian Perez Joseph

With a background in biomedical research and data science, Brian aims to utilize his quantitative background in the sciences and data programming skills to provide data-driven decision making strategies and key insights for real-world business problems.
View all posts by Brian Perez Joseph >

Leave a Comment

No comments found.

View Posts by Categories


Our Recent Popular Posts


View Posts by Tags

#python #trainwithnycdsa 2019 airbnb Alex Baransky alumni Alumni Interview Alumni Reviews Alumni Spotlight alumni story Alumnus API Application artist aws beautiful soup Best Bootcamp Best Data Science 2019 Best Data Science Bootcamp Best Data Science Bootcamp 2020 Best Ranked Big Data Book Launch Book-Signing bootcamp Bootcamp Alumni Bootcamp Prep Bundles California Cancer Research capstone Career Career Day citibike clustering Coding Course Demo Course Report D3.js data Data Analyst data science Data Science Academy Data Science Bootcamp Data science jobs Data Science Reviews Data Scientist Data Scientist Jobs data visualization Deep Learning Demo Day Discount dplyr employer networking feature engineering Finance Financial Data Science Flask gbm Get Hired ggplot2 googleVis Hadoop higgs boson Hiring hiring partner events Hiring Partners Industry Experts Instructor Blog Instructor Interview Job Job Placement Jobs Jon Krohn JP Morgan Chase Kaggle Kickstarter lasso regression Lead Data Scienctist Lead Data Scientist leaflet linear regression Logistic Regression machine learning Maps matplotlib Medical Research Meet the team meetup Networking neural network Neural networks New Courses nlp NYC NYC Data Science nyc data science academy NYC Open Data NYCDSA NYCDSA Alumni Online Online Bootcamp Online Training Open Data painter pandas Part-time Portfolio Development prediction Prework Programming PwC python python machine learning python scrapy python web scraping python webscraping Python Workshop R R language R Programming R Shiny r studio R Visualization R Workshop R-bloggers random forest Ranking recommendation recommendation system regression Remote remote data science bootcamp Scrapy scrapy visualization seaborn Selenium sentiment analysis Shiny Shiny Dashboard Spark Special Special Summer Sports statistics streaming Student Interview Student Showcase SVM Switchup Tableau team TensorFlow Testimonial tf-idf Top Data Science Bootcamp twitter visualization web scraping Weekend Course What to expect word cloud word2vec XGBoost yelp