Machine Learning Project: EDA and Customer Segmentation in Ames, Iowa Real Estate

Posted on Mar 13, 2023

Introduction

If you are reading this blog, you are more likely familiar with one of the most popular machine learning competitions from Kaggle, predicting house prices in Ames, Iowa. You can find countless amounts of work done on this topic online. For this particular project, I used both supervised and unsupervised methods however mostly focused on the marketing aspect by using the clustering technique. So, without further ado let’s get started!

Datasets

I used two datasets for this project. The Ames House Price dataset was collected from Kaggle.com and can be found by clicking this link

The State of Iowa Salary Book dataset was collected from the data.iowa.gov website and can be found here.

Exploratory Data Analysis with Machine Learning Methods

First, I started with Exploratory Data Analysis potentially to identify what features of the Ames housing dataset increase the sale price. For this step, I replaced Great Living Area with Total Living Area feature by simply adding 1st Floor, 2nd Floor, and Finished Basement square footage variables. Also, I engineered a new Aggregated Bathroom feature by adding Full Bathroom, Half Bathroom, Basement Bath, and Basement Half Bathroom variables. I used the code below to visualize the correlation between numerical features and Sale Price:

According to the results, these are the top five features that correlate with Sale Price:

  1. Overall quality
  2. Total Living Area square footage
  3. Total Basement square footage
  4. 1st Floor square footage
  5. Aggregated Bathroom count

Another great way to identify what variables influence the sales price, is to extract feature importance attribute from the random forest. This can be achieved by following these steps (make sure to label encode categorical features before fitting a model):

If we take a closer look at the top 10 predictive features influencing the sale price, we will find variables such as quality, square footage, bathroom count, year built, and neighborhood. As expected, these properties make residential real estate more attractive to buyers.

I visualized some of the most important features that affect the home's sales price.

House Overall Quality

Box plot is easiest way to represent the relationship between house overall quality and the price.

Overall house quality was the most important factor in correlation and feature importance attribute for numerical variables. From the box plot above, we can observe that on average the houses with the quality index 10 were sold for approximately  $380,000 to $530,000 versus houses with the quality index 1 and 2 sold for less than $100,000.

Total Living Area Square Footage

Total Living Area square footage is another important factor that can positively impact a real estate property's price value. As I mentioned before, this feature was engineered by adding the square footage of the 1st and 2nd floor along with the square footage of the basement making it the home's usable living space. It excludes square footage of garages, attics, lot areas, and unfinished basements.

According to the scatter plot above, there is a positive correlation between the total living area and sales price meaning that buyers in general are willing to pay more for the usable square footage. After running the simple linear regression in sklearn, I found out that the coefficient of determination for those two variables is 0.60 meaning 60% of the variance in sales price is predicted by the total living area.

Year Built

Another factor I thought worth mentioning is the year built. To best visualize the relationship between year built and sales price, I decided to break houses into 3 categories: 1) houses built before the 1990s and not remodeled, 2) houses built before the 1990s and remodeled, and 3) houses built after the 1990s. The density plot below represents the effect of the year built on the sales price.

According to the density plot, we find that houses that were built after the 1990s tend to sell at higher prices.

Customer Segmentation Analysis

As I mentioned above, I used the State of Iowa Salary Book dataset for customer segmentation analysis. It contains information such as the name, gender, county official title,  total salary received during a fiscal year and etc. of Iowa residents. After filtering the dataset by place of residency, I found that Ames is located in story county. It turns out that Iowa State University and the Department of Transportation are the two largest employers in story county. A fun fact is the salary paid in Ames ranges from $1,000.49 earned by students to $2,375,000.04 earned by a head coach in 2020. I narrowed down the total salary paid feature from $35,000 to $917,499.98 for this project. According to the histogram below, we find the right skewed salary distribution.

Feature Engineering

For this project, I segmented potential home buyers by utilizing the total salary paid feature. To find out "what house price can I afford", I engineered additional three variables derived from the residents' income:

  1. Monthly payments: the general rule states that an individual should spend around 28% of monthly income on a mortgage payment. Monthly payment = (Annual Income/12)*0.28
  2. Loan Amount: we make an assumption that the mortgage terms are 30 years and 6%. Loan amount = (payment*(1 - (1+rate/12)**(-months)))/(rate/12)
  3. Home Price: we make an assumption that a 20% down payment is required for a mortgage loan. Home Price = (Loan Amount*0.1)/0.08

Keep in mind that you will need to consider the other factors in the real-life situation such as property taxes, insurance, and credit score of a potential home buyer.

If we plot the total salary paid versus the home price in a scatter plot, you will find a perfect correlation. The reason why we got the clean results is that the home price was directly derived from the salary feature. In this case, the home price feature is the estimated number we use to determine affordability.

K Means Clustering

K-means clustering is one of the most popular unsupervised methods in machine learning. I used the elbow method to find an optimal number of clusters, which is graphically represented below:

According to the graph above, I found that k=5 is the optimal number for clustering meaning that we are going to segment our potential home buyers into five different groups and visualize the results.

Conclusion

Based on these results, you can build a marketing strategy by targeting a specific demographic. For instance, you can advertise the homes with a price range from $170,266.25 to $369,793.75 to the cluster 1 group by offering real estate with house quality 5 to 8, a total living square footage from 988 to 4676, and year built from 1879 to 2010.

I hope this blog was helpful. You can find the complete code for this project on GitHub.

About Author

Liliya Lopez

Dedicated professional with comprehensive skills in quantitative science, analysis, research, predictive models, and working with cross-functional teams to achieve goals. Known as an innovative thinker with statistics, data analytics, R and Python development, visualization, machine learning models, and...
View all posts by Liliya Lopez >

Related Articles

Leave a Comment

No comments found.

View Posts by Categories


Our Recent Popular Posts


View Posts by Tags

#python #trainwithnycdsa 2019 2020 Revenue 3-points agriculture air quality airbnb airline alcohol Alex Baransky algorithm alumni Alumni Interview Alumni Reviews Alumni Spotlight alumni story Alumnus ames dataset ames housing dataset apartment rent API Application artist aws bank loans beautiful soup Best Bootcamp Best Data Science 2019 Best Data Science Bootcamp Best Data Science Bootcamp 2020 Best Ranked Big Data Book Launch Book-Signing bootcamp Bootcamp Alumni Bootcamp Prep boston safety Bundles cake recipe California Cancer Research capstone car price Career Career Day citibike classic cars classpass clustering Coding Course Demo Course Report covid 19 credit credit card crime frequency crops D3.js data data analysis Data Analyst data analytics data for tripadvisor reviews data science Data Science Academy Data Science Bootcamp Data science jobs Data Science Reviews Data Scientist Data Scientist Jobs data visualization database Deep Learning Demo Day Discount disney dplyr drug data e-commerce economy employee employee burnout employer networking environment feature engineering Finance Financial Data Science fitness studio Flask flight delay gbm Get Hired ggplot2 googleVis H20 Hadoop hallmark holiday movie happiness healthcare frauds higgs boson Hiring hiring partner events Hiring Partners hotels housing housing data housing predictions housing price hy-vee Income Industry Experts Injuries Instructor Blog Instructor Interview insurance italki Job Job Placement Jobs Jon Krohn JP Morgan Chase Kaggle Kickstarter las vegas airport lasso regression Lead Data Scienctist Lead Data Scientist leaflet league linear regression Logistic Regression machine learning Maps market matplotlib Medical Research Meet the team meetup methal health miami beach movie music Napoli NBA netflix Networking neural network Neural networks New Courses NHL nlp NYC NYC Data Science nyc data science academy NYC Open Data nyc property NYCDSA NYCDSA Alumni Online Online Bootcamp Online Training Open Data painter pandas Part-time performance phoenix pollutants Portfolio Development precision measurement prediction Prework Programming public safety PwC python Python Data Analysis python machine learning python scrapy python web scraping python webscraping Python Workshop R R Data Analysis R language R Programming R Shiny r studio R Visualization R Workshop R-bloggers random forest Ranking recommendation recommendation system regression Remote remote data science bootcamp Scrapy scrapy visualization seaborn seafood type Selenium sentiment analysis sentiment classification Shiny Shiny Dashboard Spark Special Special Summer Sports statistics streaming Student Interview Student Showcase SVM Switchup Tableau teachers team team performance TensorFlow Testimonial tf-idf Top Data Science Bootcamp Top manufacturing companies Transfers tweets twitter videos visualization wallstreet wallstreetbets web scraping Weekend Course What to expect whiskey whiskeyadvocate wildfire word cloud word2vec XGBoost yelp youtube trending ZORI