Data Science Applied to House Data
Data Science Problem Background
Data Science is wildly adapted in real estate industry.
When it comes time to sell a house, many homeowners try to renovate their properties and improve it based on what they believe can add the most value to their houses. However, it is often difficult to estimate the price of a house by a given set of its features.
Here, using a high dimensional dataset from Kaggle that includes 77 features of a residential house other than just the number of rooms and square footage, we develop a predictive model for the sale price of houses by utilizing advanced regression techniques. In addition, we determine what are the most important features that drive the house value.
The dataset contains 79 features describing 1460 individual houses in Ames, Iowa between the years 2006 and 2010. This dataset is part of one of the most popular contests on Kaggle, involving the usage of advanced regression techniques to predict housing prices.
Data Pre-Processing:
Before starting the data analysis or modeling, we need to pre-process our data. That involves the following steps:
Train-Test split: First, we split the dataset into training and testing sets (0.7,0.3) to make sure data preprocessing is performed separately and the model is only trained on the training set and doesn’t have access to the test set.
Missing values and imputation: We examine the missing values in our dataset. Here we take a look at the before train-test split to have a more general understanding of our dataset.
Missing values in our dataset are shown in Fig.1. There are few features that have a very high proportion of missing values. As an example, if we consider the pool quality (PoolQC), the reason behind the high missing value number can be that those houses didn’t have any pool to be evaluated.
The same case can be applied to fence quality (Fence) or Type of alley access to property (Alley). With that being said, we used the following steps for missing values imputation:
- Analyze the variables to determine if some of the missing values may have a meaning of their own. For example, the missing values in the feature PoolQC probably indicate the fact that those houses have no pool. The same can be said for the garage and basement variables indicating that those houses have no basement or garage. In all these variables, 'NA' gets imputed.
- For numeric variables the median was imputed
- In the case of LotFrontage (Linear feet of street connected to property) and MasOnry Area (Masonry veneer area in square feet), the data was first grouped by neighborhoods, and then the mean of each neighborhood was imputed.
- For the remaining categorical variable, MasVnrType (Masonry veneer type) and Electrical, we imputed the mode of the variable.
We should mention that imputation has been performed on train and test set separately to prevent any data leakage between two datasets.
Fig.1 Missing value counts and imputation methods
Exploratory Data Analysis and Processing
In this step, we investigate the correlation between different (numerical/ordinal) features in our dataset (Fig 2.). As expected in such a high dimensional dataset, we see strong correlations between some of the features that need to be remedied before we proceed to the modeling. As an example, garage year built (GarageYrBlt) is highly correlated with house year built (YearBuilt), and garage area (GarageArea) is highly correlated with number of cars that can fit in (GarageCars).
The number of rooms in the house is highly correlated with the general living Area (GrLivArea).
Fig.2. Correlation matrix for dataset features
To explore our dataset more, we also analyze the correlation between the target variable (sale price) and our features and focus our analysis on the features that have the strongest correlation with the target variable (Fig.3).
Fig. 3. Correlation coefficients of different features with target variable (sale price)
The plots of SalePrice versus OveralQual (Overall quality) and GarageCars (number of cars) are shown in Fig4. The increasing trends are obvious in these plots. There is only one house with the capacity for four cars in our train set.
Fig. 4 Sale price Vs. Overall quality of the house (OverallQual) and garage space (Garage Cars)
It would also be interesting to see the plot of GrLivArea (general living area) with SalePrice (Fig. 5) since they are highly correlated. There are two points in our train datasets that seem to be outliers. They were the two biggest houses in the dataset, built in 2007 and 2008, that have the highest possible scores on Overall Quality and partial sales.
These houses were probably speculative home builds from the peak of the housing bubble. As the market started to plummet, the sale price came to reflect the neighborhood more than the size or condition of the home. After testing the models with and without these outliers, it was found that they skewed the results, so we decided to remove them from consideration.
Fig. 5
Since we are planning to implement regularized linear models, we also need to check our linear assumptions. The target variable, SalePrice, did not pass the initial test of normality. After applying a log transformation (Fig. 6), our data passed all assumptions.
Fig.6 Q-Q plots of target variable before (left) and after (right) applying the log transformation
Machine Learning Models With Data
Lasso
We implemented several machine learning models for different purposes, starting with Lasso for the purpose of empirical feature selection. In the next step, we created three predictive models, one linear model –Elastic Net– and two advanced non-linear models using Random Forest and Gradient Boosting. Lasso favors less complicated models, implementing a penalty term on model coefficients, which gradually approach zero as the penalty term increases.
By choosing the appropriate penalty strength decided by the hyperparameter lambda, certain model coefficients would be sent to zero while others remained non-zero, and predictors highly correlated with other predictors would have their overall impact regulated.
By implementing the grid search with cross-validation, we selected the Lasso model that fit the dataset well without overfitting which is shown at the intersection between the validation score and train score in Fig. 7.
Fig.7
This model reduced the number of predictors from the original 243 (including dummy variables) down to 19 (for categorical features that had nonzero coefficients for example CentralAir, we also added the other dummified columns). Among them, numerical variables that showed a high correlation with SalePrice, such as OveralQual, GarageCars, and Above GrLivArea, were included, some categorical features such as MSZoning (general zoning classification such as commercial, industrial, etc.) and CentralAir (weather it has central air conditioning or not) were also included (Fig 8).
Fig. 8 Lasso model coefficients. Actual value (left) and absolute value (right)
Elastic Net
In this step, by having the features selected from the Lasso model, we ran an Elastic Net model to predict Sale Price. Using grid search and cross-validation, we were able to find the parameters that fit well without overfitting. Our parameters for the best model were Lambda = 0.007 and L1 Ratio = 0.
It is worth noting here that since our best L1 Ratio for Elastic Net was 0.0, it ended up behaving just like a Ridge model; the grid search on Elastic Net allowed us to test the whole range of L1 Ratios before deciding what would be the optimal weights of Ridge and Lasso in Elastic Net. We will compare the scores in the following sections.
Nonlinear models
We selected random forest and gradient boosting as our nonlinear predictive models since they are well-tested ensemble models which are generally robust to overfitting. Then we implemented a grid search algorithm for both of these models to find the best parameters. We observed that these nonlinear models performed very similar to our penalized linear model but with slightly lower performance. The details of scores and parameters of all predictive models are summarized in Fig. 9.
Fig. 9
In Random Forest (or Gradient Boosting) we also have the advantage of examining the importance of different features in the model. As can be seen in Fig. 10, Overall Quality, General Living area and the age of the house are the three most important features in predicting the Sale Price.
Fig.10 Random Forest model’s feature importance
It’s worth mentioning here that these features importance scores are an indicator of each feature contribution toward the model and don’t necessarily correlate with each feature's direct impact on the price of the house. To define which features drive the value of the house, we should analyze the coefficients of the developed linear models. Having a look at Fig. 8, we realize that when deciding which renovations to make, a homeowner in Ames, Iowa might choose from the list of features in Fig.8.
We should mention that some of those features can’t be improved by a homeowner (for example year built), but some others can. For example, it might be difficult to increase the finished percentage of their basement, but we found that doing so would have a great impact on value. For other renovation ideas, they could install central air conditioning or add a fireplace.
Conclusion
In summary, we tried to answer two main questions in this project: 1. Which model better predicts the house price of the given dataset? 2. What are the most tunable features to add value to a home? Our analysis showed that a regularized linear model (Elastic Net) makes better predictions than a tree-based model (Random Forest or gradient boosting), and we were able to get a list of features ranked by value importance for homeowners looking to add value to their property with renovations.
The skills I demoed here can be learned through taking Data Science with Machine Learning bootcamp with NYC Data Science Academy.
