House Investing: Real Estate and What Affects a House Price
The skills I demoed here can be learned through taking Data Science with Machine Learning bootcamp with NYC Data Science Academy.
Introduction
A lot of people hear about the subject of real estate and generally have an intuition on what factors affect a house's sale price. Some are based on feelings whereas others are grounded in learned domain knowledge. In short, there are so many factors to consider when thinking about the end goal of predicting a house's final price. Through this project, we wanted to take a peek into the plethora of features that can influence the sale price of a home by employing machine learning models to our given dataset.
Dataset
Our dataset is a very famous one that was procured from a Kaggle competition. It is a residential housing dataset for the city of Ames, Iowa from the years of 2006 to 2010. This dataset was composed of 79 features and 1460 observations and had a column of sales prices that we used to train our models in order to predict sales prices for the test and validation datasets.
Preprocessing
Due to the number of null values in the data, the pre-processing step was incorporating a variety of methods in order to impute these null values.
At first, we focused on the null values in our training data, but we soon realized that the test data also contained null values that needed to be filled in order to make predictions. Therefore, we merged the training data and test data; however, we later realized that this caused a phenomenon known as "data leakage" to occur. This is further explained in the conclusion.
There was a total of 34 columns that contained null values.
For each column, we considered how to fill the null value by using intuition while looking at the data descriptions.
We divided all features into four parts and roughly assigned 20 each to the group members. With each portion of the dataset, we each performed simple EDA, imputed the null values, and finally combined the fragmented features to recombine the dataset.
Relationship between Columns
Then, we analyzed the relationship between columns by separating the categorical and numerical columns of each group of features. If the column had no relationship, were categorical, and the null values themselves meant "NO," the column was filled with "NA" or "None". Otherwise, if the column was numerical, the column was filled with 0.
For example, the PoolQC column is related to the Pool Area. Since pool quality assumed a null value only when pool area had a 0 value, the NA value corresponding to "no pool" was filled.
There were six categorical columns (Electrical, MSZoning, KitchenQual, Exterior1st, Exterior2nd, Sales Type) that could not contain null values due to data characteristics. When we checked the distribution of these columns excluding null values, we could see that certain values of the features were much more prevalent. Therefore, the mode was suitable for and used in the imputation of the values of these 6 columns.
For the two columns of LotFrontage and GarageYrBlt, because they could not be imputed with simple "0"s, we checked the null value counts by analyzing their correlation to other columns.
For LotFrontage, we considered two different methods of imputation:
The first method was considering LotFrontage's relationship to LotArea.
Ratio
We calculated the ratio of the mean value of LotFrontage and LotArea excluding null values and filled the null values of LotFrontage by multiplying the value of the respective observation’s LotArea to the calculated LotRatio.
The second method was relating the neighborhoods to LotFrontage.
Because houses are typically similar in neighborhoods, we grouped the data by neighborhood and found the median value of LotFrontage. Using this median value, we imputed the null values based on an observation’s respective neighborhood.
Because we didn't know which method would be better for our output, both methods were used in the process of feature engineering to determine which outputted better results; the better imputation method ended up being method 2.
The garage year built feature was imputed by relating it to the year the house was built. After calculating the difference of year built and garage year built of each observation, we rounded this value so that we could maintain integer values for year. Then, the garage year built was imputed by adding the aveDiff to the year built value.
Exploratory Data Analysis
Before we could go into the processes of feature engineering and model fitting/tuning, we had to examine the dataset that we were given.
As can be seen by the correlation heatmap in Figure 1, there were some variables that were highly correlated to one another. Due to this presence of multicollinearity and the effects it would have in our model results, we used this figure to drop some features.
For the categorical features, we had to take into consideration whether or not the features were significant with respect to the sales price. The figure above shows the categorical features that had p-values greater than 0.05 which meant that these features were not significant.
Using the intuition gained from this statistical testing process, we were able to drop some features or come up with other approaches to manipulate them such as binarizing the feature values.
On the left side of Figure 3 shows the histograms of each of the numerical features in the dataset. However, because of the skew of some of the variables, we had to standardize them. The result of this standardization process is shown in the right side of Figure 3 with less of a skew and more of a proper normal distribution. This is further explained in the feature engineering step.
We took a look into categorical variables not only through the statistical hypothesis test of chi-squared, but also by examining the feature values through the use of violin and box plots (Figure 4). We saw that there were trends in the sales price of the home with relation to the different values of the features.
All of this data analysis was later used in the process of feature engineering.
Wireframe
Figure 1 shows the modeling process of this project. After looking into the data, we developed these steps to make the feature engineering and model fitting easier to manipulate.
After loading in the file, merging datasets, and removing outliers, we put the data through a pipeline that performed feature engineering such as null imputation and standardization. This process made feature engineering seamless as it gave an efficient tool for engineering the dataset altogether rather than performing each step one by one.
For the process of model fitting, we created a grid search that generated the best hyperparameters for the models through the use of a parameter grid. We utilized multiple linear-based regressions as well as tree-based regressions.
With these different types of regressions, a stacked model was made. The stacked regression used a support vector regressor (SVR) as its meta-regressor. Using all of these models combined, by applying weights to each model, we were able to ensemble them to generate the final predictions.
Feature Engineering
When starting this project, the general thought was that the more correctly one could tune a model, the more accurate the final price would end up being. However, after repeated trial and error, we realized that the most important aspect of this project was feature engineering.
Some of our categorical variables such as BedroomAbvGr had numerical values so we had to turn them into strings and ultimately dummify the variables using one-hot encoding.
One-hot encoding was one method used to pass in categorical features into our model; however, the other method that could account for ordinal features was to factorize them so that they could have values with magnitude that accounted for the order of the values.
For features such as FireplaceQu or KitchenQual, because of the lack of data points in some of the feature values, we had to binarize these features into 0s and 1s with the 1s being the mode of the values within the feature.
Creating new Features
We also created new features by using intuition. An example of this was summing up all of the various square footage features of the house into one aggregated value of the entire square footage of the house. We also classified the data into a new category that took into account the neighborhood.
The two variables of LotFrontage and LotArea were features that were related in the imputation process. As such, we took a look into the distributions of these features and saw that they were not normally distributed.
Because of the skew within the distributions of the features, we had to incorporate methods to transform these variables. The box-cox transformation was used to produce a lambda that would be used to transform the features to be more normally distributed. In addition, the RobustScaler from scikit-learn was used as a standardizer to negate the effects of features with values that were much greater than those of other features. RobustScaler is particularly unique because it has the ability to account for outliers.
Model Fitting
After feature engineering, the first model we implemented was a stacked ensemble model.
Model stacking combines the outputs of multiple models (called base estimators) by training a higher level model (called the meta-estimator) on their outputs. Stacked regression can be used for classification or regression and uses the results of several sub-models as an input to the meta-regressor to prevent overfitting and reduce bias.
By including the random forest regressor, the accuracy of the whole model was reduced so we decided to exclude it from the final model.
We chose SVR as a meta-regressor to output the most stable score and we ensembled this stacked regression with other models without using it immediately to produce a final output. This was done to distribute weights to other individual models.
The structure of these models was not built through one step, but through multiple iterations of trial and error.
Model Tuning
After fitting our models, we tuned our parameters.
First, we reselected and modified the features. Some examples of this was combining several features together such as the size of various aspects of the house into one feature or binarizing features with multiple feature values. Not all features created or changed through feature engineering produced good results so we found the best combination of features by including or excluding features one by one, choosing ones that led to higher scores.
Second, we found the best hyperparameters for each models. Because it takes too long to find all of the best hyperparameters at once, we found the best parameters by rotating them one by one from elastic-net to lightgbm models.
Finally, we adjusted the weights among each of the models. Stacked regression had a good result by itself, but we tried to give more weight to individual models as well. To do so, we simply multiplied the results of the model by a respective weight to produce the ensembled result.
We repeated the process of feature selection, hyperparameter tuning, weight adjustment, and scoring to ultimately get a better score.
Conclusion
After finishing tuning, we got our final score to be 0.11591, which corresponded to a rank of 640 and landed us in the top 14.4% of submissions.
Here are some important lessons we learned from this project:
- Feature engineering is much more important than model tuning. Model tuning is necessary to achieve very high scores, but it must be preceded using creative methods of feature engineering.
- Feature engineering should be done through accurate analysis such as Z-score/VIF/chi^2, not merely guesswork.
- Not all logical predictions bring good results, but they can be a good starting point.
- Some complex models like XGBoost or GBM bring good results even with small effort. However, simple models like ridge or lasso can be more apt depending on the data given.
- Some models are sensitive to feature engineering. For example, support vector regression can make an incorrect prediction if standardization is implemented incorrectly.
Data Leakage
One flaw that we unintentionally enacted from the start was data leakage. Because the test dataset contained missing values that needed to be imputed, we automatically thought to combine the datasets to provide more data in the process of imputation. However, this is not at all encouraged because although it could give better results, the fact of the matter is that we ended up making predictions on data that we used in the process of manipulating our features.
For further cases, we would avoid mixing up these datasets and only use the training data in the process of feature engineering.
Further Improvements
There is the basic approach of applying other combinations of models or hyperparameters in order to improve our score.
We could also look into post-processing techniques of higher scoring submissions and apply the methods of feature engineering into our project.
About Authors
Wonchan Kim
View all posts by Wonchan Kim >
KISOO CHO
View all posts by KISOO CHO >