Which house shall we invest in?

Avatar
Avatar
Avatar
, and
Posted on Sep 6, 2019

Introduction

A lot of people hear about the subject of real estate and generally have an intuition on what factors affect a house's sale price. Some are based on feelings whereas others are grounded in learned domain knowledge. In short, there are so many factors to consider when thinking about the end goal of predicting a house's final price. Through this project, we wanted to take a peek into the plethora of features that can influence the sale price of a home by employing machine learning models to our given dataset.

Dataset

Our dataset is a very famous one that was procured from a Kaggle competition. It is a residential housing dataset for the city of Ames, Iowa from the years of 2006 to 2010. This dataset was composed of 79 features and 1460 observations and had a column of sales prices that we used to train our models in order to predict sales prices for the test and validation datasets.

Preprocessing

Due to the number of null values in the data, the pre-processing step was incorporating a variety of methods in order to impute these null values.

At first, we focused on the null values in our training data, but we soon realized that the test data also contained null values that needed to be filled in order to make predictions. Therefore, we merged the training data and test data; however, we later realized that this caused a phenomenon known as "data leakage" to occur. This is further explained in the conclusion.

There was a total of 34 columns that contained null values.

For each column, we considered how to fill the null value by using intuition while looking at the data descriptions.

We divided all features into four parts and roughly assigned 20 each to the group members. With each portion of the dataset, we each performed simple EDA, imputed the null values, and finally combined the fragmented features to recombine the dataset.

Then, we analyzed the relationship between columns by separating the categorical and numerical columns of each group of features. If the column had no relationship, were categorical, and the null values themselves meant "NO," the column was filled with "NA" or "None". Otherwise, if the column was numerical, the column was filled with 0.

For example, the PoolQC column is related to the Pool Area. Since pool quality assumed a null value only when pool area had a 0 value, the NA value corresponding to "no pool" was filled.

There were six categorical columns (Electrical, MSZoning, KitchenQual, Exterior1st, Exterior2nd, Sales Type) that could not contain null values due to data characteristics. When we checked the distribution of these columns excluding null values, we could see that certain values of the features were much more prevalent. Therefore, the mode was suitable for and used in the imputation of the values of these 6 columns.

For the two columns of LotFrontage and GarageYrBlt, because they could not be imputed with simple "0"s, we checked the null value counts by analyzing their correlation to other columns.

For LotFrontage, we considered two different methods of imputation:

The first method was considering LotFrontage's relationship to LotArea. 

We calculated the ratio of the mean value of LotFrontage and LotArea excluding null values and filled the null values of LotFrontage by multiplying the value of the respective observation’s LotArea to the calculated LotRatio.

The second method was relating the neighborhoods to LotFrontage. 

Because houses are typically similar in neighborhoods, we grouped the data by neighborhood and found the median value of LotFrontage. Using this median value, we imputed the null values based on an observation’s respective neighborhood.

Because we didn't know which method would be better for our output, both methods were used in the process of feature engineering to determine which outputted better results; the better imputation method ended up being method 2.

The garage year built feature was imputed by relating it to the year the house was built. After calculating the difference of year built and garage year built of each observation, we rounded this value so that we could maintain integer values for year. Then, the garage year built was imputed by adding the aveDiff to the year built value.

Exploratory Data Analysis

Before we could go into the processes of feature engineering and model fitting/tuning, we had to examine the dataset that we were given.

Figure 1. Correlation heatmap of variables

As can be seen by the correlation heatmap in Figure 1, there were some variables that were highly correlated to one another. Due to this presence of multicollinearity and the effects it would have in our model results, we used this figure to drop some features.

Figure 2. Chi-squared results of categorical variables that were not significant

For the categorical features, we had to take into consideration whether or not the features were significant with respect to the sales price. The figure above shows the categorical features that had p-values greater than 0.05 which meant that these features were not significant.

Using the intuition gained from this statistical testing process, we were able to drop some features or come up with other approaches to manipulate them such as binarizing the feature values.

Figure 3. Standardization outputs for numerical values

On the left side of Figure 3 shows the histograms of each of the numerical features in the dataset. However, because of the skew of some of the variables, we had to standardize them. The result of this standardization process is shown in the right side of Figure 3 with less of a skew and more of a proper normal distribution. This is further explained in the feature engineering step.

Figure 4. Violin and box plots for various categorical variables

We took a look into categorical variables not only through the statistical hypothesis test of chi-squared, but also by examining the feature values through the use of violin and box plots (Figure 4). We saw that there were trends in the sales price of the home with relation to the different values of the features.

All of this data analysis was later used in the process of feature engineering.

Wireframe

Figure 1. Wireframe of project

Figure 1 shows the modeling process of this project. After looking into the data, we developed these steps to make the feature engineering and model fitting easier to manipulate.

After loading in the file, merging datasets, and removing outliers, we put the data through a pipeline that performed feature engineering such as null imputation and standardization. This process made feature engineering seamless as it gave an efficient tool for engineering the dataset altogether rather than performing each step one by one.

For the process of model fitting, we created a grid search that generated the best hyperparameters for the models through the use of a parameter grid.  We utilized multiple linear-based regressions as well as tree-based regressions.

With these different types of regressions, a stacked model was made. The stacked regression used a support vector regressor (SVR) as its meta-regressor. Using all of these models combined, by applying weights to each model, we were able to ensemble them to generate the final predictions.

Feature Engineering

When starting this project, the general thought was that the more correctly one could tune a model, the more accurate the final price would end up being. However, after repeated trial and error, we realized that the most important aspect of this project was feature engineering.

Some of our categorical variables such as BedroomAbvGr had numerical values so we had to turn them into strings and ultimately dummify the variables using one-hot encoding.

One-hot encoding was one method used to pass in categorical features into our model; however, the other method that could account for ordinal features was to factorize them so that they could have values with magnitude that accounted for the order of the values.

For features such as FireplaceQu or KitchenQual, because of the lack of data points in some of the feature values, we had to binarize these features into 0s and 1s with the 1s being the mode of the values within the feature. 

We also created new features by using intuition. An example of this was summing up all of the various square footage features of the house into one aggregated value of the entire square footage of the house. We also classified the data into a new category that took into account the neighborhood.

Figure 5. Standardization outputs for LotFrontage and LotArea

The two variables of LotFrontage and LotArea were features that were related in the imputation process. As such, we took a look into the distributions of these features and saw that they were not normally distributed.

Because of the skew within the distributions of the features, we had to incorporate methods to transform these variables. The box-cox transformation was used to produce a lambda that would be used to transform the features to be more normally distributed. In addition, the RobustScaler from scikit-learn was used as a standardizer to negate the effects of features with values that were much greater than those of other features. RobustScaler is particularly unique because it has the ability to account for outliers.

Model Fitting

After feature engineering, the first model we implemented was a stacked ensemble model.

Model stacking combines the outputs of multiple models (called base estimators) by training a higher level model (called the meta-estimator) on their outputs. Stacked regression can be used for classification or regression and uses the results of several sub-models as an input to the meta-regressor to prevent overfitting and reduce bias.

By including the random forest regressor, the accuracy of the whole model was reduced so we decided to exclude it from the final model.

We chose SVR as a meta-regressor to output the most stable score and we ensembled this stacked regression with other models without using it immediately to produce a final output. This was done to distribute weights to other individual models.

The structure of these models was not built through one step, but through multiple iterations of trial and error.

Model Tuning

After fitting our models, we tuned our parameters.

First, we reselected and modified the features. Some examples of this was combining several features together such as the size of various aspects of the house into one feature or binarizing features with multiple feature values. Not all features created or changed through feature engineering produced good results so we found the best combination of features by including or excluding features one by one, choosing ones that led to higher scores.

Second, we found the best hyperparameters for each models. Because it takes too long to find all of the best hyperparameters at once, we found the best parameters by rotating them one by one from elastic-net to lightgbm models.

Finally, we adjusted the weights among each of the models. Stacked regression had a good result by itself, but we tried to give more weight to individual models as well. To do so, we simply multiplied the results of the model by a respective weight to produce the ensembled result.

We repeated the process of feature selection, hyperparameter tuning, weight adjustment, and scoring to ultimately get a better score.

Conclusion

After finishing tuning, we got our final score to be 0.11591, which corresponded to a rank of 640 and landed us in the top 14.4% of submissions.

Here are some important lessons we learned from this project:

  • Feature engineering is much more important than model tuning. Model tuning is necessary to achieve very high scores, but it must be preceded using creative methods of feature engineering.
  • Feature engineering should be done through accurate analysis such as Z-score/VIF/chi^2, not merely guesswork.
  • Not all logical predictions bring good results, but they can be a good starting point.
  • Some complex models like XGBoost or GBM bring good results even with small effort. However, simple models like ridge or lasso can be more apt depending on the data given.
  • Some models are sensitive to feature engineering. For example, support vector regression can make an incorrect prediction if standardization is implemented incorrectly.

One flaw that we unintentionally enacted from the start was data leakage. Because the test dataset contained missing values that needed to be imputed, we automatically thought to combine the datasets to provide more data in the process of imputation. However, this is not at all encouraged because although it could give better results, the fact of the matter is that we ended up making predictions on data that we used in the process of manipulating our features.

For further cases, we would avoid mixing up these datasets and only use the training data in the process of feature engineering.

Further Improvements

There is the basic approach of applying other combinations of models or hyperparameters in order to improve our score. 

We could also look into post-processing techniques of higher scoring submissions and apply the methods of feature engineering into our project.

About Authors

Avatar

Wonchan Kim

Wonchan graduated from the University of California, Los Angeles with a B.S. degree in Materials Engineering and a minor in Statistics in 2019. Wonchan has proven acumen in business applications and experience tailoring data analysis and machine learning...
View all posts by Wonchan Kim >

Related Articles

Leave a Comment

No comments found.

View Posts by Categories


Our Recent Popular Posts


View Posts by Tags

2019 airbnb alumni Alumni Interview Alumni Reviews Alumni Spotlight alumni story Alumnus API Application artist aws beautiful soup Best Bootcamp Best Data Science 2019 Best Data Science Bootcamp Best Data Science Bootcamp 2020 Best Ranked Big Data Book Launch Book-Signing bootcamp Bootcamp Alumni Bootcamp Prep Bundles California Cancer Research capstone Career Career Day citibike clustering Coding Course Demo Course Report D3.js data Data Analyst data science Data Science Academy Data Science Bootcamp Data science jobs Data Science Reviews Data Scientist Data Scientist Jobs data visualization Deep Learning Demo Day Discount dplyr employer networking feature engineering Finance Financial Data Science Flask gbm Get Hired ggplot2 googleVis Hadoop higgs boson Hiring hiring partner events Hiring Partners Industry Experts Job Job Placement Jobs Jon Krohn JP Morgan Chase Kaggle Kickstarter lasso regression Lead Data Scienctist Lead Data Scientist leaflet linear regression Logistic Regression machine learning Maps matplotlib Medical Research meetup Networking neural network Neural networks New Courses nlp NYC NYC Data Science nyc data science academy NYC Open Data NYCDSA NYCDSA Alumni Online Open Data painter pandas Part-time Portfolio Development prediction Prework Programming PwC python python machine learning python scrapy python web scraping python webscraping Python Workshop R R language R Programming R Shiny r studio R Visualization R Workshop R-bloggers random forest Ranking recommendation recommendation system regression Remote remote data science bootcamp Scrapy scrapy visualization seaborn Selenium sentiment analysis Shiny Shiny Dashboard Spark Special Special Summer Sports statistics streaming Student Interview Student Showcase SVM Switchup Tableau team TensorFlow Testimonial tf-idf Top Data Science Bootcamp twitter visualization web scraping Weekend Course What to expect word cloud word2vec XGBoost yelp