The path to achieving the best performing ML model for the Ames Housing Data set

Introduction

Ames, a college town in Iowa, offers a rich housing data set, meticulously collected by Dean De Cock, a professor of Statistics at Truman State University. Initially intended as a base for his student’s projects, this data set has since grown in popularity, now serving as a benchmark for budding data analysts.The success of our approach is measured by the results of the methodology and results of our machine learning predicitons 

The Data

The Ames Housing Dataset, courtesy of NYCDSA, is a comprehensive 1mb CSV file containing 82 features like 'Lot Size' for approximately 2,600 homes sold between January 2006 and July 2010.

Insights from Peers and Past Works

I extensively reviewed various published works that proved instrumental in shaping my approach. Key among them were:

• Data Is In The Eye Of The Beholder: This piece underscored the power of CatBoost and the value of retaining original features. These insights influenced my decision to use CatBoost in my models and to reconsider my feature engineering approach.
• Comprehensive data exploration with Python: Although scientifically rigorous, this manual lacked a degree of industry knowledge. Nonetheless, it solidified my understanding of data exploration techniques, which I applied throughout my project.
• Maximizing Home flipping profits using ML Techniques: This work had the best R2 performance I’ve seen within our program and served as the benchmark to build on. 

My goal is to learn from my peers, deepen my understanding of the subject matter, and contribute valuable insights to improving ML models. 

Exploratory Data Analysis (EDA)

Understanding both the data and the machine learning models is essential in building the most effective models. My curiosity is what drives me to understand the data.. With each insight came a new question. 

In the interest of brevity, and to keep the focus on the machine learning performance, I will highlight key findings from my EDA that directly influenced my modeling decisions. 

EDA Insight 1: Curiosity Fuels Discovery

Several peers used this data to advise on the optimal times to buy and sell houses, adhering to the strategy of 'buy the dip, sell the tip.' This was primarily achieved through Seasonal Trend Analysis. I engineered code to achieve the same results, and in the process came to a realization more important than seasonality: this dataset lacks essential information real estate agents rely on when valuing a property.

Real estate agents use data such as the date  the property was listed, the number of days on the market, the listing date, the number of bids, and the list price. From this standpoint alone, I realized a perfect score would not be possible, as essential data was missing. Realizing this omission and seeing the prediction performance of my peers, I calibrated my standards to achieve a 95% R2. 

It’s worth pointing out that the month a house is sold is often not the month when the transaction concludes. In 2023, 76.1% of new home purchases were financed, with the closing process taking 50 days on average. Furthermore, the average time a property stays on the market varies between 61 and 88 days, depending on the market conditions. So, if you aim to strategize when to start house hunting or listing a property, you should begin at least 110 days ahead. Data scientists using this dataset should be aware that recommendations for buy and sell date should be adjusted accordingly. 

EDA Insight 2: Single-Family Homes Take the Lead

Via EDA, we discovered most of the properties are single-family homes

Different types of homes, like townhouses or apartments, tend to be priced differently. Mixing these different types together in the data might make our predictions less accurate because the model might get confused by the variety of prices and house types.

So, to make our price predictions better, we should look more into property types, zoning, and other features so we create a ‘portfolio’ of houses with similar attributes. This way, our model acquire a higher level of precision in learning about the specific type of houses for which we are most interested in predicting prices.

EDA Insight 3: The Sales Type Matters

One part of the data that caught my attention was the extreme of the lowest sale prices; $12,789 struck me as too low a price for any home.

Digging deeper, I found out that this $12,789 property was classified as an 'Abnormal Sale.'

When we look at the distribution of sale conditions, we find that 'Normal' sales make up the lion's share, almost 94% of all sales. The other types of sales aren't very useful for predicting house prices. Why? Because those special situations don't usually need a predicted value.

Take 'Partial' sales, for instance. These are typically new houses sold before they're completed, often at a discount compared to their final value. Using data from these properties might confuse our machine learning models and make the predictions less accurate.

To make sure our model learns the most relevant patterns, we should focus on 'Normal' sales where the conditions of sale are typical and regular.

Feature Engineering: It's All in the Details

'Features' are just pieces of information that help us guess the answer we're looking for; In this case, it’s  the sale price of a house. 

Selecting the right model for the project requires considering the number of features involved.  While certain models can handle a large number of features, some only perform well when working with a  more limited set of features.

How we choose and handle these features can vary greatly from one data scientist to another.

Sometimes, we might decide to take out certain features if we think they won't help the model make good predictions. For example, we might decide to remove the 'Porch Type' feature if we believe it's not very important for determining the house price.

In other cases, we might combine features to simplify our data and speed up the process. For instance, we could combine the 'Year' and 'Date' columns into a single feature.

After looking at the work of others, especially Vinod's project, I decided to keep all the original features.

One thing I wanted to experiment with was grouping neighborhoods together based on similar characteristics. For example, not many houses were sold in the 'Green' neighborhood. Could we pair this neighborhood with another one that has similar house prices and types?

This idea comes from the real estate industry, where agents use 'comparables' or 'comps' to set house prices. Comps are recently sold properties that are similar to the one you're selling - in the same neighborhood, similar size, condition, and sold around the same time.

This approach mimics what real estate agents do when they price a house. They choose comps that closely match the house they're selling. If we could predict a house's value accurately and use that prediction as a feature, we might improve our model's accuracy. Applying   what agents do when they use comps could be a powerful tool for our model as well.

Below is a look at the closest I could get to creating a ‘comp’ price. However, this didn’t end up providing any value in my models. (Worth noting, I also tried using KNN for to identify most similar properties)

Machine Learning (ML)

During my research, I studied various models and decided on the ones I wanted to use. Each model requires different kinds of data preparation, so I will need to process the data differently for each model.

Here's the game plan for improving the model: I'm going to test each model's performance at every step of the data preparation process.

I'll provide more detail about each step as we go along, but here's the broad strokes:

1. Encoding: Some models like Catboost only need this step and can handle the rest of the data preparation that other models usually require.
2. Encoding and Handling Missing Values: These two steps will allow us to use the Random Forest model.
3. Encoding, Handling Missing Values, and Scaling: With these three steps completed, we can use all the models we planned to test.

ML Step 1: Encoding

A crucial first step in many machine learning models is "encoding". Essentially, encoding is a way of turning words into numbers so that the model can use them. For instance, in our dataset, we have a 'Street' column with two categories: 'Pave' and 'Gravel'. Through encoding, we change these categories into numerical values that the model can understand and process.

There are many ways to go about encoding. The type of encoding you choose can significantly affect your model's performance. For example, suppose we've assigned 'Pave' a 0 and 'Gravel' a 1, but in reality, paved properties are worth ten times as much as gravel ones. In that case, the model might not fully grasp the real-world impact of these categories on the house prices. A method like Target Encoding might help in such a scenario. This method assigns each category the average value of the target variable (in our case, the house price) for that category. However, this method could also lead to problems in certain cases, and the most advanced approach might involve using different encoding methods for different features.

For this project, I tested twelve encoding options and chose the best-performing one rather than tailor-picking an encoding method for each feature. This was a choice made in the interest of efficiency.

Another essential thing to keep in mind is the order of operations. You should split your data before you start encoding. That's because encoding is a common source of "data leakage". This happens when your model ends up learning from data it isn't supposed to see yet, such as future data.  Eata leakage can lead to an overly optimistic performance on your model, fooling you into thinking it'll perform just as well on new, unseen data.

After encoding the data, I tested various models and evaluated them based on their Mean Absolute Error (MAE) and R-squared (R2) on the test set. All models performed remarkably well, with none scoring lower than an R2 of 0.89. One model, Catboost with Target Encoding, achieved an impressive R2 of 0.9429, indicating that the features explain 94.29% of the house price variability. An R2 of 100% would mean perfect predictions every time.

In retrospect, using the Mean Absolute Percentage Error (MAPE) might have been beneficial, as it provides a clearer picture of the differences between the predicted and actual values in percentage terms.

ML 2: Handling Missing Values

Before we can properly use Scalers in machine learning, we have to deal with missing values, which often show up as 'NaN' - short for 'Not a Number'.

Understanding why these missing values are present requires a close look at the data. Let's take a couple of examples from our dataset. 'Lot Frontage' represents the amount of street connected to a property. When this information is missing, it could mean that the property, like an apartment, doesn't connect to the street at all. In these cases, we would fill this missing value with a '0'.

Similarly, 'Pool QC' stands for Pool Quality. A lot of these values are missing because most houses simply don't have pools. So, when the Pool QC information is not there, it's probably because there's no pool on the property.

In other words, the strategy for filling in missing values often depends on why the data is missing in the first place. Properly handling these missing values is a crucial step towards building a reliable machine learning model.

ML 3: Scaling and Encoding 

With encoding and handling of missing values complete, we can now use a scaler. This step will also allow us to include more models in our testing.

Just as with encoders, we need to take care to prevent data leakage when scaling.

I tried out six different scalers and added four more models to our lineup. This brings our total combinations of encoders, scalers, and models to a whopping 576. In the interest of space, I've only shown three encoder rows below, though there are nine more. Each model's performance is evaluated based on its R-squared (R2) score.

The Catboost model, a previous top performer, continued to do well. However, its performance was slightly lower with Target Encoding after scaling and handling missing values. With these steps complete, Helmert Encoding used with a Power Transformer scaler gave us the best performance, achieving an R2 score of 94.14%, which is still slightly less than our encoded only data set.

You’ll notice SVR performs negatively. There were additional steps that could be taken to significantly improve the model. However, considering the performance of other models, I decided not to invest in improving SVR.

ML 4: Grid Search with 5-Fold Cross-Validation

We're now at a stage where we can run a grid search for the Catboost model. This process involves tuning the model's hyperparameters, which are like the model's settings or knobs we can turn for better performance. It's similar to tweaking a car's settings to get more horsepower.

When we say we're "Fitting 5 folds for each of 135 candidates, totalling 675 fits," we're talking about the process of training and evaluating the model. The "5 folds" refers to the technique of 5-fold cross-validation, where we divide the data into five parts called folds. The model is then trained and evaluated five times, each time using a different fold as the test set and the remaining four folds as the training set. The "135 candidates" are the different combinations of hyperparameters we're trying out. The total of 675 fits comes from the number of folds multiplied by the number of hyperparameter combinations.

With this process, we achieved a best-score of 94.55%.

ML 5: Data Filtering

After identifying the best combination of model, encoder, and scaler, the next stage in our machine learning pipeline is to filter the data. This process involved testing over 100 different combinations of filtered datasets, and here, only a fraction of those tests are represented.

The data presented below is just a small subset of the entire range of trials. As we can see, we managed to achieve an even more improved R2 score of 95.52% and a Root Mean Squared Error (RMSE) of 15082, which outperforms the best scores we've seen so far.

The 'num_rows' column indicates how each dataset was filtered to remove certain properties. 'data_raw' represents the unfiltered dataset with the full 2580 properties. In contrast, the best-performing dataset, 'normal_sale_rl', contains only 1892 properties, which is approximately 700 less than the original count.

ML 6: Feature Engineering

Here we test another 50 or so combinations of normal_sale_rl. I’m only showing a bit of what we did, but the main point is that these changes helped us increase the R2.

Each combination set uses the normal_sale or normal_sale_rl as a base, and layers feature engineering on top.For example, in the top performing normale_sale_rl_cond, I concatenate all condition columns, and remove all the original condition columns. 

Conclusion

All told, we tested 12 Encoder, 6 Scalers, 8 Models, and to identify the top performing model with this data set = 576 combinations. We then performed a grid serach to get the best parameters. Finally, we filtered the data and performed feature engineering.

After conducting this research, we found that certain strategies yielded the best results we've seen so far:

1. Studying the work of other researchers in the field gave us a baseline to work off.
2. Retaining all data columns gave better results than removing some.
3. Testing 12 different data encoding methods led to a 94.29% R2 score with CatBoost.
4. Handling missing data correctly, then testing six scalers for each of the twelve encoders (without data leakage) across nine regression models resulted in a 94.14% R2 score using CatBoost.
5. Grid Search with 5-Fold Cross Validation on CatBoost improved our R2 score o 94.55%.
6. Applying the best parameters found and filtering data sets based on Exploratory Data Analysis (EDA) findings gave us an R2 score of 95.55%.
7. Layering feature engineering on top of number 6 led us to our highest score of 95.86 R2.

Future Work

There's still room for improvement and further exploration in the future. We will:

• Identify the most suitable scaler for each column: Different data may require different scaling methods to optimize the performance of machine learning models.
• Stack models: This technique involves combining multiple machine learning models to produce a final result, which can often improve accuracy by leveraging the strengths of different models.