Housing Prices: Gnome is Where the Heart is

Posted on Nov 21, 2018

Project GitHub | LinkedIn:   Niki   Moritz   Hao-Wei   Matthew   Oren

The skills we demoed here can be learned through taking Data Science with Machine Learning bootcamp with NYC Data Science Academy.

Machine Learning Project -
Your goal is to explore and analyze the “Kaggle Competition: Ames, Iowa Housing” dataset to find key features that influence sales price and develop a machine learning algorithm to predict these prices.

Dataset provided by Kaggle: https://www.kaggle.com/c/house-prices-advanced-regression-techniques

Team Lambda: Mikhail Stukalo, Lina Sprague, Anisha Luthra, Daniel Chen


Predicting housing prices is a challenge; the task involves many variables and takes some creative thinking to pinpoint the features that actually matter in order to arrive at an accurate prediction. A recent competition on Kaggle (an online data science platform) featured housing price predictions with data from Ames, Iowa. Among other things, Iowa is known for their corn production while Ames is known for Iowa State University. They also have the tallest concrete garden gnome in the world, Elwood.

Ours was a group new to Kaggle competitions, and while Kaggle had some official rules regarding the use of additional outside data (data not provided in the Kaggle dataset), one of the first things we opted to do was to pull outside data to help with our housing price predictions. This was after all, first and foremost a project we were doing for the NYC Data Science Academy. It was also our first group project.

The Group

Let me plug a paragraph in the first person perspective on a project blog that the entire group contributed to. In a competition, the group matters, and the only group that matters is your own. I was not so adamant about finding a group, but on the other hand I did not want to be left to my own devices. So that Friday after class I had texted Mike Stukalo, stating that there was a lot I could learn from him if we formed a group.

To this he texted his reply, “Sure. Let’s be a group.” Anisha Luthra was next to join the following Monday; I had thoroughly enjoyed her previous web scraping project on drug ratings from webmd.com. She was recruited in the elevator. Then, I was pleasantly surprised when Mike announced our fourth and final team member, Lina Sprague, who would have added a competitive advantage to any group. We now had a complete and awesome group: Lambda.

Now, back to the data.


The Ames Housing dataset compiled by Dean De Cock originally served as an alternative to the classic “Boston Homes” dataset, a package well known in R Studio and Python. However, the 500 observations found in the Boston Homes dataset is dwarfed in comparison. The Ames dataset contains 3,000 homes and their descriptive values as 79 variables.

These variables range from “Overall Quality” and “First Floor Square Feet” to “Alley Access” and “Roof Style”. The dataset is a mix of categorical and quantitative variables. There were also many missing values, but this would not stump our team of data scientists. We decided to add even more data.

In addition to the Ames dataset, we gathered data on schools, unemployment rate, and the labor force in Ames. We also included Fannie Mae Mortgage Rates, the Dow Jones Real Estate Index (DJREI), and corn prices in our analysis.

It made sense to include corn prices because as we all know, that is what Iowa is known for. We included fixed income and rates (unemployment, labor force, and mortgage) because these were factors to be considered when pricing in a house. Different areas have various unemployment metrics and labor force values which tend to correlate with overall income. Mortgage rates were an important consideration: people apply for mortgages when buying a home.

The DJREI provided an overview of the health of the real estate industry, there is incentive to invest in real estate when the index is strong. Schools (particularly public schools) were deemed to be important for housing prices because public schools are designated to certain districts/neighborhoods.

These were all the variables which were not originally included in the dataset but could prove invaluable in our predictions of housing prices. We treated these new features as one-month-lagged variables compared to the date a house was sold. This is because people usually try to predict future outcomes based on current data.

Imputing Missing Variables

Now that we had gathered and added data, the next step was to perform various transformations on the entire dataset (train set and test set combined). This was an absolute necessity; it would ensure that our data could lend itself to the various prediction models in a way that was accurate and precise.

In order to impute missing variables we first combined the train set and test set data. We took time to understand what the “missing” values denoted. As explained in the dataset, a missing value (NA) for AlleyAccess meant that there was no alley access to the property. It did not mean that this value was missing (not recorded); therefore we could safely replace those missing values with 'None' instead of 'NA'.

A missing value for basement features could be explained by the absence of a basement. Other missing variables could actually be computed. For a missing quantitative variable such as LotFrontage, we grouped the observations based on neighborhood and used the median value to fill in the absent value.

For imputing a missing categorical value, such as “MSZoning” or “SaleType”, we again grouped by neighborhood but calculated the mode instead to use as the replacement value. We checked to see that we now had a full dataset and converted all categorical variables into dummy/indicator variables using pandas.get_dummies() in Python3.

Feature Engineering

For our response variable SalePrice, we had a look at its distribution and saw that it was right skewed. We took the log of SalePrice in order to give it a standard normal distribution, this would allow for more accurate predictions later on.

On the scatterplot we also noticed unwanted outliers. We regarded values which were more than 5 standard deviations away from the mean to be outliers. We changed these far off values to what the value of the mean was in order to keep the observation, rather than remove it outright.

We also looked at the distributions of all the numerical features and their skewness. We considered a variable skewed if their skew() was larger than 0.75. There were 19 of these variables in total. To correct for the skew we applied a log+1 transformation to these variables.

Finally, we divided the observations back into their respective train set and test set.


We used a total of four linear models, a Random Forest Regressor, and an XGB model to make predictions on the sale prices of homes in Ames, Iowa.

The "Training RMSLE" and "Validation RMSLE" were the results we obtained from dividing the train set into its own train and validation sets. We used a test_size(validation size) of 0.3 for the train_test_split() on the train set. The test set mentioned in Imputing Missing Variables was not used until the final results testing.

Random Forest

Decision trees were used to build models of the SalePrice. For the Random Forest Regressor, we performed cross-validation to find optimal pruning parameters (min_amples_split=10 and min_samples_leaf=2) to build the final random forest model. We can see the factors with positive importance below in the bar chart below. Also shown is a scatter plot of the predicted train set results.

For our XGB model, we used boosted trees (gradient boosting) to model the prices. Below, for the graph on the left, the x-axis is the number of n_estimators for our XGBRegressor, the y-axis contains the RMSE-mean value.

Linear Regression Models - Lasso

For our first linear model we used a Lasso Regularization model fitted to the train set with all features included. Important regression coefficients are graphed on the right-side graph.

Linear Regression Models – Ridge

We then did the same with a Ridge Regularization model. Ridge Cross Validation was performed to find the optimal alpha value of 6, which returned the lowest RMSE.

Linear Regression Models on Selected Features

The features selected before by our Random Forest Regressor were used to create the following Lasso and Ridge regression models.

Lasso on Selected Features

Ridge on Selected Features

As was shown in the first chart of the Methodology section, the RMSLE results obtained from our linear regression models based on "selected features only" were quite similar to the results produced by the two linear regression models with "all features included".

With that aside, after we created all six of our models, we pickled them (import pickle) for later use.

Stacking Models

Because this is a blog, we have immediately arrived to the 'later use' stage. This is the part where we take the results we pickled to combine the predictive power of our models. We stacked the six models using the inverse of their error rate on the validation sets as weights to obtain our final prediction model.


For our final result testing, we used all of our prediction models on the test dataset given by Kaggle. We also had the actual prices for the test datset on hand to grade our own predictions.

In the end, Kaggle calculated our score based on the test dataset prediction results we submitted against an unknown dataset of their own. Our final score was close to the Predicted RMSLE of 0.1197 (based on our final stacked prediction model).

For our housing price prediction results we rounded the price up to the nearest thousand. Rounding up was chosen for a practical reason: the selected model evaluation criterion (RMSLE) penalizes underestimated values more than overestimated values.

The final resulting RMSLE for our Kaggle submission was 0.11949, an excellent result for our first machine learning group project.

All of our code can be found stored on GitHub: https://github.com/MikeStukalo/Ames


About Author

Daniel Chen

Daniel Chen is the founder of multiple startups including foreign exchange brokerages. Managed cross-border development with international companies from Europe and Asia. Has even more experience now with data science and coding. Born in 1987's Los Angeles, California...
View all posts by Daniel Chen >

Leave a Comment

No comments found.

View Posts by Categories

Our Recent Popular Posts

View Posts by Tags

#python #trainwithnycdsa 2019 2020 Revenue 3-points agriculture air quality airbnb airline alcohol Alex Baransky algorithm alumni Alumni Interview Alumni Reviews Alumni Spotlight alumni story Alumnus ames dataset ames housing dataset apartment rent API Application artist aws bank loans beautiful soup Best Bootcamp Best Data Science 2019 Best Data Science Bootcamp Best Data Science Bootcamp 2020 Best Ranked Big Data Book Launch Book-Signing bootcamp Bootcamp Alumni Bootcamp Prep boston safety Bundles cake recipe California Cancer Research capstone car price Career Career Day citibike classic cars classpass clustering Coding Course Demo Course Report covid 19 credit credit card crime frequency crops D3.js data data analysis Data Analyst data analytics data for tripadvisor reviews data science Data Science Academy Data Science Bootcamp Data science jobs Data Science Reviews Data Scientist Data Scientist Jobs data visualization database Deep Learning Demo Day Discount disney dplyr drug data e-commerce economy employee employee burnout employer networking environment feature engineering Finance Financial Data Science fitness studio Flask flight delay gbm Get Hired ggplot2 googleVis H20 Hadoop hallmark holiday movie happiness healthcare frauds higgs boson Hiring hiring partner events Hiring Partners hotels housing housing data housing predictions housing price hy-vee Income Industry Experts Injuries Instructor Blog Instructor Interview insurance italki Job Job Placement Jobs Jon Krohn JP Morgan Chase Kaggle Kickstarter las vegas airport lasso regression Lead Data Scienctist Lead Data Scientist leaflet league linear regression Logistic Regression machine learning Maps market matplotlib Medical Research Meet the team meetup methal health miami beach movie music Napoli NBA netflix Networking neural network Neural networks New Courses NHL nlp NYC NYC Data Science nyc data science academy NYC Open Data nyc property NYCDSA NYCDSA Alumni Online Online Bootcamp Online Training Open Data painter pandas Part-time performance phoenix pollutants Portfolio Development precision measurement prediction Prework Programming public safety PwC python Python Data Analysis python machine learning python scrapy python web scraping python webscraping Python Workshop R R Data Analysis R language R Programming R Shiny r studio R Visualization R Workshop R-bloggers random forest Ranking recommendation recommendation system regression Remote remote data science bootcamp Scrapy scrapy visualization seaborn seafood type Selenium sentiment analysis sentiment classification Shiny Shiny Dashboard Spark Special Special Summer Sports statistics streaming Student Interview Student Showcase SVM Switchup Tableau teachers team team performance TensorFlow Testimonial tf-idf Top Data Science Bootcamp Top manufacturing companies Transfers tweets twitter videos visualization wallstreet wallstreetbets web scraping Weekend Course What to expect whiskey whiskeyadvocate wildfire word cloud word2vec XGBoost yelp youtube trending ZORI