Ames Iowa Machine Learning Applications

Posted on Jul 6, 2020



Anthony S. Fargnoli PhD




Real estate purchases comprise the single largest investment the average American enters at one point over their lifetimes.  Real estate websites agencies and other information sources have sought machine learning approaches to model and predict housing price trends.  Often controlling for location and other factors buyers and sellers wish to capture real time market data or historical data to best optimize their transaction strategies.  Here the Ames IA pricing data set features 1460 selected single family housing transactions between 2006-2010.  The purpose of the project here was to:


  1. Identify key features that would influence pricing
  2. Apply machine learning algorithms of several classes to identify a viable model
  3. Optimize select and reduced models to arrive at a final model
  4. Enter models into Kaggle to determine position rank




 The general approach was to perform iterative progression toward the ideal model in these steps shown in the block diagram:
























  1. Data sets- Python was the sole package used for the analysis with Pandas being used to create a dataframe sets for starting point test and train sets. The initial train set contained 1460 rows with approximately 89 features with single target SalePrice data points.


  1. Feature selection was performed with visual exploratory data analysis tools including histograms frequency plots and other measures of variances. Data fields with exaggerated (i.e. >98% of distribution of one categorical data frequency) extreme missing values (i.e. data with greater than 80% of set missing) and other incomplete factors were determined not worthy of any imputation approach were eliminated.  Missingness was handled for select remaining features with SimpleImputer as the median for float and most common data for categorical to complete data sets.


  1. Model Fitting – Ridge regression Random Forest Gradient Boosting machine and the well characterized XGBoosting machine were implemented on test/train/split data from the original train set at 25% data in the test. The overall approach was to start with an easy to apply model that was parsimonious to obtain a baseline readout of model performance with the existing features.  Given that ensembling methods are more powerful and accurate for relevant scoring a shift to 3 robust ML models of Random forest Gradient Boosting Regressor and the XG Boosting Regressor were applied.  Each baseline model assessment to compare the test vs. train error rates were noted as well as a baseline cross validation scoring to assess basic variance for each. 


  1. Optimization and Final Model Predictions – After all 4 models were fit to the training set a combined GridSearch Cross validation process was applied to each model to identify optimal hyperparameters. These varied per protocol for each specific machine learning method.  Best parameters were utilized to re-train the baseline model.  A new model was re-trained to the entire data set with the optimal parameters prior to final product predictions.  Each ensembling model was used to make final predictions and entered in the official scoring for assessment. 




Per correlation analysis and basic feature plotting many variables suffered very high co-linearity hyperexaggerrated distribution with >98% in favor of one level or seemingly little bearing on housing prices.  Below is a sample of data via count analysis demonstrating significant missingness or insignificance with overlapping variables in similar categories:


FireplaceQu      730 – 4 other features for FirePlaces

GarageType       76  - Attached vs. Detached; bias either way

GarageYrBlt      78  - Largely a function of home age; Quality vs. Age

GarageFinish     78  - Arbitrary


Fence            1169

MiscFeature      1408

Alley            1352

These variables were selectively removed from further analysis.  Many were either highly co-linear with other features which had more complete sets and or high degrees of missingness or non informative bias patterns with one level feature representing over 90% of total data. 






PoolQC Fence




























Figure 1. Exploratory data analysis for selected features that were advanced in the model analysis stage.  A total of 44 features with relevant distributions and or expected impact on SalePricing were advanced. 


Missingness in these sets were minimal and a simple imputer function in Python was used to impute median in the case of float, and most frequent in the case of categorical.  Please refer to remaining Pre-processing execution in the main posted on my GitHub. 


Ridge Regression


Starting with the simplest model attempt, RidgeRegressor was fit to the test/train split train set with the following results:


The intercept is 1764731.9110The slopes are MSZoning        -761.241261  LotArea          0.452596    LotShape        -1249.027722 LandContour      3884.979427 LotConfig       -62.624721   LandSlope        6679.699746 Neighborhood     310.843423  Condition1      -668.159269  BldgType        -4953.965713 HouseStyle      -806.616131  OverallQual      13146.529312OverallCond      5354.995245 YearBuilt        334.544071  Exterior1st     -537.337348  ExterQual       -13931.468240ExterCond        1755.804705 Foundation       1225.461027 BsmtFinSF1       9.475112    TotalBsmtSF      6.426640    HeatingQC       -594.731061…………..Yr Sold         -1202.532386The training error is: 0.17666The test error is: 0.17563 GridSearch was applied to identify the hyperparameter lambda, then applied to a revised re-trained Ridge obtaining the following scores: The training error is: 0.17719The test error is: 0.17450 

Random Forest Machine Learning


The baseline Random Forest Regressor yieled an excellent initial score, indicating a much better yield than Ridge as expected, with test set error less than 9% as a starting point.


The training score is: 0.97613The test score is: 0.91970


An initial model cross validation yielded a decrease in mean score to about 0.85 suggesting an overfit training data status.  Default parameters were identified prior to GridSearchCV:


{'bootstrap': True, 'ccp_alpha': 0.0, 'criterion': 'mse', 'max_depth': None, 'max_features': 'auto', 'max_leaf_nodes': None, 'max_samples': None, 'min_impurity_decrease': 0.0, 'min_impurity_split': None, 'min_samples_leaf': 1, 'min_samples_split': 2, 'min_weight_fraction_leaf': 0.0, 'n_estimators': 100, 'n_jobs': None, 'oob_score': False, 'random_state': 0, 'verbose': 0, 'warm_start': False}


After a few attempts to optimize each of these, it was obvious it was computationally too expensive.  Thus, a refined list of major, select hyperparameters were applied for the next iteration of Random Forest:


parameters = {'max_depth': [10,50,None],

              'max_features': ['auto','sqrt'],

              'n_estimators' : [100,200,500],




Alternative approaches for Random Forest optimization as reported on Kaggle and other data science sites included RandomCVGridSearch.  This analysis was more expensive and difficult to execute, thus the simple paradigm was used above to optimize a revised fitted tree model.


The revised model yielded an improved score:


The Score on the  training data is 0.982The Score on the test data is 0.987


Gradient Boost Machine Learning


Gradient boosting ensembling method was an excellent candidate for this housing project following with 44 features and a dataset of 1460 rows.  The Gradient boosting machine builds its estimates from an assembly of weak learners which are optimized at a rate.  The base learning rate defaults were used for the first iteration to produce the following performance:


The Training Score is 0.962The Testing Score is 0.930


Five fold cross validation on the training set, demonstrating relative consistency but with a suggestion the first run of Gradient Boosting overfit the training set:


scores([0.84114232, 0.85495313, 0.7733961 , 0.85166659, 0.91937407])


A GridSearchCV with this format to opmize the major drivers of Gradient boosting was used as follows:


parameters = {'learning_rate': [0.01,0.02,0.03],

                  'subsample'    : [0.9, 0.5, 0.2],

                  'n_estimators' : [100,500,1000],

                  'max_depth'    : [4,6,8]


The best parameters across ALL searched params: {'learning_rate': 0.03, 'max_depth': 6, 'n_estimators': 500, 'subsample': 0.5}


A re-fit model with these optimized parameters yield the following performance.  Notable gains were made in reducing the training data error and a higher average CV score with more consistent variance profile. 


The Friedman_MSE on the  training data is 0.998The Friedman_MSE on the test data is 0.924


scores([0.88476961, 0.88683934, 0.80215266, 0.83565457, 0.9134895] Some reduction was noted in the testing rate, however these were less in relation to the gains in the variance reduction in the CV from the original model.   

XG Boost Machine Learning

 The XGBoost has been reported the King of Kaggle competitions for a variety of challenges and thus has gained wide popularity.  Given the positive results with Gradient boosting, it was logical to attempt this as a final model attempt to complete this project.  XG boost packages and plugins were installed and the baseline model was fitted.  Results were solid as expected, however cross validation returned some disparity combined with a high train score indicative of overfitting: The training Score is: 0.99958The test Score is: 0.91529 CV scores([0.87249273, 0.82699026, 0.79097786, 0.85860529, 0.89877748]) GridSearch CV with the following key parameters for XGBoost were applied: The best estimator across ALL searched params:  XGBRegressor(base_score=0.5, booster='gbtree', colsample_bylevel=1,             colsample_bynode=1, colsample_bytree=0.9, gamma=0, gpu_id=-1,             importance_type='gain', interaction_constraints='',             learning_rate=0.300000012, max_delta_step=0, max_depth=20,             min_child_weight=1, missing=nan, monotone_constraints='()',             n_estimators=200, n_jobs=0, num_parallel_tree=1,             objective='reg:squarederror', random_state=0, reg_alpha=0,             reg_lambda=1, scale_pos_weight=1, subsample=1, tree_method='exact',             validate_parameters=1, verbosity=None) A new iteration with these parameters yielded: The Score on the  training data is 1.000The Score on the test data is 0.907 CV scores([0.87160822, 0.84076335, 0.77042395, 0.8128964 , 0.90150892]) Given these results, while exciting to achieve a perfect score on the training data this combined with marginal loss in the CV scoring it is very likely overall without further modification the XGBoost is overfit.   

Final Model Evaluation: Kaggle Results


Each of the optimized models of Random Forest, Gradient Boosting, and XGBoost were loaded into the official Ames Iowa Kaggle competition for scoring.  The Gradient boosting model as a result was the best overall achieving a >35% percentile rank well above the other two models. 



Summary of Feature Importance


Generally, the feature importance’s between the models did not change much and below is an example from the best performing model.  The top 15 results were:

 1.  OverallQual', 0.14471587922967497),2.  GrLivArea', 0.11851858124286388),3.  ExterQual', 0.08545406813704338),4.  TotalBsmtSF', 0.08308420400688572),5.  GarageCars', 0.08129439906946702),6.  YearBuilt', 0.05259058831073286),7.  FullBath', 0.05245474540526278),8.  1stFlrSF', 0.048267085912680516),9.  BsmtFinSF1', 0.04544276425388854),10.LotArea', 0.03742499662899613),11.KitchenQual', 0.03536025676082719),12.2ndFlrSF', 0.028929776122335108),13.Fireplaces', 0.024372398428322197),14.Foundation', 0.02071006879400414),15.OpenPorchSF', 0.017767820031488368) Overall quality, total first floor living area, curb appeal via Exterior quality are all expected drivers of pricing since generally all buyers seek these.  The Year built and having additional garage space is also a major feature as expected.  Kitchen quality and upper level square footage as well as the remaining contribute to a lessor amount. The most unique feature of interest seems to be the 4th ranked Total basement square feet. The best explanation for this may be that the state of Iowa is in a hurricane corridor, unbeknownst to the majority of US, is ranked 6 on the list of US states with tornadoes.  In fact, Ames Iowa during timeline within 10 miles of city center had many tornadoes.  Basement areas in these cases could be considered lifelines that a buyer would be willing to pay for and also a function of more expensive homes with finished area.  Figure 2.  Feature importance rank results from the optimized Gradient Boost model selection.   Conclusions: ·      Ensembling methods offer much higher performance as standard for basic machine learning·      Overfitting on a relatively elementary data set appeared to be the issue for an advanced algorithm such has XGBoost·      Gradient boosting with some adjustment to the learning rate from 0.1 to 0.09 offered the best available evaluated model with highest Kaggle rank·      Improvements: Feature engineering could reduce the number of variables further, transformations applied to imbalanced sets·      GridSearch for Random forest better approached with an alternative approach 













About Author

Leave a Comment

No comments found.

View Posts by Categories

Our Recent Popular Posts

View Posts by Tags

#python #trainwithnycdsa 2019 airbnb Alex Baransky alumni Alumni Interview Alumni Reviews Alumni Spotlight alumni story Alumnus API Application artist aws beautiful soup Best Bootcamp Best Data Science 2019 Best Data Science Bootcamp Best Data Science Bootcamp 2020 Best Ranked Big Data Book Launch Book-Signing bootcamp Bootcamp Alumni Bootcamp Prep Bundles California Cancer Research capstone Career Career Day citibike clustering Coding Course Demo Course Report D3.js data Data Analyst data science Data Science Academy Data Science Bootcamp Data science jobs Data Science Reviews Data Scientist Data Scientist Jobs data visualization Deep Learning Demo Day Discount dplyr employer networking feature engineering Finance Financial Data Science Flask gbm Get Hired ggplot2 googleVis Hadoop higgs boson Hiring hiring partner events Hiring Partners Industry Experts Instructor Blog Instructor Interview Job Job Placement Jobs Jon Krohn JP Morgan Chase Kaggle Kickstarter lasso regression Lead Data Scienctist Lead Data Scientist leaflet linear regression Logistic Regression machine learning Maps matplotlib Medical Research Meet the team meetup Networking neural network Neural networks New Courses nlp NYC NYC Data Science nyc data science academy NYC Open Data NYCDSA NYCDSA Alumni Online Online Bootcamp Online Training Open Data painter pandas Part-time Portfolio Development prediction Prework Programming PwC python python machine learning python scrapy python web scraping python webscraping Python Workshop R R language R Programming R Shiny r studio R Visualization R Workshop R-bloggers random forest Ranking recommendation recommendation system regression Remote remote data science bootcamp Scrapy scrapy visualization seaborn Selenium sentiment analysis Shiny Shiny Dashboard Spark Special Special Summer Sports statistics streaming Student Interview Student Showcase SVM Switchup Tableau team TensorFlow Testimonial tf-idf Top Data Science Bootcamp twitter visualization web scraping Weekend Course What to expect word cloud word2vec XGBoost yelp