# Kaggle Higgs Boson Machine Learning Challenge

This blog encompasses a comprehensive exploratory data analysis of **Higgs Boson Machine Learning Challenge . In particular, I want to concentrate on feature engineering and selection. The response variable** in this case is binary: - either there is a Higgs Signalย or Background signal. In addition the other goal is to find if certain properties of data can help us decideย which models are more appropriate for this problem andย what should be our choice of parameters forย those models. I implement all insights learned to get an AMS score ofย 3.53929.

**Contents**

- Data
- Mystery of ย "-999.0"
- AMS Metric
- Data Preprocessing
- Response
- Features
- Featureย Reduction
- Resultant Data for Predictive Modelling
- A sample xgboost model
- Conclusions
- Future Work

**1 ย Dataย **

A description of the data is available from Kaggle's website for the competition . The data mainly consists of a training and a test file to makeย predictions and submit for evaluation. The snapshot below shows this information as it is displayed on the website.ย

**2ย Mystery of "-999.0"**

One peculiar aspect of both the training and test data is that many values areย -999.0 . The dataโs description statesย that those entries are meaningless or cannot be computedย andย ย โ999.0ย is placed for those entriesย to indicate that .

One can analyze the nature and extent of these -999.0 entries by replacing them with "NA's" and doing this missing data analysis. We starย by plotting a fraction of trainingย data which is ย -999.0 for different featuresย and the combinations of features that occurs in the dataset

The plot to the left shows there are 11 columns where values are -999.0 with three subgroups of 1, 3ย and 7ย columns . ย The combinations of features in the figureย indicates there are 6ย such combinations . Doingย ย same analysis on submission data gives exact same plot. ย This indicates that original data can be subdivided into 6ย groups, in terms ofย number of featuresย having -999.0 .

Investigating the names of featuresย with -999.0 shows that name of columns with -999.0 areย - ["DER_mass_MMC" ย ], ย ["PRI_jet_leading_pt" ย , ย "PRI_jet_leading_eta" ย ย "PRI_jet_leading_phi" ], ย [ย "DER_deltaeta_jet_jet" ย , ย "DER_mass_jet_jet" ย , ย ย "DER_prodeta_jet_jet" ย ,ย "DER_lep_eta_centrality", "PRI_jet_subleading_pt" ย "PRI_jet_subleading_eta" "PRI_jet_subleading_phi" .

Diving further into technical documentation , it is evident thatย the group with 7 featuresย isย associated with two jets where oneย or no jet events are undefinedย . Furthermore, the group of 3 featuresย isย associated with at-least one jet production which is not defined in events when there is no jet . There are also certainย observations where Higgs mass is not defined and thisย is not jet dependent . In conclusion, the original data can be subdivided into six groupsย in terms of whether or not the Higgs mass is defined and correspondingly if the event resulted in ย one jet , more than one jet, ย or no jet production at all. ย (2 x3) . We ย add two new features to incorporate this information .

**3 AMS Metric**

AMS is the particular evaluation metric for the competition . It's dependent on signal , background and weights associated with them in a peculiar manner as below.

Aย plot of AMS for complete training data range forย ย s and b is shown belowย . AMS color is saturated for score above 4 as top leaderboard scoreย isย less than 4 and we want to concentrate on what makes those models good.

The red region corresponds to high AMS scores which are linked to low false positive and high true positive rates That is expected, ย but the peculiar aspect in the figure above is that there is aย range of models which can achieve top ranked score of 4ย andย one such model would be identifying just 25 % of Higgs signal correctly, keeping the false positive rate to 3%, and still have the same AMS score of 4.

Next we investigate how much AMS score on training isย influenced by performance in 6 data subgroups we identified in the last section. The red line in the previous plotย isย produced by a perfect prediction of the trainingย data.The blue line below indicates AMSย if every prediction is said to be a signal . For each of the data subgroups we calculate the AMS score by assigning either signal or background noise toย all the events in that subgroup and using correct predictions for the rest of the data .

The figureย above shows that performance on data where the Higgs is not defined has almost no effect on the AMS score ifย everything is classified as background signal for these subgroups. ย This behavior occurs ย because there is very small signal in the data where the Higgs is not defined. Moreoverย the weight of the signals is low and the background noise has higher weights. These three factors make ย identifying everything as background noise is as good as identifying everything correctly.

Thus, classifying all events where the Higgs is not defined as backgroundย willย perform equally well in terms of the AMS score . This reduces the subgroups of data for predictive modeling to 3ย that is Higgs is defined and event resulted in either one jet or more than one jet or no jet at all.

**4 Data Preprocessing**

We prepare our fourย subgroups of data each forย training and testย ย and remove all the features which have NA's and one's which haveย becomeย redundant and scale the data . Finally we are left with only 60% of original data which is useful for predictive modeling. Doing same for submission data also leaves with 60% of predictive modeling data further cementing the idea to drop NA's , keep aside Higgs mass undefined data and splitting original data into three subgroups.

**5 Response**

We explore response of Higgs Kaggle ย which is named Label in the dataset . Plotting histogram of the weights associated with the Labels indicate Label and weights are dependent on each other . The signals exclusively have lower weights assigned to them than the background and there is no intermingling between the two Labelsย . This is a pretty good reason for Kaggle to not provideย weights for test set !

A density plot of background signal does show some distinction between three subgroups but not obvious ones so weย would leave it at that and come back if needed.

Histogram of signal shows that there are three weights or channels in which Higgs is being sought after. There are more "no jet" events with higher weights than "two or moreย jets" events which are larger in numberย but haveย lower weights .

**ย ย **

**6 Features**

*Correlationย *

We first look at the correlation among features for the three data subgroups . The order here is {2,1,0 }. Upper triangle which corresponds to DER (derived) features in all three seems to be correlated . ย These are not observations but features engineered by CERN group using particle physics. The lower rightย triangles which contains all PRIย (primitive)ย featuresย are not correlated . It could be a good idea to drop all the engineered correlated DER features .

*Principal Component Analysis ย (PCA)*

Examining the variance explained by the principal components used indicates there is aย room for theย reduction of features in all three subgroups . For example, for subgroup 2 , fifteen components out of 30 can explain 90 % of variance in the data and 24 components can explain 99% of variance .

ย Subgroup 1 ย {90,95,99} % = {11 ,13,15} ย ย ย ย ย ย ย ย ย ย ย ย ย ย Subgroup 0ย ย {90,95,99}% = {9 ,10,13}

*PCA Eigenvectors*

So our next challenge isย to identify which original features do not have any influence or have very little influence in explaining the data . We start by multiplying the PCA eigenvalues and corresponding PCA eigenvectors and plotting the projections on a heat map. The product calculated in the previous step represents the transformed variance along the PCA directions . We then sorted this along the ย horizontal axis ย by the PCA eigenvalues' , starting ย with ย lowestย on the left to theย highest onย the right . As is evident from the figure below, the lower eigenvalue transformed variance (starting from F30 and going towards F1) is zero which is what we would expect from the PCAย plot from the last section

We now sort the original features along the vertical axis with respect to their contribution to the variance. We do this forย the transformed variance productsย in descending order. ย We sum up the absolute value of the contributions and not just the contribution (as a feature can have positive or negative projectionย indicated by red and blue colors). At the end of this process, we are left with features thatย contribute the least variance displayed at the bottom.

Theย last 9 features in the above plot stand out from the rest of features as they have white blocks or zero contribution towards first four principal components. Another important observation is that they are all ย phi or eta angle features.Similarly phi and eta angles in subgroup 1 and subgroup 0 show the same behavior. In next sectionย we will see why they are least useful in explaining the data .

*Density Plotsย *

Let's look at the density plot of the last 9 whitewashed features of subgroup 2. We see theyย share a few common characteristics .ย As pointed earlier, they ย are all angle features for directions of particles and jets. In the case of the 5 phi angle features they have uniform and are identically distributed over the range for both the signal and background. This is true to some extent for eta features also but for phi it is strikingly true. Conceptuallyย it does make sense as the particles and jet would scatter off in all directions whether or not they are signals or background. Thus, the variables will follow a uniform distribution.

The plot below contrasts this uniform distribution aspect of the least influential featuresย to the density plots of the first 9 most influential features.

The same is evident for angle featuresย for subgroup 1 and subgroup 0.

**7 Featureย reduction**

The last section gave us plenty to think about with respect to which features are least influential in explaining the variance. But this particular Higgs Kaggle competition's success is determined by maximizing the AMS score.ย Thus discarding any features or for that sake any amount of data may not be a wise decision . But we can still use the above insights as a guide I propose the following approach.

*First Iteration ย ย *

- Drop ย DER features
- Drop eta features
- Drop phi ย features
- Assign Background to Higgs Undefined
- Separate Predictive modeling on 3 subgroups

*Secondย Iteration*

- Drop eta features
- Drop phi ย features
- Assign Background to Higgs Undefined
- Separate Predictive modeling on 3 subgroups

*Thirdย Iteration*

- Drop phi ย features
- Assign Background to Higgs Undefined
- Separate Predictive modeling on 3 subgroups

*Fourthย Iteration ย (if needed)*

- Assign Background to Higgs Undefined
- Separate Predictive modeling on 3 subgroups

*Fifth Iteration ย ย ย (if needed)*

- Predictive modeling on 3 subgroups and Higgs Undefined

*Sixthย Iteration ย ย ย (Nope. Do something else)*

- Brute force on full data

**8 Resultant Data for Predictiveย ****Modeling**

Letโs evaluate the approach outlined above. Using only the data which matters will reduce computation power and time needed.ย This model will beย more accurateย as it will reduce noise.ย ย Moreover,ย less data to deal with means thatย one can try more computationally expensive models like Neural networks and SVM's and try to get lucky with automatic feature engineering . ย To quantifyย this benefit, we ย plot the amount of data used at each modeling iteration.

**9 A sample xgboost Model**

We fit an xgboost decision tree model to our training data using the insights above . I chose xgboostย here due to itsย havingย low variance , low bias, high speed, and more accuracy.ย We will follow the "Third Iteration" schema ย from section 7 . ย Let's prepare the training andย testing setsย by dropping the phi variables, and assigning the background noise label to data where Higgs is not defined, and splitting the data where Higgs is defined into 3 subgroups.

"AUC"ย is the metric of choice hereย as it responds well to misclassification errors. ย The optimal number of trees to maximize the AUC score will be found by cross validation.ย We fit the whole training data toย the optimal number of trees for each dataset and make predictions for the test data to submit to Kaggle.ย

The Private AMS score for this model is **3.53929** . That's satisfactory for me at the moment considering we didn't tune any hyper-parameters except for the number of treesย and set the same threshold for all three subgroups. One need to do a grid search to find the three different thresholds.

We use "AUC" as our choice of metric as it responds well to misclassification errors. ย We use cross validation to find numbers of trees which maximizes AUC .

We fit the whole training data to most optimal number of trees for each dataset and make predictions for testย data and prepareย file for submission on Kaggle.

Private AMS score for this model isย **3.53929** . That's satisfactory for me at the moment considering we didn't tune any hyper-parameters except number of trees ย here . ย Plus we set same threshold for all three subgroups. One need to do a grid search to find the three different thresholds.

**10ย Conclusions**

- ย -999.0 are placeholders for undefined values Any attempt of imputation is plain wrong,won't work , make things bad.
- -999.0 split original data into six subgroups i.e. if Higgs mass is defined (or not) and correspondingly how many jets areย formed (0 , 1 or more than 1)
- A high AMS score requires predominantly a low false postive rate and also aย high true positive rate
- Setting everything to background for observations where Higgs mass is not definedย has a very tiny effect on AMS score
- One can effectively do predictive modeling on three subgroups of data where Higgs mass is defined and correspondingly how many jets are formed (0 , 1 or more than 1)
- The weights and the signal are dependentย onย each other.
- The Higgs is being sought in three channels in the data
- The DER variables are correlated to each other and the PRI variables are uncorrelated for most part
- The angle features of phi and eta have least influence on explaining the variance
- There is only 16 % of uncorrelated data which explains most of the variance in the data
- A simple xgboost model gives a respectable AMS ofย
**3.53929**

**11 Future Work**

- Grid search for hyperparameters of xgboost model and thresholds
- Ensemble Methods
- Stacking
- Feature engineering
- Use "iteration 1" , "2" with the least amount of uncorrelated and significant data with more computationally expensive methods such as Neural networks and SVMs to do automatic feature engineering