I’m guessing there is an easier way to get to where I ended up, but here is how I got to my solutions. First a couple of notes:
\n\n• Method 1: For each fold x, use KMeans clustering (k=25) to partition the predictions (where fold != x). Predict clusters for OOF (fold==x) and the test file. Then if the maximum prediction in a cluster was greater than 10,000 add c (c in range 0 to 20000) and test MAE for improvement for each c. For c where MAE improves the most, add c to OOF and test data in the same cluster.
\n\n• Method 2: For 10-fold CV, for each fold x, use rows fold!=x to determine the best threshold and additive value c by seeing if the MAE improves when adding c to all prediction>threshold. For this method I tested all thresholds between 0 and max(prediction), and all c between 0 and 1400.
\n\nFeature Engineering
\n\nFor additional features I built most off of three sets of the original features; the binary categorical features (each has only two levels, cat1 – cat72), multi-level categorical (more than 2 levels, cat73 – cat116), and the numeric features (cont1 – cont14).
\n\nThe additional features I created included (summary stats = minimum, mean, median, maximum, stdev, etc.):
\n\n• binary encoded categorical columns (cat1 - cat72)
\n\n• multi-level categorical columns (cat73 - cat116) label encoded
\n\n• multi-level categorical columns (cat73 - cat116) ascii value encoded
\n\n• multi-level categorical columns (cat73 - cat116) One-Hot encoded
\n\n• multi-level categorical columns (cat73 - cat116) minimum log(loss) per level
\n\n• multi-level categorical columns (cat73 - cat116) maximum log(loss) per level
\n\n• multi-level categorical columns (cat73 - cat116) mean log(loss) per level
\n\n• multi-level categorical columns (cat73 - cat116) stdev log(loss) per level
\n\n• paired categoricals, ascii value encoded
\n\n• multi-level categorical columns (cat73 - cat116) mean log(loss) per level summary stats
\n\n• binary categorical probabilities of occurrence
\n\n• multi-level categorical probabilities of occurrence
\n\n• Kinetic values (sum (probability of occurence**2) ) for categorical columns
\n\n• multi-level categorical expected values (probability * mean value)
\n\n• base numeric features summary stats
\n\n• skew-corrected, numeric columns
\n\n• skew-corrected, standard-scaled numeric columns
\n\n• Square root transformed numeric columns
\n\n• Log transformed numeric columns
\n\n• PCA features (59 features => 90% of the variance)
\n\n• distance to binary column centroid
\n\n• distance to multi-level, ascii encoded centroid
\n\n• distance to multi-level, mean(log(loss)) centroid
\n\n• distance to base numeric centroid
\n\n• distance to skew-corrected, standard-scaled centroid
\n\nFrom these features I built XGBoost, Keras, kNN, SVM, Regression, LightGBM, ExtraTrees and RandomForest models on various subsets of the features using either log(loss) or loss**(1/4) as the target feature.
\n\nFirst Level Models
\n\nI eventually narrowed these down to a set of models that gave the best 10-fold CVs and worked best in stacking. I used the out-of-fold predictions from these 28 models to train Keras, XGBoost and RandomForest models (stacking):
\n\n• 8 XGBoost models (best CV: 1126.27, PLB: ~1107)
\n\n• 4 Keras models (best CV: 1130.32, PLB: 1107.4)
\n\n• 5 ExtraTrees models (best CV: 1164.61, PLB: not submitted)
\n\n• 2 RandomForest models (best CV: 1167.07 PLB: not submitted)
\n\n• 4 k-Nearest Neighbor models (best CV: 1279.55, PLB: not submitted)
\n\n• 2 SVM models (best CV: 1145.16, PLB: not submitted)
\n\n o Both were LinearSVR\n• 3 Regression models (best CV: 1145.57, PLB: not submitted)
\n\n o 1 OLS (best CV), 1 BayesianRidge, and 1 ElasticNet\nStacking into 2nd & 3rd level models
\n\nFor choosing models to include in next-level stacking I looked at OOF correlations and their performances on various percentiles. That is, I calculated the MAE for all of the OOF predictions in 0-10th%, 10-20th, etc. and picked the ones that performed the best on a percentile range.
\n\nThe stacking combinations that led to my final two submissions (the numbers are just how I track each model output) is attached.
", "comments": [ { "id": 149921, "author_name": "Gippy", "author_url": "", "post_date": "2016-12-13T07:31:22.033000", "content": "Congrats and Thanks for sharing your solution. Can you please explain the advantage of post processing using K-Means? How did you find the cut-off 10K loss?
\n\nAlso can you please explain \"binary categorical probabilities of occurrence\". Does this mean calculating counts at every level and dividing by total rows? Thanks
", "votes": 1, "replies": [] }, { "id": 149995, "author_name": "Michael Maguire", "author_url": "", "post_date": "2016-12-13T13:36:58.143000", "content": "@ATS
\n\nI calculated on the full train. I usually try and stay away from using stats built on the target variable (especially with XGBoost) because it tends to overfit the training set, but that didn't seem to happen here.
", "votes": 0, "replies": [] }, { "id": 149994, "author_name": "Michael Maguire", "author_url": "", "post_date": "2016-12-13T13:34:15.763000", "content": "@ Gippy
\n\nThe 10K cutoff and k=25 were both trial and error. If I used a lower cutoff it started to add small amounts, like 10 or 20, to some clusters and that made the CV go way down. So I tried different settings until I got stable CVs. I had built my second method of post-processing first and I noticed that the predictions around 7-10K were only low by 200-1,000 or so, while predictions up around 20,000 could be low by 10-15K. So I built another method to add gradually bigger numbers based on prediction value. I tried regression for this but it didn't work at all, and that made me think I need to partition the predictions into blocks before testing c-values. So I went to kMeans to build the blocks. I think something like Jenks partitioning may have been better than kMeans, but kMeans was easy and quick.
\n\nYes, the probabilities of occurrence were exactly that.
", "votes": 0, "replies": [] }, { "id": 149976, "author_name": "ATS", "author_url": "", "post_date": "2016-12-13T11:08:24.300000", "content": "Thank you for impressive explanations
\n\n[quote=Michael Maguire;149881]\n• multi-level categorical columns (cat73 - cat116) minimum log(loss) per level
\n\n• multi-level categorical columns (cat73 - cat116) maximum log(loss) per level
\n\n• multi-level categorical columns (cat73 - cat116) mean log(loss) per level
\n\n• multi-level categorical columns (cat73 - cat116) stdev log(loss) per level
\n\n• multi-level categorical columns (cat73 - cat116) mean log(loss) per level summary stat\n[/quote]
\n\nDid you calc these stats using OOF of straight on full train?
", "votes": 0, "replies": [] } ], "comment_count": 4, "messages": [ { "id": 149881, "postDate": "2016-12-13T03:40:25.617Z", "content": "I’m guessing there is an easier way to get to where I ended up, but here is how I got to my solutions. First a couple of notes:
\n\n• Method 1: For each fold x, use KMeans clustering (k=25) to partition the predictions (where fold != x). Predict clusters for OOF (fold==x) and the test file. Then if the maximum prediction in a cluster was greater than 10,000 add c (c in range 0 to 20000) and test MAE for improvement for each c. For c where MAE improves the most, add c to OOF and test data in the same cluster.
\n\n• Method 2: For 10-fold CV, for each fold x, use rows fold!=x to determine the best threshold and additive value c by seeing if the MAE improves when adding c to all prediction>threshold. For this method I tested all thresholds between 0 and max(prediction), and all c between 0 and 1400.
\n\nFeature Engineering
\n\nFor additional features I built most off of three sets of the original features; the binary categorical features (each has only two levels, cat1 – cat72), multi-level categorical (more than 2 levels, cat73 – cat116), and the numeric features (cont1 – cont14).
\n\nThe additional features I created included (summary stats = minimum, mean, median, maximum, stdev, etc.):
\n\n• binary encoded categorical columns (cat1 - cat72)
\n\n• multi-level categorical columns (cat73 - cat116) label encoded
\n\n• multi-level categorical columns (cat73 - cat116) ascii value encoded
\n\n• multi-level categorical columns (cat73 - cat116) One-Hot encoded
\n\n• multi-level categorical columns (cat73 - cat116) minimum log(loss) per level
\n\n• multi-level categorical columns (cat73 - cat116) maximum log(loss) per level
\n\n• multi-level categorical columns (cat73 - cat116) mean log(loss) per level
\n\n• multi-level categorical columns (cat73 - cat116) stdev log(loss) per level
\n\n• paired categoricals, ascii value encoded
\n\n• multi-level categorical columns (cat73 - cat116) mean log(loss) per level summary stats
\n\n• binary categorical probabilities of occurrence
\n\n• multi-level categorical probabilities of occurrence
\n\n• Kinetic values (sum (probability of occurence**2) ) for categorical columns
\n\n• multi-level categorical expected values (probability * mean value)
\n\n• base numeric features summary stats
\n\n• skew-corrected, numeric columns
\n\n• skew-corrected, standard-scaled numeric columns
\n\n• Square root transformed numeric columns
\n\n• Log transformed numeric columns
\n\n• PCA features (59 features => 90% of the variance)
\n\n• distance to binary column centroid
\n\n• distance to multi-level, ascii encoded centroid
\n\n• distance to multi-level, mean(log(loss)) centroid
\n\n• distance to base numeric centroid
\n\n• distance to skew-corrected, standard-scaled centroid
\n\nFrom these features I built XGBoost, Keras, kNN, SVM, Regression, LightGBM, ExtraTrees and RandomForest models on various subsets of the features using either log(loss) or loss**(1/4) as the target feature.
\n\nFirst Level Models
\n\nI eventually narrowed these down to a set of models that gave the best 10-fold CVs and worked best in stacking. I used the out-of-fold predictions from these 28 models to train Keras, XGBoost and RandomForest models (stacking):
\n\n• 8 XGBoost models (best CV: 1126.27, PLB: ~1107)
\n\n• 4 Keras models (best CV: 1130.32, PLB: 1107.4)
\n\n• 5 ExtraTrees models (best CV: 1164.61, PLB: not submitted)
\n\n• 2 RandomForest models (best CV: 1167.07 PLB: not submitted)
\n\n• 4 k-Nearest Neighbor models (best CV: 1279.55, PLB: not submitted)
\n\n• 2 SVM models (best CV: 1145.16, PLB: not submitted)
\n\n o Both were LinearSVR\n• 3 Regression models (best CV: 1145.57, PLB: not submitted)
\n\n o 1 OLS (best CV), 1 BayesianRidge, and 1 ElasticNet\nStacking into 2nd & 3rd level models
\n\nFor choosing models to include in next-level stacking I looked at OOF correlations and their performances on various percentiles. That is, I calculated the MAE for all of the OOF predictions in 0-10th%, 10-20th, etc. and picked the ones that performed the best on a percentile range.
\n\nThe stacking combinations that led to my final two submissions (the numbers are just how I track each model output) is attached.
", "rawMarkdown": "I’m guessing there is an easier way to get to where I ended up, but here is how I got to my solutions. First a couple of notes:\r\n\r\n1.\tI used a stratified 10-fold CV for all models. The stratification was based on the normalized loss (z-score).\r\n2.\tAll the stacking models use out-of-fold predictions from the predecessor models.\r\n3.\tI did post-processing in two ways, both using a 10-fold CV to test the results and calculate the OOF predictions:\r\n\r\n•\tMethod 1: For each fold x, use KMeans clustering (k=25) to partition the predictions (where fold != x). Predict clusters for OOF (fold==x) and the test file. Then if the maximum prediction in a cluster was greater than 10,000 add c (c in range 0 to 20000) and test MAE for improvement for each c. For c where MAE improves the most, add c to OOF and test data in the same cluster.\r\n\r\n•\tMethod 2: For 10-fold CV, for each fold x, use rows fold!=x to determine the best threshold and additive value c by seeing if the MAE improves when adding c to all prediction>threshold. For this method I tested all thresholds between 0 and max(prediction), and all c between 0 and 1400.\r\n\r\n4.\tBecause my CV scores tracked well with my public LB scores I didn’t worry so much about PLB score vs. CV score when choosing two submissions as my final one. I was more worried about the post-processing overfitting the training data because my CV went down on a couple, while the PLB score went up. Ultimately, I chose one model without and post-processing and one model with. The model without any post-processing scored better on the private LB.\r\n\r\n**Feature Engineering**\r\n\r\nFor additional features I built most off of three sets of the original features; the binary categorical features (each has only two levels, cat1 – cat72), multi-level categorical (more than 2 levels, cat73 – cat116), and the numeric features (cont1 – cont14).\r\n\r\nThe additional features I created included (summary stats = minimum, mean, median, maximum, stdev, etc.):\r\n\r\n•\tbinary encoded categorical columns (cat1 - cat72)\r\n\r\n•\tmulti-level categorical columns (cat73 - cat116) label encoded\r\n\r\n•\tmulti-level categorical columns (cat73 - cat116) ascii value encoded\r\n\r\n•\tmulti-level categorical columns (cat73 - cat116) One-Hot encoded\r\n\r\n•\tmulti-level categorical columns (cat73 - cat116) minimum log(loss) per level\r\n\r\n•\tmulti-level categorical columns (cat73 - cat116) maximum log(loss) per level\r\n\r\n•\tmulti-level categorical columns (cat73 - cat116) mean log(loss) per level\r\n\r\n•\tmulti-level categorical columns (cat73 - cat116) stdev log(loss) per level\r\n\r\n•\tpaired categoricals, ascii value encoded\r\n\r\n•\tmulti-level categorical columns (cat73 - cat116) mean log(loss) per level summary stats\r\n\r\n•\tbinary categorical probabilities of occurrence\r\n\r\n•\tmulti-level categorical probabilities of occurrence\r\n\r\n•\tKinetic values (sum (probability of occurence**2) ) for categorical columns\r\n\r\n•\tmulti-level categorical expected values (probability * mean value)\r\n\r\n•\tbase numeric features summary stats\r\n\r\n•\tskew-corrected, numeric columns\r\n\r\n•\tskew-corrected, standard-scaled numeric columns\r\n\r\n•\tSquare root transformed numeric columns\r\n\r\n•\tLog transformed numeric columns\r\n\r\n•\tPCA features (59 features => 90% of the variance)\r\n\r\n•\tdistance to binary column centroid\r\n\r\n•\tdistance to multi-level, ascii encoded centroid\r\n\r\n•\tdistance to multi-level, mean(log(loss)) centroid\r\n\r\n•\tdistance to base numeric centroid\r\n\r\n•\tdistance to skew-corrected, standard-scaled centroid\r\n\r\n\r\nFrom these features I built XGBoost, Keras, kNN, SVM, Regression, LightGBM, ExtraTrees and RandomForest models on various subsets of the features using either log(loss) or loss**(1/4) as the target feature.\r\n\r\n**First Level Models**\r\n\r\nI eventually narrowed these down to a set of models that gave the best 10-fold CVs and worked best in stacking. I used the out-of-fold predictions from these 28 models to train Keras, XGBoost and RandomForest models (stacking):\r\n\r\n•\t8 XGBoost models (best CV: 1126.27, PLB: ~1107)\r\n\r\n•\t4 Keras models (best CV: 1130.32, PLB: 1107.4)\r\n\r\n•\t5 ExtraTrees models (best CV: 1164.61, PLB: not submitted)\r\n\r\n•\t2 RandomForest models (best CV: 1167.07 PLB: not submitted)\r\n\r\n•\t4 k-Nearest Neighbor models (best CV: 1279.55, PLB: not submitted)\r\n\r\n•\t2 SVM models (best CV: 1145.16, PLB: not submitted)\r\n\r\n o\tBoth were LinearSVR\r\n\r\n•\t3 Regression models (best CV: 1145.57, PLB: not submitted)\r\n\r\n o\t1 OLS (best CV), 1 BayesianRidge, and 1 ElasticNet\r\n\r\n**Stacking into 2nd & 3rd level models**\r\n\r\nFor choosing models to include in next-level stacking I looked at OOF correlations and their performances on various percentiles. That is, I calculated the MAE for all of the OOF predictions in 0-10th%, 10-20th, etc. and picked the ones that performed the best on a percentile range.\r\n\r\nThe stacking combinations that led to my final two submissions (the numbers are just how I track each model output) is attached.\r\n", "votes": 27 }, { "id": 149921, "postDate": "2016-12-13T07:31:22.033Z", "content": "Congrats and Thanks for sharing your solution. Can you please explain the advantage of post processing using K-Means? How did you find the cut-off 10K loss?
\n\nAlso can you please explain \"binary categorical probabilities of occurrence\". Does this mean calculating counts at every level and dividing by total rows? Thanks
", "rawMarkdown": "Congrats and Thanks for sharing your solution. Can you please explain the advantage of post processing using K-Means? How did you find the cut-off 10K loss? \r\n\r\nAlso can you please explain \"binary categorical probabilities of occurrence\". Does this mean calculating counts at every level and dividing by total rows? Thanks", "votes": 1 }, { "id": 149995, "postDate": "2016-12-13T13:36:58.143Z", "content": "@ATS
\n\nI calculated on the full train. I usually try and stay away from using stats built on the target variable (especially with XGBoost) because it tends to overfit the training set, but that didn't seem to happen here.
", "rawMarkdown": "@ATS\r\n\r\nI calculated on the full train. I usually try and stay away from using stats built on the target variable (especially with XGBoost) because it tends to overfit the training set, but that didn't seem to happen here." }, { "id": 149994, "postDate": "2016-12-13T13:34:15.763Z", "content": "@ Gippy
\n\nThe 10K cutoff and k=25 were both trial and error. If I used a lower cutoff it started to add small amounts, like 10 or 20, to some clusters and that made the CV go way down. So I tried different settings until I got stable CVs. I had built my second method of post-processing first and I noticed that the predictions around 7-10K were only low by 200-1,000 or so, while predictions up around 20,000 could be low by 10-15K. So I built another method to add gradually bigger numbers based on prediction value. I tried regression for this but it didn't work at all, and that made me think I need to partition the predictions into blocks before testing c-values. So I went to kMeans to build the blocks. I think something like Jenks partitioning may have been better than kMeans, but kMeans was easy and quick.
\n\nYes, the probabilities of occurrence were exactly that.
", "rawMarkdown": "@ Gippy\r\n\r\nThe 10K cutoff and k=25 were both trial and error. If I used a lower cutoff it started to add small amounts, like 10 or 20, to some clusters and that made the CV go way down. So I tried different settings until I got stable CVs. I had built my second method of post-processing first and I noticed that the predictions around 7-10K were only low by 200-1,000 or so, while predictions up around 20,000 could be low by 10-15K. So I built another method to add gradually bigger numbers based on prediction value. I tried regression for this but it didn't work at all, and that made me think I need to partition the predictions into blocks before testing c-values. So I went to kMeans to build the blocks. I think something like Jenks partitioning may have been better than kMeans, but kMeans was easy and quick.\r\n\r\nYes, the probabilities of occurrence were exactly that." }, { "id": 149976, "postDate": "2016-12-13T11:08:24.300Z", "content": "Thank you for impressive explanations
\n\n[quote=Michael Maguire;149881]\n• multi-level categorical columns (cat73 - cat116) minimum log(loss) per level
\n\n• multi-level categorical columns (cat73 - cat116) maximum log(loss) per level
\n\n• multi-level categorical columns (cat73 - cat116) mean log(loss) per level
\n\n• multi-level categorical columns (cat73 - cat116) stdev log(loss) per level
\n\n• multi-level categorical columns (cat73 - cat116) mean log(loss) per level summary stat\n[/quote]
\n\nDid you calc these stats using OOF of straight on full train?
", "rawMarkdown": "Thank you for impressive explanations\r\n\r\n[quote=Michael Maguire;149881]\r\n•\tmulti-level categorical columns (cat73 - cat116) minimum log(loss) per level\r\n\r\n•\tmulti-level categorical columns (cat73 - cat116) maximum log(loss) per level\r\n\r\n•\tmulti-level categorical columns (cat73 - cat116) mean log(loss) per level\r\n\r\n•\tmulti-level categorical columns (cat73 - cat116) stdev log(loss) per level\r\n\r\n•\tmulti-level categorical columns (cat73 - cat116) mean log(loss) per level summary stat\r\n[/quote]\r\n\r\nDid you calc these stats using OOF of straight on full train?" } ], "complete": true, "source": "live", "fetched_at": "2026-10-19T17:16:15Z", "competition_id": 5325, "competition_slug": "allstate-claims-severity", "team_id": 405536, "solution_url": "/competitions/allstate-claims-severity/writeups/michael-maguire-12th-place-solution" }