Below is the solution of team LAJMBURO (8th private LB (0.50888), 11th public (0.51001) ).
\n\nThere are two audiences intended:
\n\nTwo parts:
\n\nWe used simple level 1 models:
\n\nWe had five level 1 datasets (gbm=LightGBM, xgb=xgboost, rf=Random Forest - features are not exactly including all the features described later), I’m including their mean MCC on cross-validation (for Public LB score, add at least 0.02 - numbers might be off a bit):
\n\nLevel 2 data set was a bit special, and included:
\n\nLevel 2 stack models (giving the weaker model a stronger weight was better):
\n\nWe had a very minor issue at overfitting LB:
\n\nIn the case it were to overfit severely, we also made a backup set which is leakage-free (using features not using the label directly) and without a multiplicand. For instance, features like the Unsupervised Nearest Neighbors was removed. A single xgboost provided 0.506 on Public/Private LB, good enough for a gold medal.
\n\nOrder:
\n\nPath:
\n\nDatetime:
\n\nNumeric:
\n\nCategorical:
\n\nEarning 0.500+ LB is possible in 10 minutes using a single model (LightGBM) under 16GB RAM with appropriate features. But creating these features do need more than 16GB RAM…
\n\nxgboost can outperform LightGBM if tuned appropriately (my tuned xgboost was scoring 0.483 CV, which confirmed itself on LB, beating all our cross-validated model submissions including LightGBM) but it is much slower than LightGBM! (and was eating 58GB RAM.. mmm yummy! when you know LightGBM used only 16GB at most!)
\n\nIf you are not interested into reading about the team (and only were interested in the model/feature), you can skip everything below (there might be the "validation method" and "submission method" which you might want to read).
\n\nMultiple parts:
\n\nIt is just the concatenation of the initials of the team members! We didn’t have a hard time to find it… because I put it temporarily as a placeholder name. And then we didn’t even chat about the name...
\n\nI had so many people applying after my Red Hat forum post that I had to make a selection. For a team of 6 (that's big for Kaggle - but "big" is not that big, really) working like in a real project, it requires a lot of balancing and diversity in skills. Therefore, using the 2 to 20+ lines people sent me when “applying” (either directly by email for those who got my email, through Skype, through text, or through Kaggle), I selected 4 people (I had already one in my team) according to their skills:
\n\nFinally, we had: (including a diversity of R/Python)
\n\nAnd from my experience in making hackathons with mandatory teams, I do know that learners:
\n\nIn addition, we all have a RL (a real life) out of the world of Kaggle, therefore availabilities could be somewhat scarce and rare.
\n\nHence, typically we were 2 to 4 working on this competition. Thus, that’s not really a big team, and it was extremely easy to handle.
\n\nThere are two team events that happened:
\n\nAnd frankly, we were stuck with jayjay’s set for about 15 days. Literally, it took us that time before we started to see real improvements: there is a strange feature conditioning which made all our sets we had to not play well together (by pair - with jayjay’s). When we started to add new features (not from any existing feature set we had), real improvements kicked in. And there was only 1 week left when we started seeing good improvements…
\n\nWe used 5-fold cross validation for our supervised models. I generated a fold very early during this competition so we could all have the same basis to work on this data. But then we found my folds might not be stable enough! Therefore, someone created “stable folds” to use. However, what we found is that “too stable” is “not good enough”. The issue was probably lying in the duplicates. Therefore, we decided to stick with my initial folds, which are actually very good as we could spot fold instability very easily!
\n\nThis one is a very long text to explain. There is no magic behind submitting (compute threshold from CV), but not wasting submissions is another thing. How to submit on the LB was not very tricky. Not tricky due to the metric to optimize: either 0 or 1s. Add into the factors the fold instability... That was.. literally doom. I worked on a “submission creator” which took CV prediction (out of fold), CV test predictions, and a full model predictions, and which output a lot of different metrics in addition to a diagnostics file and 10 submission files.
\n\nEach of these 10 submission files had their positive count printed in the diagnostics. Here is an example of diagnostics.
\n\n Fold 1 converged after 0836 iterations.\n Fold 2 converged after 0571 iterations.\n Fold 3 converged after 0499 iterations.\n Fold 4 converged after 0585 iterations.\n Fold 5 converged after 0566 iterations.\n Iterations: 611.40 + 129.874\n\n\n Fold 1: AUC=0.9125954\n Fold 2: AUC=0.9221191\n Fold 3: AUC=0.9188253\n Fold 4: AUC=0.9200112\n Fold 5: AUC=0.9183175\n AUC: 0.9183737 + 0.0035463\n Average AUC using all data: 0.9037691\n\n\n Fold 1: MCC=0.4756098 (0534 [38.84%] positives), threshold=0.3749080 => True positives = 76.592%\n Fold 2: MCC=0.4769881 (0437 [31.76%] positives), threshold=0.5057880 => True positives = 84.897%\n Fold 3: MCC=0.4801341 (0560 [40.70%] positives), threshold=0.3444360 => True positives = 75.536%\n Fold 4: MCC=0.4737958 (0605 [43.97%] positives), threshold=0.3298900 => True positives = 71.736%\n Fold 5: MCC=0.4829743 (0615 [44.69%] positives), threshold=0.3086210 => True positives = 72.520%\n MCC: 0.4779004 + 0.0036627\n Threshold: 0.3727286 + 0.0781904\n Positives: 550.20 + 071.37\n Detection Rate %: 39.991 + 05.185\n True positives %: 76.256 + 05.237\n Undetected positives: 0959.20 + 0028.86\n Average MCC on all data (5 fold): 0.4734361, threshold=0.3727286\n Average MCC using all data: 0.4747755, threshold=0.3478090\n\n\n Submission overfitted threshold on all MCC positives: 2896\n\n Submission average validated threshold on all MCC positives: 2736\n\n Submission average of overfit+validated threshold positives: 2814\n\n Submission with all data overfitted threshold on all MCC positives: 2933. Threshold=0.347809\n\n Submission with all data average validated threshold on all MCC positives: 2803. Threshold=0.3727286\n\n Submission with all data average of overfit+validated threshold positives: 2867. Threshold=0.3602688\n\n Submission with all data by taking the amount of positives of overfitted threshold on all MCC positives: 2896. Threshold=0.354801\n\n Submission with all data by taking the amount of positives of average validated threshold on all MCC positives: 2736. Threshold=0.385727\n\n Submission with all data by taking the amount of positives of average of overfit+validated threshold positives: 2814. Threshold=0.370321\n\n Submission with all data by taking the sum of positives of validated positives: 2751. Threshold=0.383094\n\n Cross-validated used features list (all used features to copy & paste):\n\n c("sameL0_next", "sameL1_next", "CATEGORICAL_Last_____1", "S24.311", \n"sameL3_next", "GF1", "GF0", "FOR30_Sum_S", "CATEGORICAL_out_out_L3_S32_F3854_class2", \n (censored because way too long)\n\n Cross-validated multipresence used features list (all used features to copy & paste):\n\n c("sameL0_next", "sameL1_next", "CATEGORICAL_Last_____1", "S24.311", \n"sameL3_next", "GF1", "GF0", "FOR30_Sum_S", "CATEGORICAL_out_out_L3_S32_F3854_class2", \n (censored because way too long)\nDoes it look like Chinese to interpret what is wrong and what is OK? At first sight, yes. I made a small tutorial (by message on a chat… un-obviously) to help my team (those who are modeling and/or testing internally features) decipher the diagnostics file. Here are good excerpts of what I wrote:
\n\n\n\n\nMy folds have extreme feature variance on each fold (not intentional,\n but it seems to help a lot), so if your features don't work/generalize\n on several folds, it will decrease the stability.
\n \nCV variance is OK, but too much variance is not. I tested over 20\n combos on 3-4-5-6 CV folds locally to test how variance can increase:
\n \n\n
\n \n- at 6 fold, standard deviation (sd) is about 0.01
\n- 5 fold, sd about 0.006
\n- 4 fold, sd about 0.006 (not sure?)
\n- 3 fold, sd about 0.004
\nAt the top of the diagnostics, you have all diagnostic things to\n analyze the model, so you can check for consistency "out-of-fold" (out\n of fold predictions) and "all-of-out-of-fold" (when taking all out of\n fold predictions together): AUC, MCC, threshold, positive rate,\n detection rate, true positives, undetected positives (false\n negatives), MCC using 5-fold average threshold, MCC using all data.
\n \nIf the difference between both is high, most of thresholding methods\n we use will fail (and we have a fallback for this in this scenario).\n For instance, 0.92 avg AUC out of fold, 0.85 AUC all data is not\n stable for submission. You can have high AUC and "trash" MCC. But we\n use AUC to monitor if predictions remain stable on each fold so we\n don't get any surprize when training on all data (because from that\n point we go nearly "blind"). If your probabilities predicted are\n stable, AUC on all data should be close to the average per fold
\n \nWhen your folds are unstable (due to XYZ feature), the difference\n increases as a [0, 1] scale for a fold might not be much more\n different against the same [0, 1] scale for another fold.
\n \nMy script creates 10 submissions:
\n \n\n
\n \n- 1st one: use threshold using all data, on CV averaged prediction = mostly for debugging purposes
\n- 2nd one: use average threshold from CV, on CV averaged prediction = mostly for debugging purposes
\n- 3rd one: take mean of 1st and 2nd, on CV averaged prediction = mostly for debugging purposes
\nThen you have 3 more submissions created (4th, 5th, 6th) which uses\n the predicted probabilities on testing data using the same\n thresholding techniques of 1st, 2nd, and 3rd. By default, the 4th\n submission is the best to submit (it's not that "overfitted" - it\n overfits more and more if your fold stability is lower)
\n \nThen you have 7th 8th 9th, which uses the positive count found using\n the 1st 2nd 3rd methods but on the predictions using the model with\n all data together (those one should be avoided most of the times, but\n they are left as is if needed).
\n \nAnd the final one the 10th: it takes the positive count of the CV and\n uses that same count on the test data from the model using all data.\n It assumes the hypothesis the positive count in train = positive count\n in test, which might not be true. Obviously, no one knows if true or\n not. This is the submission you must use when your folds are clearly\n unstable (it is the fallback method for submitting with an unstable\n CV).
\n \nIf you see on these debug a number like 10,000 or more (usually it\n sits around 2000-4000) it means your folds are unstable (captain\n obvious is here) and you need to look out what happened. It means also\n your CV models / full model predictions are unstable too (so only 10th\n should be used, unless the one you pick has stable predictions = value\n around 2000-4000). Therefore, only the fallback submission should be\n valid from that point, in most cases.
\n \nIf you make a submission using 1st, 2nd, or 3rd set, you MUST\n approximately get what you had in CV. If you don’t, the feature you\n added is bad enough to be conditioned by something in the folds. When\n you want to test 1 or more submissions from a single model, you use\n this order if your model is stable according to the diagnostics and\n your sight of it:
\n \n\n
\n \n- 4th (nearly always the best)
\n- 6th (rare you need 2 submissions - only if you clearly think your model is good - acts as a confirmatory response vs your CV if 4th\n overshot the LB)
\n- 5th (very rare you will need 3 submissions to test a single model - typically happens when we don't have enough to submit, this should\n always score lower on LB than 4th/6th no matter what you do, unless\n luck)
\nIf unstable: - 10th (and only this one)
\n \nAverage MCC on all data (5 fold) <- use 5 fold threshold, average, and\n compute MCC using that average threshold
\n \nAverage MCC using all data <- compute threshold on all data, use that\n threshold on all data
\n \nCross-validated multipresence features are in case you have features which are not picked up by your model if you input LightGBM CV output from my package. You can't compare both using
\nfeaturelist1[which(!(featurelist1 %in% featurelist2))]in R.
Mainly the most important names for a fully decentralized team working on a Kaggle competition in a (non or not-non) project-mode:
\n\nThe team experience so far? Really great! So much fun and no tension in the team, with people listening to each other => a nice ambiance to spend time on Kaggle. And people wanting to learn from others is always a wonderful experience, as it is the TEST/challenge which checks whether your knowledge was correct or not correct / biased (yea, you can learn wrong something, it may happen).
\n\nAh, and I think this is the post which will put me to Master tier in Discussions. 199 medals, now 200. A good double Master tier from this competition. Anyways, great work from everyone in this competition! (not only my team but the others! - look at all these great kernels and especially the pathing one!)
\n\nThere are 5 new competition Masters according to the top 12 private LB! (Alexey Noskov, Laurae, Shubin, Michael Maguire, Alexandru Papiu) - congratulations!
", "messages": [ { "id": "144191", "postDate": "11/12/2016 22:29:34", "content": "Below is the solution of team LAJMBURO (8th private LB (0.50888), 11th public (0.51001) ).
\n\nThere are two audiences intended:
\n\nTwo parts:
\n\nWe used simple level 1 models:
\n\nWe had five level 1 datasets (gbm=LightGBM, xgb=xgboost, rf=Random Forest - features are not exactly including all the features described later), I’m including their mean MCC on cross-validation (for Public LB score, add at least 0.02 - numbers might be off a bit):
\n\nLevel 2 data set was a bit special, and included:
\n\nLevel 2 stack models (giving the weaker model a stronger weight was better):
\n\nWe had a very minor issue at overfitting LB:
\n\nIn the case it were to overfit severely, we also made a backup set which is leakage-free (using features not using the label directly) and without a multiplicand. For instance, features like the Unsupervised Nearest Neighbors was removed. A single xgboost provided 0.506 on Public/Private LB, good enough for a gold medal.
\n\nOrder:
\n\nPath:
\n\nDatetime:
\n\nNumeric:
\n\nCategorical:
\n\nEarning 0.500+ LB is possible in 10 minutes using a single model (LightGBM) under 16GB RAM with appropriate features. But creating these features do need more than 16GB RAM…
\n\nxgboost can outperform LightGBM if tuned appropriately (my tuned xgboost was scoring 0.483 CV, which confirmed itself on LB, beating all our cross-validated model submissions including LightGBM) but it is much slower than LightGBM! (and was eating 58GB RAM.. mmm yummy! when you know LightGBM used only 16GB at most!)
\n\nIf you are not interested into reading about the team (and only were interested in the model/feature), you can skip everything below (there might be the "validation method" and "submission method" which you might want to read).
\n\nMultiple parts:
\n\nIt is just the concatenation of the initials of the team members! We didn’t have a hard time to find it… because I put it temporarily as a placeholder name. And then we didn’t even chat about the name...
\n\nI had so many people applying after my Red Hat forum post that I had to make a selection. For a team of 6 (that's big for Kaggle - but "big" is not that big, really) working like in a real project, it requires a lot of balancing and diversity in skills. Therefore, using the 2 to 20+ lines people sent me when “applying” (either directly by email for those who got my email, through Skype, through text, or through Kaggle), I selected 4 people (I had already one in my team) according to their skills:
\n\nFinally, we had: (including a diversity of R/Python)
\n\nAnd from my experience in making hackathons with mandatory teams, I do know that learners:
\n\nIn addition, we all have a RL (a real life) out of the world of Kaggle, therefore availabilities could be somewhat scarce and rare.
\n\nHence, typically we were 2 to 4 working on this competition. Thus, that’s not really a big team, and it was extremely easy to handle.
\n\nThere are two team events that happened:
\n\nAnd frankly, we were stuck with jayjay’s set for about 15 days. Literally, it took us that time before we started to see real improvements: there is a strange feature conditioning which made all our sets we had to not play well together (by pair - with jayjay’s). When we started to add new features (not from any existing feature set we had), real improvements kicked in. And there was only 1 week left when we started seeing good improvements…
\n\nWe used 5-fold cross validation for our supervised models. I generated a fold very early during this competition so we could all have the same basis to work on this data. But then we found my folds might not be stable enough! Therefore, someone created “stable folds” to use. However, what we found is that “too stable” is “not good enough”. The issue was probably lying in the duplicates. Therefore, we decided to stick with my initial folds, which are actually very good as we could spot fold instability very easily!
\n\nThis one is a very long text to explain. There is no magic behind submitting (compute threshold from CV), but not wasting submissions is another thing. How to submit on the LB was not very tricky. Not tricky due to the metric to optimize: either 0 or 1s. Add into the factors the fold instability... That was.. literally doom. I worked on a “submission creator” which took CV prediction (out of fold), CV test predictions, and a full model predictions, and which output a lot of different metrics in addition to a diagnostics file and 10 submission files.
\n\nEach of these 10 submission files had their positive count printed in the diagnostics. Here is an example of diagnostics.
\n\n Fold 1 converged after 0836 iterations.\n Fold 2 converged after 0571 iterations.\n Fold 3 converged after 0499 iterations.\n Fold 4 converged after 0585 iterations.\n Fold 5 converged after 0566 iterations.\n Iterations: 611.40 + 129.874\n\n\n Fold 1: AUC=0.9125954\n Fold 2: AUC=0.9221191\n Fold 3: AUC=0.9188253\n Fold 4: AUC=0.9200112\n Fold 5: AUC=0.9183175\n AUC: 0.9183737 + 0.0035463\n Average AUC using all data: 0.9037691\n\n\n Fold 1: MCC=0.4756098 (0534 [38.84%] positives), threshold=0.3749080 => True positives = 76.592%\n Fold 2: MCC=0.4769881 (0437 [31.76%] positives), threshold=0.5057880 => True positives = 84.897%\n Fold 3: MCC=0.4801341 (0560 [40.70%] positives), threshold=0.3444360 => True positives = 75.536%\n Fold 4: MCC=0.4737958 (0605 [43.97%] positives), threshold=0.3298900 => True positives = 71.736%\n Fold 5: MCC=0.4829743 (0615 [44.69%] positives), threshold=0.3086210 => True positives = 72.520%\n MCC: 0.4779004 + 0.0036627\n Threshold: 0.3727286 + 0.0781904\n Positives: 550.20 + 071.37\n Detection Rate %: 39.991 + 05.185\n True positives %: 76.256 + 05.237\n Undetected positives: 0959.20 + 0028.86\n Average MCC on all data (5 fold): 0.4734361, threshold=0.3727286\n Average MCC using all data: 0.4747755, threshold=0.3478090\n\n\n Submission overfitted threshold on all MCC positives: 2896\n\n Submission average validated threshold on all MCC positives: 2736\n\n Submission average of overfit+validated threshold positives: 2814\n\n Submission with all data overfitted threshold on all MCC positives: 2933. Threshold=0.347809\n\n Submission with all data average validated threshold on all MCC positives: 2803. Threshold=0.3727286\n\n Submission with all data average of overfit+validated threshold positives: 2867. Threshold=0.3602688\n\n Submission with all data by taking the amount of positives of overfitted threshold on all MCC positives: 2896. Threshold=0.354801\n\n Submission with all data by taking the amount of positives of average validated threshold on all MCC positives: 2736. Threshold=0.385727\n\n Submission with all data by taking the amount of positives of average of overfit+validated threshold positives: 2814. Threshold=0.370321\n\n Submission with all data by taking the sum of positives of validated positives: 2751. Threshold=0.383094\n\n Cross-validated used features list (all used features to copy & paste):\n\n c("sameL0_next", "sameL1_next", "CATEGORICAL_Last_____1", "S24.311", \n"sameL3_next", "GF1", "GF0", "FOR30_Sum_S", "CATEGORICAL_out_out_L3_S32_F3854_class2", \n (censored because way too long)\n\n Cross-validated multipresence used features list (all used features to copy & paste):\n\n c("sameL0_next", "sameL1_next", "CATEGORICAL_Last_____1", "S24.311", \n"sameL3_next", "GF1", "GF0", "FOR30_Sum_S", "CATEGORICAL_out_out_L3_S32_F3854_class2", \n (censored because way too long)\nDoes it look like Chinese to interpret what is wrong and what is OK? At first sight, yes. I made a small tutorial (by message on a chat… un-obviously) to help my team (those who are modeling and/or testing internally features) decipher the diagnostics file. Here are good excerpts of what I wrote:
\n\n\n\n\nMy folds have extreme feature variance on each fold (not intentional,\n but it seems to help a lot), so if your features don't work/generalize\n on several folds, it will decrease the stability.
\n \nCV variance is OK, but too much variance is not. I tested over 20\n combos on 3-4-5-6 CV folds locally to test how variance can increase:
\n \n\n
\n \n- at 6 fold, standard deviation (sd) is about 0.01
\n- 5 fold, sd about 0.006
\n- 4 fold, sd about 0.006 (not sure?)
\n- 3 fold, sd about 0.004
\nAt the top of the diagnostics, you have all diagnostic things to\n analyze the model, so you can check for consistency "out-of-fold" (out\n of fold predictions) and "all-of-out-of-fold" (when taking all out of\n fold predictions together): AUC, MCC, threshold, positive rate,\n detection rate, true positives, undetected positives (false\n negatives), MCC using 5-fold average threshold, MCC using all data.
\n \nIf the difference between both is high, most of thresholding methods\n we use will fail (and we have a fallback for this in this scenario).\n For instance, 0.92 avg AUC out of fold, 0.85 AUC all data is not\n stable for submission. You can have high AUC and "trash" MCC. But we\n use AUC to monitor if predictions remain stable on each fold so we\n don't get any surprize when training on all data (because from that\n point we go nearly "blind"). If your probabilities predicted are\n stable, AUC on all data should be close to the average per fold
\n \nWhen your folds are unstable (due to XYZ feature), the difference\n increases as a [0, 1] scale for a fold might not be much more\n different against the same [0, 1] scale for another fold.
\n \nMy script creates 10 submissions:
\n \n\n
\n \n- 1st one: use threshold using all data, on CV averaged prediction = mostly for debugging purposes
\n- 2nd one: use average threshold from CV, on CV averaged prediction = mostly for debugging purposes
\n- 3rd one: take mean of 1st and 2nd, on CV averaged prediction = mostly for debugging purposes
\nThen you have 3 more submissions created (4th, 5th, 6th) which uses\n the predicted probabilities on testing data using the same\n thresholding techniques of 1st, 2nd, and 3rd. By default, the 4th\n submission is the best to submit (it's not that "overfitted" - it\n overfits more and more if your fold stability is lower)
\n \nThen you have 7th 8th 9th, which uses the positive count found using\n the 1st 2nd 3rd methods but on the predictions using the model with\n all data together (those one should be avoided most of the times, but\n they are left as is if needed).
\n \nAnd the final one the 10th: it takes the positive count of the CV and\n uses that same count on the test data from the model using all data.\n It assumes the hypothesis the positive count in train = positive count\n in test, which might not be true. Obviously, no one knows if true or\n not. This is the submission you must use when your folds are clearly\n unstable (it is the fallback method for submitting with an unstable\n CV).
\n \nIf you see on these debug a number like 10,000 or more (usually it\n sits around 2000-4000) it means your folds are unstable (captain\n obvious is here) and you need to look out what happened. It means also\n your CV models / full model predictions are unstable too (so only 10th\n should be used, unless the one you pick has stable predictions = value\n around 2000-4000). Therefore, only the fallback submission should be\n valid from that point, in most cases.
\n \nIf you make a submission using 1st, 2nd, or 3rd set, you MUST\n approximately get what you had in CV. If you don’t, the feature you\n added is bad enough to be conditioned by something in the folds. When\n you want to test 1 or more submissions from a single model, you use\n this order if your model is stable according to the diagnostics and\n your sight of it:
\n \n\n
\n \n- 4th (nearly always the best)
\n- 6th (rare you need 2 submissions - only if you clearly think your model is good - acts as a confirmatory response vs your CV if 4th\n overshot the LB)
\n- 5th (very rare you will need 3 submissions to test a single model - typically happens when we don't have enough to submit, this should\n always score lower on LB than 4th/6th no matter what you do, unless\n luck)
\nIf unstable: - 10th (and only this one)
\n \nAverage MCC on all data (5 fold) <- use 5 fold threshold, average, and\n compute MCC using that average threshold
\n \nAverage MCC using all data <- compute threshold on all data, use that\n threshold on all data
\n \nCross-validated multipresence features are in case you have features which are not picked up by your model if you input LightGBM CV output from my package. You can't compare both using
\nfeaturelist1[which(!(featurelist1 %in% featurelist2))]in R.
Mainly the most important names for a fully decentralized team working on a Kaggle competition in a (non or not-non) project-mode:
\n\nThe team experience so far? Really great! So much fun and no tension in the team, with people listening to each other => a nice ambiance to spend time on Kaggle. And people wanting to learn from others is always a wonderful experience, as it is the TEST/challenge which checks whether your knowledge was correct or not correct / biased (yea, you can learn wrong something, it may happen).
\n\nAh, and I think this is the post which will put me to Master tier in Discussions. 199 medals, now 200. A good double Master tier from this competition. Anyways, great work from everyone in this competition! (not only my team but the others! - look at all these great kernels and especially the pathing one!)
\n\nThere are 5 new competition Masters according to the top 12 private LB! (Alexey Noskov, Laurae, Shubin, Michael Maguire, Alexandru Papiu) - congratulations!
", "rawMarkdown": "Below is the solution of team LAJMBURO (8th private LB (0.50888), 11th public (0.51001) ).\r\n\r\nThere are two audiences intended:\r\n\r\n* If you are interested in only the models/features, then read the part 1 until \"Features\" included and you can skip everything else (maybe you might want to read \"Validation method\" and \"Submission method\" from part 2 of this post.\r\n* If you are interested at how a team of 8 is working, then read from “the name of the team…” which is part 2\r\n\r\n---\r\n\r\n## Part 1: Models & Features\r\n\r\nTwo parts:\r\n\r\n* Modeling and data\r\n* Features\r\n\r\n### Modeling and data\r\n\r\nWe used simple **level 1 models**:\r\n\r\n* LightGBM\r\n* xgboost\r\n* Random Forest\r\n* Neural Networks (didn’t get picked up on level 2, so we removed it)\r\n\r\nWe had **five level 1 datasets** (gbm=LightGBM, xgb=xgboost, rf=Random Forest - features are not exactly including all the features described later), I’m including their **mean MCC on cross-validation** (for Public LB score, add at least 0.02 - numbers might be off a bit):\r\n\r\n* Data set 1 (0.477 gbm): order, raw numeric, date, categorical\r\n* Data set 2 (0.482 gbm, 0.477 xgb, 0.473 rf): order, path, raw numeric, date\r\n* Data set 3 (0.479 gbm, 0.473 xgb): order, path, numeric, date, refined categorical\r\n* Data set 4 (0.469 xgb, 0.442 rf): has features sorted by numeric values + date features + faron’s magic features, path, unsupervised nearest neighbors (L1 = Manhattan / L2 = Euclidean distances) per label\r\n* Data set 5 (0.43 xgb): has faron’s magic features, path, unsupervised nearest neighbors\r\n\r\n**Level 2 data set** was a bit special, and included:\r\n\r\n* Level 1 predictions (we had 12 predictions from level 1)\r\n* Data set 5\r\n* Duplicate feature (count and position)\r\n\r\n**Level 2 stack models** (giving the weaker model a stronger weight was better):\r\n\r\n* 30% weighted xgboost gbtree (~0.488 CV)\r\n* 70% weighted Random Forest (~0.485 CV)\r\n\r\nWe had a very **minor issue at overfitting LB**:\r\n\r\n* We used a threshold multiplicand of 0.92 from the cross-validated threshold (which seemed the best multiplicand), 0.51001 Public LB, 0.50888 Private LB\r\n* If we were using the threshold from cross-validation without a multiplicand, we would have had, 0.50923 Public LB, 0.50954 Private LB. But that's not an issue at all overall.\r\n\r\nIn the case it were to overfit severely, we also made a backup set which is leakage-free (using features not using the label directly) and without a multiplicand. For instance, features like the Unsupervised Nearest Neighbors was removed. A single xgboost provided 0.506 on Public/Private LB, good enough for a gold medal.\r\n\r\n### Features:\r\n\r\n**Order**:\r\n\r\n* Faron’s magic features (sorted by max time, min time)\r\n* Magic features sorted by numeric values\r\n* Same timestamps previous/next sample after sorting by tmax\r\n* Previous / next response for duplicates\r\n\r\n**Path**:\r\n\r\n* Redefine station and line numbers based on timestamps to make sure timestamp was constant for each station\r\n* Cluster of samples by path and calculated per sample the absolute and relative difference between numeric / time difference (max - min) features and cluster mean. \r\n* Path based error rate (leave one out method) but it did not improve score\r\n* Merge features based on path\r\n* Entry / exit station\r\n* Previous and next station for S29 - S37 (one-hot encoded)\r\n\r\n**Datetime**:\r\n\r\n* Timestamp per station, per line, merged per path\r\n* Max-min per station, per line, merged per path\r\n* Kurtosis + kurtosis per line (very strong, surprisingly)\r\n* Lead/lag response rate statistics after sorting by tmax, id\r\n* Lead/lag response rate statistics after sorting by tmax, numeric, id\r\n* Timestamp based error rate (leave one out method) but it did not improve score\r\n* Timestamp label-based density (overfit)\r\n\r\n**Numeric**:\r\n\r\n* Use supervised decision trees to try to predict the numeric value using the date, per station => the split points allow to define thresholds, which clustered the values by group of dates (over 4000 features) => used xgboost per station to shrink it to ~120 features as adding 4000 features didn’t help at all\r\n* Numeric after removing time trend (did not improve score)\r\n* Raw numeric data (a specific selection of them)\r\n\r\n**Categorical**:\r\n\r\n* Couple of features one-hot encoded added a little bit\r\n\r\n### In the end\r\n\r\nEarning 0.500+ LB is possible in 10 minutes using a single model (LightGBM) under 16GB RAM with appropriate features. But creating these features do need more than 16GB RAM…\r\n\r\nxgboost can outperform LightGBM if tuned appropriately (my tuned xgboost was scoring 0.483 CV, which confirmed itself on LB, beating all our cross-validated model submissions including LightGBM) but it is much slower than LightGBM! (and was eating 58GB RAM.. mmm yummy! when you know LightGBM used only 16GB at most!)\r\n\r\n---\r\n\r\n---\r\n\r\nIf you are not interested into reading about the team (and only were interested in the model/feature), you can skip everything below (there might be the \"validation method\" and \"submission method\" which you might want to read).\r\n\r\n---\r\n\r\n---\r\n\r\n## Part 2: The team\r\n\r\nMultiple parts:\r\n\r\n* The name of the team\r\n* Team composition (this is what happens when you are play the recruiter for a team project)\r\n* Validation method\r\n* Stuff we used in the team\r\n\r\n### The name of the team…\r\n\r\nIt is just the concatenation of the initials of the team members! We didn’t have a hard time to find it… because I put it temporarily as a placeholder name. And then we didn’t even chat about the name...\r\n\r\n### Team composition\r\n\r\nI had so many people applying after my Red Hat forum post that I had to make a selection. For a team of 6 (that's big for Kaggle - but \"big\" is not that big, really) working like in a real project, it requires a lot of balancing and diversity in skills. Therefore, using the 2 to 20+ lines people sent me when “applying” (either directly by email for those who got my email, through Skype, through text, or through Kaggle), I selected 4 people (I had already one in my team) according to their skills:\r\n\r\n**Finally, we had:** (including a diversity of R/Python)\r\n\r\n* One who can go “ham” for trying to push on LB (Shubin) - great competition spirit\r\n* One with a lot of “let’s try this” features (Joost) - it helped a lot\r\n* One who can analyze methodically what he has and what he gets as feedback to improve (Michael) - very good at avoiding going too fast\r\n* One who can help providing better insights using R packages over Python packages (Rodolphe) - great at providing alternatives\r\n* One who is more about the organization and providing a good kickstart with sample test scripts for the team (Alex, cf see [what][1] he can do in the House Prices competition)\r\n\r\nAnd from my experience in making hackathons with mandatory teams, I do know that learners:\r\n\r\n* Are starting very fresh in mind and are extremely committed early\r\n* Are (not always for any person) slowly dropping out as time passes by and commitment requires to increase (which is exactly what must not happen when kaggling)\r\n\r\nIn addition, we all have a RL (a real life) out of the world of Kaggle, therefore availabilities could be somewhat scarce and rare.\r\n\r\nHence, typically we were 2 to 4 working on this competition. Thus, that’s not really a big team, and it was extremely easy to handle.\r\n\r\nThere are two team events that happened:\r\n\r\n* Adding jayjay to our team, when he asked for a team (while he was in top 15 Public LB with 0.49+), as he had a solid feature set (we knew it could be a bit overfitting before even merging with him, as our team experienced massive differences by just changing the threshold for MCC)\r\n* Adding onthehilluu to our team, to think more about pushing on LB (feature insight) and sharing knowledge/experience (when we merged with jayjay and managed to push a bit on LB with his single model, the initial objective of sharing experience was left behind, and the new objective was set to get a gold medal, i.e top 12)\r\n\r\nAnd frankly, we were stuck with jayjay’s set for about 15 days. Literally, it took us that time before we started to see real improvements: there is a strange feature conditioning which made all our sets we had to not play well together (by pair - with jayjay’s). When we started to add new features (not from any existing feature set we had), real improvements kicked in. And there was only 1 week left when we started seeing good improvements…\r\n\r\n### Validation method\r\n\r\nWe used 5-fold cross validation for our supervised models. I generated a fold very early during this competition so we could all have the same basis to work on this data. But then we found my folds might not be stable enough! Therefore, someone created “stable folds” to use. However, what we found is that “too stable” is “not good enough”. The issue was probably lying in the duplicates. Therefore, we decided to stick with my initial folds, which are actually very good as we could spot fold instability very easily!\r\n\r\n### Submission method\r\n\r\nThis one is a very long text to explain. There is no magic behind submitting (compute threshold from CV), but not wasting submissions is another thing. How to submit on the LB was not very tricky. Not tricky due to the metric to optimize: either 0 or 1s. Add into the factors the fold instability... That was.. literally doom. I worked on a “submission creator” which took CV prediction (out of fold), CV test predictions, and a full model predictions, and which output a lot of different metrics in addition to a diagnostics file and 10 submission files.\r\n\r\nEach of these 10 submission files had their positive count printed in the diagnostics. Here is an example of diagnostics.\r\n\r\n Fold 1 converged after 0836 iterations.\r\n Fold 2 converged after 0571 iterations.\r\n Fold 3 converged after 0499 iterations.\r\n Fold 4 converged after 0585 iterations.\r\n Fold 5 converged after 0566 iterations.\r\n Iterations: 611.40 + 129.874\r\n \r\n \r\n Fold 1: AUC=0.9125954\r\n Fold 2: AUC=0.9221191\r\n Fold 3: AUC=0.9188253\r\n Fold 4: AUC=0.9200112\r\n Fold 5: AUC=0.9183175\r\n AUC: 0.9183737 + 0.0035463\r\n Average AUC using all data: 0.9037691\r\n \r\n \r\n Fold 1: MCC=0.4756098 (0534 [38.84%] positives), threshold=0.3749080 => True positives = 76.592%\r\n Fold 2: MCC=0.4769881 (0437 [31.76%] positives), threshold=0.5057880 => True positives = 84.897%\r\n Fold 3: MCC=0.4801341 (0560 [40.70%] positives), threshold=0.3444360 => True positives = 75.536%\r\n Fold 4: MCC=0.4737958 (0605 [43.97%] positives), threshold=0.3298900 => True positives = 71.736%\r\n Fold 5: MCC=0.4829743 (0615 [44.69%] positives), threshold=0.3086210 => True positives = 72.520%\r\n MCC: 0.4779004 + 0.0036627\r\n Threshold: 0.3727286 + 0.0781904\r\n Positives: 550.20 + 071.37\r\n Detection Rate %: 39.991 + 05.185\r\n True positives %: 76.256 + 05.237\r\n Undetected positives: 0959.20 + 0028.86\r\n Average MCC on all data (5 fold): 0.4734361, threshold=0.3727286\r\n Average MCC using all data: 0.4747755, threshold=0.3478090\r\n \r\n \r\n Submission overfitted threshold on all MCC positives: 2896\r\n \r\n Submission average validated threshold on all MCC positives: 2736\r\n \r\n Submission average of overfit+validated threshold positives: 2814\r\n \r\n Submission with all data overfitted threshold on all MCC positives: 2933. Threshold=0.347809\r\n \r\n Submission with all data average validated threshold on all MCC positives: 2803. Threshold=0.3727286\r\n \r\n Submission with all data average of overfit+validated threshold positives: 2867. Threshold=0.3602688\r\n \r\n Submission with all data by taking the amount of positives of overfitted threshold on all MCC positives: 2896. Threshold=0.354801\r\n \r\n Submission with all data by taking the amount of positives of average validated threshold on all MCC positives: 2736. Threshold=0.385727\r\n \r\n Submission with all data by taking the amount of positives of average of overfit+validated threshold positives: 2814. Threshold=0.370321\r\n \r\n Submission with all data by taking the sum of positives of validated positives: 2751. Threshold=0.383094\r\n \r\n Cross-validated used features list (all used features to copy & paste):\r\n \r\n c(\"sameL0_next\", \"sameL1_next\", \"CATEGORICAL_Last_____1\", \"S24.311\", \r\n \"sameL3_next\", \"GF1\", \"GF0\", \"FOR30_Sum_S\", \"CATEGORICAL_out_out_L3_S32_F3854_class2\", \r\n (censored because way too long)\r\n \r\n Cross-validated multipresence used features list (all used features to copy & paste):\r\n \r\n c(\"sameL0_next\", \"sameL1_next\", \"CATEGORICAL_Last_____1\", \"S24.311\", \r\n \"sameL3_next\", \"GF1\", \"GF0\", \"FOR30_Sum_S\", \"CATEGORICAL_out_out_L3_S32_F3854_class2\", \r\n (censored because way too long)\r\n\r\nDoes it look like Chinese to interpret what is wrong and what is OK? At first sight, yes. I made a small tutorial (by message on a chat… un-obviously) to help my team (those who are modeling and/or testing internally features) decipher the diagnostics file. Here are good excerpts of what I wrote:\r\n\r\n> My folds have extreme feature variance on each fold (not intentional,\r\n> but it seems to help a lot), so if your features don't work/generalize\r\n> on several folds, it will decrease the stability.\r\n> \r\n> CV variance is OK, but too much variance is not. I tested over 20\r\n> combos on 3-4-5-6 CV folds locally to test how variance can increase:\r\n> \r\n> * at 6 fold, standard deviation (sd) is about 0.01\r\n> * 5 fold, sd about 0.006\r\n> * 4 fold, sd about 0.006 (not sure?)\r\n> * 3 fold, sd about 0.004\r\n> \r\n> At the top of the diagnostics, you have all diagnostic things to\r\n> analyze the model, so you can check for consistency \"out-of-fold\" (out\r\n> of fold predictions) and \"all-of-out-of-fold\" (when taking all out of\r\n> fold predictions together): AUC, MCC, threshold, positive rate,\r\n> detection rate, true positives, undetected positives (false\r\n> negatives), MCC using 5-fold average threshold, MCC using all data.\r\n> \r\n> If the difference between both is high, most of thresholding methods\r\n> we use will fail (and we have a fallback for this in this scenario).\r\n> For instance, 0.92 avg AUC out of fold, 0.85 AUC all data is not\r\n> stable for submission. You can have high AUC and \"trash\" MCC. But we\r\n> use AUC to monitor if predictions remain stable on each fold so we\r\n> don't get any surprize when training on all data (because from that\r\n> point we go nearly \"blind\"). If your probabilities predicted are\r\n> stable, AUC on all data should be close to the average per fold\r\n> \r\n> When your folds are unstable (due to XYZ feature), the difference\r\n> increases as a [0, 1] scale for a fold might not be much more\r\n> different against the same [0, 1] scale for another fold.\r\n> \r\n> My script creates 10 submissions:\r\n> \r\n> * 1st one: use threshold using all data, on CV averaged prediction = mostly for debugging purposes\r\n> * 2nd one: use average threshold from CV, on CV averaged prediction = mostly for debugging purposes\r\n> * 3rd one: take mean of 1st and 2nd, on CV averaged prediction = mostly for debugging purposes\r\n> \r\n> Then you have 3 more submissions created (4th, 5th, 6th) which uses\r\n> the predicted probabilities on testing data using the same\r\n> thresholding techniques of 1st, 2nd, and 3rd. By default, the 4th\r\n> submission is the best to submit (it's not that \"overfitted\" - it\r\n> overfits more and more if your fold stability is lower)\r\n> \r\n> Then you have 7th 8th 9th, which uses the positive count found using\r\n> the 1st 2nd 3rd methods but on the predictions using the model with\r\n> all data together (those one should be avoided most of the times, but\r\n> they are left as is if needed).\r\n> \r\n> And the final one the 10th: it takes the positive count of the CV and\r\n> uses that same count on the test data from the model using all data.\r\n> It assumes the hypothesis the positive count in train = positive count\r\n> in test, which might not be true. Obviously, no one knows if true or\r\n> not. This is the submission you must use when your folds are clearly\r\n> unstable (it is the fallback method for submitting with an unstable\r\n> CV).\r\n> \r\n> If you see on these debug a number like 10,000 or more (usually it\r\n> sits around 2000-4000) it means your folds are unstable (captain\r\n> obvious is here) and you need to look out what happened. It means also\r\n> your CV models / full model predictions are unstable too (so only 10th\r\n> should be used, unless the one you pick has stable predictions = value\r\n> around 2000-4000). Therefore, only the fallback submission should be\r\n> valid from that point, in most cases.\r\n> \r\n> If you make a submission using 1st, 2nd, or 3rd set, you MUST\r\n> approximately get what you had in CV. If you don’t, the feature you\r\n> added is bad enough to be conditioned by something in the folds. When\r\n> you want to test 1 or more submissions from a single model, you use\r\n> this order if your model is stable according to the diagnostics and\r\n> your sight of it:\r\n> \r\n> * 4th (nearly always the best)\r\n> * 6th (rare you need 2 submissions - only if you clearly think your model is good - acts as a confirmatory response vs your CV if 4th\r\n> overshot the LB)\r\n> * 5th (very rare you will need 3 submissions to test a single model - typically happens when we don't have enough to submit, this should\r\n> always score lower on LB than 4th/6th no matter what you do, unless\r\n> luck)\r\n> \r\n> If unstable: - 10th (and only this one)\r\n> \r\n> Average MCC on all data (5 fold) <- use 5 fold threshold, average, and\r\n> compute MCC using that average threshold\r\n> \r\n> Average MCC using all data <- compute threshold on all data, use that\r\n> threshold on all data\r\n> \r\n> Cross-validated multipresence features are in case you have features which are not picked up by your model if you input LightGBM CV output from my package. You can't compare both using `featurelist1[which(!(featurelist1 %in% featurelist2))]` in R.\r\n\r\n### Stuff we used in the team\r\n\r\nMainly the most important names for a fully decentralized team working on a Kaggle competition in a (non or not-non) project-mode:\r\n\r\n* R programming language\r\n* Python programming language\r\n* [LightGBM][2] through the [Laurae package][3] (fastest R implementation at that time - you can do a 5-fold CV on 1000+ features on less than 40 minutes, I/O included)\r\n* [xgboost][4] (because “unidimensional machine learning at Kaggle is not Kaggle without xgboost”)\r\n* scikit-learn Random Forest (eats RAM but it works in Python without having to go through Java)\r\n* H2O Random Forest (when in R you want to use parallelized and fast Random Forests…. T_T)\r\n* Keras Neural Networks (who needs a neural network in the brain?)\r\n* data.table (fread/fwrite and super fast data manipulation, please)\r\n* Markdown, Rmarkdown, hacked pandoc (welcome Visual Studio executable hacks, by Laurae), iPython notebooks (yea, yea, we did reporting in the team to share insights / do “bookkeeping” - in a real project-mode)\r\n* Computers (who does not need a computer to compete at Kaggle? Use a phone?)\r\n* Monitors (because we are not blind)\r\n* Windows / Linux (because you need an operating system on your computer…)\r\nAWS / Google Cloud Platform (when you needed more horsepower on your car)\r\n* SSH FTP & Filezilla (how do you share data securely with the 1Gbps network from Laurae with all the team, when you have 10GB files?)\r\n* RStudio / Spyder (who needs an IDE to program?)\r\n* Remote Desktop (to share sessions on a server, troubleshooting, etc.)\r\n* Skype / Phone (could use Slack too, but we didn’t in the end as Skype was faster)\r\n* GitHub (private repository, to share code privately)\r\n* An internet browser to go on websites (it’s obvious you need to use a browser somewhere)\r\n\r\nThe team experience so far? Really great! So much fun and no tension in the team, with people listening to each other => a nice ambiance to spend time on Kaggle. And people wanting to learn from others is always a wonderful experience, as it is the **TEST/challenge which checks whether your knowledge was correct or not correct / biased** (yea, you can learn wrong something, it may happen).\r\n\r\n--- \r\n\r\nAh, and I think this is the post which will put me to Master tier in Discussions. 199 medals, now 200. A good double Master tier from this competition. Anyways, great work from everyone in this competition! (not only my team but the others! - look at all these great kernels and especially the pathing one!)\r\n\r\nThere are 5 new competition Masters according to the top 12 private LB! (Alexey Noskov, Laurae, Shubin, Michael Maguire, Alexandru Papiu) - congratulations!\r\n\r\n\r\n [1]: https://www.kaggle.com/apapiu/house-prices-advanced-regression-techniques/regularized-linear-models\r\n [2]: https://github.com/Microsoft/LightGBM\r\n [3]: https://github.com/Laurae2/Laurae\r\n [4]: https://github.com/dmlc/xgboost", "votes": null }, { "id": "144194", "postDate": "11/12/2016 22:52:06", "content": "@Laurae, Thanks a lot for pulling this team together! As you describe, it was a great experience for us all. We would never have achieved this result without your end-to-end knowledge and Kaggle experience. Your detailed post is another great example of the almost endless time and effort you put into sharing your knowledge with the Kaggle community. Thanks again!
", "rawMarkdown": "Laurae, Thanks a lot for pulling this team together! As you describe, it was a great experience for us all. We would never have achieved this result without your end-to-end knowledge and Kaggle experience. Your detailed post is another great example of the almost endless time and effort you put into sharing your knowledge with the Kaggle community. Thanks again!", "votes": null }, { "id": "144255", "postDate": "11/13/2016 11:36:52", "content": "Thanks for sharing.
\n\nClever idea to create a real data science project team. Results well deserved. Congrats on becoming a master.
\n\nedited after I read @Laurae's RedHat post
", "rawMarkdown": "Thanks for sharing. \r\n\r\nClever idea to create a real data science project team. Results well deserved. Congrats on becoming a master.\r\n\r\n*edited after I read @Laurae's RedHat post*", "votes": null }, { "id": "145282", "postDate": "11/17/2016 17:11:51", "content": "Always willing to share Laurae, thanks for all you gave, I learned a lot.
", "rawMarkdown": "Always willing to share Laurae, thanks for all you gave, I learned a lot.", "votes": null }, { "id": "146011", "postDate": "11/22/2016 09:09:16", "content": "Thanks for sharing! \nLong story and learned a lot
", "rawMarkdown": "Thanks for sharing! \r\nLong story and learned a lot", "votes": null } ], "comments": [ { "id": 144194, "author_name": "joostgp", "author_url": "", "post_date": "11/12/2016 22:52:06", "content": "@Laurae, Thanks a lot for pulling this team together! As you describe, it was a great experience for us all. We would never have achieved this result without your end-to-end knowledge and Kaggle experience. Your detailed post is another great example of the almost endless time and effort you put into sharing your knowledge with the Kaggle community. Thanks again!
", "votes": null, "replies": [] }, { "id": 144255, "author_name": "cpmpml", "author_url": "", "post_date": "11/13/2016 11:36:52", "content": "Thanks for sharing.
\n\nClever idea to create a real data science project team. Results well deserved. Congrats on becoming a master.
\n\nedited after I read @Laurae's RedHat post
", "votes": null, "replies": [] }, { "id": 145282, "author_name": "rodolphelampe", "author_url": "", "post_date": "11/17/2016 17:11:51", "content": "Always willing to share Laurae, thanks for all you gave, I learned a lot.
", "votes": null, "replies": [] }, { "id": 146011, "author_name": "fancyhelloworld", "author_url": "", "post_date": "11/22/2016 09:09:16", "content": "Thanks for sharing! \nLong story and learned a lot
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "144191": "Below is the solution of team LAJMBURO (8th private LB (0.50888), 11th public (0.51001) ).\r\n\r\nThere are two audiences intended:\r\n\r\n* If you are interested in only the models/features, then read the part 1 until \"Features\" included and you can skip everything else (maybe you might want to read \"Validation method\" and \"Submission method\" from part 2 of this post.\r\n* If you are interested at how a team of 8 is working, then read from “the name of the team…” which is part 2\r\n\r\n---\r\n\r\n## Part 1: Models & Features\r\n\r\nTwo parts:\r\n\r\n* Modeling and data\r\n* Features\r\n\r\n### Modeling and data\r\n\r\nWe used simple **level 1 models**:\r\n\r\n* LightGBM\r\n* xgboost\r\n* Random Forest\r\n* Neural Networks (didn’t get picked up on level 2, so we removed it)\r\n\r\nWe had **five level 1 datasets** (gbm=LightGBM, xgb=xgboost, rf=Random Forest - features are not exactly including all the features described later), I’m including their **mean MCC on cross-validation** (for Public LB score, add at least 0.02 - numbers might be off a bit):\r\n\r\n* Data set 1 (0.477 gbm): order, raw numeric, date, categorical\r\n* Data set 2 (0.482 gbm, 0.477 xgb, 0.473 rf): order, path, raw numeric, date\r\n* Data set 3 (0.479 gbm, 0.473 xgb): order, path, numeric, date, refined categorical\r\n* Data set 4 (0.469 xgb, 0.442 rf): has features sorted by numeric values + date features + faron’s magic features, path, unsupervised nearest neighbors (L1 = Manhattan / L2 = Euclidean distances) per label\r\n* Data set 5 (0.43 xgb): has faron’s magic features, path, unsupervised nearest neighbors\r\n\r\n**Level 2 data set** was a bit special, and included:\r\n\r\n* Level 1 predictions (we had 12 predictions from level 1)\r\n* Data set 5\r\n* Duplicate feature (count and position)\r\n\r\n**Level 2 stack models** (giving the weaker model a stronger weight was better):\r\n\r\n* 30% weighted xgboost gbtree (~0.488 CV)\r\n* 70% weighted Random Forest (~0.485 CV)\r\n\r\nWe had a very **minor issue at overfitting LB**:\r\n\r\n* We used a threshold multiplicand of 0.92 from the cross-validated threshold (which seemed the best multiplicand), 0.51001 Public LB, 0.50888 Private LB\r\n* If we were using the threshold from cross-validation without a multiplicand, we would have had, 0.50923 Public LB, 0.50954 Private LB. But that's not an issue at all overall.\r\n\r\nIn the case it were to overfit severely, we also made a backup set which is leakage-free (using features not using the label directly) and without a multiplicand. For instance, features like the Unsupervised Nearest Neighbors was removed. A single xgboost provided 0.506 on Public/Private LB, good enough for a gold medal.\r\n\r\n### Features:\r\n\r\n**Order**:\r\n\r\n* Faron’s magic features (sorted by max time, min time)\r\n* Magic features sorted by numeric values\r\n* Same timestamps previous/next sample after sorting by tmax\r\n* Previous / next response for duplicates\r\n\r\n**Path**:\r\n\r\n* Redefine station and line numbers based on timestamps to make sure timestamp was constant for each station\r\n* Cluster of samples by path and calculated per sample the absolute and relative difference between numeric / time difference (max - min) features and cluster mean. \r\n* Path based error rate (leave one out method) but it did not improve score\r\n* Merge features based on path\r\n* Entry / exit station\r\n* Previous and next station for S29 - S37 (one-hot encoded)\r\n\r\n**Datetime**:\r\n\r\n* Timestamp per station, per line, merged per path\r\n* Max-min per station, per line, merged per path\r\n* Kurtosis + kurtosis per line (very strong, surprisingly)\r\n* Lead/lag response rate statistics after sorting by tmax, id\r\n* Lead/lag response rate statistics after sorting by tmax, numeric, id\r\n* Timestamp based error rate (leave one out method) but it did not improve score\r\n* Timestamp label-based density (overfit)\r\n\r\n**Numeric**:\r\n\r\n* Use supervised decision trees to try to predict the numeric value using the date, per station => the split points allow to define thresholds, which clustered the values by group of dates (over 4000 features) => used xgboost per station to shrink it to ~120 features as adding 4000 features didn’t help at all\r\n* Numeric after removing time trend (did not improve score)\r\n* Raw numeric data (a specific selection of them)\r\n\r\n**Categorical**:\r\n\r\n* Couple of features one-hot encoded added a little bit\r\n\r\n### In the end\r\n\r\nEarning 0.500+ LB is possible in 10 minutes using a single model (LightGBM) under 16GB RAM with appropriate features. But creating these features do need more than 16GB RAM…\r\n\r\nxgboost can outperform LightGBM if tuned appropriately (my tuned xgboost was scoring 0.483 CV, which confirmed itself on LB, beating all our cross-validated model submissions including LightGBM) but it is much slower than LightGBM! (and was eating 58GB RAM.. mmm yummy! when you know LightGBM used only 16GB at most!)\r\n\r\n---\r\n\r\n---\r\n\r\nIf you are not interested into reading about the team (and only were interested in the model/feature), you can skip everything below (there might be the \"validation method\" and \"submission method\" which you might want to read).\r\n\r\n---\r\n\r\n---\r\n\r\n## Part 2: The team\r\n\r\nMultiple parts:\r\n\r\n* The name of the team\r\n* Team composition (this is what happens when you are play the recruiter for a team project)\r\n* Validation method\r\n* Stuff we used in the team\r\n\r\n### The name of the team…\r\n\r\nIt is just the concatenation of the initials of the team members! We didn’t have a hard time to find it… because I put it temporarily as a placeholder name. And then we didn’t even chat about the name...\r\n\r\n### Team composition\r\n\r\nI had so many people applying after my Red Hat forum post that I had to make a selection. For a team of 6 (that's big for Kaggle - but \"big\" is not that big, really) working like in a real project, it requires a lot of balancing and diversity in skills. Therefore, using the 2 to 20+ lines people sent me when “applying” (either directly by email for those who got my email, through Skype, through text, or through Kaggle), I selected 4 people (I had already one in my team) according to their skills:\r\n\r\n**Finally, we had:** (including a diversity of R/Python)\r\n\r\n* One who can go “ham” for trying to push on LB (Shubin) - great competition spirit\r\n* One with a lot of “let’s try this” features (Joost) - it helped a lot\r\n* One who can analyze methodically what he has and what he gets as feedback to improve (Michael) - very good at avoiding going too fast\r\n* One who can help providing better insights using R packages over Python packages (Rodolphe) - great at providing alternatives\r\n* One who is more about the organization and providing a good kickstart with sample test scripts for the team (Alex, cf see [what][1] he can do in the House Prices competition)\r\n\r\nAnd from my experience in making hackathons with mandatory teams, I do know that learners:\r\n\r\n* Are starting very fresh in mind and are extremely committed early\r\n* Are (not always for any person) slowly dropping out as time passes by and commitment requires to increase (which is exactly what must not happen when kaggling)\r\n\r\nIn addition, we all have a RL (a real life) out of the world of Kaggle, therefore availabilities could be somewhat scarce and rare.\r\n\r\nHence, typically we were 2 to 4 working on this competition. Thus, that’s not really a big team, and it was extremely easy to handle.\r\n\r\nThere are two team events that happened:\r\n\r\n* Adding jayjay to our team, when he asked for a team (while he was in top 15 Public LB with 0.49+), as he had a solid feature set (we knew it could be a bit overfitting before even merging with him, as our team experienced massive differences by just changing the threshold for MCC)\r\n* Adding onthehilluu to our team, to think more about pushing on LB (feature insight) and sharing knowledge/experience (when we merged with jayjay and managed to push a bit on LB with his single model, the initial objective of sharing experience was left behind, and the new objective was set to get a gold medal, i.e top 12)\r\n\r\nAnd frankly, we were stuck with jayjay’s set for about 15 days. Literally, it took us that time before we started to see real improvements: there is a strange feature conditioning which made all our sets we had to not play well together (by pair - with jayjay’s). When we started to add new features (not from any existing feature set we had), real improvements kicked in. And there was only 1 week left when we started seeing good improvements…\r\n\r\n### Validation method\r\n\r\nWe used 5-fold cross validation for our supervised models. I generated a fold very early during this competition so we could all have the same basis to work on this data. But then we found my folds might not be stable enough! Therefore, someone created “stable folds” to use. However, what we found is that “too stable” is “not good enough”. The issue was probably lying in the duplicates. Therefore, we decided to stick with my initial folds, which are actually very good as we could spot fold instability very easily!\r\n\r\n### Submission method\r\n\r\nThis one is a very long text to explain. There is no magic behind submitting (compute threshold from CV), but not wasting submissions is another thing. How to submit on the LB was not very tricky. Not tricky due to the metric to optimize: either 0 or 1s. Add into the factors the fold instability... That was.. literally doom. I worked on a “submission creator” which took CV prediction (out of fold), CV test predictions, and a full model predictions, and which output a lot of different metrics in addition to a diagnostics file and 10 submission files.\r\n\r\nEach of these 10 submission files had their positive count printed in the diagnostics. Here is an example of diagnostics.\r\n\r\n Fold 1 converged after 0836 iterations.\r\n Fold 2 converged after 0571 iterations.\r\n Fold 3 converged after 0499 iterations.\r\n Fold 4 converged after 0585 iterations.\r\n Fold 5 converged after 0566 iterations.\r\n Iterations: 611.40 + 129.874\r\n \r\n \r\n Fold 1: AUC=0.9125954\r\n Fold 2: AUC=0.9221191\r\n Fold 3: AUC=0.9188253\r\n Fold 4: AUC=0.9200112\r\n Fold 5: AUC=0.9183175\r\n AUC: 0.9183737 + 0.0035463\r\n Average AUC using all data: 0.9037691\r\n \r\n \r\n Fold 1: MCC=0.4756098 (0534 [38.84%] positives), threshold=0.3749080 => True positives = 76.592%\r\n Fold 2: MCC=0.4769881 (0437 [31.76%] positives), threshold=0.5057880 => True positives = 84.897%\r\n Fold 3: MCC=0.4801341 (0560 [40.70%] positives), threshold=0.3444360 => True positives = 75.536%\r\n Fold 4: MCC=0.4737958 (0605 [43.97%] positives), threshold=0.3298900 => True positives = 71.736%\r\n Fold 5: MCC=0.4829743 (0615 [44.69%] positives), threshold=0.3086210 => True positives = 72.520%\r\n MCC: 0.4779004 + 0.0036627\r\n Threshold: 0.3727286 + 0.0781904\r\n Positives: 550.20 + 071.37\r\n Detection Rate %: 39.991 + 05.185\r\n True positives %: 76.256 + 05.237\r\n Undetected positives: 0959.20 + 0028.86\r\n Average MCC on all data (5 fold): 0.4734361, threshold=0.3727286\r\n Average MCC using all data: 0.4747755, threshold=0.3478090\r\n \r\n \r\n Submission overfitted threshold on all MCC positives: 2896\r\n \r\n Submission average validated threshold on all MCC positives: 2736\r\n \r\n Submission average of overfit+validated threshold positives: 2814\r\n \r\n Submission with all data overfitted threshold on all MCC positives: 2933. Threshold=0.347809\r\n \r\n Submission with all data average validated threshold on all MCC positives: 2803. Threshold=0.3727286\r\n \r\n Submission with all data average of overfit+validated threshold positives: 2867. Threshold=0.3602688\r\n \r\n Submission with all data by taking the amount of positives of overfitted threshold on all MCC positives: 2896. Threshold=0.354801\r\n \r\n Submission with all data by taking the amount of positives of average validated threshold on all MCC positives: 2736. Threshold=0.385727\r\n \r\n Submission with all data by taking the amount of positives of average of overfit+validated threshold positives: 2814. Threshold=0.370321\r\n \r\n Submission with all data by taking the sum of positives of validated positives: 2751. Threshold=0.383094\r\n \r\n Cross-validated used features list (all used features to copy & paste):\r\n \r\n c(\"sameL0_next\", \"sameL1_next\", \"CATEGORICAL_Last_____1\", \"S24.311\", \r\n \"sameL3_next\", \"GF1\", \"GF0\", \"FOR30_Sum_S\", \"CATEGORICAL_out_out_L3_S32_F3854_class2\", \r\n (censored because way too long)\r\n \r\n Cross-validated multipresence used features list (all used features to copy & paste):\r\n \r\n c(\"sameL0_next\", \"sameL1_next\", \"CATEGORICAL_Last_____1\", \"S24.311\", \r\n \"sameL3_next\", \"GF1\", \"GF0\", \"FOR30_Sum_S\", \"CATEGORICAL_out_out_L3_S32_F3854_class2\", \r\n (censored because way too long)\r\n\r\nDoes it look like Chinese to interpret what is wrong and what is OK? At first sight, yes. I made a small tutorial (by message on a chat… un-obviously) to help my team (those who are modeling and/or testing internally features) decipher the diagnostics file. Here are good excerpts of what I wrote:\r\n\r\n> My folds have extreme feature variance on each fold (not intentional,\r\n> but it seems to help a lot), so if your features don't work/generalize\r\n> on several folds, it will decrease the stability.\r\n> \r\n> CV variance is OK, but too much variance is not. I tested over 20\r\n> combos on 3-4-5-6 CV folds locally to test how variance can increase:\r\n> \r\n> * at 6 fold, standard deviation (sd) is about 0.01\r\n> * 5 fold, sd about 0.006\r\n> * 4 fold, sd about 0.006 (not sure?)\r\n> * 3 fold, sd about 0.004\r\n> \r\n> At the top of the diagnostics, you have all diagnostic things to\r\n> analyze the model, so you can check for consistency \"out-of-fold\" (out\r\n> of fold predictions) and \"all-of-out-of-fold\" (when taking all out of\r\n> fold predictions together): AUC, MCC, threshold, positive rate,\r\n> detection rate, true positives, undetected positives (false\r\n> negatives), MCC using 5-fold average threshold, MCC using all data.\r\n> \r\n> If the difference between both is high, most of thresholding methods\r\n> we use will fail (and we have a fallback for this in this scenario).\r\n> For instance, 0.92 avg AUC out of fold, 0.85 AUC all data is not\r\n> stable for submission. You can have high AUC and \"trash\" MCC. But we\r\n> use AUC to monitor if predictions remain stable on each fold so we\r\n> don't get any surprize when training on all data (because from that\r\n> point we go nearly \"blind\"). If your probabilities predicted are\r\n> stable, AUC on all data should be close to the average per fold\r\n> \r\n> When your folds are unstable (due to XYZ feature), the difference\r\n> increases as a [0, 1] scale for a fold might not be much more\r\n> different against the same [0, 1] scale for another fold.\r\n> \r\n> My script creates 10 submissions:\r\n> \r\n> * 1st one: use threshold using all data, on CV averaged prediction = mostly for debugging purposes\r\n> * 2nd one: use average threshold from CV, on CV averaged prediction = mostly for debugging purposes\r\n> * 3rd one: take mean of 1st and 2nd, on CV averaged prediction = mostly for debugging purposes\r\n> \r\n> Then you have 3 more submissions created (4th, 5th, 6th) which uses\r\n> the predicted probabilities on testing data using the same\r\n> thresholding techniques of 1st, 2nd, and 3rd. By default, the 4th\r\n> submission is the best to submit (it's not that \"overfitted\" - it\r\n> overfits more and more if your fold stability is lower)\r\n> \r\n> Then you have 7th 8th 9th, which uses the positive count found using\r\n> the 1st 2nd 3rd methods but on the predictions using the model with\r\n> all data together (those one should be avoided most of the times, but\r\n> they are left as is if needed).\r\n> \r\n> And the final one the 10th: it takes the positive count of the CV and\r\n> uses that same count on the test data from the model using all data.\r\n> It assumes the hypothesis the positive count in train = positive count\r\n> in test, which might not be true. Obviously, no one knows if true or\r\n> not. This is the submission you must use when your folds are clearly\r\n> unstable (it is the fallback method for submitting with an unstable\r\n> CV).\r\n> \r\n> If you see on these debug a number like 10,000 or more (usually it\r\n> sits around 2000-4000) it means your folds are unstable (captain\r\n> obvious is here) and you need to look out what happened. It means also\r\n> your CV models / full model predictions are unstable too (so only 10th\r\n> should be used, unless the one you pick has stable predictions = value\r\n> around 2000-4000). Therefore, only the fallback submission should be\r\n> valid from that point, in most cases.\r\n> \r\n> If you make a submission using 1st, 2nd, or 3rd set, you MUST\r\n> approximately get what you had in CV. If you don’t, the feature you\r\n> added is bad enough to be conditioned by something in the folds. When\r\n> you want to test 1 or more submissions from a single model, you use\r\n> this order if your model is stable according to the diagnostics and\r\n> your sight of it:\r\n> \r\n> * 4th (nearly always the best)\r\n> * 6th (rare you need 2 submissions - only if you clearly think your model is good - acts as a confirmatory response vs your CV if 4th\r\n> overshot the LB)\r\n> * 5th (very rare you will need 3 submissions to test a single model - typically happens when we don't have enough to submit, this should\r\n> always score lower on LB than 4th/6th no matter what you do, unless\r\n> luck)\r\n> \r\n> If unstable: - 10th (and only this one)\r\n> \r\n> Average MCC on all data (5 fold) <- use 5 fold threshold, average, and\r\n> compute MCC using that average threshold\r\n> \r\n> Average MCC using all data <- compute threshold on all data, use that\r\n> threshold on all data\r\n> \r\n> Cross-validated multipresence features are in case you have features which are not picked up by your model if you input LightGBM CV output from my package. You can't compare both using `featurelist1[which(!(featurelist1 %in% featurelist2))]` in R.\r\n\r\n### Stuff we used in the team\r\n\r\nMainly the most important names for a fully decentralized team working on a Kaggle competition in a (non or not-non) project-mode:\r\n\r\n* R programming language\r\n* Python programming language\r\n* [LightGBM][2] through the [Laurae package][3] (fastest R implementation at that time - you can do a 5-fold CV on 1000+ features on less than 40 minutes, I/O included)\r\n* [xgboost][4] (because “unidimensional machine learning at Kaggle is not Kaggle without xgboost”)\r\n* scikit-learn Random Forest (eats RAM but it works in Python without having to go through Java)\r\n* H2O Random Forest (when in R you want to use parallelized and fast Random Forests…. T_T)\r\n* Keras Neural Networks (who needs a neural network in the brain?)\r\n* data.table (fread/fwrite and super fast data manipulation, please)\r\n* Markdown, Rmarkdown, hacked pandoc (welcome Visual Studio executable hacks, by Laurae), iPython notebooks (yea, yea, we did reporting in the team to share insights / do “bookkeeping” - in a real project-mode)\r\n* Computers (who does not need a computer to compete at Kaggle? Use a phone?)\r\n* Monitors (because we are not blind)\r\n* Windows / Linux (because you need an operating system on your computer…)\r\nAWS / Google Cloud Platform (when you needed more horsepower on your car)\r\n* SSH FTP & Filezilla (how do you share data securely with the 1Gbps network from Laurae with all the team, when you have 10GB files?)\r\n* RStudio / Spyder (who needs an IDE to program?)\r\n* Remote Desktop (to share sessions on a server, troubleshooting, etc.)\r\n* Skype / Phone (could use Slack too, but we didn’t in the end as Skype was faster)\r\n* GitHub (private repository, to share code privately)\r\n* An internet browser to go on websites (it’s obvious you need to use a browser somewhere)\r\n\r\nThe team experience so far? Really great! So much fun and no tension in the team, with people listening to each other => a nice ambiance to spend time on Kaggle. And people wanting to learn from others is always a wonderful experience, as it is the **TEST/challenge which checks whether your knowledge was correct or not correct / biased** (yea, you can learn wrong something, it may happen).\r\n\r\n--- \r\n\r\nAh, and I think this is the post which will put me to Master tier in Discussions. 199 medals, now 200. A good double Master tier from this competition. Anyways, great work from everyone in this competition! (not only my team but the others! - look at all these great kernels and especially the pathing one!)\r\n\r\nThere are 5 new competition Masters according to the top 12 private LB! (Alexey Noskov, Laurae, Shubin, Michael Maguire, Alexandru Papiu) - congratulations!\r\n\r\n\r\n [1]: https://www.kaggle.com/apapiu/house-prices-advanced-regression-techniques/regularized-linear-models\r\n [2]: https://github.com/Microsoft/LightGBM\r\n [3]: https://github.com/Laurae2/Laurae\r\n [4]: https://github.com/dmlc/xgboost", "144194": "Laurae, Thanks a lot for pulling this team together! As you describe, it was a great experience for us all. We would never have achieved this result without your end-to-end knowledge and Kaggle experience. Your detailed post is another great example of the almost endless time and effort you put into sharing your knowledge with the Kaggle community. Thanks again!", "144255": "Thanks for sharing. \r\n\r\nClever idea to create a real data science project team. Results well deserved. Congrats on becoming a master.\r\n\r\n*edited after I read @Laurae's RedHat post*", "145282": "Always willing to share Laurae, thanks for all you gave, I learned a lot.", "146011": "Thanks for sharing! \r\nLong story and learned a lot" }, "source": "meta" }