![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F2535866%2F98201b802ab2215e4598b18500334198%2Flevel%201%20correlations.png?generation=1579827039622041&alt=media\")
all running totals\n - duration of assessments, activities, games, and clips\n - correct move percentage (event_id 3021 / (3021 + 3020))\n - rate of correct moves (3021 / (assessment duration + game duration))\n - proportion of time in assessments\n - difference between two event_ids for right (3021 and 3121) and wrong moves (3020 and 3120)\n - I considered learning curve features as the linear regression of correct move percentage within each game world's activities, because each game world teaches a different topic. Players with a high value are at the start of the learning curve and learning quickly, while players with low values have plateaued, having thoroughly learned the topic. The target is the last assessment's result, not average performance, so it makes sense to measure the upward trend. I found that many players would have only had two points, so I did not implement this.
\n\nThis is running k-fold predictions n times.
\n\nI chose models based on gradient boosting type and then Bayesian optimized the hyperparameters with Optuna. I can set n arbitrarily high to use up the 9 hours. Scores here are using QWK rounded to the train set's proportions because it was more stable than OptimizedRounder. (GBDT, GOSS, DART, and RF are boosting types in LGB.)\n>Starting Keras\n[ 7 N 5 K] score: 0.59346, total fit time: 2 hour 1 min, average fit time: 3 min\nStarting gbdt\n[ 20 N 5 K] score: 0.59890, total fit time: 46 min, average fit time: 27 sec\nStarting goss\n[ 30 N 5 K] score: 0.59882, total fit time: 23 min, average fit time: 9 sec\nStarting dart\n[ 5 N 5 K] score: 0.59875, total fit time: 1 hour 9 min, average fit time: 2 min 47 sec\nStarting rf\n[ 10 N 5 K] score: 0.58182, total fit time: 24 min, average fit time: 29 sec\nStarting CatBoost\n[ 1 N 5 K] score: 0.59514, total fit time: 26 min, average fit time: 5 min\nStarting XGB\n[ 5 N 5 K] score: 0.60283, total fit time: 1 hour 18 min, average fit time: 3 min
\n\nA neural network particularly benefits from higher n and is less correlated with other algorithms.
\n\n
When I Optuna optimized with larger n, it responded by lowering regularization parameters, making slightly overfitted models. Using k of 5 rather than 10 or more produces better results probably also from overfitting. Overfitted models perform better in the ensemble because models reach nuances and retain their strengths while their weak areas cancel out.
\n\nI selected the last one.
\n\nlevel 2
\n\nI stacked level 1 predictions with most manually created features and used LGB and linear regression. Old_qwk is rounding to the train set's proportions and new_qwk is the OptimizedRounder.\n> LGB level 2 RMSE: 0.96753, old_qwk: 0.60749, new_qwk: 0.61472 boundaries: [0.98744333 1.62733313 2.29026382]\nLR level 2 RMSE: 0.96682, old_qwk: 0.61039, new_qwk: 0.61572 boundaries: [1.01671793 1.49101464 2.28878774]
\n\nI also used TensorFlow's GradientTape with each weight directly SGD optimizing a weighted average (similar to this kernel).\n> SGD level 2 RMSE: 0.96523, old_qwk: 0.61146, new_qwk: 0.61610 boundaries: [1.01336522 1.50943782 2.27812701]\nweights: keras[0.3242] gbdt[0.0434] goss[0.0296] dart[-0.2894] rf[0.3581] cat[-0.0589] xgb[0.5994], total[1.0064]
\n\nIt commonly gives negative weights for more than four inputs, something a human would never do. I thought about this for a while, looked it up online and didn't find anything, and concluded it likely has an equivalent all-positive solution, and it seems to work, so I keep it.
\n\nlevel 3
\n\nSGD weighted average of level 2\n> SGD level 3 RMSE: 0.96533, old_qwk: 0.61089, new_qwk: 0.61606 boundaries: [0.99402423 1.69609969 2.17251209]\nweights: SGD2[0.4853] LR2[0.1978] LGB2[0.3187], total[1.0018]
\n\nAll weights are positive this time. LGB started off at around 0.2 and rose to 0.3 after Optuna. My level two and three ensembles give similar results. Although SGD level 2 had slightly better scores than SGD3, SGD3 had more stable boundaries, so I used SGD3.
\n\nOn the second to last day, I submitted four combinations of clustering features into game worlds and normalizing test with train, and found that adjusting the test set did not do well. On the last day, I submitted four again with a different seed. That brought me to 26 entries. One entry had the best RMSE and QWK in both optimizations and I picked that one even though the public score was low.
\n\n| entry | RMSE (lower is better) | proportional QWK (higher is better) | OptimizedRounder QWK (selected) | public score | private score |\n| --- | --- |\n| +clu +adj 1 | 0.96954 | 0.60516 | 0.59382 |0.522 | 0.524 |\n|\n|+clu +adj 2 | 0.96854 | 0.60573 | 0.61122 | 0.519 | 0.527 |\n|\n|+clu -adj 1 | 0.96768 | 0.60663 | 0.59914 | 0.541 | 0.547 |\n|\n|+clu -adj 2 | 0.96907 | 0.60416 | 0.61302 | 0.538 | 0.551 |\n|\n|-clu +adj 1| 0.96894 | 0.60516 | 0.61102 | 0.531 | 0.536 |\n|\n|-clu +adj 2 | 0.96732 | 0.60709 | 0.60954| 0.529 | 0.537 |\n|\n|-clu -adj 2 | 0.96761| 0.60752| 0.61302 | 0.526 | 0.545 |\n|\n|-clu -adj 1 | 0.96533 | 0.61089 | 0.61606 | 0.525 | 0.548 |
\n\nIn hindsight, that was the one to pick. Clustering would have lowered the private score by 0.004 on average and adjusting 0.018. (The other submission I selected was elsewhere, 0.546 on both public and private.)
\n\nThere is a quirk with QWK that I was not aware of until the competition ended. James Trotman quoted in his shake-up thread from Gรกbor S:
\n\n> - this yielded a very interesting insight: given two calibrations with\nroughly equal average performance (over all folds), if calibration A\ndoes better on the public test set, calibration B is very likely to\noutperform A on the private set (this appears to be a quirk of the\nkappa metric)
\n\nThat is exactly what I observe here every time: the entry with the lower public score has the higher private score; moreover, the difference is roughly the same amount.
\n\nGradient descent can handle many more hyperparameters than Bayesian methods. The authors of Gradient-based Hyperparameter Optimization through Reversible Learning used SGD to optimize thousands of neural network hyperparameters at a time, with a separate learning rate for every epoch. I did not get a Keras model to run in a TensorFlow custom train function as I did for a weighted average equation. With gradient descent, you can simultaneously optimize hyperparameters of every model of an ensemble, while creating many different models of diverse strengths. I'll be doing more of this in the future.
", "comments": [ { "id": 728287, "author_name": "James Trotman", "author_url": "", "post_date": "2020-01-24T15:09:37.053000", "content": "\n\n\nIt commonly gives negative weights for more than four inputs, something a human would never do. I thought about this for a while, looked it up online and didn't find anything, and concluded it likely has an equivalent all-positive solution, and it seems to work, so I keep it.
\n
I think Kaggle has a weirdly low Google pagerank weight or something, because I can recall a few vaguely \"holy wars\" type discussions on Kaggle about whether negative weights are good or bad (Weights in an Ensemble and Trading ensembling tips & more). I find if you cache predictions from various epochs during NN training then linearly weight them, early epochs get negative weights and later ones are positive. It generally leads to a small performance increase, but it's one of those very Kaggle specific things. (Feels like too much attention to the wrong detail!)
\n\nCongrats on the ๐ฅ and ๐ for the write up!
", "votes": 1, "replies": [] }, { "id": 728141, "author_name": "Miyabon", "author_url": "", "post_date": "2020-01-24T12:46:50.173000", "content": "Congrats silver medal & Expert๐ Thanks for sharing!!๐ ๐
", "votes": 0, "replies": [] } ], "comment_count": 2, "messages": [ { "id": 727993, "postDate": "2020-01-24T10:00:00.703Z", "content": "all running totals\n - duration of assessments, activities, games, and clips\n - correct move percentage (event_id 3021 / (3021 + 3020))\n - rate of correct moves (3021 / (assessment duration + game duration))\n - proportion of time in assessments\n - difference between two event_ids for right (3021 and 3121) and wrong moves (3020 and 3120)\n - I considered learning curve features as the linear regression of correct move percentage within each game world's activities, because each game world teaches a different topic. Players with a high value are at the start of the learning curve and learning quickly, while players with low values have plateaued, having thoroughly learned the topic. The target is the last assessment's result, not average performance, so it makes sense to measure the upward trend. I found that many players would have only had two points, so I did not implement this.
\n\nThis is running k-fold predictions n times.
\n\nI chose models based on gradient boosting type and then Bayesian optimized the hyperparameters with Optuna. I can set n arbitrarily high to use up the 9 hours. Scores here are using QWK rounded to the train set's proportions because it was more stable than OptimizedRounder. (GBDT, GOSS, DART, and RF are boosting types in LGB.)\n>Starting Keras\n[ 7 N 5 K] score: 0.59346, total fit time: 2 hour 1 min, average fit time: 3 min\nStarting gbdt\n[ 20 N 5 K] score: 0.59890, total fit time: 46 min, average fit time: 27 sec\nStarting goss\n[ 30 N 5 K] score: 0.59882, total fit time: 23 min, average fit time: 9 sec\nStarting dart\n[ 5 N 5 K] score: 0.59875, total fit time: 1 hour 9 min, average fit time: 2 min 47 sec\nStarting rf\n[ 10 N 5 K] score: 0.58182, total fit time: 24 min, average fit time: 29 sec\nStarting CatBoost\n[ 1 N 5 K] score: 0.59514, total fit time: 26 min, average fit time: 5 min\nStarting XGB\n[ 5 N 5 K] score: 0.60283, total fit time: 1 hour 18 min, average fit time: 3 min
\n\nA neural network particularly benefits from higher n and is less correlated with other algorithms.
\n\n
When I Optuna optimized with larger n, it responded by lowering regularization parameters, making slightly overfitted models. Using k of 5 rather than 10 or more produces better results probably also from overfitting. Overfitted models perform better in the ensemble because models reach nuances and retain their strengths while their weak areas cancel out.
\n\nI selected the last one.
\n\nlevel 2
\n\nI stacked level 1 predictions with most manually created features and used LGB and linear regression. Old_qwk is rounding to the train set's proportions and new_qwk is the OptimizedRounder.\n> LGB level 2 RMSE: 0.96753, old_qwk: 0.60749, new_qwk: 0.61472 boundaries: [0.98744333 1.62733313 2.29026382]\nLR level 2 RMSE: 0.96682, old_qwk: 0.61039, new_qwk: 0.61572 boundaries: [1.01671793 1.49101464 2.28878774]
\n\nI also used TensorFlow's GradientTape with each weight directly SGD optimizing a weighted average (similar to this kernel).\n> SGD level 2 RMSE: 0.96523, old_qwk: 0.61146, new_qwk: 0.61610 boundaries: [1.01336522 1.50943782 2.27812701]\nweights: keras[0.3242] gbdt[0.0434] goss[0.0296] dart[-0.2894] rf[0.3581] cat[-0.0589] xgb[0.5994], total[1.0064]
\n\nIt commonly gives negative weights for more than four inputs, something a human would never do. I thought about this for a while, looked it up online and didn't find anything, and concluded it likely has an equivalent all-positive solution, and it seems to work, so I keep it.
\n\nlevel 3
\n\nSGD weighted average of level 2\n> SGD level 3 RMSE: 0.96533, old_qwk: 0.61089, new_qwk: 0.61606 boundaries: [0.99402423 1.69609969 2.17251209]\nweights: SGD2[0.4853] LR2[0.1978] LGB2[0.3187], total[1.0018]
\n\nAll weights are positive this time. LGB started off at around 0.2 and rose to 0.3 after Optuna. My level two and three ensembles give similar results. Although SGD level 2 had slightly better scores than SGD3, SGD3 had more stable boundaries, so I used SGD3.
\n\nOn the second to last day, I submitted four combinations of clustering features into game worlds and normalizing test with train, and found that adjusting the test set did not do well. On the last day, I submitted four again with a different seed. That brought me to 26 entries. One entry had the best RMSE and QWK in both optimizations and I picked that one even though the public score was low.
\n\n| entry | RMSE (lower is better) | proportional QWK (higher is better) | OptimizedRounder QWK (selected) | public score | private score |\n| --- | --- |\n| +clu +adj 1 | 0.96954 | 0.60516 | 0.59382 |0.522 | 0.524 |\n|\n|+clu +adj 2 | 0.96854 | 0.60573 | 0.61122 | 0.519 | 0.527 |\n|\n|+clu -adj 1 | 0.96768 | 0.60663 | 0.59914 | 0.541 | 0.547 |\n|\n|+clu -adj 2 | 0.96907 | 0.60416 | 0.61302 | 0.538 | 0.551 |\n|\n|-clu +adj 1| 0.96894 | 0.60516 | 0.61102 | 0.531 | 0.536 |\n|\n|-clu +adj 2 | 0.96732 | 0.60709 | 0.60954| 0.529 | 0.537 |\n|\n|-clu -adj 2 | 0.96761| 0.60752| 0.61302 | 0.526 | 0.545 |\n|\n|-clu -adj 1 | 0.96533 | 0.61089 | 0.61606 | 0.525 | 0.548 |
\n\nIn hindsight, that was the one to pick. Clustering would have lowered the private score by 0.004 on average and adjusting 0.018. (The other submission I selected was elsewhere, 0.546 on both public and private.)
\n\nThere is a quirk with QWK that I was not aware of until the competition ended. James Trotman quoted in his shake-up thread from Gรกbor S:
\n\n> - this yielded a very interesting insight: given two calibrations with\nroughly equal average performance (over all folds), if calibration A\ndoes better on the public test set, calibration B is very likely to\noutperform A on the private set (this appears to be a quirk of the\nkappa metric)
\n\nThat is exactly what I observe here every time: the entry with the lower public score has the higher private score; moreover, the difference is roughly the same amount.
\n\nGradient descent can handle many more hyperparameters than Bayesian methods. The authors of Gradient-based Hyperparameter Optimization through Reversible Learning used SGD to optimize thousands of neural network hyperparameters at a time, with a separate learning rate for every epoch. I did not get a Keras model to run in a TensorFlow custom train function as I did for a weighted average equation. With gradient descent, you can simultaneously optimize hyperparameters of every model of an ensemble, while creating many different models of diverse strengths. I'll be doing more of this in the future.
", "rawMarkdown": "#Solution\n\n### New Features\nall running totals\n - duration of assessments, activities, games, and clips\n - correct move percentage (event_id 3021 / (3021 + 3020))\n - rate of correct moves (3021 / (assessment duration + game duration))\n - proportion of time in assessments\n - difference between two event_ids for right (3021 and 3121) and wrong moves (3020 and 3120)\n - I considered [learning curve](https://en.wikipedia.org/wiki/Learning_curve) features as the linear regression of correct move percentage within each game world's activities, because each game world teaches a different topic. Players with a high value are at the start of the learning curve and learning quickly, while players with low values have plateaued, having thoroughly learned the topic. The target is the last assessment's result, not average performance, so it makes sense to measure the upward trend. I found that many players would have only had two points, so I did not implement this.\n\n###[NK-fold Prediction](https://www.kaggle.com/c/dont-overfit-ii/discussion/80242)\n\nThis is running *k*-fold predictions *n* times.\n\nI chose models based on gradient boosting type and then Bayesian optimized the hyperparameters with Optuna. I can set *n* arbitrarily high to use up the 9 hours. Scores here are using QWK rounded to the train set's proportions because it was more stable than OptimizedRounder. (GBDT, GOSS, DART, and RF are boosting types in LGB.)\n>Starting Keras\n[ 7 N 5 K] score: 0.59346, total fit time: 2 hour 1 min, average fit time: 3 min\nStarting gbdt\n[ 20 N 5 K] score: 0.59890, total fit time: 46 min, average fit time: 27 sec\nStarting goss\n[ 30 N 5 K] score: 0.59882, total fit time: 23 min, average fit time: 9 sec\nStarting dart\n[ 5 N 5 K] score: 0.59875, total fit time: 1 hour 9 min, average fit time: 2 min 47 sec\nStarting rf\n[ 10 N 5 K] score: 0.58182, total fit time: 24 min, average fit time: 29 sec\nStarting CatBoost\n[ 1 N 5 K] score: 0.59514, total fit time: 26 min, average fit time: 5 min\nStarting XGB\n[ 5 N 5 K] score: 0.60283, total fit time: 1 hour 18 min, average fit time: 3 min\n\nA neural network particularly benefits from higher *n* and is less correlated with other algorithms.\n\n\n\n### NK-predict and Optuna Interactions\n\nWhen I Optuna optimized with larger *n*, it responded by lowering regularization parameters, making slightly overfitted models. Using *k* of 5 rather than 10 or more produces better results probably also from overfitting. Overfitted models perform better in the ensemble because models reach nuances and retain their strengths while their weak areas cancel out.\n\n### Keras Tuning\n - dense layers [192, 256, 384, 512]. I arrange ascending powers of two with \"half-steps\" in between, and pick where the first layer should be. I find this useful because there are more complex relationships than simple ones, so you make room for complexity in later layers. Preliminary Optuna testing gives layers in the 200's with a smaller last layer, so I might change this heuristic in the future.\n - dropout 0.45. Optuna recommended over 0.60 for 400 features but for time I chose 0.45.\n - scaled with the [Yeo-Johnson power transformation](https://scikit-learn.org/stable/modules/generated/sklearn.preprocessing.PowerTransformer.html) over MinMaxScaler, StandardScaler, RobustScaler, MaxAbsScaler, and QuantileTransformer, all in sklearn. I don't currently have the results but [here](https://www.kdnuggets.com/2019/04/normalization-vs-standardization-quantitative-analysis.html) is a detailed comparison of other datasets and algorithms.\n - compared layer order\n> Starting LayerNormalization Activation\n[ 10 N 5 K] score: 0.591645, total fit time: 1 hour 22 min, average fit time: 3 min\nStarting BatchNormalization Activation\n[ 10 N 5 K] score: 0.591235, total fit time: 1 hour 19 min, average fit time: 3 min\nStarting Activation LayerNormalization\n[ 10 N 5 K] score: 0.589965, total fit time: 1 hour 23 min, average fit time: 3 min\nStarting Activation BatchNormalization\n[ 10 N 5 K] score: 0.593635, total fit time: 1 hour 13 min, average fit time: 2 min 56 sec\n\nI selected the last one.\n\n### 3 Level Ensemble\n\n**level 2**\n\nI stacked level 1 predictions with most manually created features and used LGB and linear regression. Old\\_qwk is rounding to the train set's proportions and new_qwk is the OptimizedRounder.\n> LGB level 2 RMSE: 0.96753, old\\_qwk: 0.60749, new\\_qwk: 0.61472 boundaries: [0.98744333 1.62733313 2.29026382]\nLR level 2 RMSE: 0.96682, old\\_qwk: 0.61039, new\\_qwk: 0.61572 boundaries: [1.01671793 1.49101464 2.28878774]\n\nI also used TensorFlow's GradientTape with each weight directly SGD optimizing a weighted average (similar to [this kernel](https://www.kaggle.com/barnrang/ashrae-leak-validation-gradient-descent-search)).\n> SGD level 2 RMSE: 0.96523, old\\_qwk: 0.61146, new\\_qwk: 0.61610 boundaries: [1.01336522 1.50943782 2.27812701]\nweights: keras[0.3242] gbdt[0.0434] goss[0.0296] dart[-0.2894] rf[0.3581] cat[-0.0589] xgb[0.5994], total[1.0064]\n\nIt commonly gives negative weights for more than four inputs, something a human would never do. I thought about this for a while, looked it up online and didn't find anything, and concluded it likely has an equivalent all-positive solution, and it seems to work, so I keep it.\n\n**level 3**\n\nSGD weighted average of level 2\n> SGD level 3 RMSE: 0.96533, old\\_qwk: 0.61089, new\\_qwk: 0.61606 boundaries: [0.99402423 1.69609969 2.17251209]\nweights: SGD2[0.4853] LR2[0.1978] LGB2[0.3187], total[1.0018]\n\nAll weights are positive this time. LGB started off at around 0.2 and rose to 0.3 after Optuna. My level two and three ensembles give similar results. Although SGD level 2 had slightly better scores than SGD3, SGD3 had more stable boundaries, so I used SGD3.\n\n### Miscellaneous\n- didn't normalize the test set's column mean with train, which was good because it would have lowered the private score by an average 0.018\n- used OptimizedRounder over rounding to train's proportions for its higher median and variance\n- tried clustering columns by game world, didn't help for me\n\n### The Last Two Days and Shake-up\n\nOn the second to last day, I submitted four combinations of clustering features into game worlds and normalizing test with train, and found that adjusting the test set did not do well. On the last day, I submitted four again with a different seed. That brought me to 26 entries. One entry had the best RMSE and QWK in both optimizations and I picked that one even though the public score was low.\n\n| entry | **RMSE (lower is better)** | **proportional QWK (higher is better)** | **OptimizedRounder QWK (selected)** | **public score** | **private score** |\n| --- | --- |\n| +clu +adj 1 | 0.96954 | 0.60516 | 0.59382 |0.522 | 0.524 |\n|\n|+clu +adj 2 | 0.96854 | 0.60573 | 0.61122 | 0.519 | 0.527 |\n|\n|+clu -adj 1 | 0.96768 | 0.60663 | 0.59914 | 0.541 | 0.547 |\n|\n|+clu -adj 2 | 0.96907 | 0.60416 | 0.61302 | 0.538 | 0.551 |\n|\n|-clu +adj 1| 0.96894 | 0.60516 | 0.61102 | 0.531 | 0.536 |\n|\n|-clu +adj 2 | 0.96732 | 0.60709 | 0.60954| 0.529 | 0.537 |\n|\n|-clu -adj 2 | 0.96761| 0.60752| 0.61302 | 0.526 | 0.545 |\n|\n|***-clu -adj 1*** | ***0.96533*** | ***0.61089*** | ***0.61606*** | ***0.525*** | ***0.548*** |\n\nIn hindsight, that was the one to pick. Clustering would have lowered the private score by 0.004 on average and adjusting 0.018. (The other submission I selected was elsewhere, 0.546 on both public and private.)\n\nThere is a quirk with QWK that I was not aware of until the competition ended. James Trotman [quoted](https://www.kaggle.com/c/data-science-bowl-2019/discussion/127203#727005) in his shake-up thread from [Gรกbor S](https://www.kaggle.com/c/prudential-life-insurance-assessment/discussion/19010#108362):\n\n> - this yielded a very interesting insight: given two calibrations with\nroughly equal average performance (over all folds), if calibration A\ndoes better on the public test set, calibration B is very likely to\noutperform A on the private set (this appears to be a quirk of the\nkappa metric)\n\nThat is exactly what I observe here every time: the entry with the lower public score has the higher private score; moreover, the difference is roughly the same amount.\n\n### Gradient Descent Hyperparameter Optimization\n\nGradient descent can handle many more hyperparameters than Bayesian methods. The authors of [Gradient-based Hyperparameter Optimization through Reversible Learning](https://arxiv.org/pdf/1502.03492.pdf) used SGD to optimize thousands of neural network hyperparameters at a time, with a separate learning rate for every epoch. I did not get a Keras model to run in a TensorFlow custom train function as I did for a weighted average equation. With gradient descent, you can simultaneously optimize hyperparameters of every model of an ensemble, while creating many different models of diverse strengths. I'll be doing more of this in the future.", "votes": 11 }, { "id": 728287, "postDate": "2020-01-24T15:09:37.053Z", "content": "\n\n\nIt commonly gives negative weights for more than four inputs, something a human would never do. I thought about this for a while, looked it up online and didn't find anything, and concluded it likely has an equivalent all-positive solution, and it seems to work, so I keep it.
\n
I think Kaggle has a weirdly low Google pagerank weight or something, because I can recall a few vaguely \"holy wars\" type discussions on Kaggle about whether negative weights are good or bad (Weights in an Ensemble and Trading ensembling tips & more). I find if you cache predictions from various epochs during NN training then linearly weight them, early epochs get negative weights and later ones are positive. It generally leads to a small performance increase, but it's one of those very Kaggle specific things. (Feels like too much attention to the wrong detail!)
\n\nCongrats on the ๐ฅ and ๐ for the write up!
", "rawMarkdown": "> It commonly gives negative weights for more than four inputs, something a human would never do. I thought about this for a while, looked it up online and ***didn't find anything***, and concluded it likely has an equivalent all-positive solution, and it seems to work, so I keep it.\n\nI think Kaggle has a weirdly low Google pagerank weight or something, because I can recall a few vaguely \"holy wars\" type discussions on Kaggle about whether negative weights are good or bad ([Weights in an Ensemble][1] and [Trading ensembling tips][2] & more). I find if you cache predictions from various epochs during NN training then linearly weight them, early epochs get negative weights and later ones are positive. It generally leads to a small performance increase, but it's one of those very Kaggle specific things. (Feels like too much attention to the wrong detail!)\n\nCongrats on the ๐ฅ and ๐ for the write up!\n\n[1]: https://www.kaggle.com/c/allstate-claims-severity/discussion/25240 \"Weights in an Ensemble\"\n[2]: https://www.kaggle.com/c/microsoft-malware-prediction/discussion/80368 \"Trading ensembling tips\"\n\n", "votes": 1 }, { "id": 728141, "postDate": "2020-01-24T12:46:50.173Z", "content": "Congrats silver medal & Expert๐ Thanks for sharing!!๐ ๐
", "rawMarkdown": "Congrats silver medal & Expert๐ Thanks for sharing!!๐ ๐ " } ], "complete": true, "source": "live", "fetched_at": "2026-10-23T23:49:38Z", "competition_id": 16531, "competition_slug": "data-science-bowl-2019", "team_id": 4150428, "solution_url": "/competitions/data-science-bowl-2019/writeups/charles-lin-solo-top-3-solution-silver-medal" }