{
  "id": 127408,
  "title": "40th place solution 0.553 - 0.555, +1050 positions",
  "content": "<h3>I guess that many of those who are reading it understand how difficult it was to be at 1000+ position all the competition doing the right things at the same time. 😏</h3>\n\n<h2>Of course I did not do:</h2>\n\n<ul>\n<li>Any adjustment of test set features (adjustment by ratio of mean values was used in almost any public kernel).</li>\n<li>Tuning of round thresholds for final predictions (when we convert continuous values to discreat labels) to fit the train distribution of classes (that was used in almost any public kernel).</li>\n</ul>\n\n<p><strong>It saved me from overfitting to public test distribution which was of course biased</strong> (only 14% of data, it was clearly seen from my own experiments and this discussion: <a href=\"https://www.kaggle.com/c/data-science-bowl-2019/discussion/122767\">https://www.kaggle.com/c/data-science-bowl-2019/discussion/122767</a> )\n<strong>But at the same time it holds me at 1000+ LB position...</strong></p>\n\n<p>I solved regression problem. It was the right way because kappa measure penalize the different errors unequally. Besides some popular features available in public kernels I used features based on “misses” and “rounds” and found them helpful. Additional custom features based on counters were used. After feature selection only 505 features were remained.</p>\n\n<h2>Training:</h2>\n\n<ul>\n<li>Using labeled test samples for training.</li>\n<li><strong>Custom RandGroupKfold</strong> was used because sklearn GroupKfold does not have \"seed\" parameter to provide randomness.</li>\n<li>Ensamble of 15 LGBM models (3 CV iterations with 5 folds each, \"feature fraction\" parameter was changing at different folds).</li>\n<li><strong>Adversarial validation</strong> via selection of random samples with unique IDs was used (200 iterations). <strong>Median score</strong> for each model was calculated and saved (it was used as a weight at model voting stage).</li>\n<li>Round <strong>thresholds optimization</strong> via OptimizedRounder was implemented <strong>for each model individualy</strong> based on whole train dataset. So each of 15 models has its own set of round thresholds what was used for final prediction.</li>\n<li><strong>Prediction confidence values were calculated based on ratio between the distance to nearest round value and interval length between adjacent round values</strong>. These confidences were used as weights at model voting stage.</li>\n<li>Model voting stage was done with weights based on <strong>median fold score</strong> and <strong>confidence values.</strong></li>\n</ul>\n\n<p>I provided full results repeatability using SortedList, SortedDict, SortedSet (from sortedcontainers package) for features with \"set\" and \"dict\" as well as \"seed\" values for all models. It helped me a lot during model selection and submission stages.</p>\n\n<h2>Things that did not help:</h2>\n\n<ul>\n<li>Training dataset augmentation.</li>\n<li>Using kappa measures as evaluation approach for stopping criteria in LGBM, CATBoost and XGBoost (it was very unstable).</li>\n<li>Ensamble LGBM with CATBoost, XGBoost and NN models (results were nearly the same as LGBM standalone model has but training time increased a lot).</li>\n</ul>\n\n<h3>Thanks for reading and good luck in future competitions!</h3>",
  "comments": [
    {
      "id": 727519,
      "author_name": "isma",
      "author_url": "",
      "post_date": "2020-01-23T19:06:08.180000",
      "content": "<p>I feel you. I gained as well quite a bit, ~1.700 spots, and to be honest, for a long while I gave up trying because I did not really understand how I was achieving continuously higher CV and higher local holdout scores but my LB was desperately not progressing - I basically came to the conclusion that I was overfitting somehow or that I had a data leak that I could not uncover.\nThat's a good lesson for me that LB could be quite deceiving especially when relying on such a small test set.</p>",
      "votes": 2,
      "replies": [
        {
          "id": 728376,
          "author_name": "DN",
          "author_url": "",
          "post_date": "2020-01-24T16:50:59.757000",
          "content": "<p>Totally argee with you!</p>",
          "votes": 0,
          "replies": []
        }
      ]
    },
    {
      "id": 728146,
      "author_name": "Miyabon",
      "author_url": "",
      "post_date": "2020-01-24T13:01:52.913000",
      "content": "<p>Congrats &amp; Thanks for sharing!🎉 😄 👍 </p>",
      "votes": 1,
      "replies": []
    },
    {
      "id": 727747,
      "author_name": "Ailurophile",
      "author_url": "",
      "post_date": "2020-01-24T02:19:39.113000",
      "content": "<p>Congratulations\nThanks for Sharing your Approach &amp; Insights!! <a href=\"/dmitrynovikov\">@dmitrynovikov</a> </p>",
      "votes": 0,
      "replies": []
    }
  ],
  "comment_count": 4,
  "messages": [
    {
      "id": 727445,
      "postDate": "2020-01-23T18:16:06.397Z",
      "content": "<h3>I guess that many of those who are reading it understand how difficult it was to be at 1000+ position all the competition doing the right things at the same time. 😏</h3>\n\n<h2>Of course I did not do:</h2>\n\n<ul>\n<li>Any adjustment of test set features (adjustment by ratio of mean values was used in almost any public kernel).</li>\n<li>Tuning of round thresholds for final predictions (when we convert continuous values to discreat labels) to fit the train distribution of classes (that was used in almost any public kernel).</li>\n</ul>\n\n<p><strong>It saved me from overfitting to public test distribution which was of course biased</strong> (only 14% of data, it was clearly seen from my own experiments and this discussion: <a href=\"https://www.kaggle.com/c/data-science-bowl-2019/discussion/122767\">https://www.kaggle.com/c/data-science-bowl-2019/discussion/122767</a> )\n<strong>But at the same time it holds me at 1000+ LB position...</strong></p>\n\n<p>I solved regression problem. It was the right way because kappa measure penalize the different errors unequally. Besides some popular features available in public kernels I used features based on “misses” and “rounds” and found them helpful. Additional custom features based on counters were used. After feature selection only 505 features were remained.</p>\n\n<h2>Training:</h2>\n\n<ul>\n<li>Using labeled test samples for training.</li>\n<li><strong>Custom RandGroupKfold</strong> was used because sklearn GroupKfold does not have \"seed\" parameter to provide randomness.</li>\n<li>Ensamble of 15 LGBM models (3 CV iterations with 5 folds each, \"feature fraction\" parameter was changing at different folds).</li>\n<li><strong>Adversarial validation</strong> via selection of random samples with unique IDs was used (200 iterations). <strong>Median score</strong> for each model was calculated and saved (it was used as a weight at model voting stage).</li>\n<li>Round <strong>thresholds optimization</strong> via OptimizedRounder was implemented <strong>for each model individualy</strong> based on whole train dataset. So each of 15 models has its own set of round thresholds what was used for final prediction.</li>\n<li><strong>Prediction confidence values were calculated based on ratio between the distance to nearest round value and interval length between adjacent round values</strong>. These confidences were used as weights at model voting stage.</li>\n<li>Model voting stage was done with weights based on <strong>median fold score</strong> and <strong>confidence values.</strong></li>\n</ul>\n\n<p>I provided full results repeatability using SortedList, SortedDict, SortedSet (from sortedcontainers package) for features with \"set\" and \"dict\" as well as \"seed\" values for all models. It helped me a lot during model selection and submission stages.</p>\n\n<h2>Things that did not help:</h2>\n\n<ul>\n<li>Training dataset augmentation.</li>\n<li>Using kappa measures as evaluation approach for stopping criteria in LGBM, CATBoost and XGBoost (it was very unstable).</li>\n<li>Ensamble LGBM with CATBoost, XGBoost and NN models (results were nearly the same as LGBM standalone model has but training time increased a lot).</li>\n</ul>\n\n<h3>Thanks for reading and good luck in future competitions!</h3>",
      "rawMarkdown": "### I guess that many of those who are reading it understand how difficult it was to be at 1000+ position all the competition doing the right things at the same time. 😏 \n\n## Of course I did not do:\n- Any adjustment of test set features (adjustment by ratio of mean values was used in almost any public kernel).\n- Tuning of round thresholds for final predictions (when we convert continuous values to discreat labels) to fit the train distribution of classes (that was used in almost any public kernel).\n\n**It saved me from overfitting to public test distribution which was of course biased** (only 14% of data, it was clearly seen from my own experiments and this discussion: https://www.kaggle.com/c/data-science-bowl-2019/discussion/122767 )\n**But at the same time it holds me at 1000+ LB position...**\n\nI solved regression problem. It was the right way because kappa measure penalize the different errors unequally. Besides some popular features available in public kernels I used features based on “misses” and “rounds” and found them helpful. Additional custom features based on counters were used. After feature selection only 505 features were remained.\n\n## Training:\n- Using labeled test samples for training.\n- **Custom RandGroupKfold** was used because sklearn GroupKfold does not have \"seed\" parameter to provide randomness.\n- Ensamble of 15 LGBM models (3 CV iterations with 5 folds each, \"feature fraction\" parameter was changing at different folds).\n- **Adversarial validation** via selection of random samples with unique IDs was used (200 iterations). **Median score** for each model was calculated and saved (it was used as a weight at model voting stage).\n- Round **thresholds optimization** via OptimizedRounder was implemented **for each model individualy** based on whole train dataset. So each of 15 models has its own set of round thresholds what was used for final prediction.\n- **Prediction confidence values were calculated based on ratio between the distance to nearest round value and interval length between adjacent round values**. These confidences were used as weights at model voting stage.\n- Model voting stage was done with weights based on **median fold score** and **confidence values.**\n\n\nI provided full results repeatability using SortedList, SortedDict, SortedSet (from sortedcontainers package) for features with \"set\" and \"dict\" as well as \"seed\" values for all models. It helped me a lot during model selection and submission stages.\n\n## Things that did not help:\n- Training dataset augmentation.\n- Using kappa measures as evaluation approach for stopping criteria in LGBM, CATBoost and XGBoost (it was very unstable).\n- Ensamble LGBM with CATBoost, XGBoost and NN models (results were nearly the same as LGBM standalone model has but training time increased a lot).\n\t\n### Thanks for reading and good luck in future competitions!",
      "votes": 13
    },
    {
      "id": 727519,
      "postDate": "2020-01-23T19:06:08.180Z",
      "content": "<p>I feel you. I gained as well quite a bit, ~1.700 spots, and to be honest, for a long while I gave up trying because I did not really understand how I was achieving continuously higher CV and higher local holdout scores but my LB was desperately not progressing - I basically came to the conclusion that I was overfitting somehow or that I had a data leak that I could not uncover.\nThat's a good lesson for me that LB could be quite deceiving especially when relying on such a small test set.</p>",
      "rawMarkdown": "I feel you. I gained as well quite a bit, ~1.700 spots, and to be honest, for a long while I gave up trying because I did not really understand how I was achieving continuously higher CV and higher local holdout scores but my LB was desperately not progressing - I basically came to the conclusion that I was overfitting somehow or that I had a data leak that I could not uncover.\nThat's a good lesson for me that LB could be quite deceiving especially when relying on such a small test set.",
      "votes": 2,
      "replies": [
        {
          "id": 728376,
          "postDate": "2020-01-24T16:50:59.757Z",
          "content": "<p>Totally argee with you!</p>",
          "rawMarkdown": "Totally argee with you!"
        }
      ]
    },
    {
      "id": 728146,
      "postDate": "2020-01-24T13:01:52.913Z",
      "content": "<p>Congrats &amp; Thanks for sharing!🎉 😄 👍 </p>",
      "rawMarkdown": "Congrats &amp; Thanks for sharing!🎉 😄 👍 ",
      "votes": 1
    },
    {
      "id": 727747,
      "postDate": "2020-01-24T02:19:39.113Z",
      "content": "<p>Congratulations\nThanks for Sharing your Approach &amp; Insights!! <a href=\"/dmitrynovikov\">@dmitrynovikov</a> </p>",
      "rawMarkdown": "Congratulations\nThanks for Sharing your Approach &amp; Insights!! @dmitrynovikov "
    }
  ],
  "complete": true,
  "source": "live",
  "fetched_at": "2026-10-23T06:53:56Z",
  "competition_id": 16531,
  "competition_slug": "data-science-bowl-2019",
  "team_id": 4205458,
  "solution_url": "/competitions/data-science-bowl-2019/writeups/dn-40th-place-solution-0-553-0-555-1050-positions"
}