{
  "id": 341780,
  "title": "Need-to-know: Rerun competitions and datasets",
  "url": "/competitions/AI4Code/discussion/341780",
  "author_name": "",
  "post_date": "2022-08-04T09:19:49.554490400Z",
  "votes": 15,
  "comment_count": 3,
  "views": 0,
  "content": "<p>Below is just a reminder for some and crucial for new Kagglers. I happily inform this now than comment a failed notebook after deadline.</p>\n<p>This competition is a competition with several of reruns as evaluation after the deadline, in contrary to the more usual approach, with the private test-set calculated directly when submitting the notebook and the result is showed after deadline. <br>\nTo be extra clear, this means that Kaggle is running the selected notebooks again after the deadline in the same way as when you submitted the kernel the first time.</p>\n<p>Again, what you need to know with a rerun competition and using public datasets, is that if the public dataset is deleted for some reason your submitted notebook will fail in the evaluation period and also if you delete your own used dataset during the evaluation period.</p>\n<p>Having said that, I have never seen a public dataset deleted during evaluation period, leaves no guarantees though, but what can happen is that it is removed, for some different reasons, before deadline in a public notebook, and also that you also mistakenly delete your own private dataset during the long period of evaluation period, when cleaning among datasets.</p>\n<p>Personally, I think it would be great with a rule that a public dataset can’t be deleted after deadline until the final private leaderboard is finished. Or maybe it’s so already, haven’t tried or seen that rule in text.</p>\n<p>Alright, I hope now it’s clearer how it works. 👍</p>",
  "messages": [
    {
      "id": "1884075",
      "postDate": "08/04/2022 09:19:49",
      "content": "<p>Below is just a reminder for some and crucial for new Kagglers. I happily inform this now than comment a failed notebook after deadline.</p>\n<p>This competition is a competition with several of reruns as evaluation after the deadline, in contrary to the more usual approach, with the private test-set calculated directly when submitting the notebook and the result is showed after deadline. <br>\nTo be extra clear, this means that Kaggle is running the selected notebooks again after the deadline in the same way as when you submitted the kernel the first time.</p>\n<p>Again, what you need to know with a rerun competition and using public datasets, is that if the public dataset is deleted for some reason your submitted notebook will fail in the evaluation period and also if you delete your own used dataset during the evaluation period.</p>\n<p>Having said that, I have never seen a public dataset deleted during evaluation period, leaves no guarantees though, but what can happen is that it is removed, for some different reasons, before deadline in a public notebook, and also that you also mistakenly delete your own private dataset during the long period of evaluation period, when cleaning among datasets.</p>\n<p>Personally, I think it would be great with a rule that a public dataset can’t be deleted after deadline until the final private leaderboard is finished. Or maybe it’s so already, haven’t tried or seen that rule in text.</p>\n<p>Alright, I hope now it’s clearer how it works. 👍</p>",
      "rawMarkdown": "Below is just a reminder for some and crucial for new Kagglers. I happily inform this now than comment a failed notebook after deadline.\n\nThis competition is a competition with several of reruns as evaluation after the deadline, in contrary to the more usual approach, with the private test-set calculated directly when submitting the notebook and the result is showed after deadline. \nTo be extra clear, this means that Kaggle is running the selected notebooks again after the deadline in the same way as when you submitted the kernel the first time.\n\nAgain, what you need to know with a rerun competition and using public datasets, is that if the public dataset is deleted for some reason your submitted notebook will fail in the evaluation period and also if you delete your own used dataset during the evaluation period.\n\nHaving said that, I have never seen a public dataset deleted during evaluation period, leaves no guarantees though, but what can happen is that it is removed, for some different reasons, before deadline in a public notebook, and also that you also mistakenly delete your own private dataset during the long period of evaluation period, when cleaning among datasets.\n\nPersonally, I think it would be great with a rule that a public dataset can’t be deleted after deadline until the final private leaderboard is finished. Or maybe it’s so already, haven’t tried or seen that rule in text.\n\nAlright, I hope now it’s clearer how it works. 👍",
      "votes": null
    },
    {
      "id": "1884613",
      "postDate": "08/04/2022 14:27:33",
      "content": "<p>If I'm reading that right, the controversial 0.8533 notebook <em>won't even run</em> against test data/private-LB, since it has three deleted data sources.</p>",
      "rawMarkdown": "If I'm reading that right, the controversial 0.8533 notebook *won't even run* against test data/private-LB, since it has three deleted data sources.",
      "votes": null
    },
    {
      "id": "1889840",
      "postDate": "08/08/2022 11:38:18",
      "content": "<p>Public notebooks with private datasets (not your own private ds) that the code is using will fail in a rerun competition.</p>",
      "rawMarkdown": "Public notebooks with private datasets (not your own private ds) that the code is using will fail in a rerun competition.",
      "votes": null
    },
    {
      "id": "1889843",
      "postDate": "08/08/2022 11:40:49",
      "content": "<p>I also posted this request to the Kaggle Team <a href=\"https://www.kaggle.com/discussions/product-feedback/341844\" target=\"_blank\">https://www.kaggle.com/discussions/product-feedback/341844</a></p>",
      "rawMarkdown": "I also posted this request to the Kaggle Team https://www.kaggle.com/discussions/product-feedback/341844",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 1884613,
      "author_name": "happycube",
      "author_url": "",
      "post_date": "08/04/2022 14:27:33",
      "content": "<p>If I'm reading that right, the controversial 0.8533 notebook <em>won't even run</em> against test data/private-LB, since it has three deleted data sources.</p>",
      "votes": null,
      "replies": [
        {
          "id": 1889840,
          "author_name": "kirderf",
          "author_url": "",
          "post_date": "08/08/2022 11:38:18",
          "content": "<p>Public notebooks with private datasets (not your own private ds) that the code is using will fail in a rerun competition.</p>",
          "votes": null,
          "replies": []
        }
      ]
    },
    {
      "id": 1889843,
      "author_name": "kirderf",
      "author_url": "",
      "post_date": "08/08/2022 11:40:49",
      "content": "<p>I also posted this request to the Kaggle Team <a href=\"https://www.kaggle.com/discussions/product-feedback/341844\" target=\"_blank\">https://www.kaggle.com/discussions/product-feedback/341844</a></p>",
      "votes": null,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "1884075": "Below is just a reminder for some and crucial for new Kagglers. I happily inform this now than comment a failed notebook after deadline.\n\nThis competition is a competition with several of reruns as evaluation after the deadline, in contrary to the more usual approach, with the private test-set calculated directly when submitting the notebook and the result is showed after deadline. \nTo be extra clear, this means that Kaggle is running the selected notebooks again after the deadline in the same way as when you submitted the kernel the first time.\n\nAgain, what you need to know with a rerun competition and using public datasets, is that if the public dataset is deleted for some reason your submitted notebook will fail in the evaluation period and also if you delete your own used dataset during the evaluation period.\n\nHaving said that, I have never seen a public dataset deleted during evaluation period, leaves no guarantees though, but what can happen is that it is removed, for some different reasons, before deadline in a public notebook, and also that you also mistakenly delete your own private dataset during the long period of evaluation period, when cleaning among datasets.\n\nPersonally, I think it would be great with a rule that a public dataset can’t be deleted after deadline until the final private leaderboard is finished. Or maybe it’s so already, haven’t tried or seen that rule in text.\n\nAlright, I hope now it’s clearer how it works. 👍",
    "1884613": "If I'm reading that right, the controversial 0.8533 notebook *won't even run* against test data/private-LB, since it has three deleted data sources.",
    "1889840": "Public notebooks with private datasets (not your own private ds) that the code is using will fail in a rerun competition.",
    "1889843": "I also posted this request to the Kaggle Team https://www.kaggle.com/discussions/product-feedback/341844"
  },
  "source": "meta"
}