{
  "id": 237483,
  "title": "Time Constraints",
  "url": "/competitions/coleridgeinitiative-show-us-the-data/discussion/237483",
  "author_name": "",
  "post_date": "2021-05-08T23:52:45.264373Z",
  "votes": 3,
  "comment_count": 2,
  "views": 0,
  "content": "<p>I see that the intro mentioned </p>\n<p>\"Note that the hidden test set has roughly ~8000 publications, many times the size of the public test set. Plan your compute time accordingly.\"</p>\n<p>And the public test set is only 12% of the total data and the allowed notebook time is 9 hours. So if i submitted and got my results back on the public score in say 2 hours, does that mean I'm going to exceed the time limit( 2 hours * (100 / 12)) = 16.66 &gt; 9 hours)?</p>",
  "messages": [
    {
      "id": "1298501",
      "postDate": "05/08/2021 23:52:45",
      "content": "<p>I see that the intro mentioned </p>\n<p>\"Note that the hidden test set has roughly ~8000 publications, many times the size of the public test set. Plan your compute time accordingly.\"</p>\n<p>And the public test set is only 12% of the total data and the allowed notebook time is 9 hours. So if i submitted and got my results back on the public score in say 2 hours, does that mean I'm going to exceed the time limit( 2 hours * (100 / 12)) = 16.66 &gt; 9 hours)?</p>",
      "rawMarkdown": "I see that the intro mentioned \n\n\"Note that the hidden test set has roughly ~8000 publications, many times the size of the public test set. Plan your compute time accordingly.\"\n\nAnd the public test set is only 12% of the total data and the allowed notebook time is 9 hours. So if i submitted and got my results back on the public score in say 2 hours, does that mean I'm going to exceed the time limit( 2 hours * (100 / 12)) = 16.66 > 9 hours)?",
      "votes": null
    },
    {
      "id": "1298524",
      "postDate": "05/09/2021 00:44:56",
      "content": "<p>I believe the \"public test set\" is referring to these four files that are provided:</p>\n<pre><code>2100032a-7c33-4bff-97ef-690822c43466.json\n2f392438-e215-4169-bebf-21ac4ff253e1.json\n3f316b38-1a24-45a9-8d8c-4e05a42257c6.json\n8e6996b4-ca08-4c0b-bed2-aaf07a4c6a60.json\n</code></pre>\n<p>However, I see where the phrase \"public test set\" has been used in other contexts to refer to the portion of the hidden test set used to calculate the public LB score, which can understandably lead to confusion.</p>\n<p>My understanding is that the \"hidden test set\" refers to the files that a notebook is run against on submission. On submission, your notebook is run against <em>all</em> the \"hidden test set\" but only a portion of the results are used for the public LB score. The remainder of the results are used for the private LB score that gets updated at the same time. We just don't see the private LB score until the end of the competition.</p>\n<p>So, in summary, I don't think any additional time for computation needs to be factored in.</p>",
      "rawMarkdown": "I believe the \"public test set\" is referring to these four files that are provided:\n\n```\n2100032a-7c33-4bff-97ef-690822c43466.json\n2f392438-e215-4169-bebf-21ac4ff253e1.json\n3f316b38-1a24-45a9-8d8c-4e05a42257c6.json\n8e6996b4-ca08-4c0b-bed2-aaf07a4c6a60.json\n```\n\nHowever, I see where the phrase \"public test set\" has been used in other contexts to refer to the portion of the hidden test set used to calculate the public LB score, which can understandably lead to confusion.\n\nMy understanding is that the \"hidden test set\" refers to the files that a notebook is run against on submission. On submission, your notebook is run against *all* the \"hidden test set\" but only a portion of the results are used for the public LB score. The remainder of the results are used for the private LB score that gets updated at the same time. We just don't see the private LB score until the end of the competition.\n\nSo, in summary, I don't think any additional time for computation needs to be factored in.",
      "votes": null
    },
    {
      "id": "1298528",
      "postDate": "05/09/2021 01:16:40",
      "content": "<p>Thanks for the explanation!</p>",
      "rawMarkdown": "Thanks for the explanation!",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 1298524,
      "author_name": "trentb",
      "author_url": "",
      "post_date": "05/09/2021 00:44:56",
      "content": "<p>I believe the \"public test set\" is referring to these four files that are provided:</p>\n<pre><code>2100032a-7c33-4bff-97ef-690822c43466.json\n2f392438-e215-4169-bebf-21ac4ff253e1.json\n3f316b38-1a24-45a9-8d8c-4e05a42257c6.json\n8e6996b4-ca08-4c0b-bed2-aaf07a4c6a60.json\n</code></pre>\n<p>However, I see where the phrase \"public test set\" has been used in other contexts to refer to the portion of the hidden test set used to calculate the public LB score, which can understandably lead to confusion.</p>\n<p>My understanding is that the \"hidden test set\" refers to the files that a notebook is run against on submission. On submission, your notebook is run against <em>all</em> the \"hidden test set\" but only a portion of the results are used for the public LB score. The remainder of the results are used for the private LB score that gets updated at the same time. We just don't see the private LB score until the end of the competition.</p>\n<p>So, in summary, I don't think any additional time for computation needs to be factored in.</p>",
      "votes": null,
      "replies": [
        {
          "id": 1298528,
          "author_name": "willyboy1",
          "author_url": "",
          "post_date": "05/09/2021 01:16:40",
          "content": "<p>Thanks for the explanation!</p>",
          "votes": null,
          "replies": []
        }
      ]
    }
  ],
  "raw_markdown_by_id": {
    "1298501": "I see that the intro mentioned \n\n\"Note that the hidden test set has roughly ~8000 publications, many times the size of the public test set. Plan your compute time accordingly.\"\n\nAnd the public test set is only 12% of the total data and the allowed notebook time is 9 hours. So if i submitted and got my results back on the public score in say 2 hours, does that mean I'm going to exceed the time limit( 2 hours * (100 / 12)) = 16.66 > 9 hours)?",
    "1298524": "I believe the \"public test set\" is referring to these four files that are provided:\n\n```\n2100032a-7c33-4bff-97ef-690822c43466.json\n2f392438-e215-4169-bebf-21ac4ff253e1.json\n3f316b38-1a24-45a9-8d8c-4e05a42257c6.json\n8e6996b4-ca08-4c0b-bed2-aaf07a4c6a60.json\n```\n\nHowever, I see where the phrase \"public test set\" has been used in other contexts to refer to the portion of the hidden test set used to calculate the public LB score, which can understandably lead to confusion.\n\nMy understanding is that the \"hidden test set\" refers to the files that a notebook is run against on submission. On submission, your notebook is run against *all* the \"hidden test set\" but only a portion of the results are used for the public LB score. The remainder of the results are used for the private LB score that gets updated at the same time. We just don't see the private LB score until the end of the competition.\n\nSo, in summary, I don't think any additional time for computation needs to be factored in.",
    "1298528": "Thanks for the explanation!"
  },
  "source": "meta"
}