What if the submission run in 9 hours for current test dataset but exceed 9 hours for replaced test dataset?
", "messages": [ { "id": 1868411, "postDate": "2022-07-24T00:40:25.930Z", "content": "What if the submission run in 9 hours for current test dataset but exceed 9 hours for replaced test dataset?
", "rawMarkdown": "What if the submission run in 9 hours for current test dataset but exceed 9 hours for replaced test dataset?", "votes": 8 }, { "id": 1868907, "postDate": "2022-07-24T10:32:07.430Z", "content": "I have asked the same question here: https://www.kaggle.com/competitions/AI4Code/discussion/332516
\nIt'll be great if evaluation on private dataset can have more quotas (both memory and time) to prevent the unexepcted failures…
", "rawMarkdown": "I have asked the same question here: https://www.kaggle.com/competitions/AI4Code/discussion/332516\n\nIt'll be great if evaluation on private dataset can have more quotas (both memory and time) to prevent the unexepcted failures...", "votes": 3, "replies": [ { "id": 1869078, "postDate": "2022-07-24T13:18:07.527Z", "content": "I suppose that the future test dataset will be +- the same size as the nowadays test dataset. If not, it will be funny for ensemblers 😄👀.
", "rawMarkdown": "I suppose that the future test dataset will be +- the same size as the nowadays test dataset. If not, it will be funny for ensemblers 😄👀. " }, { "id": 1869179, "postDate": "2022-07-24T14:51:53.373Z", "content": "It's not only about the number of notebooks but also the number of cells, so the difference has a chance to be not a minor one. Currently I am keeping the inference time <= 6 hours to prevent this issue…
", "rawMarkdown": "It's not only about the number of notebooks but also the number of cells, so the difference has a chance to be not a minor one. Currently I am keeping the inference time <= 6 hours to prevent this issue...", "votes": 3 }, { "id": 1870404, "postDate": "2022-07-25T14:25:52.237Z", "content": "In order to be reasonable, we will check that the second-stage test set is approximately similar in size to the current one (i.e. it might be larger but will not be excessively larger). However, it's still your responsibility to make sure it runs in the alloted time. You are wise to leave a buffer or build in timing controls that make sure you don't run overtime.
", "rawMarkdown": "In order to be reasonable, we will check that the second-stage test set is approximately similar in size to the current one (i.e. it might be larger but will not be excessively larger). However, it's still your responsibility to make sure it runs in the alloted time. You are wise to leave a buffer or build in timing controls that make sure you don't run overtime.", "votes": 3, "replies": [ { "id": 1870799, "postDate": "2022-07-25T21:31:53.667Z", "content": "Thanks!
\nMay I also suggest excluding unrealistic notebooks, like ones with thousands of cells?
\nI'm testing my solution to make sure it can handle such inputs, but I'd expect it may break some notebooks.
@wcukierski , got it, thanks!
", "rawMarkdown": "@wcukierski , got it, thanks!" }, { "id": 1870811, "postDate": "2022-07-25T21:41:16.207Z", "content": "I just suggest we could have similar overall markdown and code cell ratio.
", "rawMarkdown": "I just suggest we could have similar overall markdown and code cell ratio. " }, { "id": 1883990, "postDate": "2022-08-04T08:28:41.623Z", "content": "@wcukierski why wouldn't you just use the same number of notebooks (or a bit less)?
", "rawMarkdown": "@wcukierski why wouldn't you just use the same number of notebooks (or a bit less)?", "votes": 2 } ] }, { "id": 1868740, "postDate": "2022-07-24T07:28:24.953Z", "content": "I don't think this will count then as the condition says that the code needs to run within 9 hours
", "rawMarkdown": "I don't think this will count then as the condition says that the code needs to run within 9 hours" }, { "id": 1869195, "postDate": "2022-07-24T15:10:35.853Z", "rawMarkdown": "", "isDeleted": true }, { "id": 1874332, "postDate": "2022-07-28T07:58:34Z", "content": "Thanks for sharing!
", "rawMarkdown": "Thanks for sharing!" } ], "comments": [ { "id": 1868907, "author_name": "Chenglu", "author_url": "", "post_date": "2022-07-24T10:32:07.430000", "content": "I have asked the same question here: https://www.kaggle.com/competitions/AI4Code/discussion/332516
\nIt'll be great if evaluation on private dataset can have more quotas (both memory and time) to prevent the unexepcted failures…
", "votes": 3, "replies": [ { "id": 1869078, "author_name": "Vadim Irtlach", "author_url": "", "post_date": "2022-07-24T13:18:07.527000", "content": "I suppose that the future test dataset will be +- the same size as the nowadays test dataset. If not, it will be funny for ensemblers 😄👀.
", "votes": 0, "replies": [] }, { "id": 1869179, "author_name": "Chenglu", "author_url": "", "post_date": "2022-07-24T14:51:53.373000", "content": "It's not only about the number of notebooks but also the number of cells, so the difference has a chance to be not a minor one. Currently I am keeping the inference time <= 6 hours to prevent this issue…
", "votes": 3, "replies": [] }, { "id": 1870404, "author_name": "Will Cukierski", "author_url": "", "post_date": "2022-07-25T14:25:52.237000", "content": "In order to be reasonable, we will check that the second-stage test set is approximately similar in size to the current one (i.e. it might be larger but will not be excessively larger). However, it's still your responsibility to make sure it runs in the alloted time. You are wise to leave a buffer or build in timing controls that make sure you don't run overtime.
", "votes": 3, "replies": [ { "id": 1870799, "author_name": "Dmytro Poplavskiy", "author_url": "", "post_date": "2022-07-25T21:31:53.667000", "content": "Thanks!
\nMay I also suggest excluding unrealistic notebooks, like ones with thousands of cells?
\nI'm testing my solution to make sure it can handle such inputs, but I'd expect it may break some notebooks.
@wcukierski , got it, thanks!
", "votes": 0, "replies": [] }, { "id": 1870811, "author_name": "gezi", "author_url": "", "post_date": "2022-07-25T21:41:16.207000", "content": "I just suggest we could have similar overall markdown and code cell ratio.
", "votes": 0, "replies": [] }, { "id": 1883990, "author_name": "CroDoc", "author_url": "", "post_date": "2022-08-04T08:28:41.623000", "content": "@wcukierski why wouldn't you just use the same number of notebooks (or a bit less)?
", "votes": 2, "replies": [] } ] }, { "id": 1868740, "author_name": "Ravi Ramakrishnan", "author_url": "", "post_date": "2022-07-24T07:28:24.953000", "content": "I don't think this will count then as the condition says that the code needs to run within 9 hours
", "votes": 0, "replies": [] }, { "id": 1869195, "author_name": "", "author_url": "", "post_date": "2022-07-24T15:10:35.853000", "content": "", "votes": 0, "replies": [] }, { "id": 1874332, "author_name": "wilson136", "author_url": "", "post_date": "2022-07-28T07:58:34", "content": "Thanks for sharing!
", "votes": 0, "replies": [] } ], "raw_markdown_by_id": { "1868411": "What if the submission run in 9 hours for current test dataset but exceed 9 hours for replaced test dataset?", "1868907": "I have asked the same question here: https://www.kaggle.com/competitions/AI4Code/discussion/332516\n\nIt'll be great if evaluation on private dataset can have more quotas (both memory and time) to prevent the unexepcted failures...", "1868740": "I don't think this will count then as the condition says that the code needs to run within 9 hours", "1869195": "", "1874332": "Thanks for sharing!" } }