This kernel competition is different from others as in other kernel competition, the LB and private LB are evaluate in the same time, if there occued an exception on the private test dataset, the submission will failed, but in this game, it's not since there are no private test dataset for now.
\nThere are some factors that may make the private test dataset raise exceptions but not the LB test dataset. Such as the dataset size differences.
\nIs there any advises on this one? Or if exception really occurs in the future, what is the sequences?
", "messages": [ { "id": "1828812", "postDate": "06/22/2022 04:14:29", "content": "This kernel competition is different from others as in other kernel competition, the LB and private LB are evaluate in the same time, if there occued an exception on the private test dataset, the submission will failed, but in this game, it's not since there are no private test dataset for now.
\nThere are some factors that may make the private test dataset raise exceptions but not the LB test dataset. Such as the dataset size differences.
\nIs there any advises on this one? Or if exception really occurs in the future, what is the sequences?
", "rawMarkdown": "This kernel competition is different from others as in other kernel competition, the LB and private LB are evaluate in the same time, if there occued an exception on the private test dataset, the submission will failed, but in this game, it's not since there are no private test dataset for now.\n\nThere are some factors that may make the private test dataset raise exceptions but not the LB test dataset. Such as the dataset size differences.\n\nIs there any advises on this one? Or if exception really occurs in the future, what is the sequences?", "votes": null }, { "id": "1828817", "postDate": "06/22/2022 04:28:05", "content": "The final LB is based off 90 days of notebooks that don't exist yet, presumingly the size should be similar to the 90 day sample used for the 'public' LB.
\nHopefully there won't be anything especially tricky in it - I figure if there was one that caused a lot of notebooks to crash, it could be dropped. (But that might not help if your code's unique…)
\nAs for load testing, try feeding 30K of train directory .jsons? If your code can handle that many it should be OK with either LB.
", "rawMarkdown": "The final LB is based off 90 days of notebooks that don't exist yet, presumingly the size should be similar to the 90 day sample used for the 'public' LB. \n\nHopefully there won't be anything especially tricky in it - I figure if there was one that caused a lot of notebooks to crash, it could be dropped. (But that might not help if your code's unique...)\n\nAs for load testing, try feeding 30K of train directory .jsons? If your code can handle that many it should be OK with either LB.", "votes": null }, { "id": "1833623", "postDate": "06/26/2022 07:14:09", "content": "Since timing is already taken care of in the current LB submission, it should \"ideally\" be unrestricted for the second one, I feel (my opinion). I dont see any reason why kernels will fail on the second one after passing the first (unless some really wierd bot-generated kernels or something, but this should be weeded out by organisers in first place). Of course we should aim to keep an hour or so as buffer, just in case
\nMore concerning would be notebook failing due to data issues during pre-processing. I think we have to be careful there. would be a pity if high scoring notebook loses out due to some unanticipated formatting issue.
", "rawMarkdown": "Since timing is already taken care of in the current LB submission, it should \"ideally\" be unrestricted for the second one, I feel (my opinion). I dont see any reason why kernels will fail on the second one after passing the first (unless some really wierd bot-generated kernels or something, but this should be weeded out by organisers in first place). Of course we should aim to keep an hour or so as buffer, just in case\n\nMore concerning would be notebook failing due to data issues during pre-processing. I think we have to be careful there. would be a pity if high scoring notebook loses out due to some unanticipated formatting issue.", "votes": null }, { "id": "1833682", "postDate": "06/26/2022 08:14:19", "content": "Yes, and I already wrap my preprocessing code with a try ... except ... block to prevent the issue you mentioned.
You might consider using function decorators as described in this post for this.
", "rawMarkdown": "You might consider using function decorators as described [in this post](https://www.kaggle.com/c/nfl-big-data-bowl-2020/discussion/120375#688496) for this.", "votes": null }, { "id": "1836257", "postDate": "06/28/2022 13:43:02", "content": "Great tip, thanks!
", "rawMarkdown": "Great tip, thanks!", "votes": null } ], "comments": [ { "id": 1828817, "author_name": "happycube", "author_url": "", "post_date": "06/22/2022 04:28:05", "content": "The final LB is based off 90 days of notebooks that don't exist yet, presumingly the size should be similar to the 90 day sample used for the 'public' LB.
\nHopefully there won't be anything especially tricky in it - I figure if there was one that caused a lot of notebooks to crash, it could be dropped. (But that might not help if your code's unique…)
\nAs for load testing, try feeding 30K of train directory .jsons? If your code can handle that many it should be OK with either LB.
", "votes": null, "replies": [] }, { "id": 1833623, "author_name": "allohvk", "author_url": "", "post_date": "06/26/2022 07:14:09", "content": "Since timing is already taken care of in the current LB submission, it should \"ideally\" be unrestricted for the second one, I feel (my opinion). I dont see any reason why kernels will fail on the second one after passing the first (unless some really wierd bot-generated kernels or something, but this should be weeded out by organisers in first place). Of course we should aim to keep an hour or so as buffer, just in case
\nMore concerning would be notebook failing due to data issues during pre-processing. I think we have to be careful there. would be a pity if high scoring notebook loses out due to some unanticipated formatting issue.
", "votes": null, "replies": [ { "id": 1833682, "author_name": "snaker", "author_url": "", "post_date": "06/26/2022 08:14:19", "content": "Yes, and I already wrap my preprocessing code with a try ... except ... block to prevent the issue you mentioned.
You might consider using function decorators as described in this post for this.
", "votes": null, "replies": [] }, { "id": 1836257, "author_name": "snaker", "author_url": "", "post_date": "06/28/2022 13:43:02", "content": "Great tip, thanks!
", "votes": null, "replies": [] } ] } ], "raw_markdown_by_id": { "1828812": "This kernel competition is different from others as in other kernel competition, the LB and private LB are evaluate in the same time, if there occued an exception on the private test dataset, the submission will failed, but in this game, it's not since there are no private test dataset for now.\n\nThere are some factors that may make the private test dataset raise exceptions but not the LB test dataset. Such as the dataset size differences.\n\nIs there any advises on this one? Or if exception really occurs in the future, what is the sequences?", "1828817": "The final LB is based off 90 days of notebooks that don't exist yet, presumingly the size should be similar to the 90 day sample used for the 'public' LB. \n\nHopefully there won't be anything especially tricky in it - I figure if there was one that caused a lot of notebooks to crash, it could be dropped. (But that might not help if your code's unique...)\n\nAs for load testing, try feeding 30K of train directory .jsons? If your code can handle that many it should be OK with either LB.", "1833623": "Since timing is already taken care of in the current LB submission, it should \"ideally\" be unrestricted for the second one, I feel (my opinion). I dont see any reason why kernels will fail on the second one after passing the first (unless some really wierd bot-generated kernels or something, but this should be weeded out by organisers in first place). Of course we should aim to keep an hour or so as buffer, just in case\n\nMore concerning would be notebook failing due to data issues during pre-processing. I think we have to be careful there. would be a pity if high scoring notebook loses out due to some unanticipated formatting issue.", "1833682": "Yes, and I already wrap my preprocessing code with a `try ... except ...` block to prevent the issue you mentioned.", "1835553": "You might consider using function decorators as described [in this post](https://www.kaggle.com/c/nfl-big-data-bowl-2020/discussion/120375#688496) for this.", "1836257": "Great tip, thanks!" }, "source": "meta" }