Some notebooks are giving out of memory error while submission. The training is fine but submission gives the error. But these error are also counted as valid submission and the submission count increases. Can this be avoided @organizers?
", "messages": [ { "id": "2444572", "postDate": "09/18/2023 10:51:24", "content": "Some notebooks are giving out of memory error while submission. The training is fine but submission gives the error. But these error are also counted as valid submission and the submission count increases. Can this be avoided @organizers?
", "rawMarkdown": "Some notebooks are giving out of memory error while submission. The training is fine but submission gives the error. But these error are also counted as valid submission and the submission count increases. Can this be avoided @organizers?", "votes": null }, { "id": "2444602", "postDate": "09/18/2023 11:07:38", "content": "Upvoted! This will be very helpful if getting implemented. Wonder the consideration about why counting it as a submission though.
", "rawMarkdown": "Upvoted! This will be very helpful if getting implemented. Wonder the consideration about why counting it as a submission though.", "votes": null }, { "id": "2444615", "postDate": "09/18/2023 11:17:34", "content": "🥲 Exactly !!! The notebooks become out of memory during submissions even after adding delete statements and gc.collect() in proper places. And before you know it, you have used 4/5 submisisons in errors. Something needs to be done about it!
", "rawMarkdown": "🥲 Exactly !!! The notebooks become out of memory during submissions even after adding delete statements and gc.collect() in proper places. And before you know it, you have used 4/5 submisisons in errors. Something needs to be done about it!", "votes": null }, { "id": "2444734", "postDate": "09/18/2023 12:55:32", "content": "Organizers are afraid of extracting important elements of the test set( eg actual lenght) if multiple failed submissions are used.
", "rawMarkdown": "Organizers are afraid of extracting important elements of the test set( eg actual lenght) if multiple failed submissions are used.", "votes": null }, { "id": "2444872", "postDate": "09/18/2023 14:23:48", "content": "That's right. In the competition ICR that I previously participated in, there was a limit of one submission per day, and the purpose was to prohibit so-called probing. Please see here in details. Participants repeatedly commented on this no-count error during submission, but Kaggle did not respond at all.
", "rawMarkdown": "That's right. In the competition [ICR](https://www.kaggle.com/competitions/icr-identify-age-related-conditions) that I previously participated in, there was a limit of one submission per day, and the purpose was to prohibit so-called probing. Please see [here](https://www.kaggle.com/competitions/icr-identify-age-related-conditions/discussion/425130) in details. Participants repeatedly commented on this no-count error during submission, but Kaggle did not respond at all.", "votes": null }, { "id": "2444939", "postDate": "09/18/2023 14:48:24", "content": "This has been happening a lot when submitting relatively more complex models. Lost a lot of submissions from memory errors after scoring for over an hour. Actually checked the logs and had out of memory error after 6 hours waiting for the scoring even just performing inference on the notebook!
", "rawMarkdown": "This has been happening a lot when submitting relatively more complex models. Lost a lot of submissions from memory errors after scoring for over an hour. Actually checked the logs and had out of memory error after 6 hours waiting for the scoring even just performing inference on the notebook!", "votes": null }, { "id": "2444942", "postDate": "09/18/2023 14:49:20", "content": "Am with you guys on this. I has not been funny at all after waiting hours for scoring.
", "rawMarkdown": "Am with you guys on this. I has not been funny at all after waiting hours for scoring.", "votes": null }, { "id": "2445233", "postDate": "09/18/2023 17:08:37", "content": "Hi @abhranta, consider feature extraction and prediction only on a set of series_id at once, may be use half of test series_id at once. This worked for me.
\nHere is a reference to predicting a single series at once and combining results
", "rawMarkdown": "Hi @abhranta, consider feature extraction and prediction only on a set of series_id at once, may be use half of test series_id at once. This worked for me.\n\nHere is a [reference](https://www.kaggle.com/code/anilreddyvv/tackling-out-of-memory-issue-on-test-set) to predicting a single series at once and combining results", "votes": null }, { "id": "2446793", "postDate": "09/19/2023 16:06:48", "content": "I empathize with your frustration regarding out of memory errors being considered as submissions, I experienced similar frustration before this discussion in the past ICR competition.
\nOut of memory error counting as a submission is necessary to prevent leaderboard probing.
\nWithout this policy in place, individuals could easily exploit the system by using an if statement in a notebook which would raise the OOM error if specific conditions are true in the dataset, all without using up an official submission.
\nIt more difficult for us, but it can't be avoided. @abhranta @rimbax @aidlre001
", "rawMarkdown": "I empathize with your frustration regarding out of memory errors being considered as submissions, I experienced similar frustration before [this ](https://www.kaggle.com/competitions/icr-identify-age-related-conditions/discussion/409801) discussion in the past ICR competition. \n\nOut of memory error counting as a submission is necessary to prevent leaderboard probing. \n\nWithout this policy in place, individuals could easily exploit the system by using an if statement in a notebook which would raise the OOM error if specific conditions are true in the dataset, all without using up an official submission. \n\nIt more difficult for us, but it can't be avoided. @abhranta @rimbax @aidlre001", "votes": null }, { "id": "2446800", "postDate": "09/19/2023 16:10:28", "content": "I mean, its fine if they just show that the notebook ran out of memory but don't count it as valid solution right? They don't need to count them as submissions too
", "rawMarkdown": "I mean, its fine if they just show that the notebook ran out of memory but don't count it as valid solution right? They don't need to count them as submissions too", "votes": null }, { "id": "2446809", "postDate": "09/19/2023 16:16:50", "content": "No, I can do something like this:
\nif something_I_want_to_check: throw error
\nelse: submit normally (or throw other error type)
and I will get info without a submission being counted (if condition it's true, but that can definitely be exploited).
", "rawMarkdown": "No, I can do something like this:\n\n`if something_I_want_to_check: throw error`\n`else: submit normally (or throw other error type)`\n\nand I will get info without a submission being counted (if condition it's true, but that can definitely be exploited).", "votes": null } ], "comments": [ { "id": 2444602, "author_name": "rimbax", "author_url": "", "post_date": "09/18/2023 11:07:38", "content": "Upvoted! This will be very helpful if getting implemented. Wonder the consideration about why counting it as a submission though.
", "votes": null, "replies": [ { "id": 2444615, "author_name": "abhranta", "author_url": "", "post_date": "09/18/2023 11:17:34", "content": "🥲 Exactly !!! The notebooks become out of memory during submissions even after adding delete statements and gc.collect() in proper places. And before you know it, you have used 4/5 submisisons in errors. Something needs to be done about it!
", "votes": null, "replies": [ { "id": 2444942, "author_name": "aidlre001", "author_url": "", "post_date": "09/18/2023 14:49:20", "content": "Am with you guys on this. I has not been funny at all after waiting hours for scoring.
", "votes": null, "replies": [] } ] } ] }, { "id": 2444734, "author_name": "georgem", "author_url": "", "post_date": "09/18/2023 12:55:32", "content": "Organizers are afraid of extracting important elements of the test set( eg actual lenght) if multiple failed submissions are used.
", "votes": null, "replies": [ { "id": 2444872, "author_name": "mitsuyasuhoshino", "author_url": "", "post_date": "09/18/2023 14:23:48", "content": "That's right. In the competition ICR that I previously participated in, there was a limit of one submission per day, and the purpose was to prohibit so-called probing. Please see here in details. Participants repeatedly commented on this no-count error during submission, but Kaggle did not respond at all.
", "votes": null, "replies": [] } ] }, { "id": 2444939, "author_name": "aidlre001", "author_url": "", "post_date": "09/18/2023 14:48:24", "content": "This has been happening a lot when submitting relatively more complex models. Lost a lot of submissions from memory errors after scoring for over an hour. Actually checked the logs and had out of memory error after 6 hours waiting for the scoring even just performing inference on the notebook!
", "votes": null, "replies": [] }, { "id": 2445233, "author_name": "anilreddyvv", "author_url": "", "post_date": "09/18/2023 17:08:37", "content": "Hi @abhranta, consider feature extraction and prediction only on a set of series_id at once, may be use half of test series_id at once. This worked for me.
\nHere is a reference to predicting a single series at once and combining results
", "votes": null, "replies": [] }, { "id": 2446793, "author_name": "annafabris", "author_url": "", "post_date": "09/19/2023 16:06:48", "content": "I empathize with your frustration regarding out of memory errors being considered as submissions, I experienced similar frustration before this discussion in the past ICR competition.
\nOut of memory error counting as a submission is necessary to prevent leaderboard probing.
\nWithout this policy in place, individuals could easily exploit the system by using an if statement in a notebook which would raise the OOM error if specific conditions are true in the dataset, all without using up an official submission.
\nIt more difficult for us, but it can't be avoided. @abhranta @rimbax @aidlre001
", "votes": null, "replies": [ { "id": 2446800, "author_name": "abhranta", "author_url": "", "post_date": "09/19/2023 16:10:28", "content": "I mean, its fine if they just show that the notebook ran out of memory but don't count it as valid solution right? They don't need to count them as submissions too
", "votes": null, "replies": [ { "id": 2446809, "author_name": "annafabris", "author_url": "", "post_date": "09/19/2023 16:16:50", "content": "No, I can do something like this:
\nif something_I_want_to_check: throw error
\nelse: submit normally (or throw other error type)
and I will get info without a submission being counted (if condition it's true, but that can definitely be exploited).
", "votes": null, "replies": [] } ] } ] } ], "raw_markdown_by_id": { "2444572": "Some notebooks are giving out of memory error while submission. The training is fine but submission gives the error. But these error are also counted as valid submission and the submission count increases. Can this be avoided @organizers?", "2444602": "Upvoted! This will be very helpful if getting implemented. Wonder the consideration about why counting it as a submission though.", "2444615": "🥲 Exactly !!! The notebooks become out of memory during submissions even after adding delete statements and gc.collect() in proper places. And before you know it, you have used 4/5 submisisons in errors. Something needs to be done about it!", "2444734": "Organizers are afraid of extracting important elements of the test set( eg actual lenght) if multiple failed submissions are used.", "2444872": "That's right. In the competition [ICR](https://www.kaggle.com/competitions/icr-identify-age-related-conditions) that I previously participated in, there was a limit of one submission per day, and the purpose was to prohibit so-called probing. Please see [here](https://www.kaggle.com/competitions/icr-identify-age-related-conditions/discussion/425130) in details. Participants repeatedly commented on this no-count error during submission, but Kaggle did not respond at all.", "2444939": "This has been happening a lot when submitting relatively more complex models. Lost a lot of submissions from memory errors after scoring for over an hour. Actually checked the logs and had out of memory error after 6 hours waiting for the scoring even just performing inference on the notebook!", "2444942": "Am with you guys on this. I has not been funny at all after waiting hours for scoring.", "2445233": "Hi @abhranta, consider feature extraction and prediction only on a set of series_id at once, may be use half of test series_id at once. This worked for me.\n\nHere is a [reference](https://www.kaggle.com/code/anilreddyvv/tackling-out-of-memory-issue-on-test-set) to predicting a single series at once and combining results", "2446793": "I empathize with your frustration regarding out of memory errors being considered as submissions, I experienced similar frustration before [this ](https://www.kaggle.com/competitions/icr-identify-age-related-conditions/discussion/409801) discussion in the past ICR competition. \n\nOut of memory error counting as a submission is necessary to prevent leaderboard probing. \n\nWithout this policy in place, individuals could easily exploit the system by using an if statement in a notebook which would raise the OOM error if specific conditions are true in the dataset, all without using up an official submission. \n\nIt more difficult for us, but it can't be avoided. @abhranta @rimbax @aidlre001", "2446800": "I mean, its fine if they just show that the notebook ran out of memory but don't count it as valid solution right? They don't need to count them as submissions too", "2446809": "No, I can do something like this:\n\n`if something_I_want_to_check: throw error`\n`else: submit normally (or throw other error type)`\n\nand I will get info without a submission being counted (if condition it's true, but that can definitely be exploited)." }, "source": "meta" }