In our team we have noticed a few submission.csv not found errors on recent subs. The file gets generated ok on the 10 rows of visible test data. Also, we adjusted thresholds to test that it still gets generated when some rows of the submission file predict no bboxes. There are no meaningful error messages and we have tried to test a variety of potential scenarios that might be happening without finding any errors.
\n\nAt least in some cases the error occurred after a few minutes and quicker than we expected the submission should have taken, but we don't get those errors on the visible test data or on the validation data and it is not obvious how to debug.
\n\nI understand that one option is to re-purpose some public code but it would be nice to understand how to debug our existing code.
\n\nDoes anyone have any suggestions what could be causing this, or, how to debug?
", "messages": [ { "id": "941521", "postDate": "07/23/2020 09:08:56", "content": "In our team we have noticed a few submission.csv not found errors on recent subs. The file gets generated ok on the 10 rows of visible test data. Also, we adjusted thresholds to test that it still gets generated when some rows of the submission file predict no bboxes. There are no meaningful error messages and we have tried to test a variety of potential scenarios that might be happening without finding any errors.
\n\nAt least in some cases the error occurred after a few minutes and quicker than we expected the submission should have taken, but we don't get those errors on the visible test data or on the validation data and it is not obvious how to debug.
\n\nI understand that one option is to re-purpose some public code but it would be nice to understand how to debug our existing code.
\n\nDoes anyone have any suggestions what could be causing this, or, how to debug?
", "rawMarkdown": "In our team we have noticed a few submission.csv not found errors on recent subs. The file gets generated ok on the 10 rows of visible test data. Also, we adjusted thresholds to test that it still gets generated when some rows of the submission file predict no bboxes. There are no meaningful error messages and we have tried to test a variety of potential scenarios that might be happening without finding any errors.\n\nAt least in some cases the error occurred after a few minutes and quicker than we expected the submission should have taken, but we don't get those errors on the visible test data or on the validation data and it is not obvious how to debug.\n\nI understand that one option is to re-purpose some public code but it would be nice to understand how to debug our existing code.\n\nDoes anyone have any suggestions what could be causing this, or, how to debug?", "votes": null }, { "id": "941801", "postDate": "07/23/2020 12:24:53", "content": "@datahobbit , this submission.csv not found error is faced by many of us in this challenge. It is very difficult to reproduce the issue with available 10 images. After spending so much time to fix it I share my steps to resolve this issue-
\n\nIn my local system with the given 10 test images, I also added different resolution images(with and without wheat) and a text file in the folder. I wrapped my prediction code inside the try block and in except block with printing the exception I returned empty string. In this I was able to reproduce some error in my code and after resolving those issues I was able to submit successfully.
", "rawMarkdown": "datahobbit , this submission.csv not found error is faced by many of us in this challenge. It is very difficult to reproduce the issue with available 10 images. After spending so much time to fix it I share my steps to resolve this issue-\n\n In my local system with the given 10 test images, I also added different resolution images(with and without wheat) and a text file in the folder. I wrapped my prediction code inside the try block and in except block with printing the exception I returned empty string. In this I was able to reproduce some error in my code and after resolving those issues I was able to submit successfully.", "votes": null }, { "id": "942338", "postDate": "07/23/2020 17:47:54", "content": "OK thanks for the good suggestions Prateek.
", "rawMarkdown": "OK thanks for the good suggestions Prateek.", "votes": null }, { "id": "942604", "postDate": "07/23/2020 21:20:45", "content": "If you suspect that the amount of test image data has something to do with it then you could try generating a submission based on the train dataset and treat it as the test data.
\n\nJust as a matter of testing how it works with a larger dataset.
\n\nI've had a similar issue in an earlier competition and after using the train data as test data to generate a submission I discovered the kernel was running out of memory. Hope this helps.
", "rawMarkdown": "If you suspect that the amount of test image data has something to do with it then you could try generating a submission based on the train dataset and treat it as the test data.\n\nJust as a matter of testing how it works with a larger dataset.\n\nI've had a similar issue in an earlier competition and after using the train data as test data to generate a submission I discovered the kernel was running out of memory. Hope this helps.", "votes": null }, { "id": "942720", "postDate": "07/24/2020 00:48:16", "content": "@datahobbit I experienced a similar issue even though my code was able to handle differently sized images and images with no wheat heads.
\n\nI couldn't identify the root cause, but I was able to work around the problem by separating the training and inference kernels and pruning the inference kernel to remove all unnecessary code.
", "rawMarkdown": "datahobbit I experienced a similar issue even though my code was able to handle differently sized images and images with no wheat heads.\n\nI couldn't identify the root cause, but I was able to work around the problem by separating the training and inference kernels and pruning the inference kernel to remove all unnecessary code.", "votes": null }, { "id": "943870", "postDate": "07/24/2020 16:33:48", "content": "Thanks for the suggestions, through a combination of these and internal team investigation we have resolved the issue. Like Antony said it still isn't exactly clear what was causing it though...
", "rawMarkdown": "Thanks for the suggestions, through a combination of these and internal team investigation we have resolved the issue. Like Antony said it still isn't exactly clear what was causing it though...", "votes": null } ], "comments": [ { "id": 941801, "author_name": "iamprateek", "author_url": "", "post_date": "07/23/2020 12:24:53", "content": "@datahobbit , this submission.csv not found error is faced by many of us in this challenge. It is very difficult to reproduce the issue with available 10 images. After spending so much time to fix it I share my steps to resolve this issue-
\n\nIn my local system with the given 10 test images, I also added different resolution images(with and without wheat) and a text file in the folder. I wrapped my prediction code inside the try block and in except block with printing the exception I returned empty string. In this I was able to reproduce some error in my code and after resolving those issues I was able to submit successfully.
", "votes": null, "replies": [ { "id": 942338, "author_name": "datahobbit", "author_url": "", "post_date": "07/23/2020 17:47:54", "content": "OK thanks for the good suggestions Prateek.
", "votes": null, "replies": [] } ] }, { "id": 942604, "author_name": "rsmits", "author_url": "", "post_date": "07/23/2020 21:20:45", "content": "If you suspect that the amount of test image data has something to do with it then you could try generating a submission based on the train dataset and treat it as the test data.
\n\nJust as a matter of testing how it works with a larger dataset.
\n\nI've had a similar issue in an earlier competition and after using the train data as test data to generate a submission I discovered the kernel was running out of memory. Hope this helps.
", "votes": null, "replies": [] }, { "id": 942720, "author_name": "aemulcahy", "author_url": "", "post_date": "07/24/2020 00:48:16", "content": "@datahobbit I experienced a similar issue even though my code was able to handle differently sized images and images with no wheat heads.
\n\nI couldn't identify the root cause, but I was able to work around the problem by separating the training and inference kernels and pruning the inference kernel to remove all unnecessary code.
", "votes": null, "replies": [] }, { "id": 943870, "author_name": "datahobbit", "author_url": "", "post_date": "07/24/2020 16:33:48", "content": "Thanks for the suggestions, through a combination of these and internal team investigation we have resolved the issue. Like Antony said it still isn't exactly clear what was causing it though...
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "941521": "In our team we have noticed a few submission.csv not found errors on recent subs. The file gets generated ok on the 10 rows of visible test data. Also, we adjusted thresholds to test that it still gets generated when some rows of the submission file predict no bboxes. There are no meaningful error messages and we have tried to test a variety of potential scenarios that might be happening without finding any errors.\n\nAt least in some cases the error occurred after a few minutes and quicker than we expected the submission should have taken, but we don't get those errors on the visible test data or on the validation data and it is not obvious how to debug.\n\nI understand that one option is to re-purpose some public code but it would be nice to understand how to debug our existing code.\n\nDoes anyone have any suggestions what could be causing this, or, how to debug?", "941801": "datahobbit , this submission.csv not found error is faced by many of us in this challenge. It is very difficult to reproduce the issue with available 10 images. After spending so much time to fix it I share my steps to resolve this issue-\n\n In my local system with the given 10 test images, I also added different resolution images(with and without wheat) and a text file in the folder. I wrapped my prediction code inside the try block and in except block with printing the exception I returned empty string. In this I was able to reproduce some error in my code and after resolving those issues I was able to submit successfully.", "942338": "OK thanks for the good suggestions Prateek.", "942604": "If you suspect that the amount of test image data has something to do with it then you could try generating a submission based on the train dataset and treat it as the test data.\n\nJust as a matter of testing how it works with a larger dataset.\n\nI've had a similar issue in an earlier competition and after using the train data as test data to generate a submission I discovered the kernel was running out of memory. Hope this helps.", "942720": "datahobbit I experienced a similar issue even though my code was able to handle differently sized images and images with no wheat heads.\n\nI couldn't identify the root cause, but I was able to work around the problem by separating the training and inference kernels and pruning the inference kernel to remove all unnecessary code.", "943870": "Thanks for the suggestions, through a combination of these and internal team investigation we have resolved the issue. Like Antony said it still isn't exactly clear what was causing it though..." }, "source": "meta" }