![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F3867543%2F48039ef44f8be71577b4e65f2e35363f%2FCapture.JPG?generation=1589105085435894&alt=media\")
Dear challengers and hosts,
\n\nThis is my first participation, and like some a number of other people I am also having trouble with my submissions, with either \"Submission scoring error\" or \"submission CSV not found\" status, despite the fact that my submission.csv seems correctly generated and formatted (see code output below). I checked the provided example notebooks and don't get where my errors could come from.
\n\n@bendvd , could you provide supplementary information about the files that are modified between the coding competition test set and scoring test set, as well as the way the submission file is parsed ? Also, about the path in which the submission.csv file has to be generated in ?
\n\nI am editing this message incrementally, but so far here are the tracks we went to :
\n\nWe can read elsewhere in this forum that in the submission file the image id order may have to be similar to the one in the sample_submission file; how that can be true if the scoring test set is not the same as the coding test set ? EDIT : according to https://www.kaggle.com/c/global-wheat-detection/discussion/148678#834579 by @chittalpatel and @kaushal2896 it doesn't seem to be the case.
I also tried to submit notebooks getting the scoring test set images either by reading the sample_submission file or listing the test folder files as discussed here https://www.kaggle.com/c/global-wheat-detection/discussion/148678#840750 , without success.
For some contestants it would seem that this issue may be caused by test set image size issues (https://www.kaggle.com/c/global-wheat-detection/discussion/149949 by @greatgamedota ) but to the exception of really abnormal images i doubt it to be the main cause of trouble, as the method i am trying to set up should not be sensible to variations in image size. For the sake of verification i modified input image size and my notebook still worked properly.
Managing exceptions thrown by empty predictions could be the key... i'll let you know
I am curious to receive feedback from experienced synchronous competition participants about this.
\n\nThanks for your help and all the best !\nJ.
\n\n
Related topics in which participants are having trouble too (so i'm not pointed out for not having taken time to read the forums haha) :\n- https://www.kaggle.com/c/global-wheat-detection/discussion/149847 by @rilllove \n- https://www.kaggle.com/c/global-wheat-detection/discussion/149809 by @spytensor \n- https://www.kaggle.com/c/global-wheat-detection/discussion/149630 by @rilllove \n- https://www.kaggle.com/c/global-wheat-detection/discussion/148678 by @chittalpatel \n- useful thread to try solving these errors by @bamps53 : https://www.kaggle.com/c/global-wheat-detection/discussion/148873
", "messages": [ { "id": "840730", "postDate": "05/10/2020 10:10:30", "content": "Dear challengers and hosts,
\n\nThis is my first participation, and like some a number of other people I am also having trouble with my submissions, with either \"Submission scoring error\" or \"submission CSV not found\" status, despite the fact that my submission.csv seems correctly generated and formatted (see code output below). I checked the provided example notebooks and don't get where my errors could come from.
\n\n@bendvd , could you provide supplementary information about the files that are modified between the coding competition test set and scoring test set, as well as the way the submission file is parsed ? Also, about the path in which the submission.csv file has to be generated in ?
\n\nI am editing this message incrementally, but so far here are the tracks we went to :
\n\nWe can read elsewhere in this forum that in the submission file the image id order may have to be similar to the one in the sample_submission file; how that can be true if the scoring test set is not the same as the coding test set ? EDIT : according to https://www.kaggle.com/c/global-wheat-detection/discussion/148678#834579 by @chittalpatel and @kaushal2896 it doesn't seem to be the case.
I also tried to submit notebooks getting the scoring test set images either by reading the sample_submission file or listing the test folder files as discussed here https://www.kaggle.com/c/global-wheat-detection/discussion/148678#840750 , without success.
For some contestants it would seem that this issue may be caused by test set image size issues (https://www.kaggle.com/c/global-wheat-detection/discussion/149949 by @greatgamedota ) but to the exception of really abnormal images i doubt it to be the main cause of trouble, as the method i am trying to set up should not be sensible to variations in image size. For the sake of verification i modified input image size and my notebook still worked properly.
Managing exceptions thrown by empty predictions could be the key... i'll let you know
I am curious to receive feedback from experienced synchronous competition participants about this.
\n\nThanks for your help and all the best !\nJ.
\n\n
Related topics in which participants are having trouble too (so i'm not pointed out for not having taken time to read the forums haha) :\n- https://www.kaggle.com/c/global-wheat-detection/discussion/149847 by @rilllove \n- https://www.kaggle.com/c/global-wheat-detection/discussion/149809 by @spytensor \n- https://www.kaggle.com/c/global-wheat-detection/discussion/149630 by @rilllove \n- https://www.kaggle.com/c/global-wheat-detection/discussion/148678 by @chittalpatel \n- useful thread to try solving these errors by @bamps53 : https://www.kaggle.com/c/global-wheat-detection/discussion/148873
", "rawMarkdown": "Dear challengers and hosts,\n\nThis is my first participation, and like some a number of other people I am also having trouble with my submissions, with either \"Submission scoring error\" or \"submission CSV not found\" status, despite the fact that my submission.csv seems correctly generated and formatted (see code output below). I checked the provided example notebooks and don't get where my errors could come from.\n\n@bendvd , could you provide supplementary information about the files that are modified between the coding competition test set and scoring test set, as well as the way the submission file is parsed ? Also, about the path in which the submission.csv file has to be generated in ?\n\n\nI am editing this message incrementally, but so far here are the tracks we went to :\n\n- We can read elsewhere in this forum that in the submission file the image id order may have to be similar to the one in the sample_submission file; how that can be true if the scoring test set is not the same as the coding test set ? EDIT : according to https://www.kaggle.com/c/global-wheat-detection/discussion/148678#834579 by @chittalpatel and @kaushal2896 it doesn't seem to be the case.\n\n- I also tried to submit notebooks getting the scoring test set images either by reading the sample_submission file or listing the test folder files as discussed here https://www.kaggle.com/c/global-wheat-detection/discussion/148678#840750 , without success.\n\n- For some contestants it would seem that this issue may be caused by test set image size issues (https://www.kaggle.com/c/global-wheat-detection/discussion/149949 by @greatgamedota ) but to the exception of really abnormal images i doubt it to be the main cause of trouble, as the method i am trying to set up should not be sensible to variations in image size. For the sake of verification i modified input image size and my notebook still worked properly.\n\n- Managing exceptions thrown by empty predictions could be the key... i'll let you know\n\n\nI am curious to receive feedback from experienced synchronous competition participants about this. \n\nThanks for your help and all the best !\nJ.\n\n\n\nRelated topics in which participants are having trouble too (so i'm not pointed out for not having taken time to read the forums haha) :\n- https://www.kaggle.com/c/global-wheat-detection/discussion/149847 by @rilllove \n- https://www.kaggle.com/c/global-wheat-detection/discussion/149809 by @spytensor \n- https://www.kaggle.com/c/global-wheat-detection/discussion/149630 by @rilllove \n- https://www.kaggle.com/c/global-wheat-detection/discussion/148678 by @chittalpatel \n- useful thread to try solving these errors by @bamps53 : https://www.kaggle.com/c/global-wheat-detection/discussion/148873", "votes": null }, { "id": "840774", "postDate": "05/10/2020 11:14:05", "content": "Same issue for me, some times I got 'Submission scoring error', some times for 'submission CSV not found', I can also see the submission.csv file in my Output and the format is correct, I try commit 10 times but still can't fix it.
", "rawMarkdown": "Same issue for me, some times I got 'Submission scoring error', some times for 'submission CSV not found', I can also see the submission.csv file in my Output and the format is correct, I try commit 10 times but still can't fix it.", "votes": null }, { "id": "840804", "postDate": "05/10/2020 11:55:05", "content": "Exactly! If order is to be maintained, then how it's possible to predict on hidden test set whose image IDs are not given in sample_submission.csv (I've seen test.csv in other such competitions but it's not there in our case). Also in almost all public kernels (including the kernel of competition host), authors are taking reference of given sample submission csv file and doing this will only predict 10 images from test set.
\n\nI even tried to predict on images directly read from \"test\" directory but that's not working. Here is my inference kernel if anyone wants to take a look: https://www.kaggle.com/kaushal2896/wheat-detection-fasterrcnn-inference
\n\nAnd this is my today's submission history:\n![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F1905996%2F252b4dce5b14ddc4c8e68f66858578a5%2FScreenshot%202020-05-10%20at%205.22.40%20PM.png?generation=1589111602252864&alt=media\")
Here is another thread for this issue: https://www.kaggle.com/c/global-wheat-detection/discussion/148678#840750 \nbut this doesn't makes sense to me.
\n\nI've tried many possible approaches to create the submission file but all failed!
\n\nSUPER FRUSTRATING!
\n\nEDIT:
\n\nSee this kernel: https://www.kaggle.com/ateplyuk/gwd-starter-efficientdet-inference\nFrom that we can see that order doesn't matter.
\n\nThanks!
", "rawMarkdown": "Exactly! If order is to be maintained, then how it's possible to predict on hidden test set whose image IDs are not given in sample\\_submission.csv (I've seen test.csv in other such competitions but it's not there in our case). Also in almost all public kernels (including the kernel of competition host), authors are taking reference of given sample submission csv file and doing this will only predict 10 images from test set.\n\nI even tried to predict on images directly read from \"test\" directory but that's not working. Here is my inference kernel if anyone wants to take a look: https://www.kaggle.com/kaushal2896/wheat-detection-fasterrcnn-inference\n\nAnd this is my today's submission history:\n\n\nHere is another thread for this issue: https://www.kaggle.com/c/global-wheat-detection/discussion/148678#840750 \nbut this doesn't makes sense to me.\n\nI've tried many possible approaches to create the submission file but all failed!\n\nSUPER FRUSTRATING!\n\nEDIT:\n\nSee this kernel: https://www.kaggle.com/ateplyuk/gwd-starter-efficientdet-inference\nFrom that we can see that order doesn't matter.\n\nThanks!", "votes": null }, { "id": "841199", "postDate": "05/10/2020 17:04:54", "content": "I'm too beginner like you @codename5281 and would really like to know the clear and informative description of test data and submission procedures for this competition.
", "rawMarkdown": "I'm too beginner like you @codename5281 and would really like to know the clear and informative description of test data and submission procedures for this competition.", "votes": null }, { "id": "841201", "postDate": "05/10/2020 17:05:38", "content": "I struggled with the format for awhile and I would like to share my findings:\n- Order doesn't matter, you can get the list of names from either sample_submission.csv or listing the test image directory\n- Confidence must be a float, doesn't matter or many digits of precision you give\n- X,Y,Width,Height must be integers\n- I'm not 100% sure but make sure Width and Height are always >0\n- For empty predictions you must predict and empty string '' NOT np.nan
Finally \"Submission CSV Not Found\" means either your submission isn't formatted correctly OR your kernal had an error while submitting.
", "rawMarkdown": "I struggled with the format for awhile and I would like to share my findings:\n- Order doesn't matter, you can get the list of names from either `sample_submission.csv` or listing the test image directory\n- Confidence must be a float, doesn't matter or many digits of precision you give\n- X,Y,Width,Height must be integers\n- I'm not 100% sure but make sure Width and Height are always >0\n- For empty predictions you must predict and empty string `''` NOT `np.nan`\n\nFinally \"Submission CSV Not Found\" means either your submission isn't formatted correctly OR your kernal had an error while submitting.", "votes": null }, { "id": "841219", "postDate": "05/10/2020 17:16:40", "content": "Thanks for sharing !\nOn my side, except for the empty predictions case everything should be OK. And i don't see why the submitted kernel would throw more errors than the commited one.\nEDIT: i'm more and more convinced that the issue could come from managing empty predictions. Can't wait for refreshed submission credits to let you know !
", "rawMarkdown": "Thanks for sharing !\nOn my side, except for the empty predictions case everything should be OK. And i don't see why the submitted kernel would throw more errors than the commited one.\nEDIT: i'm more and more convinced that the issue could come from managing empty predictions. Can't wait for refreshed submission credits to let you know !", "votes": null }, { "id": "841228", "postDate": "05/10/2020 17:18:59", "content": "Data has been described quite in details in the competition preprint paper available at https://arxiv.org/ftp/arxiv/papers/2005/2005.02162.pdf and in published exploratory data analysis (EDA) notebooks.\nFor all the other details, let's stick with the findings of other challengers while waiting for any eventual precisions from the host ! :)
", "rawMarkdown": "Data has been described quite in details in the competition preprint paper available at https://arxiv.org/ftp/arxiv/papers/2005/2005.02162.pdf and in published exploratory data analysis (EDA) notebooks.\nFor all the other details, let's stick with the findings of other challengers while waiting for any eventual precisions from the host ! :)", "votes": null }, { "id": "842132", "postDate": "05/11/2020 08:09:43", "content": "An update on this topic !
\n\nFollowing suggestions from @greatgamedota i finally succeded (after 26 trials, how good is that ?!) to submit !\nSome clues :\n- starting from sample_submission.csv did not work for me, as if it did not contain the right list of files on which perform detection for submission. Starting from listing the files in /kaggle/input/global-wheat-detection/test/ did the trick.\n- adding code to catch exceptions in the case where the method i'm using did not detect heads did help.
\n\nHere you go ! Now i'm heading back to model training :)
", "rawMarkdown": "An update on this topic !\n\nFollowing suggestions from @greatgamedota i finally succeded (after 26 trials, how good is that ?!) to submit !\nSome clues :\n- starting from sample_submission.csv did not work for me, as if it did not contain the right list of files on which perform detection for submission. Starting from listing the files in /kaggle/input/global-wheat-detection/test/ did the trick.\n- adding code to catch exceptions in the case where the method i'm using did not detect heads did help.\n\nHere you go ! Now i'm heading back to model training :)", "votes": null }, { "id": "851517", "postDate": "05/17/2020 17:23:12", "content": "I'm still facing the same problem.
\n\nI tried to create a notebook that only take the submission file from the \"global-wheat-detection-first-submission\" kernel and save it as submission.csv file.\nI double checked that the output csv file of my kernel is identical to the original kernel output. \nI still get failure on submission. \nWhen I tried to fork the original kernel and submit it everything is working fine.
\n\nAny idea what i'm doing wrong?
", "rawMarkdown": "I'm still facing the same problem. \n\nI tried to create a notebook that only take the submission file from the \"global-wheat-detection-first-submission\" kernel and save it as submission.csv file.\nI double checked that the output csv file of my kernel is identical to the original kernel output. \nI still get failure on submission. \nWhen I tried to fork the original kernel and submit it everything is working fine.\n\nAny idea what i'm doing wrong?", "votes": null }, { "id": "895324", "postDate": "06/21/2020 09:02:58", "content": "how to solve the 'Submission scoring error ' on the hidden private dataset. I am still getting the error even everything seems to look fine when I see the result with the available test set
", "rawMarkdown": "how to solve the 'Submission scoring error ' on the hidden private dataset. I am still getting the error even everything seems to look fine when I see the result with the available test set", "votes": null }, { "id": "895365", "postDate": "06/21/2020 09:35:45", "content": "@kaushal2896 could you please let me know how did you solve the submission scoring error
", "rawMarkdown": "kaushal2896 could you please let me know how did you solve the submission scoring error", "votes": null }, { "id": "904020", "postDate": "06/27/2020 09:29:29", "content": "hi there, have you solve the problem? I met the same problem too and I want to know how you solved it, thanks for your reply
", "rawMarkdown": "hi there, have you solve the problem? I met the same problem too and I want to know how you solved it, thanks for your reply", "votes": null }, { "id": "904085", "postDate": "06/27/2020 10:52:34", "content": "@kalyanpichuka and @shuhaoqiu I think your script should correctly form the output when there are no objects are detected in the given image. This was missing in my case. Therefore I took the reference of top FasterRCNN inference public kernel (https://www.kaggle.com/pestipeti/pytorch-starter-fasterrcnn-inference) and now it's working. Otherwise the order is not important and also you can directly iterate through either the submission CSV or the test directory for reading the test data.
", "rawMarkdown": "kalyanpichuka and @shuhaoqiu I think your script should correctly form the output when there are no objects are detected in the given image. This was missing in my case. Therefore I took the reference of top FasterRCNN inference public kernel (https://www.kaggle.com/pestipeti/pytorch-starter-fasterrcnn-inference) and now it's working. Otherwise the order is not important and also you can directly iterate through either the submission CSV or the test directory for reading the test data.", "votes": null }, { "id": "904815", "postDate": "06/28/2020 00:55:23", "content": "Thank you very much, it's exactly the problem!
", "rawMarkdown": "Thank you very much, it's exactly the problem!", "votes": null }, { "id": "907870", "postDate": "06/30/2020 07:57:18", "content": "Thanks @kaushal2896 for the response
", "rawMarkdown": "Thanks @kaushal2896 for the response", "votes": null }, { "id": "909333", "postDate": "06/30/2020 14:13:10", "content": "Thank you! Your findings were very helpful!
", "rawMarkdown": "Thank you! Your findings were very helpful!", "votes": null }, { "id": "909453", "postDate": "06/30/2020 15:46:52", "content": "Thanks a lot for sharing!\nYou mentioned \"For empty predictions you must predict and empty string '' NOT np.nan\"\nDoes this mean that even if there are no predicted bounding boxes for image it should be listed in submission.csv anyway? Something like:\n|image_id|PredictionString|\n|---------|:---------------:|\n|ag632h2438||
", "rawMarkdown": "Thanks a lot for sharing!\nYou mentioned \"For empty predictions you must predict and empty string '' NOT np.nan\"\nDoes this mean that even if there are no predicted bounding boxes for image it should be listed in submission.csv anyway? Something like:\n|image_id|PredictionString|\n|---------|:---------------:|\n|ag632h2438||", "votes": null }, { "id": "909613", "postDate": "06/30/2020 17:25:00", "content": "yes else itll error out saying there isnt enough rows in the csv
", "rawMarkdown": "yes else itll error out saying there isnt enough rows in the csv", "votes": null }, { "id": "912825", "postDate": "07/02/2020 18:48:12", "content": "Hi there, you said if there are no boxes, then we should leave it empty with empty string\nbut aren't there only 10 images to predict? and my prediction has all of them filled hence no empty boxes\nstill the error which questioner mentioned
", "rawMarkdown": "Hi there, you said if there are no boxes, then we should leave it empty with empty string\nbut aren't there only 10 images to predict? and my prediction has all of them filled hence no empty boxes\nstill the error which questioner mentioned", "votes": null }, { "id": "912832", "postDate": "07/02/2020 18:55:25", "content": "those 10 images aren't the full test set, when you submit they are replaced with all the test images which is a lot more than 10
", "rawMarkdown": "those 10 images aren't the full test set, when you submit they are replaced with all the test images which is a lot more than 10", "votes": null }, { "id": "912846", "postDate": "07/02/2020 19:05:05", "content": "Can you please suggest a notebook which is making predictions on that hidden test data.
", "rawMarkdown": "Can you please suggest a notebook which is making predictions on that hidden test data.", "votes": null }, { "id": "912851", "postDate": "07/02/2020 19:07:28", "content": "Pretty much any of the public inference notebooks will do. Some examples:\n- https://www.kaggle.com/pestipeti/pytorch-starter-fasterrcnn-inference\n- https://www.kaggle.com/shonenkov/inference-efficientdet\n- https://www.kaggle.com/trungthanhnguyen0502/gwd-inference\n- https://www.kaggle.com/orkatz2/yolov5-fake-or-real-single-model-l-b-0-753
", "rawMarkdown": "Pretty much any of the public inference notebooks will do. Some examples:\n- https://www.kaggle.com/pestipeti/pytorch-starter-fasterrcnn-inference\n- https://www.kaggle.com/shonenkov/inference-efficientdet\n- https://www.kaggle.com/trungthanhnguyen0502/gwd-inference\n- https://www.kaggle.com/orkatz2/yolov5-fake-or-real-single-model-l-b-0-753", "votes": null }, { "id": "912858", "postDate": "07/02/2020 19:10:55", "content": "Thank you
", "rawMarkdown": "Thank you", "votes": null }, { "id": "930066", "postDate": "07/15/2020 07:05:22", "content": "Hello,\nShould the submission.csv contain one unique row per image_id with all predictionString in one line as given in this link (https://www.kaggle.com/pestipeti/pytorch-starter-fasterrcnn-inference),\nOR predictionString of format(1.0,0,0,50,50) on multiple lines for same image_id.\nI tried both, also handle the case where it returned no bounding box(replacing the predictionString=\"\")\nBut still getting Submission scoring error.\nCan anyone please help me.\nThank You.
", "rawMarkdown": "Hello,\nShould the submission.csv contain one unique row per image_id with all predictionString in one line as given in this link (https://www.kaggle.com/pestipeti/pytorch-starter-fasterrcnn-inference),\nOR predictionString of format(1.0,0,0,50,50) on multiple lines for same image_id.\nI tried both, also handle the case where it returned no bounding box(replacing the predictionString=\"\")\nBut still getting Submission scoring error.\nCan anyone please help me.\nThank You.", "votes": null }, { "id": "930071", "postDate": "07/15/2020 07:09:30", "content": "There should be one row per imageid with all predictions concatenated in one PredictionString.
", "rawMarkdown": "There should be one row per imageid with all predictions concatenated in one PredictionString.", "votes": null }, { "id": "931127", "postDate": "07/16/2020 02:16:33", "content": "Hello,\nThank you for clarifying the doubt.\n![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F2652991%2Fe2c2c8a618901eb338fbc5c64253a87b%2FCapture.PNG?generation=1594865680473349&alt=media\")
I have attached the image of my submission.csv file.\nIt contains 1 unique row for every image_id, but still getting Submission scoring error.
\n\nThank you.
", "rawMarkdown": "Hello,\nThank you for clarifying the doubt.\n\n\nI have attached the image of my submission.csv file.\nIt contains 1 unique row for every image_id, but still getting Submission scoring error.\n\nThank you.", "votes": null }, { "id": "931149", "postDate": "07/16/2020 02:49:11", "content": "If you've already checked all the above suggestions, perhaps a cell in your submission notebook is failing when run on the private test set, leading to a broken submission.csv file. Like perhaps your post processing (WBF or likewise) fails when it encounters a prediction with no box? I would try my best to simplify your submission process, removing extra steps, to try and find where the problem lies. Sorry I don't have any better suggestions that I can think of
", "rawMarkdown": "If you've already checked all the above suggestions, perhaps a cell in your submission notebook is failing when run on the private test set, leading to a broken submission.csv file. Like perhaps your post processing (WBF or likewise) fails when it encounters a prediction with no box? I would try my best to simplify your submission process, removing extra steps, to try and find where the problem lies. Sorry I don't have any better suggestions that I can think of", "votes": null }, { "id": "956537", "postDate": "08/03/2020 15:34:07", "content": "@matthewmasters i get submission scoring error with 1024 size image. \nhttps://www.kaggle.com/pestipeti/pytorch-starter-fasterrcnn-inference\nm doing same as here. Could No object detected could be an issue. If yes how to handle that..
\n\nI suppose there are two problems\n1) Image size more than 1024, can we resolve it with just doing resizing 1024 ?\n2) No object detected ,what to do for this ?
", "rawMarkdown": "matthewmasters i get submission scoring error with 1024 size image. \nhttps://www.kaggle.com/pestipeti/pytorch-starter-fasterrcnn-inference\nm doing same as here. Could No object detected could be an issue. If yes how to handle that.. \n\nI suppose there are two problems\n1) Image size more than 1024, can we resolve it with just doing resizing 1024 ?\n2) No object detected ,what to do for this ?", "votes": null }, { "id": "956575", "postDate": "08/03/2020 15:55:43", "content": "@kalyanpichuka could u help on how u resolved submission scoring error with img size 1024. \nAs per posts i see there are two issues causing the issue\n1) Img size not consistent ,as long as we were submiting wih 512 ,we were resizing,now how can we handle if we use full img size\n2) how to handle blank predictions are they already handled by almost all inference kernels based on Alexes ?
", "rawMarkdown": "kalyanpichuka could u help on how u resolved submission scoring error with img size 1024. \nAs per posts i see there are two issues causing the issue\n1) Img size not consistent ,as long as we were submiting wih 512 ,we were resizing,now how can we handle if we use full img size\n2) how to handle blank predictions are they already handled by almost all inference kernels based on Alexes ?", "votes": null }, { "id": "956585", "postDate": "08/03/2020 16:00:46", "content": "@codename5281
\n1) could let me know how u resolved sub scoring error when submitting with 1024 img size ,how do we check for img size 1024 \n2) How we know which image is not 1024 so we rescale its prediction wrt to its original size ?\nLooking for a help thanks in advance\n?
Same issue for me, some times I got 'Submission scoring error', some times for 'submission CSV not found', I can also see the submission.csv file in my Output and the format is correct, I try commit 10 times but still can't fix it.
", "votes": null, "replies": [] }, { "id": 840804, "author_name": "kaushal2896", "author_url": "", "post_date": "05/10/2020 11:55:05", "content": "Exactly! If order is to be maintained, then how it's possible to predict on hidden test set whose image IDs are not given in sample_submission.csv (I've seen test.csv in other such competitions but it's not there in our case). Also in almost all public kernels (including the kernel of competition host), authors are taking reference of given sample submission csv file and doing this will only predict 10 images from test set.
\n\nI even tried to predict on images directly read from \"test\" directory but that's not working. Here is my inference kernel if anyone wants to take a look: https://www.kaggle.com/kaushal2896/wheat-detection-fasterrcnn-inference
\n\nAnd this is my today's submission history:\n
Here is another thread for this issue: https://www.kaggle.com/c/global-wheat-detection/discussion/148678#840750 \nbut this doesn't makes sense to me.
\n\nI've tried many possible approaches to create the submission file but all failed!
\n\nSUPER FRUSTRATING!
\n\nEDIT:
\n\nSee this kernel: https://www.kaggle.com/ateplyuk/gwd-starter-efficientdet-inference\nFrom that we can see that order doesn't matter.
\n\nThanks!
", "votes": null, "replies": [ { "id": 895365, "author_name": "kalyanpichuka", "author_url": "", "post_date": "06/21/2020 09:35:45", "content": "@kaushal2896 could you please let me know how did you solve the submission scoring error
", "votes": null, "replies": [] }, { "id": 904020, "author_name": "shuhaoqiu", "author_url": "", "post_date": "06/27/2020 09:29:29", "content": "hi there, have you solve the problem? I met the same problem too and I want to know how you solved it, thanks for your reply
", "votes": null, "replies": [] }, { "id": 904085, "author_name": "kaushal2896", "author_url": "", "post_date": "06/27/2020 10:52:34", "content": "@kalyanpichuka and @shuhaoqiu I think your script should correctly form the output when there are no objects are detected in the given image. This was missing in my case. Therefore I took the reference of top FasterRCNN inference public kernel (https://www.kaggle.com/pestipeti/pytorch-starter-fasterrcnn-inference) and now it's working. Otherwise the order is not important and also you can directly iterate through either the submission CSV or the test directory for reading the test data.
", "votes": null, "replies": [] }, { "id": 904815, "author_name": "shuhaoqiu", "author_url": "", "post_date": "06/28/2020 00:55:23", "content": "Thank you very much, it's exactly the problem!
", "votes": null, "replies": [] }, { "id": 907870, "author_name": "kalyanpichuka", "author_url": "", "post_date": "06/30/2020 07:57:18", "content": "Thanks @kaushal2896 for the response
", "votes": null, "replies": [] }, { "id": 930066, "author_name": "kiranrgupta26", "author_url": "", "post_date": "07/15/2020 07:05:22", "content": "Hello,\nShould the submission.csv contain one unique row per image_id with all predictionString in one line as given in this link (https://www.kaggle.com/pestipeti/pytorch-starter-fasterrcnn-inference),\nOR predictionString of format(1.0,0,0,50,50) on multiple lines for same image_id.\nI tried both, also handle the case where it returned no bounding box(replacing the predictionString=\"\")\nBut still getting Submission scoring error.\nCan anyone please help me.\nThank You.
", "votes": null, "replies": [] }, { "id": 930071, "author_name": "matthewmasters", "author_url": "", "post_date": "07/15/2020 07:09:30", "content": "There should be one row per imageid with all predictions concatenated in one PredictionString.
", "votes": null, "replies": [] }, { "id": 931127, "author_name": "kiranrgupta26", "author_url": "", "post_date": "07/16/2020 02:16:33", "content": "Hello,\nThank you for clarifying the doubt.\n
I have attached the image of my submission.csv file.\nIt contains 1 unique row for every image_id, but still getting Submission scoring error.
\n\nThank you.
", "votes": null, "replies": [] }, { "id": 931149, "author_name": "matthewmasters", "author_url": "", "post_date": "07/16/2020 02:49:11", "content": "If you've already checked all the above suggestions, perhaps a cell in your submission notebook is failing when run on the private test set, leading to a broken submission.csv file. Like perhaps your post processing (WBF or likewise) fails when it encounters a prediction with no box? I would try my best to simplify your submission process, removing extra steps, to try and find where the problem lies. Sorry I don't have any better suggestions that I can think of
", "votes": null, "replies": [] }, { "id": 956537, "author_name": "jaideepvalani", "author_url": "", "post_date": "08/03/2020 15:34:07", "content": "@matthewmasters i get submission scoring error with 1024 size image. \nhttps://www.kaggle.com/pestipeti/pytorch-starter-fasterrcnn-inference\nm doing same as here. Could No object detected could be an issue. If yes how to handle that..
\n\nI suppose there are two problems\n1) Image size more than 1024, can we resolve it with just doing resizing 1024 ?\n2) No object detected ,what to do for this ?
", "votes": null, "replies": [] } ] }, { "id": 841199, "author_name": "chittalpatel", "author_url": "", "post_date": "05/10/2020 17:04:54", "content": "I'm too beginner like you @codename5281 and would really like to know the clear and informative description of test data and submission procedures for this competition.
", "votes": null, "replies": [ { "id": 841228, "author_name": "codename5281", "author_url": "", "post_date": "05/10/2020 17:18:59", "content": "Data has been described quite in details in the competition preprint paper available at https://arxiv.org/ftp/arxiv/papers/2005/2005.02162.pdf and in published exploratory data analysis (EDA) notebooks.\nFor all the other details, let's stick with the findings of other challengers while waiting for any eventual precisions from the host ! :)
", "votes": null, "replies": [] } ] }, { "id": 841201, "author_name": "greatgamedota", "author_url": "", "post_date": "05/10/2020 17:05:38", "content": "I struggled with the format for awhile and I would like to share my findings:\n- Order doesn't matter, you can get the list of names from either sample_submission.csv or listing the test image directory\n- Confidence must be a float, doesn't matter or many digits of precision you give\n- X,Y,Width,Height must be integers\n- I'm not 100% sure but make sure Width and Height are always >0\n- For empty predictions you must predict and empty string '' NOT np.nan
Finally \"Submission CSV Not Found\" means either your submission isn't formatted correctly OR your kernal had an error while submitting.
", "votes": null, "replies": [ { "id": 841219, "author_name": "codename5281", "author_url": "", "post_date": "05/10/2020 17:16:40", "content": "Thanks for sharing !\nOn my side, except for the empty predictions case everything should be OK. And i don't see why the submitted kernel would throw more errors than the commited one.\nEDIT: i'm more and more convinced that the issue could come from managing empty predictions. Can't wait for refreshed submission credits to let you know !
", "votes": null, "replies": [] }, { "id": 909333, "author_name": "kseniar", "author_url": "", "post_date": "06/30/2020 14:13:10", "content": "Thank you! Your findings were very helpful!
", "votes": null, "replies": [] }, { "id": 909453, "author_name": "scherkhan", "author_url": "", "post_date": "06/30/2020 15:46:52", "content": "Thanks a lot for sharing!\nYou mentioned \"For empty predictions you must predict and empty string '' NOT np.nan\"\nDoes this mean that even if there are no predicted bounding boxes for image it should be listed in submission.csv anyway? Something like:\n|image_id|PredictionString|\n|---------|:---------------:|\n|ag632h2438||
", "votes": null, "replies": [] }, { "id": 909613, "author_name": "greatgamedota", "author_url": "", "post_date": "06/30/2020 17:25:00", "content": "yes else itll error out saying there isnt enough rows in the csv
", "votes": null, "replies": [] }, { "id": 912825, "author_name": "muhammad4hmed", "author_url": "", "post_date": "07/02/2020 18:48:12", "content": "Hi there, you said if there are no boxes, then we should leave it empty with empty string\nbut aren't there only 10 images to predict? and my prediction has all of them filled hence no empty boxes\nstill the error which questioner mentioned
", "votes": null, "replies": [] }, { "id": 912832, "author_name": "greatgamedota", "author_url": "", "post_date": "07/02/2020 18:55:25", "content": "those 10 images aren't the full test set, when you submit they are replaced with all the test images which is a lot more than 10
", "votes": null, "replies": [] }, { "id": 912846, "author_name": "muhammad4hmed", "author_url": "", "post_date": "07/02/2020 19:05:05", "content": "Can you please suggest a notebook which is making predictions on that hidden test data.
", "votes": null, "replies": [] }, { "id": 912851, "author_name": "matthewmasters", "author_url": "", "post_date": "07/02/2020 19:07:28", "content": "Pretty much any of the public inference notebooks will do. Some examples:\n- https://www.kaggle.com/pestipeti/pytorch-starter-fasterrcnn-inference\n- https://www.kaggle.com/shonenkov/inference-efficientdet\n- https://www.kaggle.com/trungthanhnguyen0502/gwd-inference\n- https://www.kaggle.com/orkatz2/yolov5-fake-or-real-single-model-l-b-0-753
", "votes": null, "replies": [] }, { "id": 912858, "author_name": "muhammad4hmed", "author_url": "", "post_date": "07/02/2020 19:10:55", "content": "Thank you
", "votes": null, "replies": [] } ] }, { "id": 842132, "author_name": "codename5281", "author_url": "", "post_date": "05/11/2020 08:09:43", "content": "An update on this topic !
\n\nFollowing suggestions from @greatgamedota i finally succeded (after 26 trials, how good is that ?!) to submit !\nSome clues :\n- starting from sample_submission.csv did not work for me, as if it did not contain the right list of files on which perform detection for submission. Starting from listing the files in /kaggle/input/global-wheat-detection/test/ did the trick.\n- adding code to catch exceptions in the case where the method i'm using did not detect heads did help.
\n\nHere you go ! Now i'm heading back to model training :)
", "votes": null, "replies": [ { "id": 956585, "author_name": "jaideepvalani", "author_url": "", "post_date": "08/03/2020 16:00:46", "content": "@codename5281
\n1) could let me know how u resolved sub scoring error when submitting with 1024 img size ,how do we check for img size 1024 \n2) How we know which image is not 1024 so we rescale its prediction wrt to its original size ?\nLooking for a help thanks in advance\n?
I'm still facing the same problem.
\n\nI tried to create a notebook that only take the submission file from the \"global-wheat-detection-first-submission\" kernel and save it as submission.csv file.\nI double checked that the output csv file of my kernel is identical to the original kernel output. \nI still get failure on submission. \nWhen I tried to fork the original kernel and submit it everything is working fine.
\n\nAny idea what i'm doing wrong?
", "votes": null, "replies": [] }, { "id": 895324, "author_name": "kalyanpichuka", "author_url": "", "post_date": "06/21/2020 09:02:58", "content": "how to solve the 'Submission scoring error ' on the hidden private dataset. I am still getting the error even everything seems to look fine when I see the result with the available test set
", "votes": null, "replies": [ { "id": 956575, "author_name": "jaideepvalani", "author_url": "", "post_date": "08/03/2020 15:55:43", "content": "@kalyanpichuka could u help on how u resolved submission scoring error with img size 1024. \nAs per posts i see there are two issues causing the issue\n1) Img size not consistent ,as long as we were submiting wih 512 ,we were resizing,now how can we handle if we use full img size\n2) how to handle blank predictions are they already handled by almost all inference kernels based on Alexes ?
", "votes": null, "replies": [] } ] } ], "raw_markdown_by_id": { "840730": "Dear challengers and hosts,\n\nThis is my first participation, and like some a number of other people I am also having trouble with my submissions, with either \"Submission scoring error\" or \"submission CSV not found\" status, despite the fact that my submission.csv seems correctly generated and formatted (see code output below). I checked the provided example notebooks and don't get where my errors could come from.\n\n@bendvd , could you provide supplementary information about the files that are modified between the coding competition test set and scoring test set, as well as the way the submission file is parsed ? Also, about the path in which the submission.csv file has to be generated in ?\n\n\nI am editing this message incrementally, but so far here are the tracks we went to :\n\n- We can read elsewhere in this forum that in the submission file the image id order may have to be similar to the one in the sample_submission file; how that can be true if the scoring test set is not the same as the coding test set ? EDIT : according to https://www.kaggle.com/c/global-wheat-detection/discussion/148678#834579 by @chittalpatel and @kaushal2896 it doesn't seem to be the case.\n\n- I also tried to submit notebooks getting the scoring test set images either by reading the sample_submission file or listing the test folder files as discussed here https://www.kaggle.com/c/global-wheat-detection/discussion/148678#840750 , without success.\n\n- For some contestants it would seem that this issue may be caused by test set image size issues (https://www.kaggle.com/c/global-wheat-detection/discussion/149949 by @greatgamedota ) but to the exception of really abnormal images i doubt it to be the main cause of trouble, as the method i am trying to set up should not be sensible to variations in image size. For the sake of verification i modified input image size and my notebook still worked properly.\n\n- Managing exceptions thrown by empty predictions could be the key... i'll let you know\n\n\nI am curious to receive feedback from experienced synchronous competition participants about this. \n\nThanks for your help and all the best !\nJ.\n\n\n\nRelated topics in which participants are having trouble too (so i'm not pointed out for not having taken time to read the forums haha) :\n- https://www.kaggle.com/c/global-wheat-detection/discussion/149847 by @rilllove \n- https://www.kaggle.com/c/global-wheat-detection/discussion/149809 by @spytensor \n- https://www.kaggle.com/c/global-wheat-detection/discussion/149630 by @rilllove \n- https://www.kaggle.com/c/global-wheat-detection/discussion/148678 by @chittalpatel \n- useful thread to try solving these errors by @bamps53 : https://www.kaggle.com/c/global-wheat-detection/discussion/148873", "840774": "Same issue for me, some times I got 'Submission scoring error', some times for 'submission CSV not found', I can also see the submission.csv file in my Output and the format is correct, I try commit 10 times but still can't fix it.", "840804": "Exactly! If order is to be maintained, then how it's possible to predict on hidden test set whose image IDs are not given in sample\\_submission.csv (I've seen test.csv in other such competitions but it's not there in our case). Also in almost all public kernels (including the kernel of competition host), authors are taking reference of given sample submission csv file and doing this will only predict 10 images from test set.\n\nI even tried to predict on images directly read from \"test\" directory but that's not working. Here is my inference kernel if anyone wants to take a look: https://www.kaggle.com/kaushal2896/wheat-detection-fasterrcnn-inference\n\nAnd this is my today's submission history:\n\n\nHere is another thread for this issue: https://www.kaggle.com/c/global-wheat-detection/discussion/148678#840750 \nbut this doesn't makes sense to me.\n\nI've tried many possible approaches to create the submission file but all failed!\n\nSUPER FRUSTRATING!\n\nEDIT:\n\nSee this kernel: https://www.kaggle.com/ateplyuk/gwd-starter-efficientdet-inference\nFrom that we can see that order doesn't matter.\n\nThanks!", "841199": "I'm too beginner like you @codename5281 and would really like to know the clear and informative description of test data and submission procedures for this competition.", "841201": "I struggled with the format for awhile and I would like to share my findings:\n- Order doesn't matter, you can get the list of names from either `sample_submission.csv` or listing the test image directory\n- Confidence must be a float, doesn't matter or many digits of precision you give\n- X,Y,Width,Height must be integers\n- I'm not 100% sure but make sure Width and Height are always >0\n- For empty predictions you must predict and empty string `''` NOT `np.nan`\n\nFinally \"Submission CSV Not Found\" means either your submission isn't formatted correctly OR your kernal had an error while submitting.", "841219": "Thanks for sharing !\nOn my side, except for the empty predictions case everything should be OK. And i don't see why the submitted kernel would throw more errors than the commited one.\nEDIT: i'm more and more convinced that the issue could come from managing empty predictions. Can't wait for refreshed submission credits to let you know !", "841228": "Data has been described quite in details in the competition preprint paper available at https://arxiv.org/ftp/arxiv/papers/2005/2005.02162.pdf and in published exploratory data analysis (EDA) notebooks.\nFor all the other details, let's stick with the findings of other challengers while waiting for any eventual precisions from the host ! :)", "842132": "An update on this topic !\n\nFollowing suggestions from @greatgamedota i finally succeded (after 26 trials, how good is that ?!) to submit !\nSome clues :\n- starting from sample_submission.csv did not work for me, as if it did not contain the right list of files on which perform detection for submission. Starting from listing the files in /kaggle/input/global-wheat-detection/test/ did the trick.\n- adding code to catch exceptions in the case where the method i'm using did not detect heads did help.\n\nHere you go ! Now i'm heading back to model training :)", "851517": "I'm still facing the same problem. \n\nI tried to create a notebook that only take the submission file from the \"global-wheat-detection-first-submission\" kernel and save it as submission.csv file.\nI double checked that the output csv file of my kernel is identical to the original kernel output. \nI still get failure on submission. \nWhen I tried to fork the original kernel and submit it everything is working fine.\n\nAny idea what i'm doing wrong?", "895324": "how to solve the 'Submission scoring error ' on the hidden private dataset. I am still getting the error even everything seems to look fine when I see the result with the available test set", "895365": "kaushal2896 could you please let me know how did you solve the submission scoring error", "904020": "hi there, have you solve the problem? I met the same problem too and I want to know how you solved it, thanks for your reply", "904085": "kalyanpichuka and @shuhaoqiu I think your script should correctly form the output when there are no objects are detected in the given image. This was missing in my case. Therefore I took the reference of top FasterRCNN inference public kernel (https://www.kaggle.com/pestipeti/pytorch-starter-fasterrcnn-inference) and now it's working. Otherwise the order is not important and also you can directly iterate through either the submission CSV or the test directory for reading the test data.", "904815": "Thank you very much, it's exactly the problem!", "907870": "Thanks @kaushal2896 for the response", "909333": "Thank you! Your findings were very helpful!", "909453": "Thanks a lot for sharing!\nYou mentioned \"For empty predictions you must predict and empty string '' NOT np.nan\"\nDoes this mean that even if there are no predicted bounding boxes for image it should be listed in submission.csv anyway? Something like:\n|image_id|PredictionString|\n|---------|:---------------:|\n|ag632h2438||", "909613": "yes else itll error out saying there isnt enough rows in the csv", "912825": "Hi there, you said if there are no boxes, then we should leave it empty with empty string\nbut aren't there only 10 images to predict? and my prediction has all of them filled hence no empty boxes\nstill the error which questioner mentioned", "912832": "those 10 images aren't the full test set, when you submit they are replaced with all the test images which is a lot more than 10", "912846": "Can you please suggest a notebook which is making predictions on that hidden test data.", "912851": "Pretty much any of the public inference notebooks will do. Some examples:\n- https://www.kaggle.com/pestipeti/pytorch-starter-fasterrcnn-inference\n- https://www.kaggle.com/shonenkov/inference-efficientdet\n- https://www.kaggle.com/trungthanhnguyen0502/gwd-inference\n- https://www.kaggle.com/orkatz2/yolov5-fake-or-real-single-model-l-b-0-753", "912858": "Thank you", "930066": "Hello,\nShould the submission.csv contain one unique row per image_id with all predictionString in one line as given in this link (https://www.kaggle.com/pestipeti/pytorch-starter-fasterrcnn-inference),\nOR predictionString of format(1.0,0,0,50,50) on multiple lines for same image_id.\nI tried both, also handle the case where it returned no bounding box(replacing the predictionString=\"\")\nBut still getting Submission scoring error.\nCan anyone please help me.\nThank You.", "930071": "There should be one row per imageid with all predictions concatenated in one PredictionString.", "931127": "Hello,\nThank you for clarifying the doubt.\n\n\nI have attached the image of my submission.csv file.\nIt contains 1 unique row for every image_id, but still getting Submission scoring error.\n\nThank you.", "931149": "If you've already checked all the above suggestions, perhaps a cell in your submission notebook is failing when run on the private test set, leading to a broken submission.csv file. Like perhaps your post processing (WBF or likewise) fails when it encounters a prediction with no box? I would try my best to simplify your submission process, removing extra steps, to try and find where the problem lies. Sorry I don't have any better suggestions that I can think of", "956537": "matthewmasters i get submission scoring error with 1024 size image. \nhttps://www.kaggle.com/pestipeti/pytorch-starter-fasterrcnn-inference\nm doing same as here. Could No object detected could be an issue. If yes how to handle that.. \n\nI suppose there are two problems\n1) Image size more than 1024, can we resolve it with just doing resizing 1024 ?\n2) No object detected ,what to do for this ?", "956575": "kalyanpichuka could u help on how u resolved submission scoring error with img size 1024. \nAs per posts i see there are two issues causing the issue\n1) Img size not consistent ,as long as we were submiting wih 512 ,we were resizing,now how can we handle if we use full img size\n2) how to handle blank predictions are they already handled by almost all inference kernels based on Alexes ?", "956585": "codename5281 \n1) could let me know how u resolved sub scoring error when submitting with 1024 img size ,how do we check for img size 1024 \n2) How we know which image is not 1024 so we rescale its prediction wrt to its original size ?\nLooking for a help thanks in advance\n?" }, "source": "meta" }