![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F1231059%2Fd0e863a47c42babf181085f7d9f2ad7f%2Fkaggle.jpg?generation=1593769076252761&alt=media\")
\nSource: https://www.kaggle.com/c/global-wheat-detection/discussion/163291#912847Hi everyone, I am new to kaggle. This is my first competition and I am confused with the current status of the competition.
\n\nSo the Kaggle announce that they are going to fix the private datasets. So the current leader-board seems invalid as the private datasets change. \nWith my new models, How I am going to understand if the model is performing well ?\nAmi I going to check the leaderboard or my local CV :C ?\nIt will be very helpful if any experienced kaggler give me some ideas.
", "messages": [ { "id": "912517", "postDate": "07/02/2020 14:36:58", "content": "Hi everyone, I am new to kaggle. This is my first competition and I am confused with the current status of the competition.
\n\nSo the Kaggle announce that they are going to fix the private datasets. So the current leader-board seems invalid as the private datasets change. \nWith my new models, How I am going to understand if the model is performing well ?\nAmi I going to check the leaderboard or my local CV :C ?\nIt will be very helpful if any experienced kaggler give me some ideas.
", "rawMarkdown": "Hi everyone, I am new to kaggle. This is my first competition and I am confused with the current status of the competition.\n\nSo the Kaggle announce that they are going to fix the private datasets. So the current leader-board seems invalid as the private datasets change. \nWith my new models, How I am going to understand if the model is performing well ?\nAmi I going to check the leaderboard or my local CV :C ?\nIt will be very helpful if any experienced kaggler give me some ideas.", "votes": null }, { "id": "912548", "postDate": "07/02/2020 15:01:50", "content": "Just to clarify: they WILL NOT be changing any of the test data that is used to calculate the current leaderboard. However, they WILL be updating the 38% of test data that is used in the final scoring.
\n\nWhat this means for you:\n - Your LB scores won't change.\n - Your LB score might be misleading. Since they are only correcting the final portion of the test dataset, this means the errors that are present in the remaining 62% will remain. Therefor, the score you obtain on the LB might appear better or worse than it actually is.
\n\nThe general takeaway can be summed up here: \n> It (the competition update) doesn't help us at all develop a better solution. But hopefully it will at least result in the better, more generalized models winning.
", "rawMarkdown": "Just to clarify: they WILL NOT be changing any of the test data that is used to calculate the current leaderboard. However, they WILL be updating the 38% of test data that is used in the final scoring.\n\nWhat this means for you:\n - Your LB scores won't change.\n - Your LB score might be misleading. Since they are only correcting the final portion of the test dataset, this means the errors that are present in the remaining 62% will remain. Therefor, the score you obtain on the LB might appear better or worse than it actually is.\n\nThe general takeaway can be summed up [here](https://www.kaggle.com/c/global-wheat-detection/discussion/163291#911173): \n> It (the competition update) doesn't help us at all develop a better solution. But hopefully it will at least result in the better, more generalized models winning.", "votes": null }, { "id": "912551", "postDate": "07/02/2020 15:04:09", "content": "I think CV with train dataset without bbox noise is most helpful for evaluating model because the test data for private LB should have no noisy bboxes.
", "rawMarkdown": "I think CV with train dataset without bbox noise is most helpful for evaluating model because the test data for private LB should have no noisy bboxes.", "votes": null }, { "id": "912562", "postDate": "07/02/2020 15:12:07", "content": "My god. This is so messed up :C
", "rawMarkdown": "My god. This is so messed up :C", "votes": null }, { "id": "912651", "postDate": "07/02/2020 16:18:59", "content": "@umepon0626 does this mean we should create our own bounding boxes?
", "rawMarkdown": "umepon0626 does this mean we should create our own bounding boxes?", "votes": null }, { "id": "912774", "postDate": "07/02/2020 18:01:12", "content": "Not sure why I got downvoted when I am asking a legitimate question. It has been a long time since I have done anything related to object detection so I am asking about a best practice here.
", "rawMarkdown": "Not sure why I got downvoted when I am asking a legitimate question. It has been a long time since I have done anything related to object detection so I am asking about a best practice here.", "votes": null }, { "id": "912779", "postDate": "07/02/2020 18:05:49", "content": "I think, instead of creating new bbox. Maybe we can focus on fixing the outlier bboxes.
", "rawMarkdown": "I think, instead of creating new bbox. Maybe we can focus on fixing the outlier bboxes.", "votes": null }, { "id": "912785", "postDate": "07/02/2020 18:09:26", "content": "thank you @ahmmadrashid - I honestly wasn't sure of the suggestion but that makes sense.
", "rawMarkdown": "thank you @ahmmadrashid - I honestly wasn't sure of the suggestion but that makes sense.", "votes": null }, { "id": "913098", "postDate": "07/03/2020 01:04:47", "content": "there are many discussion about noisy bboxes.(ex.https://www.kaggle.com/c/global-wheat-detection/discussion/159578)
\n\nThe official will not improve our train dataset and test dataset for Public LB.\nSo, we have to fix our train dataset manually in order to make good score in Private LB.
", "rawMarkdown": "there are many discussion about noisy bboxes.(ex.https://www.kaggle.com/c/global-wheat-detection/discussion/159578)\n\nThe official will not improve our train dataset and test dataset for Public LB.\nSo, we have to fix our train dataset manually in order to make good score in Private LB.", "votes": null }, { "id": "913527", "postDate": "07/03/2020 09:04:17", "content": "Yes and we are sorry for the inconvenience. It would have been easier to not do the mistake in first place and upload a cleaner dataset. There were serious technical concerns about changing the whole dataset during a competition. We believe our update restore the fairness of the competition while limiting the technical risk.
\n\nRegarding data cleaning, it is in my opinion always something to do in any data science project. Even if the dataset was more correct, \"right\" boxes are \"noisy\" in some way, so in order to generalize, the training dataset is not the only parameter to optimize. Stuff like domain-adversarial training, pseudo-labelling can also help !
", "rawMarkdown": "Yes and we are sorry for the inconvenience. It would have been easier to not do the mistake in first place and upload a cleaner dataset. There were serious technical concerns about changing the whole dataset during a competition. We believe our update restore the fairness of the competition while limiting the technical risk.\n\nRegarding data cleaning, it is in my opinion always something to do in any data science project. Even if the dataset was more correct, \"right\" boxes are \"noisy\" in some way, so in order to generalize, the training dataset is not the only parameter to optimize. Stuff like domain-adversarial training, pseudo-labelling can also help !", "votes": null }, { "id": "913582", "postDate": "07/03/2020 09:42:54", "content": "Hi @bendvd, thank you for updating the private test dataset. I just want to verify if the 38% test data actually have a tight bbox to the wheat heads? Will the bbox be anything closer to the below image with blue rectangle ? This will help us better understand the dataset.\n\nSource: https://www.kaggle.com/c/global-wheat-detection/discussion/163291#912847
Just to clarify: they WILL NOT be changing any of the test data that is used to calculate the current leaderboard. However, they WILL be updating the 38% of test data that is used in the final scoring.
\n\nWhat this means for you:\n - Your LB scores won't change.\n - Your LB score might be misleading. Since they are only correcting the final portion of the test dataset, this means the errors that are present in the remaining 62% will remain. Therefor, the score you obtain on the LB might appear better or worse than it actually is.
\n\nThe general takeaway can be summed up here: \n> It (the competition update) doesn't help us at all develop a better solution. But hopefully it will at least result in the better, more generalized models winning.
", "votes": null, "replies": [ { "id": 912562, "author_name": "ahmmadrashid", "author_url": "", "post_date": "07/02/2020 15:12:07", "content": "My god. This is so messed up :C
", "votes": null, "replies": [] }, { "id": 913527, "author_name": "bendvd", "author_url": "", "post_date": "07/03/2020 09:04:17", "content": "Yes and we are sorry for the inconvenience. It would have been easier to not do the mistake in first place and upload a cleaner dataset. There were serious technical concerns about changing the whole dataset during a competition. We believe our update restore the fairness of the competition while limiting the technical risk.
\n\nRegarding data cleaning, it is in my opinion always something to do in any data science project. Even if the dataset was more correct, \"right\" boxes are \"noisy\" in some way, so in order to generalize, the training dataset is not the only parameter to optimize. Stuff like domain-adversarial training, pseudo-labelling can also help !
", "votes": null, "replies": [] }, { "id": 913582, "author_name": "bendang", "author_url": "", "post_date": "07/03/2020 09:42:54", "content": "Hi @bendvd, thank you for updating the private test dataset. I just want to verify if the 38% test data actually have a tight bbox to the wheat heads? Will the bbox be anything closer to the below image with blue rectangle ? This will help us better understand the dataset.\n\nSource: https://www.kaggle.com/c/global-wheat-detection/discussion/163291#912847
I think CV with train dataset without bbox noise is most helpful for evaluating model because the test data for private LB should have no noisy bboxes.
", "votes": null, "replies": [ { "id": 912651, "author_name": "rdizzl3", "author_url": "", "post_date": "07/02/2020 16:18:59", "content": "@umepon0626 does this mean we should create our own bounding boxes?
", "votes": null, "replies": [] }, { "id": 912774, "author_name": "rdizzl3", "author_url": "", "post_date": "07/02/2020 18:01:12", "content": "Not sure why I got downvoted when I am asking a legitimate question. It has been a long time since I have done anything related to object detection so I am asking about a best practice here.
", "votes": null, "replies": [] }, { "id": 912779, "author_name": "ahmmadrashid", "author_url": "", "post_date": "07/02/2020 18:05:49", "content": "I think, instead of creating new bbox. Maybe we can focus on fixing the outlier bboxes.
", "votes": null, "replies": [] }, { "id": 912785, "author_name": "rdizzl3", "author_url": "", "post_date": "07/02/2020 18:09:26", "content": "thank you @ahmmadrashid - I honestly wasn't sure of the suggestion but that makes sense.
", "votes": null, "replies": [] }, { "id": 913098, "author_name": "umepon0626", "author_url": "", "post_date": "07/03/2020 01:04:47", "content": "there are many discussion about noisy bboxes.(ex.https://www.kaggle.com/c/global-wheat-detection/discussion/159578)
\n\nThe official will not improve our train dataset and test dataset for Public LB.\nSo, we have to fix our train dataset manually in order to make good score in Private LB.
", "votes": null, "replies": [] } ] } ], "raw_markdown_by_id": { "912517": "Hi everyone, I am new to kaggle. This is my first competition and I am confused with the current status of the competition.\n\nSo the Kaggle announce that they are going to fix the private datasets. So the current leader-board seems invalid as the private datasets change. \nWith my new models, How I am going to understand if the model is performing well ?\nAmi I going to check the leaderboard or my local CV :C ?\nIt will be very helpful if any experienced kaggler give me some ideas.", "912548": "Just to clarify: they WILL NOT be changing any of the test data that is used to calculate the current leaderboard. However, they WILL be updating the 38% of test data that is used in the final scoring.\n\nWhat this means for you:\n - Your LB scores won't change.\n - Your LB score might be misleading. Since they are only correcting the final portion of the test dataset, this means the errors that are present in the remaining 62% will remain. Therefor, the score you obtain on the LB might appear better or worse than it actually is.\n\nThe general takeaway can be summed up [here](https://www.kaggle.com/c/global-wheat-detection/discussion/163291#911173): \n> It (the competition update) doesn't help us at all develop a better solution. But hopefully it will at least result in the better, more generalized models winning.", "912551": "I think CV with train dataset without bbox noise is most helpful for evaluating model because the test data for private LB should have no noisy bboxes.", "912562": "My god. This is so messed up :C", "912651": "umepon0626 does this mean we should create our own bounding boxes?", "912774": "Not sure why I got downvoted when I am asking a legitimate question. It has been a long time since I have done anything related to object detection so I am asking about a best practice here.", "912779": "I think, instead of creating new bbox. Maybe we can focus on fixing the outlier bboxes.", "912785": "thank you @ahmmadrashid - I honestly wasn't sure of the suggestion but that makes sense.", "913098": "there are many discussion about noisy bboxes.(ex.https://www.kaggle.com/c/global-wheat-detection/discussion/159578)\n\nThe official will not improve our train dataset and test dataset for Public LB.\nSo, we have to fix our train dataset manually in order to make good score in Private LB.", "913527": "Yes and we are sorry for the inconvenience. It would have been easier to not do the mistake in first place and upload a cleaner dataset. There were serious technical concerns about changing the whole dataset during a competition. We believe our update restore the fairness of the competition while limiting the technical risk.\n\nRegarding data cleaning, it is in my opinion always something to do in any data science project. Even if the dataset was more correct, \"right\" boxes are \"noisy\" in some way, so in order to generalize, the training dataset is not the only parameter to optimize. Stuff like domain-adversarial training, pseudo-labelling can also help !", "913582": "Hi @bendvd, thank you for updating the private test dataset. I just want to verify if the 38% test data actually have a tight bbox to the wheat heads? Will the bbox be anything closer to the below image with **blue** rectangle ? This will help us better understand the dataset.\n\nSource: https://www.kaggle.com/c/global-wheat-detection/discussion/163291#912847" }, "source": "meta" }