This topic really applies to all competitions.
\n\nAfter the whole debacle (that's still on going) with DFDC competition I would like to make a small topic about competition hosts' lack of interaction while a competition is on going. I haven't been competing on Kaggle for very long and so far most of the competitions I've joined the host is around the first week or so answering questions us competitors have. However very quickly the hosts (and Kaggle) go silent and stop all interaction. This wouldn't be that big of a deal if the competition was perfect with no problems, however the amazing minds that compete here are quick to find pitfalls and errors with these competitions.
\n\nFor example this competition: people found that there are a lot of mislabeled bboxes that are either to large or too small (as was discussed here). As well as an apparent error with test images not being 1024px by 1024px (discussed here). These are problems found pretty early on in the competition yet the hosts and Kaggle have yet to fix the data. There was even a post about it nearly a month ago that went completely unanswered here.
\n\nThis competition is not an outlier: In DFDC many questions about external data went unanswered which caused some of the top teams to be disqualified or penalized. In Baidu Autonomous Driving Competition an error was found with the evaluation metric which took months to fix and even then competitors were still confused on exactly how it worked.
\n\nI bring this up to mostly force someone to listen and actually fix the issues with this competition. But because of DFDC, I hope that Kaggle will be open to hear about other issues plaguing Kaggle and hopefully think about ways to make Kaggle better.
", "messages": [ { "id": "891095", "postDate": "06/17/2020 22:26:54", "content": "This topic really applies to all competitions.
\n\nAfter the whole debacle (that's still on going) with DFDC competition I would like to make a small topic about competition hosts' lack of interaction while a competition is on going. I haven't been competing on Kaggle for very long and so far most of the competitions I've joined the host is around the first week or so answering questions us competitors have. However very quickly the hosts (and Kaggle) go silent and stop all interaction. This wouldn't be that big of a deal if the competition was perfect with no problems, however the amazing minds that compete here are quick to find pitfalls and errors with these competitions.
\n\nFor example this competition: people found that there are a lot of mislabeled bboxes that are either to large or too small (as was discussed here). As well as an apparent error with test images not being 1024px by 1024px (discussed here). These are problems found pretty early on in the competition yet the hosts and Kaggle have yet to fix the data. There was even a post about it nearly a month ago that went completely unanswered here.
\n\nThis competition is not an outlier: In DFDC many questions about external data went unanswered which caused some of the top teams to be disqualified or penalized. In Baidu Autonomous Driving Competition an error was found with the evaluation metric which took months to fix and even then competitors were still confused on exactly how it worked.
\n\nI bring this up to mostly force someone to listen and actually fix the issues with this competition. But because of DFDC, I hope that Kaggle will be open to hear about other issues plaguing Kaggle and hopefully think about ways to make Kaggle better.
", "rawMarkdown": "This topic really applies to all competitions.\n\nAfter the whole debacle (that's still on going) with DFDC competition I would like to make a small topic about competition hosts' lack of interaction while a competition is on going. I haven't been competing on Kaggle for very long and so far most of the competitions I've joined the host is around the first week or so answering questions us competitors have. However very quickly the hosts (and Kaggle) go silent and stop all interaction. This wouldn't be that big of a deal if the competition was perfect with no problems, however the amazing minds that compete here are quick to find pitfalls and errors with these competitions.\n\nFor example this competition: people found that there are a lot of mislabeled bboxes that are either to large or too small (as was discussed [here](https://www.kaggle.com/c/global-wheat-detection/discussion/149032)). As well as an apparent error with test images not being 1024px by 1024px (discussed [here](https://www.kaggle.com/c/global-wheat-detection/discussion/149949)). These are problems found pretty early on in the competition yet the hosts and Kaggle have yet to fix the data. There was even a post about it nearly a month ago that went completely unanswered [here](https://www.kaggle.com/c/global-wheat-detection/discussion/154685).\n\nThis competition is not an outlier: In DFDC many questions about external data went unanswered which caused some of the top teams to be disqualified or penalized. In Baidu Autonomous Driving Competition an error was found with the evaluation metric which took months to fix and even then competitors were still confused on exactly how it worked.\n\nI bring this up to mostly force someone to listen and actually fix the issues with this competition. But because of DFDC, I hope that Kaggle will be open to hear about other issues plaguing Kaggle and hopefully think about ways to make Kaggle better.", "votes": null }, { "id": "892160", "postDate": "06/18/2020 17:30:01", "content": "Really hoping they would modify the evaluation metric.
\n\n\n\n\nFrom the competition description: \"These images are used to estimate the density and size of wheat heads in different varieties”. I think that perfectly accurate bounding boxes won’t affect the ability to do so. What’s most important is that the wheat heads are more or less localised with high precision and recall.
\n
https://www.kaggle.com/c/global-wheat-detection/discussion/150781
", "rawMarkdown": "Really hoping they would modify the evaluation metric.\n> From the competition description: \"These images are used to estimate the density and size of wheat heads in different varieties”. I think that perfectly accurate bounding boxes won’t affect the ability to do so. What’s most important is that the wheat heads are more or less localised with high precision and recall.\n\n https://www.kaggle.com/c/global-wheat-detection/discussion/150781", "votes": null }, { "id": "892175", "postDate": "06/18/2020 17:42:57", "content": "Maybe prediciting accurate bounding boxes won't really matter in the estimation of density (We can predict the right number of bboxes with bad dimensions and still have the same estimation of the density.) However, predicting accurate bounding boxes will certainly improve the size estimation of wheat heads.
", "rawMarkdown": "Maybe prediciting accurate bounding boxes won't really matter in the estimation of density (We can predict the right number of bboxes with bad dimensions and still have the same estimation of the density.) However, predicting accurate bounding boxes will certainly improve the size estimation of wheat heads.", "votes": null }, { "id": "892322", "postDate": "06/18/2020 19:59:42", "content": "I agree that they should have changed the metric.. or at least put more consideration into it before launching the competition. It's unfortunate that it won't happen as I suspect the winner of the competition won't actually be the most generalized solution but rather the most overfit.
", "rawMarkdown": "I agree that they should have changed the metric.. or at least put more consideration into it before launching the competition. It's unfortunate that it won't happen as I suspect the winner of the competition won't actually be the most generalized solution but rather the most overfit.", "votes": null }, { "id": "895984", "postDate": "06/21/2020 18:22:49", "content": "@ffares I thought about this too and tend to agree in general. I also wonder how much the distance of the camera to each individual wheat head also affects the size estimate.
", "rawMarkdown": "ffares I thought about this too and tend to agree in general. I also wonder how much the distance of the camera to each individual wheat head also affects the size estimate.", "votes": null }, { "id": "896014", "postDate": "06/21/2020 18:46:54", "content": "@alexandersoare For sure! Predicting accurate bboxes is essential but not enough. We should consider the distance to the canopy, the field view, the focal length and the Ground Sampling Distance. I agree with you that the distance of the camera to each individual wheat head affects the size estimate. In the end we have to make some approximations because the wheat head in the center of the image doesn't have the same angle with the one in the border of the image. However, in average we can have pretty good estimation if we assume that wheat heads in the same field would have approximately the same size.
", "rawMarkdown": "alexandersoare For sure! Predicting accurate bboxes is essential but not enough. We should consider the distance to the canopy, the field view, the focal length and the Ground Sampling Distance. I agree with you that the distance of the camera to each individual wheat head affects the size estimate. In the end we have to make some approximations because the wheat head in the center of the image doesn't have the same angle with the one in the border of the image. However, in average we can have pretty good estimation if we assume that wheat heads in the same field would have approximately the same size.", "votes": null }, { "id": "896934", "postDate": "06/22/2020 14:27:58", "content": "Dear all,
\n\nI really understand your point. The problem comes from an export from our bbox tool that have merge two bboxes into one (if you look at all the huge bboxes, the two wheat heads are in the corner) . A corrected version of the training dataset and test set is ready but there are several concerns from the technical team that it could cause problem.
\n\nI agree that we failed as organizers to provide the best experience for competitors, but we are currently trying our best to correct the situation.
\n\nBest,
\n\nEtienne
", "rawMarkdown": "Dear all,\n\nI really understand your point. The problem comes from an export from our bbox tool that have merge two bboxes into one (if you look at all the huge bboxes, the two wheat heads are in the corner) . A corrected version of the training dataset and test set is ready but there are several concerns from the technical team that it could cause problem. \n\nI agree that we failed as organizers to provide the best experience for competitors, but we are currently trying our best to correct the situation. \n\nBest,\n\nEtienne", "votes": null }, { "id": "896940", "postDate": "06/22/2020 14:32:04", "content": "Choosing an evaluation metric is critical and always difficult. Your feedbacks matter a lot to us, as it could help us to shape a future competition. However, it is difficult to change it after launch.
\n\nLocalization of wheat head is important not only for counting but for estimation of other parameters such as size, spikelets counting, health status etc...
\n\nWhatever the metric is, the risk of having an overfit algorithm on the test set is always a limit of such competition. The bias between the train and test set is however quite huge, so a novel approach could win in my opinion.
", "rawMarkdown": "Choosing an evaluation metric is critical and always difficult. Your feedbacks matter a lot to us, as it could help us to shape a future competition. However, it is difficult to change it after launch.\n\nLocalization of wheat head is important not only for counting but for estimation of other parameters such as size, spikelets counting, health status etc...\n\n Whatever the metric is, the risk of having an overfit algorithm on the test set is always a limit of such competition. The bias between the train and test set is however quite huge, so a novel approach could win in my opinion.", "votes": null }, { "id": "897032", "postDate": "06/22/2020 15:28:06", "content": "Thank you very much for the responses! I understand it can be difficult to be active in these times and I appreciate the effort your putting in. I hope you can fix things to the best of your ability.
", "rawMarkdown": "Thank you very much for the responses! I understand it can be difficult to be active in these times and I appreciate the effort your putting in. I hope you can fix things to the best of your ability.", "votes": null }, { "id": "897292", "postDate": "06/22/2020 18:56:53", "content": "Hello there - Wanted to jump in here and let you know that we do take your concerns seriously. As Etienne mentions in this string we are looking at alternatives. There is a lot of complexity that goes into setting up a competition. Thank you for your patience and understanding.
", "rawMarkdown": "Hello there - Wanted to jump in here and let you know that we do take your concerns seriously. As Etienne mentions in this string we are looking at alternatives. There is a lot of complexity that goes into setting up a competition. Thank you for your patience and understanding.", "votes": null }, { "id": "898644", "postDate": "06/23/2020 16:43:03", "content": "@bendvd It would be great to know that a data update is coming or not , as after two NLP competitions many others like me are looking forward to start with this competition and it's really crucial to know whether data will update or not . And if it will then by what time and will it mean extension of dealine, all previous will stay or will become null void? I would really appreciate if you take out the time to answer thank you
", "rawMarkdown": "bendvd It would be great to know that a data update is coming or not , as after two NLP competitions many others like me are looking forward to start with this competition and it's really crucial to know whether data will update or not . And if it will then by what time and will it mean extension of dealine, all previous will stay or will become null void? I would really appreciate if you take out the time to answer thank you", "votes": null }, { "id": "899251", "postDate": "06/24/2020 05:32:56", "content": "@tanulsingh077 thank you very much for your question. We are currently investigating what would be the best solution to solve the bug.
", "rawMarkdown": "tanulsingh077 thank you very much for your question. We are currently investigating what would be the best solution to solve the bug.", "votes": null }, { "id": "910838", "postDate": "07/01/2020 11:53:58", "content": "Waiting solution to solve the bug.
", "rawMarkdown": "Waiting solution to solve the bug.", "votes": null }, { "id": "910878", "postDate": "07/01/2020 12:26:52", "content": "Thanks for these details. Any update on the date of \"data release\"? 🤔\nAlso, would this impact the duration of the competition as was the case for other competitions?
", "rawMarkdown": "Thanks for these details. Any update on the date of \"data release\"? 🤔\nAlso, would this impact the duration of the competition as was the case for other competitions?", "votes": null }, { "id": "910951", "postDate": "07/01/2020 13:18:20", "content": "@bendvd Could you clarify, pls. When train and test datasets will fixed?\nOr this data won't change until end competition?\nit is very important for us.
", "rawMarkdown": "bendvd Could you clarify, pls. When train and test datasets will fixed?\nOr this data won't change until end competition?\nit is very important for us.", "votes": null } ], "comments": [ { "id": 892160, "author_name": "bendang", "author_url": "", "post_date": "06/18/2020 17:30:01", "content": "Really hoping they would modify the evaluation metric.
\n\n\n\n\nFrom the competition description: \"These images are used to estimate the density and size of wheat heads in different varieties”. I think that perfectly accurate bounding boxes won’t affect the ability to do so. What’s most important is that the wheat heads are more or less localised with high precision and recall.
\n
https://www.kaggle.com/c/global-wheat-detection/discussion/150781
", "votes": null, "replies": [ { "id": 892175, "author_name": "ffares", "author_url": "", "post_date": "06/18/2020 17:42:57", "content": "Maybe prediciting accurate bounding boxes won't really matter in the estimation of density (We can predict the right number of bboxes with bad dimensions and still have the same estimation of the density.) However, predicting accurate bounding boxes will certainly improve the size estimation of wheat heads.
", "votes": null, "replies": [] }, { "id": 892322, "author_name": "matthewmasters", "author_url": "", "post_date": "06/18/2020 19:59:42", "content": "I agree that they should have changed the metric.. or at least put more consideration into it before launching the competition. It's unfortunate that it won't happen as I suspect the winner of the competition won't actually be the most generalized solution but rather the most overfit.
", "votes": null, "replies": [] }, { "id": 895984, "author_name": "alexandersoare", "author_url": "", "post_date": "06/21/2020 18:22:49", "content": "@ffares I thought about this too and tend to agree in general. I also wonder how much the distance of the camera to each individual wheat head also affects the size estimate.
", "votes": null, "replies": [] }, { "id": 896014, "author_name": "ffares", "author_url": "", "post_date": "06/21/2020 18:46:54", "content": "@alexandersoare For sure! Predicting accurate bboxes is essential but not enough. We should consider the distance to the canopy, the field view, the focal length and the Ground Sampling Distance. I agree with you that the distance of the camera to each individual wheat head affects the size estimate. In the end we have to make some approximations because the wheat head in the center of the image doesn't have the same angle with the one in the border of the image. However, in average we can have pretty good estimation if we assume that wheat heads in the same field would have approximately the same size.
", "votes": null, "replies": [] }, { "id": 896940, "author_name": "bendvd", "author_url": "", "post_date": "06/22/2020 14:32:04", "content": "Choosing an evaluation metric is critical and always difficult. Your feedbacks matter a lot to us, as it could help us to shape a future competition. However, it is difficult to change it after launch.
\n\nLocalization of wheat head is important not only for counting but for estimation of other parameters such as size, spikelets counting, health status etc...
\n\nWhatever the metric is, the risk of having an overfit algorithm on the test set is always a limit of such competition. The bias between the train and test set is however quite huge, so a novel approach could win in my opinion.
", "votes": null, "replies": [] } ] }, { "id": 896934, "author_name": "bendvd", "author_url": "", "post_date": "06/22/2020 14:27:58", "content": "Dear all,
\n\nI really understand your point. The problem comes from an export from our bbox tool that have merge two bboxes into one (if you look at all the huge bboxes, the two wheat heads are in the corner) . A corrected version of the training dataset and test set is ready but there are several concerns from the technical team that it could cause problem.
\n\nI agree that we failed as organizers to provide the best experience for competitors, but we are currently trying our best to correct the situation.
\n\nBest,
\n\nEtienne
", "votes": null, "replies": [ { "id": 897032, "author_name": "greatgamedota", "author_url": "", "post_date": "06/22/2020 15:28:06", "content": "Thank you very much for the responses! I understand it can be difficult to be active in these times and I appreciate the effort your putting in. I hope you can fix things to the best of your ability.
", "votes": null, "replies": [] }, { "id": 898644, "author_name": "tanulsingh077", "author_url": "", "post_date": "06/23/2020 16:43:03", "content": "@bendvd It would be great to know that a data update is coming or not , as after two NLP competitions many others like me are looking forward to start with this competition and it's really crucial to know whether data will update or not . And if it will then by what time and will it mean extension of dealine, all previous will stay or will become null void? I would really appreciate if you take out the time to answer thank you
", "votes": null, "replies": [] }, { "id": 899251, "author_name": "bendvd", "author_url": "", "post_date": "06/24/2020 05:32:56", "content": "@tanulsingh077 thank you very much for your question. We are currently investigating what would be the best solution to solve the bug.
", "votes": null, "replies": [] }, { "id": 910878, "author_name": "yassinealouini", "author_url": "", "post_date": "07/01/2020 12:26:52", "content": "Thanks for these details. Any update on the date of \"data release\"? 🤔\nAlso, would this impact the duration of the competition as was the case for other competitions?
", "votes": null, "replies": [] }, { "id": 910951, "author_name": "alexeyfil", "author_url": "", "post_date": "07/01/2020 13:18:20", "content": "@bendvd Could you clarify, pls. When train and test datasets will fixed?\nOr this data won't change until end competition?\nit is very important for us.
", "votes": null, "replies": [] } ] }, { "id": 897292, "author_name": "maggiemd", "author_url": "", "post_date": "06/22/2020 18:56:53", "content": "Hello there - Wanted to jump in here and let you know that we do take your concerns seriously. As Etienne mentions in this string we are looking at alternatives. There is a lot of complexity that goes into setting up a competition. Thank you for your patience and understanding.
", "votes": null, "replies": [] }, { "id": 910838, "author_name": "bixiaopeng", "author_url": "", "post_date": "07/01/2020 11:53:58", "content": "Waiting solution to solve the bug.
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "891095": "This topic really applies to all competitions.\n\nAfter the whole debacle (that's still on going) with DFDC competition I would like to make a small topic about competition hosts' lack of interaction while a competition is on going. I haven't been competing on Kaggle for very long and so far most of the competitions I've joined the host is around the first week or so answering questions us competitors have. However very quickly the hosts (and Kaggle) go silent and stop all interaction. This wouldn't be that big of a deal if the competition was perfect with no problems, however the amazing minds that compete here are quick to find pitfalls and errors with these competitions.\n\nFor example this competition: people found that there are a lot of mislabeled bboxes that are either to large or too small (as was discussed [here](https://www.kaggle.com/c/global-wheat-detection/discussion/149032)). As well as an apparent error with test images not being 1024px by 1024px (discussed [here](https://www.kaggle.com/c/global-wheat-detection/discussion/149949)). These are problems found pretty early on in the competition yet the hosts and Kaggle have yet to fix the data. There was even a post about it nearly a month ago that went completely unanswered [here](https://www.kaggle.com/c/global-wheat-detection/discussion/154685).\n\nThis competition is not an outlier: In DFDC many questions about external data went unanswered which caused some of the top teams to be disqualified or penalized. In Baidu Autonomous Driving Competition an error was found with the evaluation metric which took months to fix and even then competitors were still confused on exactly how it worked.\n\nI bring this up to mostly force someone to listen and actually fix the issues with this competition. But because of DFDC, I hope that Kaggle will be open to hear about other issues plaguing Kaggle and hopefully think about ways to make Kaggle better.", "892160": "Really hoping they would modify the evaluation metric.\n> From the competition description: \"These images are used to estimate the density and size of wheat heads in different varieties”. I think that perfectly accurate bounding boxes won’t affect the ability to do so. What’s most important is that the wheat heads are more or less localised with high precision and recall.\n\n https://www.kaggle.com/c/global-wheat-detection/discussion/150781", "892175": "Maybe prediciting accurate bounding boxes won't really matter in the estimation of density (We can predict the right number of bboxes with bad dimensions and still have the same estimation of the density.) However, predicting accurate bounding boxes will certainly improve the size estimation of wheat heads.", "892322": "I agree that they should have changed the metric.. or at least put more consideration into it before launching the competition. It's unfortunate that it won't happen as I suspect the winner of the competition won't actually be the most generalized solution but rather the most overfit.", "895984": "ffares I thought about this too and tend to agree in general. I also wonder how much the distance of the camera to each individual wheat head also affects the size estimate.", "896014": "alexandersoare For sure! Predicting accurate bboxes is essential but not enough. We should consider the distance to the canopy, the field view, the focal length and the Ground Sampling Distance. I agree with you that the distance of the camera to each individual wheat head affects the size estimate. In the end we have to make some approximations because the wheat head in the center of the image doesn't have the same angle with the one in the border of the image. However, in average we can have pretty good estimation if we assume that wheat heads in the same field would have approximately the same size.", "896934": "Dear all,\n\nI really understand your point. The problem comes from an export from our bbox tool that have merge two bboxes into one (if you look at all the huge bboxes, the two wheat heads are in the corner) . A corrected version of the training dataset and test set is ready but there are several concerns from the technical team that it could cause problem. \n\nI agree that we failed as organizers to provide the best experience for competitors, but we are currently trying our best to correct the situation. \n\nBest,\n\nEtienne", "896940": "Choosing an evaluation metric is critical and always difficult. Your feedbacks matter a lot to us, as it could help us to shape a future competition. However, it is difficult to change it after launch.\n\nLocalization of wheat head is important not only for counting but for estimation of other parameters such as size, spikelets counting, health status etc...\n\n Whatever the metric is, the risk of having an overfit algorithm on the test set is always a limit of such competition. The bias between the train and test set is however quite huge, so a novel approach could win in my opinion.", "897032": "Thank you very much for the responses! I understand it can be difficult to be active in these times and I appreciate the effort your putting in. I hope you can fix things to the best of your ability.", "897292": "Hello there - Wanted to jump in here and let you know that we do take your concerns seriously. As Etienne mentions in this string we are looking at alternatives. There is a lot of complexity that goes into setting up a competition. Thank you for your patience and understanding.", "898644": "bendvd It would be great to know that a data update is coming or not , as after two NLP competitions many others like me are looking forward to start with this competition and it's really crucial to know whether data will update or not . And if it will then by what time and will it mean extension of dealine, all previous will stay or will become null void? I would really appreciate if you take out the time to answer thank you", "899251": "tanulsingh077 thank you very much for your question. We are currently investigating what would be the best solution to solve the bug.", "910838": "Waiting solution to solve the bug.", "910878": "Thanks for these details. Any update on the date of \"data release\"? 🤔\nAlso, would this impact the duration of the competition as was the case for other competitions?", "910951": "bendvd Could you clarify, pls. When train and test datasets will fixed?\nOr this data won't change until end competition?\nit is very important for us." }, "source": "meta" }