This competition just finished, thank you Kaggle and the host. However, we know there are some problems with this competition, and they brought bad experience. \nFirstly, the dataset is badly labelled. the community reported this problem many days ago, but the host only agreed to fix the private dataset. I don't understand why they don't fix the training set as well. This measure makes the private test set has a different distribution from other data sets. The consequence is the huge shake-up we see now.\nSecondly, the license problem or more precisely, the YOLOv5 problem. Kaggle is an open-source community. I don't know why the host decided to ban GPL. This issue was reported about a month ago when YOLOv5-based solution had occupied the top of LB. The host admitted their choice of license has some problems, but they refused to change the rule, and the reason is \"cannot change the rule in the middle\". However, as the new rule explanation was added. The rule was actually changed in the middle of the competition. YOLOv5 seems to be the best solution to this game, banning this brought a large impact. This \"rule\" was actually added in the middle of the competition. My team focused on YOLOv5 before that, and our best YOLOv5-based solution which was submitted a month ago got 0.7133 with the private set. This submission is enough to get a 12th position on the private LB. We didn't use that cause we just want to play by the rule. I don't agree with the host's decision, but once we attended a contest, we need to obey its rule, right? However, many teams are not compliant with the rule are still on the private LB. If this situation keeps to the final LB, I will be really unhappy. :(
To prevent similar issues appear in the future contest, I have the following suggestions to Kaggle.\n1. Kaggle should check the rule in advance, the host must demonstrate they have the ability to enforce the rule.\nIt doesn't matter if the host wants to ban something or make some special regulations. But if they are not able to check everyone, the LB will become a paradox. If you cheat and get a high score, you are easily checked and disqualified, if you don't cheat you just lose. Kaggle should check the competition regulation in advance to prevent this situation. Most people here, or at least my team were not coming to Kaggle to play this kind of paradox games.
\n\nKaggle now is the best place for the data science community. I sincerely hope it will be better.
", "messages": [ { "id": 958756, "postDate": "2020-08-05T06:13:03.817Z", "content": "This competition just finished, thank you Kaggle and the host. However, we know there are some problems with this competition, and they brought bad experience. \nFirstly, the dataset is badly labelled. the community reported this problem many days ago, but the host only agreed to fix the private dataset. I don't understand why they don't fix the training set as well. This measure makes the private test set has a different distribution from other data sets. The consequence is the huge shake-up we see now.\nSecondly, the license problem or more precisely, the YOLOv5 problem. Kaggle is an open-source community. I don't know why the host decided to ban GPL. This issue was reported about a month ago when YOLOv5-based solution had occupied the top of LB. The host admitted their choice of license has some problems, but they refused to change the rule, and the reason is \"cannot change the rule in the middle\". However, as the new rule explanation was added. The rule was actually changed in the middle of the competition. YOLOv5 seems to be the best solution to this game, banning this brought a large impact. This \"rule\" was actually added in the middle of the competition. My team focused on YOLOv5 before that, and our best YOLOv5-based solution which was submitted a month ago got 0.7133 with the private set. This submission is enough to get a 12th position on the private LB. We didn't use that cause we just want to play by the rule. I don't agree with the host's decision, but once we attended a contest, we need to obey its rule, right? However, many teams are not compliant with the rule are still on the private LB. If this situation keeps to the final LB, I will be really unhappy. :(
To prevent similar issues appear in the future contest, I have the following suggestions to Kaggle.\n1. Kaggle should check the rule in advance, the host must demonstrate they have the ability to enforce the rule.\nIt doesn't matter if the host wants to ban something or make some special regulations. But if they are not able to check everyone, the LB will become a paradox. If you cheat and get a high score, you are easily checked and disqualified, if you don't cheat you just lose. Kaggle should check the competition regulation in advance to prevent this situation. Most people here, or at least my team were not coming to Kaggle to play this kind of paradox games.
\n\nKaggle now is the best place for the data science community. I sincerely hope it will be better.
", "rawMarkdown": "This competition just finished, thank you Kaggle and the host. However, we know there are some problems with this competition, and they brought bad experience. \n**Firstly, the dataset is badly labelled.** the community reported this problem many days ago, but the host only agreed to fix the private dataset. I don't understand why they don't fix the training set as well. This measure makes the private test set has a different distribution from other data sets. The consequence is the huge shake-up we see now.\n**Secondly, the license problem or more precisely, the YOLOv5 problem.** Kaggle is an open-source community. I don't know why the host decided to ban GPL. This issue was reported about a month ago when YOLOv5-based solution had occupied the top of LB. The host admitted their choice of license has some problems, but they refused to change the rule, and the reason is \"cannot change the rule in the middle\". However, as the new rule explanation was added. The rule was actually changed in the middle of the competition. YOLOv5 seems to be the best solution to this game, banning this brought a large impact. This \"rule\" was actually added in the middle of the competition. My team focused on YOLOv5 before that, and our best YOLOv5-based solution which was submitted a month ago got `0.7133` with the private set. This submission is enough to get a 12th position on the private LB. We didn't use that cause we just want to play by the rule. I don't agree with the host's decision, but once we attended a contest, we need to obey its rule, right? However, many teams are not compliant with the rule are still on the private LB. If this situation keeps to the final LB, I will be really unhappy. :(\n\nTo prevent similar issues appear in the future contest, I have the following suggestions to Kaggle.\n1. **Kaggle should check the rule in advance, the host must demonstrate they have the ability to enforce the rule.**\nIt doesn't matter if the host wants to ban something or make some special regulations. But if they are not able to check everyone, the LB will become a paradox. If you cheat and get a high score, you are easily checked and disqualified, if you don't cheat you just lose. Kaggle should check the competition regulation in advance to prevent this situation. Most people here, or at least my team were not coming to Kaggle to play this kind of paradox games.\n\n2. **host should have a plan to change the rule during the competition.**\nThe bugs in the dataset or the competition rule were very likely found for a long competition. There should be a rule in the rules saying under what circumstance and to what extent the host can modify the rule. e.g. The host can interact with the community to fix the problems in the dataset, metrics or regulations by the first half of the competition. Then everything will be frozen in the second half. \n\nKaggle now is the best place for the data science community. I sincerely hope it will be better. ", "votes": 18 }, { "id": 958796, "postDate": "2020-08-05T06:31:19.673Z", "content": "The issue you have mentioned is true. \nPersonally I was interested in this competition from the very beginning but the rules paradox soon made me demotivated to act in this competition.\nkaggle authority must look into this deeply.\nAlso, this competition was hosted by an educational institution. \nIt was quite unfortunate something like this from any institution of that calibre.\nBut the main point is clarification. \nClarification of rules from A-Z is the must so that no one can put any counter question like - \" what is that rule meant to be? \"
", "rawMarkdown": "The issue you have mentioned is true. \nPersonally I was interested in this competition from the very beginning but the rules paradox soon made me demotivated to act in this competition.\nkaggle authority must look into this deeply.\nAlso, this competition was hosted by an educational institution. \nIt was quite unfortunate something like this from any institution of that calibre.\nBut the main point is **clarification**. \nClarification of rules from A-Z is the must so that no one can put any counter question like - \" what is that rule meant to be? \" ", "votes": 5, "replies": [ { "id": 958839, "postDate": "2020-08-05T07:03:02.537Z", "content": "Yeah, most people here are not lawyers, the complex rules and licenses need to be clarified.
", "rawMarkdown": "Yeah, most people here are not lawyers, the complex rules and licenses need to be clarified.", "votes": 1 }, { "id": 958928, "postDate": "2020-08-05T08:14:37.653Z", "content": "kaggle should introduce a pop up alert notification system specifically when a rule of any particular competition got changed while it is running. Also, hosts should run a simulation inside their hut before launching any competition which I guess most of them don't do.
", "rawMarkdown": "kaggle should introduce a pop up alert notification system specifically when a rule of any particular competition got changed while it is running. Also, hosts should run a simulation inside their hut before launching any competition which I guess most of them don't do." } ] }, { "id": 958767, "postDate": "2020-08-05T06:15:17.763Z", "content": "I was also a bit disappointed with the activity of the organisers. Took a while to clarify yolov5 stuff, and they still haven't given a straight answer on how the leaderboard will be checked for compliance. 100% agree with your points also.
", "rawMarkdown": "I was also a bit disappointed with the activity of the organisers. Took a while to clarify yolov5 stuff, and they still haven't given a straight answer on how the leaderboard will be checked for compliance. 100% agree with your points also.", "votes": 3 }, { "id": 960673, "postDate": "2020-08-06T15:34:12.620Z", "content": "The main issue is the inactive attitude of the host and their ambiguous attitude towards this challenge. They are from well known educational institutes we expect at least some clear answers.
", "rawMarkdown": "The main issue is the inactive attitude of the host and their ambiguous attitude towards this challenge. They are from well known educational institutes we expect at least some clear answers. ", "votes": 1 }, { "id": 958989, "postDate": "2020-08-05T09:08:52.170Z", "content": "Sadly I have to agree here. Some competition specifics really reduced the motivation for many Wheat-Detector-Kagglers including myself. \nIn general, the interaction between the hosts and the Kagglers was not enough. See the gold-post here.
\n\n1) Even if the rules where always the same: It took **weeks **to get an answer from Kaggle/the hosts if/how the rules are enforced or even: What a non-compliant submission results in: removal of the team? Of the submission?? Not being able to winning the prize? Not getting medals? If this all is so easy and obvious to answer, why was there no answer for weeks ;)?\n2) The badly labeled dataset was also not dealt with properly: It took far to long until a decision was made and then only the unseen part of the private dataset was changed.
\n\nLets keep in mind: It's not about complaining though, but about improving. \nI would be thankful and glad if the Kaggle Team will improve this for future competitions.\nAnyway: this competition was a lot of fun! Thanks to all organizers and participants!
", "rawMarkdown": "Sadly I have to agree here. Some competition specifics really reduced the motivation for many Wheat-Detector-Kagglers including myself. \nIn general, the interaction between the hosts and the Kagglers was not enough. See the gold-post [here](https://www.kaggle.com/c/global-wheat-detection/discussion/159585).\n\n1) Even if the rules where always the same: It took **weeks **to get an answer from Kaggle/the hosts if/how the rules are enforced or even: What a non-compliant submission results in: removal of the team? Of the submission?? Not being able to winning the prize? Not getting medals? If this all is so easy and obvious to answer, why was there no answer for weeks ;)?\n2) The badly labeled dataset was also not dealt with properly: It took far to long until a decision was made and then only the unseen part of the private dataset was changed.\n\nLets keep in mind: It's not about complaining though, but about improving. \nI would be thankful and glad if the Kaggle Team will improve this for future competitions.\nAnyway: this competition was a lot of fun! Thanks to all organizers and participants!", "votes": 1 }, { "id": 958863, "postDate": "2020-08-05T07:27:17.280Z", "content": "I hear your frustration with the competition's issues and agree with most of your points. The communication on these issues was never super clear and left a lot of people unsure. But just to clarify one thing, the rules were always the same. The rules always stated that an MIT license was required. They just didn't explicitly state that licenses like GPLv3 aren't compatible which led to a lot of confusion.
", "rawMarkdown": "I hear your frustration with the competition's issues and agree with most of your points. The communication on these issues was never super clear and left a lot of people unsure. But just to clarify one thing, the rules were always the same. The rules always stated that an MIT license was required. They just didn't explicitly state that licenses like GPLv3 aren't compatible which led to a lot of confusion.", "votes": 2, "replies": [ { "id": 958896, "postDate": "2020-08-05T07:51:17.683Z", "content": "Yes, but I guess that the host would not be aware of this problem till the end if the community didn't report that issue. The rule text was never changed, but if the execution way of the rule is changed that would be the thing actually influence our behavior. Keeping the rules doctrinally doesn't give us fairness, that is why I suggest there should be \"a rule to change rules\".
", "rawMarkdown": "Yes, but I guess that the host would not be aware of this problem till the end if the community didn't report that issue. The rule text was never changed, but if the execution way of the rule is changed that would be the thing actually influence our behavior. Keeping the rules doctrinally doesn't give us fairness, that is why I suggest there should be \"a rule to change rules\".", "votes": 1 } ] }, { "id": 959753, "postDate": "2020-08-05T21:30:08.477Z", "content": "kaggle is company, GPL-based source code is impossible to use for commercial purpose.
", "rawMarkdown": "kaggle is company, GPL-based source code is impossible to use for commercial purpose.", "votes": -2 }, { "id": 958799, "postDate": "2020-08-05T06:32:57.127Z", "rawMarkdown": "", "isDeleted": true, "replies": [ { "id": 958824, "postDate": "2020-08-05T06:46:19.063Z", "content": "The main sponsor of this competition is a academic institution, so I think it would be better to keep all things open. But you are right, host paid $15,000 and they have the complete right to set any rules whatever they like. The important thing is, how the host enforce the rules. The requirement is the same for all candidates, not only the golden zone, or say, if I choose the YOLO submission I would be in the golden zone. I write this post is to urge the admin giving a clear and consistent policy to everyone.\nThank you for your comments.
", "rawMarkdown": "The main sponsor of this competition is a academic institution, so I think it would be better to keep all things open. But you are right, host paid $15,000 and they have the complete right to set any rules whatever they like. The important thing is, how the host enforce the rules. The requirement is the same for all candidates, not only the golden zone, or say, if I choose the YOLO submission I would be in the golden zone. I write this post is to urge the admin giving a clear and consistent policy to everyone.\nThank you for your comments.", "votes": 1 }, { "id": 959000, "postDate": "2020-08-05T09:26:18.977Z", "rawMarkdown": "", "isDeleted": true } ] } ], "comments": [ { "id": 958796, "author_name": "745H1N", "author_url": "", "post_date": "2020-08-05T06:31:19.673000", "content": "The issue you have mentioned is true. \nPersonally I was interested in this competition from the very beginning but the rules paradox soon made me demotivated to act in this competition.\nkaggle authority must look into this deeply.\nAlso, this competition was hosted by an educational institution. \nIt was quite unfortunate something like this from any institution of that calibre.\nBut the main point is clarification. \nClarification of rules from A-Z is the must so that no one can put any counter question like - \" what is that rule meant to be? \"
", "votes": 5, "replies": [ { "id": 958839, "author_name": "Qingyuan Wang", "author_url": "", "post_date": "2020-08-05T07:03:02.537000", "content": "Yeah, most people here are not lawyers, the complex rules and licenses need to be clarified.
", "votes": 1, "replies": [] }, { "id": 958928, "author_name": "745H1N", "author_url": "", "post_date": "2020-08-05T08:14:37.653000", "content": "kaggle should introduce a pop up alert notification system specifically when a rule of any particular competition got changed while it is running. Also, hosts should run a simulation inside their hut before launching any competition which I guess most of them don't do.
", "votes": 0, "replies": [] } ] }, { "id": 958767, "author_name": "Stanley Zheng", "author_url": "", "post_date": "2020-08-05T06:15:17.763000", "content": "I was also a bit disappointed with the activity of the organisers. Took a while to clarify yolov5 stuff, and they still haven't given a straight answer on how the leaderboard will be checked for compliance. 100% agree with your points also.
", "votes": 3, "replies": [] }, { "id": 960673, "author_name": "Rifat Rashid", "author_url": "", "post_date": "2020-08-06T15:34:12.620000", "content": "The main issue is the inactive attitude of the host and their ambiguous attitude towards this challenge. They are from well known educational institutes we expect at least some clear answers.
", "votes": 1, "replies": [] }, { "id": 958989, "author_name": "from coffee import *", "author_url": "", "post_date": "2020-08-05T09:08:52.170000", "content": "Sadly I have to agree here. Some competition specifics really reduced the motivation for many Wheat-Detector-Kagglers including myself. \nIn general, the interaction between the hosts and the Kagglers was not enough. See the gold-post here.
\n\n1) Even if the rules where always the same: It took **weeks **to get an answer from Kaggle/the hosts if/how the rules are enforced or even: What a non-compliant submission results in: removal of the team? Of the submission?? Not being able to winning the prize? Not getting medals? If this all is so easy and obvious to answer, why was there no answer for weeks ;)?\n2) The badly labeled dataset was also not dealt with properly: It took far to long until a decision was made and then only the unseen part of the private dataset was changed.
\n\nLets keep in mind: It's not about complaining though, but about improving. \nI would be thankful and glad if the Kaggle Team will improve this for future competitions.\nAnyway: this competition was a lot of fun! Thanks to all organizers and participants!
", "votes": 1, "replies": [] }, { "id": 958863, "author_name": "Matt", "author_url": "", "post_date": "2020-08-05T07:27:17.280000", "content": "I hear your frustration with the competition's issues and agree with most of your points. The communication on these issues was never super clear and left a lot of people unsure. But just to clarify one thing, the rules were always the same. The rules always stated that an MIT license was required. They just didn't explicitly state that licenses like GPLv3 aren't compatible which led to a lot of confusion.
", "votes": 2, "replies": [ { "id": 958896, "author_name": "Qingyuan Wang", "author_url": "", "post_date": "2020-08-05T07:51:17.683000", "content": "Yes, but I guess that the host would not be aware of this problem till the end if the community didn't report that issue. The rule text was never changed, but if the execution way of the rule is changed that would be the thing actually influence our behavior. Keeping the rules doctrinally doesn't give us fairness, that is why I suggest there should be \"a rule to change rules\".
", "votes": 1, "replies": [] } ] }, { "id": 959753, "author_name": "Ji Wong Park", "author_url": "", "post_date": "2020-08-05T21:30:08.477000", "content": "kaggle is company, GPL-based source code is impossible to use for commercial purpose.
", "votes": -2, "replies": [] }, { "id": 958799, "author_name": "", "author_url": "", "post_date": "2020-08-05T06:32:57.127000", "content": "", "votes": 0, "replies": [ { "id": 958824, "author_name": "Qingyuan Wang", "author_url": "", "post_date": "2020-08-05T06:46:19.063000", "content": "The main sponsor of this competition is a academic institution, so I think it would be better to keep all things open. But you are right, host paid $15,000 and they have the complete right to set any rules whatever they like. The important thing is, how the host enforce the rules. The requirement is the same for all candidates, not only the golden zone, or say, if I choose the YOLO submission I would be in the golden zone. I write this post is to urge the admin giving a clear and consistent policy to everyone.\nThank you for your comments.
", "votes": 1, "replies": [] }, { "id": 959000, "author_name": "", "author_url": "", "post_date": "2020-08-05T09:26:18.977000", "content": "", "votes": 0, "replies": [] } ] } ], "raw_markdown_by_id": { "958756": "This competition just finished, thank you Kaggle and the host. However, we know there are some problems with this competition, and they brought bad experience. \n**Firstly, the dataset is badly labelled.** the community reported this problem many days ago, but the host only agreed to fix the private dataset. I don't understand why they don't fix the training set as well. This measure makes the private test set has a different distribution from other data sets. The consequence is the huge shake-up we see now.\n**Secondly, the license problem or more precisely, the YOLOv5 problem.** Kaggle is an open-source community. I don't know why the host decided to ban GPL. This issue was reported about a month ago when YOLOv5-based solution had occupied the top of LB. The host admitted their choice of license has some problems, but they refused to change the rule, and the reason is \"cannot change the rule in the middle\". However, as the new rule explanation was added. The rule was actually changed in the middle of the competition. YOLOv5 seems to be the best solution to this game, banning this brought a large impact. This \"rule\" was actually added in the middle of the competition. My team focused on YOLOv5 before that, and our best YOLOv5-based solution which was submitted a month ago got `0.7133` with the private set. This submission is enough to get a 12th position on the private LB. We didn't use that cause we just want to play by the rule. I don't agree with the host's decision, but once we attended a contest, we need to obey its rule, right? However, many teams are not compliant with the rule are still on the private LB. If this situation keeps to the final LB, I will be really unhappy. :(\n\nTo prevent similar issues appear in the future contest, I have the following suggestions to Kaggle.\n1. **Kaggle should check the rule in advance, the host must demonstrate they have the ability to enforce the rule.**\nIt doesn't matter if the host wants to ban something or make some special regulations. But if they are not able to check everyone, the LB will become a paradox. If you cheat and get a high score, you are easily checked and disqualified, if you don't cheat you just lose. Kaggle should check the competition regulation in advance to prevent this situation. Most people here, or at least my team were not coming to Kaggle to play this kind of paradox games.\n\n2. **host should have a plan to change the rule during the competition.**\nThe bugs in the dataset or the competition rule were very likely found for a long competition. There should be a rule in the rules saying under what circumstance and to what extent the host can modify the rule. e.g. The host can interact with the community to fix the problems in the dataset, metrics or regulations by the first half of the competition. Then everything will be frozen in the second half. \n\nKaggle now is the best place for the data science community. I sincerely hope it will be better. ", "958796": "The issue you have mentioned is true. \nPersonally I was interested in this competition from the very beginning but the rules paradox soon made me demotivated to act in this competition.\nkaggle authority must look into this deeply.\nAlso, this competition was hosted by an educational institution. \nIt was quite unfortunate something like this from any institution of that calibre.\nBut the main point is **clarification**. \nClarification of rules from A-Z is the must so that no one can put any counter question like - \" what is that rule meant to be? \" ", "958767": "I was also a bit disappointed with the activity of the organisers. Took a while to clarify yolov5 stuff, and they still haven't given a straight answer on how the leaderboard will be checked for compliance. 100% agree with your points also.", "960673": "The main issue is the inactive attitude of the host and their ambiguous attitude towards this challenge. They are from well known educational institutes we expect at least some clear answers. ", "958989": "Sadly I have to agree here. Some competition specifics really reduced the motivation for many Wheat-Detector-Kagglers including myself. \nIn general, the interaction between the hosts and the Kagglers was not enough. See the gold-post [here](https://www.kaggle.com/c/global-wheat-detection/discussion/159585).\n\n1) Even if the rules where always the same: It took **weeks **to get an answer from Kaggle/the hosts if/how the rules are enforced or even: What a non-compliant submission results in: removal of the team? Of the submission?? Not being able to winning the prize? Not getting medals? If this all is so easy and obvious to answer, why was there no answer for weeks ;)?\n2) The badly labeled dataset was also not dealt with properly: It took far to long until a decision was made and then only the unseen part of the private dataset was changed.\n\nLets keep in mind: It's not about complaining though, but about improving. \nI would be thankful and glad if the Kaggle Team will improve this for future competitions.\nAnyway: this competition was a lot of fun! Thanks to all organizers and participants!", "958863": "I hear your frustration with the competition's issues and agree with most of your points. The communication on these issues was never super clear and left a lot of people unsure. But just to clarify one thing, the rules were always the same. The rules always stated that an MIT license was required. They just didn't explicitly state that licenses like GPLv3 aren't compatible which led to a lot of confusion.", "959753": "kaggle is company, GPL-based source code is impossible to use for commercial purpose.", "958799": "" } }