The organizers made it clear that they will check \"winning\" submissions for licence compliance, e.g.: probably the first 10 or 20, but It would be way too much work to check a few hundred more.
\nWhile in a perfect world we could expect everyone to select only compliant final submissions (YOLOv5 free), unfortunately I think we all know this is not going to happen. The urge to reach nice shiny medals is too hard to resist for some.
\nI feel like in the end we will have many many YOLOv5 based submissions around positions 20-500 or so. I think it would be quite disappointing for those who put in a lot of effort, using compliant code but still end up quite far behind on the leaderboard because of hundreds of YOLOv5 based submissions.
\nWould it be possible to simply programatically scan the input folders of all solutions and invalidate those which use YOLOv5 source code? Of course If you try hard you could avoid being detected, but it could eliminate 95-99% percent of non-compliant solutions, and only people trying hard to willingly cheat would survive this cleaning. It may even require less work than manually checking tens of winning solutions, so Kaggle could save some effort too.
\nI think it would make the competition much more motivating, knowing that our honest efforts will probably have results.
\nPlease consider this solution,. thanks.
\nUdpate: I understand that the organizers did not comment on this issue during the competition, as it could have helped cheaters to avoid detection. However, now that submissions are over, I think it would be nice to react to a topic with ~ 70 upvotes. Thanks!
", "messages": [ { "id": 936502, "postDate": "2020-07-20T09:18:28.503Z", "content": "The organizers made it clear that they will check \"winning\" submissions for licence compliance, e.g.: probably the first 10 or 20, but It would be way too much work to check a few hundred more.
\nWhile in a perfect world we could expect everyone to select only compliant final submissions (YOLOv5 free), unfortunately I think we all know this is not going to happen. The urge to reach nice shiny medals is too hard to resist for some.
\nI feel like in the end we will have many many YOLOv5 based submissions around positions 20-500 or so. I think it would be quite disappointing for those who put in a lot of effort, using compliant code but still end up quite far behind on the leaderboard because of hundreds of YOLOv5 based submissions.
\nWould it be possible to simply programatically scan the input folders of all solutions and invalidate those which use YOLOv5 source code? Of course If you try hard you could avoid being detected, but it could eliminate 95-99% percent of non-compliant solutions, and only people trying hard to willingly cheat would survive this cleaning. It may even require less work than manually checking tens of winning solutions, so Kaggle could save some effort too.
\nI think it would make the competition much more motivating, knowing that our honest efforts will probably have results.
\nPlease consider this solution,. thanks.
\nUdpate: I understand that the organizers did not comment on this issue during the competition, as it could have helped cheaters to avoid detection. However, now that submissions are over, I think it would be nice to react to a topic with ~ 70 upvotes. Thanks!
", "rawMarkdown": "The organizers made it clear that they will check \"winning\" submissions for licence compliance, e.g.: probably the first 10 or 20, but It would be way too much work to check a few hundred more. \n\nWhile in a perfect world we could expect everyone to select only compliant final submissions (YOLOv5 free), unfortunately I think we all know this is not going to happen. The urge to reach nice shiny medals is too hard to resist for some.\n\nI feel like in the end we will have many many YOLOv5 based submissions around positions 20-500 or so. I think it would be quite disappointing for those who put in a lot of effort, using compliant code but still end up quite far behind on the leaderboard because of hundreds of YOLOv5 based submissions.\n\nWould it be possible to simply programatically scan the input folders of all solutions and invalidate those which use YOLOv5 source code? Of course If you try hard you could avoid being detected, but it could eliminate 95-99% percent of non-compliant solutions, and only people trying hard to willingly cheat would survive this cleaning. It may even require less work than manually checking tens of winning solutions, so Kaggle could save some effort too.\n\nI think it would make the competition much more motivating, knowing that our honest efforts will probably have results.\n\nPlease consider this solution,. thanks.\n\n---\nUdpate: I understand that the organizers did not comment on this issue during the competition, as it could have helped cheaters to avoid detection. However, now that submissions are over, I think it would be nice to react to a topic with ~ 70 upvotes. Thanks!", "votes": 72 }, { "id": 936609, "postDate": "2020-07-20T10:59:36.983Z", "content": "Kaggle should make everything (including your nice suggestion) to detect as much as possible of non-compliant users. It is not about the prize, nor the gold medals, it's about supporting those who deserve to be on the top, those who followed the rules.
", "rawMarkdown": "Kaggle should make everything (including your nice suggestion) to detect as much as possible of non-compliant users. It is not about the prize, nor the gold medals, it's about supporting those who deserve to be on the top, those who followed the rules. ", "votes": 19 }, { "id": 936562, "postDate": "2020-07-20T10:15:07.723Z", "content": "I cannot agree with you more.
", "rawMarkdown": "I cannot agree with you more.", "votes": 8 }, { "id": 943122, "postDate": "2020-07-24T06:46:35.340Z", "content": "Maybe , Kaggle team has to finetune CodeBERT on YoloV5 source code, and make classification if the submitted source/input really similar to the originanl YoloV5 repo , or not ?
\n\nHow about this @riblidezso ? 😁
", "rawMarkdown": "Maybe , Kaggle team has to finetune CodeBERT on YoloV5 source code, and make classification if the submitted source/input really similar to the originanl YoloV5 repo , or not ?\n\nHow about this @riblidezso ? 😁 ", "votes": 7 }, { "id": 937209, "postDate": "2020-07-20T20:13:12.437Z", "content": "I agree completely with your viewpoint. At this point, we just have to put our trust into the Kaggle compliance team that they will do their best. Even if they are using such an automated system, I doubt they would want to come out and say that they are. Those who try to game the system and submit using YoloV5 will just try to use this information to their advantage and avoid detection by changing the inference code.
", "rawMarkdown": "I agree completely with your viewpoint. At this point, we just have to put our trust into the Kaggle compliance team that they will do their best. Even if they are using such an automated system, I doubt they would want to come out and say that they are. Those who try to game the system and submit using YoloV5 will just try to use this information to their advantage and avoid detection by changing the inference code.", "votes": 5 }, { "id": 937874, "postDate": "2020-07-21T08:05:24.903Z", "content": "This would trully be a nice idea for a next competition. Automatically detect/classify the frameworks and licenses used. Likely already done by someone on this planet ;-)
", "rawMarkdown": "This would trully be a nice idea for a next competition. Automatically detect/classify the frameworks and licenses used. Likely already done by someone on this planet ;-)", "votes": 3 }, { "id": 973510, "postDate": "2020-08-17T10:39:43.377Z", "content": "@philculliton could you comment? Is there an automatic mechanism for removal, or are you going from the top line by line on the leaderboard? Thanks!
", "rawMarkdown": "@philculliton could you comment? Is there an automatic mechanism for removal, or are you going from the top line by line on the leaderboard? Thanks!", "votes": 2, "replies": [ { "id": 973606, "postDate": "2020-08-17T11:52:21.063Z", "content": "@riblidezso We should first ask are you going to remove Yolov5 submissions at all? Because not in a single discussion thread anyone has mentioned this.
", "rawMarkdown": "@riblidezso We should first ask are you going to remove Yolov5 submissions at all? Because not in a single discussion thread anyone has mentioned this.", "replies": [ { "id": 973615, "postDate": "2020-08-17T12:00:46.623Z", "content": "They did say they will check \"winning submissions\" so the question is how far down they will go on the leaderboard. Maybe they are going down to the last medal solution manually, that's why it takes so long, but I guess anyway you are right, any communication about this issue would be appreciated :)
", "rawMarkdown": "They did say they will check \"winning submissions\" so the question is how far down they will go on the leaderboard. Maybe they are going down to the last medal solution manually, that's why it takes so long, but I guess anyway you are right, any communication about this issue would be appreciated :)" }, { "id": 973760, "postDate": "2020-08-17T13:55:44.630Z", "content": "@riblidezso yes I'm talking about removing YoloV5 for entire medal zone only. Removing only for winners won't be fair at all.
", "rawMarkdown": "@riblidezso yes I'm talking about removing YoloV5 for entire medal zone only. Removing only for winners won't be fair at all." } ] } ] }, { "id": 948724, "postDate": "2020-07-28T07:03:39.220Z", "content": "I'm pretty confident in Kaggle's skills and determination in keeping the competitions fair. I believe that there is already a system in place to detect suspicious activities in general (multiple accounts, etc.) As for non-compliant solutions, it will become very obvious to which team is using Yolov5-based solutions when they open-source their code and the organizer is trying to replicate their results. They even banned a Grandmaster for cheating regarding a kernel competition few months back.
", "rawMarkdown": "I'm pretty confident in Kaggle's skills and determination in keeping the competitions fair. I believe that there is already a system in place to detect suspicious activities in general (multiple accounts, etc.) As for non-compliant solutions, it will become very obvious to which team is using Yolov5-based solutions when they open-source their code and the organizer is trying to replicate their results. They even banned a Grandmaster for cheating regarding a kernel competition few months back.", "votes": 1, "replies": [ { "id": 948869, "postDate": "2020-07-28T09:34:31.910Z", "content": "I made this post because I think the previous communications in this competition were not very reassuring. Especially when they said that only \"winning\" solutions need to be compliant, which incentives people to give a shot to Yolov5 for a high LB position which still goes under the radar. \nI wanted to make it clear that only checking winning/open sourced solutions will result in a competition where many honest competitors are demotivated and disappointed. You know, most of us have no chance to win, but some positive feedback is important in the learning process. Let's hope the best.
", "rawMarkdown": "I made this post because I think the previous communications in this competition were not very reassuring. Especially when they said that only \"winning\" solutions need to be compliant, which incentives people to give a shot to Yolov5 for a high LB position which still goes under the radar. \nI wanted to make it clear that only checking winning/open sourced solutions will result in a competition where many honest competitors are demotivated and disappointed. You know, most of us have no chance to win, but some positive feedback is important in the learning process. Let's hope the best.", "votes": 5 } ] }, { "id": 945601, "postDate": "2020-07-26T02:14:04.087Z", "content": "I agree, but this would have to be extremely carefully done. I'm using the darknet implementation of YOLOV4 (public domain) for my submission and I'm sure many others are too.
", "rawMarkdown": "I agree, but this would have to be extremely carefully done. I'm using the darknet implementation of YOLOV4 (public domain) for my submission and I'm sure many others are too. ", "votes": 1, "replies": [ { "id": 948545, "postDate": "2020-07-28T03:58:19.233Z", "content": "@stanleyjzheng What was your baseline score with YOLOv4? I achieved 0.6626 and then with WBF I got 0.6815.
", "rawMarkdown": "@stanleyjzheng What was your baseline score with YOLOv4? I achieved 0.6626 and then with WBF I got 0.6815. ", "votes": 1 }, { "id": 949330, "postDate": "2020-07-28T15:06:23.207Z", "rawMarkdown": "", "isDeleted": true }, { "id": 949375, "postDate": "2020-07-28T15:42:44.413Z", "content": "Yes I trained the model with image resolution 704x704 and by following AlexeyAB Repo of training custom dataset.
", "rawMarkdown": "Yes I trained the model with image resolution 704x704 and by following AlexeyAB Repo of training custom dataset." } ] }, { "id": 944922, "postDate": "2020-07-25T13:07:09.417Z", "content": "I totally agree!
", "rawMarkdown": "I totally agree!", "votes": 1 }, { "id": 944861, "postDate": "2020-07-25T11:53:21.740Z", "content": "I am new to kaggle and I worked very hard for a good prediction. And somehow I managed to drop 400 positions in a span of 2-3 days. Now I know why :(
", "rawMarkdown": "I am new to kaggle and I worked very hard for a good prediction. And somehow I managed to drop 400 positions in a span of 2-3 days. Now I know why :(", "votes": 1, "replies": [ { "id": 947095, "postDate": "2020-07-27T04:34:51.430Z", "content": "welcome to Kaggle, wait until the last days, people will drop like flies :)
", "rawMarkdown": "welcome to Kaggle, wait until the last days, people will drop like flies :)", "votes": 1 }, { "id": 947272, "postDate": "2020-07-27T06:54:33.143Z", "content": "I hope so @alincijov
", "rawMarkdown": "I hope so @alincijov " } ] }, { "id": 944800, "postDate": "2020-07-25T11:07:11.597Z", "content": "I agree.
", "rawMarkdown": "I agree.", "votes": 1 }, { "id": 943273, "postDate": "2020-07-24T08:49:56.947Z", "content": "I totally agree. I spent many hours working on a compliant solution using other models, and probably there are 10's or 100's of participants above me using YOLO v5. This shoud be a fair competition and I think all medal positions should be checked
", "rawMarkdown": "I totally agree. I spent many hours working on a compliant solution using other models, and probably there are 10's or 100's of participants above me using YOLO v5. This shoud be a fair competition and I think all medal positions should be checked", "votes": 1 }, { "id": 939148, "postDate": "2020-07-22T04:08:43.790Z", "content": "100% agree kaggle admins should take some steps
", "rawMarkdown": "100% agree kaggle admins should take some steps", "votes": 1 }, { "id": 937969, "postDate": "2020-07-21T09:21:37.573Z", "content": "agree with request. If someone want to try yolov5 to expand knowledge, thats ok, but this should not be marked as final submission
", "rawMarkdown": "agree with request. If someone want to try yolov5 to expand knowledge, thats ok, but this should not be marked as final submission", "votes": 1 }, { "id": 948278, "postDate": "2020-07-27T19:20:07.650Z", "content": "Personally I believe that the biggest issue with yolov5 is the possibility it was used as a generator of bounding boxes for some of the test cases. I saw some other post insinuating it and I believe that this explains the great advantage of this implementation.
\n\nAlso the organizers mentioned that they changed the private test set in the sense that they corrected it... What if they just recalculated the bounding boxes for the \"yolov5 inferenced cases\" and now this big advantage is simply lost in the private leaderboard. I don't say that this is a solution,I don't even have a solid proof for this speculation. It is just an opinion...
", "rawMarkdown": "Personally I believe that the biggest issue with yolov5 is the possibility it was used as a generator of bounding boxes for some of the test cases. I saw some other post insinuating it and I believe that this explains the great advantage of this implementation. \n\nAlso the organizers mentioned that they changed the private test set in the sense that they corrected it... What if they just recalculated the bounding boxes for the \"yolov5 inferenced cases\" and now this big advantage is simply lost in the private leaderboard. I don't say that this is a solution,I don't even have a solid proof for this speculation. It is just an opinion...", "votes": 2, "replies": [ { "id": 948284, "postDate": "2020-07-27T19:25:19.257Z", "content": "I don't think that alone was the reason YoloV5 excelled. They only did the 'semi-supervised' labeling on one of the sources from the test set (Tokyo_2 which makes up less than 10%) and it was YoloV3 (which has actually shown to have pretty poor performance) not V5.
Thank you @matthewmasters for the clarification. There must be some correlation though. Or let me rephrase it a correlation on labelling (Tokyo_2) with the output of yolov5 would explain a lot :). Don't get me wrong it is a great piece of software! But the difference with the competition is huge and with now parameter tweaking.
\n\nIn my opinion, the biggest issue of automatic annotation using a software, is the ratio of the bounding boxes. This has to do with the anchor boxes and the detection settings. Maybe the test set annotations are more \"yolov5\" compatible. Again this is a speculation :)
", "rawMarkdown": "Thank you @matthewmasters for the clarification. There must be some correlation though. Or let me rephrase it a correlation on labelling (Tokyo_2) with the output of yolov5 would explain a lot :). Don't get me wrong it is a great piece of software! But the difference with the competition is huge and with now parameter tweaking. \n\nIn my opinion, the biggest issue of automatic annotation using a software, is the ratio of the bounding boxes. This has to do with the anchor boxes and the detection settings. Maybe the test set annotations are more \"yolov5\" compatible. Again this is a speculation :) ", "votes": 2 } ] }, { "id": 947096, "postDate": "2020-07-27T04:36:51.283Z", "content": "All this AI/ML and Kaggle has no tech in place to detect when rules are being broken ?\nAlso look at the submission of those unknown novice accounts ;) fishy business going on
", "rawMarkdown": "All this AI/ML and Kaggle has no tech in place to detect when rules are being broken ?\nAlso look at the submission of those unknown novice accounts ;) fishy business going on", "votes": 2 }, { "id": 936943, "postDate": "2020-07-20T16:06:23.593Z", "content": "What exactly is the problem with yolov5? Sorry I just joined this competition?
", "rawMarkdown": "What exactly is the problem with yolov5? Sorry I just joined this competition?", "votes": 2, "replies": [ { "id": 936959, "postDate": "2020-07-20T16:16:45.027Z", "content": "The competition rules do not allow it because of its licence.
", "rawMarkdown": "The competition rules do not allow it because of its licence." }, { "id": 948754, "postDate": "2020-07-28T07:36:06.217Z", "content": "license and regulations derp derp :(
", "rawMarkdown": "license and regulations derp derp :(" } ] }, { "id": 948269, "postDate": "2020-07-27T19:11:31.870Z", "content": "Yes, I agree!
", "rawMarkdown": "Yes, I agree!" }, { "id": 948464, "postDate": "2020-07-28T01:21:32.987Z", "content": "The reason for this post is obviously jealousy
", "rawMarkdown": "The reason for this post is obviously jealousy", "votes": -21 }, { "id": 954634, "postDate": "2020-08-02T00:09:06.750Z", "content": "How to remove the submitted one? I don't find it
", "rawMarkdown": "How to remove the submitted one? I don't find it", "replies": [ { "id": 954728, "postDate": "2020-08-02T03:35:08.610Z", "content": "You can't remove it and don't need to remove it.Just don't select it as final submission.
", "rawMarkdown": "You can't remove it and don't need to remove it.Just don't select it as final submission.", "votes": 3 } ] }, { "id": 939238, "postDate": "2020-07-22T05:43:51.910Z", "content": "Agree, now that hosts banned yolov5, they should have some methods to enforce this rule. Otherwise it will introduce new unfairness.
", "rawMarkdown": "Agree, now that hosts banned yolov5, they should have some methods to enforce this rule. Otherwise it will introduce new unfairness." }, { "id": 937258, "postDate": "2020-07-20T21:30:38.837Z", "content": "Without a good automatically removing, it's like a prisoner's dilemma. 🙂
", "rawMarkdown": "Without a good automatically removing, it's like a prisoner's dilemma. 🙂 ", "replies": [ { "id": 946285, "postDate": "2020-07-26T13:46:11.813Z", "content": "As, have stop working with YOLOv5. I dont even know what is our position in leaderboard
", "rawMarkdown": "As, have stop working with YOLOv5. I dont even know what is our position in leaderboard", "votes": 1 }, { "id": 946719, "postDate": "2020-07-26T19:06:16.797Z", "content": "Maybe you'll find my thread (Leaderboard of Compliant Submissions) helpful.
", "rawMarkdown": "Maybe you'll find my thread ([Leaderboard of Compliant Submissions](https://www.kaggle.com/c/global-wheat-detection/discussion/166793)) helpful." } ] }, { "id": 937016, "postDate": "2020-07-20T17:03:43.480Z", "content": "Completely agreed.. Instead of living in a delusion of if's and but's, better to drop all YOLO submissions..
\n\nOne approach could be- Download the .py version of all notebooks and as in yolov5, everyone is cloning a Github repo related to YOLO version model training. And using regular expression find all those solutions that posses the mentioned condition and drop them.
\n\nWhat you guys think about the above approach..
", "rawMarkdown": "Completely agreed.. Instead of living in a delusion of if's and but's, better to drop all YOLO submissions..\n\nOne approach could be- Download the .py version of all notebooks and as in yolov5, everyone is cloning a Github repo related to YOLO version model training. And using regular expression find all those solutions that posses the mentioned condition and drop them. \n\nWhat you guys think about the above approach..", "replies": [ { "id": 943148, "postDate": "2020-07-24T07:01:28.120Z", "content": "@vin1234 Yes agree. Not only check the top solutions for YoloV5 where there is price money involved..but check all selected final sumissions on YoloV5 to have a fair distribution of the leaderboard.
\n\nI'am not sure if YoloV5 submissions should be dropped if they are not selected for the final leaderboard. If someone trully wants to experiment with YoloV5 but does not use it for its final submissions .. then I guess that should be fine.
", "rawMarkdown": "@vin1234 Yes agree. Not only check the top solutions for YoloV5 where there is price money involved..but check all selected final sumissions on YoloV5 to have a fair distribution of the leaderboard.\n\nI'am not sure if YoloV5 submissions should be dropped if they are not selected for the final leaderboard. If someone trully wants to experiment with YoloV5 but does not use it for its final submissions .. then I guess that should be fine." }, { "id": 945603, "postDate": "2020-07-26T02:15:23.660Z", "content": "I think it's important to make a distinction between the versions of YOLO. YOLOV5 was made by a different person and licenced GPL3. The darknet implementation of YOLOV4 is public domain, and therefore allowed.
", "rawMarkdown": "I think it's important to make a distinction between the versions of YOLO. YOLOV5 was made by a different person and licenced GPL3. The darknet implementation of YOLOV4 is public domain, and therefore allowed. ", "votes": 1, "replies": [ { "id": 946219, "postDate": "2020-07-26T12:50:05.907Z", "content": "Don't worry, YOLOv5 has unique source code, which is super easy to tell apart from other YOLO versions, there is no way compliant YOLO(v3/v4) based submissions will be invalidated.
", "rawMarkdown": "Don't worry, YOLOv5 has unique source code, which is super easy to tell apart from other YOLO versions, there is no way compliant YOLO(v3/v4) based submissions will be invalidated.", "votes": 1 } ] } ] }, { "id": 939865, "postDate": "2020-07-22T14:34:30.750Z", "rawMarkdown": "", "isDeleted": true, "replies": [ { "id": 939866, "postDate": "2020-07-22T14:37:32.927Z", "content": "The rules differ competition from competition. This one required an MIT license and therefor cannot use anything under GPLv3.
", "rawMarkdown": "The rules differ competition from competition. This one required an MIT license and therefor cannot use anything under GPLv3.", "votes": 1 }, { "id": 940558, "postDate": "2020-07-23T02:46:22.103Z", "rawMarkdown": "", "isDeleted": true } ] }, { "id": 937677, "postDate": "2020-07-21T05:54:01.947Z", "rawMarkdown": "", "votes": -3, "isDeleted": true }, { "id": 937333, "postDate": "2020-07-20T23:58:42.273Z", "rawMarkdown": "", "votes": -2, "isDeleted": true } ], "comments": [ { "id": 936609, "author_name": "Fares FOURATI", "author_url": "", "post_date": "2020-07-20T10:59:36.983000", "content": "Kaggle should make everything (including your nice suggestion) to detect as much as possible of non-compliant users. It is not about the prize, nor the gold medals, it's about supporting those who deserve to be on the top, those who followed the rules.
", "votes": 19, "replies": [] }, { "id": 936562, "author_name": "nvnn", "author_url": "", "post_date": "2020-07-20T10:15:07.723000", "content": "I cannot agree with you more.
", "votes": 8, "replies": [] }, { "id": 943122, "author_name": "Neuron Engineer", "author_url": "", "post_date": "2020-07-24T06:46:35.340000", "content": "Maybe , Kaggle team has to finetune CodeBERT on YoloV5 source code, and make classification if the submitted source/input really similar to the originanl YoloV5 repo , or not ?
\n\nHow about this @riblidezso ? 😁
", "votes": 7, "replies": [] }, { "id": 937209, "author_name": "Matt", "author_url": "", "post_date": "2020-07-20T20:13:12.437000", "content": "I agree completely with your viewpoint. At this point, we just have to put our trust into the Kaggle compliance team that they will do their best. Even if they are using such an automated system, I doubt they would want to come out and say that they are. Those who try to game the system and submit using YoloV5 will just try to use this information to their advantage and avoid detection by changing the inference code.
", "votes": 5, "replies": [] }, { "id": 937874, "author_name": "Robin Smits", "author_url": "", "post_date": "2020-07-21T08:05:24.903000", "content": "This would trully be a nice idea for a next competition. Automatically detect/classify the frameworks and licenses used. Likely already done by someone on this planet ;-)
", "votes": 3, "replies": [] }, { "id": 973510, "author_name": "Dezső Ribli", "author_url": "", "post_date": "2020-08-17T10:39:43.377000", "content": "@philculliton could you comment? Is there an automatic mechanism for removal, or are you going from the top line by line on the leaderboard? Thanks!
", "votes": 2, "replies": [ { "id": 973606, "author_name": "Kaushal Shah", "author_url": "", "post_date": "2020-08-17T11:52:21.063000", "content": "@riblidezso We should first ask are you going to remove Yolov5 submissions at all? Because not in a single discussion thread anyone has mentioned this.
", "votes": 0, "replies": [ { "id": 973615, "author_name": "Dezső Ribli", "author_url": "", "post_date": "2020-08-17T12:00:46.623000", "content": "They did say they will check \"winning submissions\" so the question is how far down they will go on the leaderboard. Maybe they are going down to the last medal solution manually, that's why it takes so long, but I guess anyway you are right, any communication about this issue would be appreciated :)
", "votes": 0, "replies": [] }, { "id": 973760, "author_name": "Kaushal Shah", "author_url": "", "post_date": "2020-08-17T13:55:44.630000", "content": "@riblidezso yes I'm talking about removing YoloV5 for entire medal zone only. Removing only for winners won't be fair at all.
", "votes": 0, "replies": [] } ] } ] }, { "id": 948724, "author_name": "Peiyuan Liao", "author_url": "", "post_date": "2020-07-28T07:03:39.220000", "content": "I'm pretty confident in Kaggle's skills and determination in keeping the competitions fair. I believe that there is already a system in place to detect suspicious activities in general (multiple accounts, etc.) As for non-compliant solutions, it will become very obvious to which team is using Yolov5-based solutions when they open-source their code and the organizer is trying to replicate their results. They even banned a Grandmaster for cheating regarding a kernel competition few months back.
", "votes": 1, "replies": [ { "id": 948869, "author_name": "Dezső Ribli", "author_url": "", "post_date": "2020-07-28T09:34:31.910000", "content": "I made this post because I think the previous communications in this competition were not very reassuring. Especially when they said that only \"winning\" solutions need to be compliant, which incentives people to give a shot to Yolov5 for a high LB position which still goes under the radar. \nI wanted to make it clear that only checking winning/open sourced solutions will result in a competition where many honest competitors are demotivated and disappointed. You know, most of us have no chance to win, but some positive feedback is important in the learning process. Let's hope the best.
", "votes": 5, "replies": [] } ] }, { "id": 945601, "author_name": "Stanley Zheng", "author_url": "", "post_date": "2020-07-26T02:14:04.087000", "content": "I agree, but this would have to be extremely carefully done. I'm using the darknet implementation of YOLOV4 (public domain) for my submission and I'm sure many others are too.
", "votes": 1, "replies": [ { "id": 948545, "author_name": "Prateek Gupta", "author_url": "", "post_date": "2020-07-28T03:58:19.233000", "content": "@stanleyjzheng What was your baseline score with YOLOv4? I achieved 0.6626 and then with WBF I got 0.6815.
", "votes": 1, "replies": [] }, { "id": 949330, "author_name": "", "author_url": "", "post_date": "2020-07-28T15:06:23.207000", "content": "", "votes": 0, "replies": [] }, { "id": 949375, "author_name": "Prateek Gupta", "author_url": "", "post_date": "2020-07-28T15:42:44.413000", "content": "Yes I trained the model with image resolution 704x704 and by following AlexeyAB Repo of training custom dataset.
", "votes": 0, "replies": [] } ] }, { "id": 944922, "author_name": "Aidyn Ubingazhibov", "author_url": "", "post_date": "2020-07-25T13:07:09.417000", "content": "I totally agree!
", "votes": 1, "replies": [] }, { "id": 944861, "author_name": "SADMAN ARAF", "author_url": "", "post_date": "2020-07-25T11:53:21.740000", "content": "I am new to kaggle and I worked very hard for a good prediction. And somehow I managed to drop 400 positions in a span of 2-3 days. Now I know why :(
", "votes": 1, "replies": [ { "id": 947095, "author_name": "Alin Cijov", "author_url": "", "post_date": "2020-07-27T04:34:51.430000", "content": "welcome to Kaggle, wait until the last days, people will drop like flies :)
", "votes": 1, "replies": [] }, { "id": 947272, "author_name": "SADMAN ARAF", "author_url": "", "post_date": "2020-07-27T06:54:33.143000", "content": "I hope so @alincijov
", "votes": 0, "replies": [] } ] }, { "id": 944800, "author_name": "Araik Tamazian", "author_url": "", "post_date": "2020-07-25T11:07:11.597000", "content": "I agree.
", "votes": 1, "replies": [] }, { "id": 943273, "author_name": "Alberto Benayas", "author_url": "", "post_date": "2020-07-24T08:49:56.947000", "content": "I totally agree. I spent many hours working on a compliant solution using other models, and probably there are 10's or 100's of participants above me using YOLO v5. This shoud be a fair competition and I think all medal positions should be checked
", "votes": 1, "replies": [] }, { "id": 939148, "author_name": "Rifat Rashid", "author_url": "", "post_date": "2020-07-22T04:08:43.790000", "content": "100% agree kaggle admins should take some steps
", "votes": 1, "replies": [] }, { "id": 937969, "author_name": "Marcin Czelej", "author_url": "", "post_date": "2020-07-21T09:21:37.573000", "content": "agree with request. If someone want to try yolov5 to expand knowledge, thats ok, but this should not be marked as final submission
", "votes": 1, "replies": [] }, { "id": 948278, "author_name": "Costas Voglis", "author_url": "", "post_date": "2020-07-27T19:20:07.650000", "content": "Personally I believe that the biggest issue with yolov5 is the possibility it was used as a generator of bounding boxes for some of the test cases. I saw some other post insinuating it and I believe that this explains the great advantage of this implementation.
\n\nAlso the organizers mentioned that they changed the private test set in the sense that they corrected it... What if they just recalculated the bounding boxes for the \"yolov5 inferenced cases\" and now this big advantage is simply lost in the private leaderboard. I don't say that this is a solution,I don't even have a solid proof for this speculation. It is just an opinion...
", "votes": 2, "replies": [ { "id": 948284, "author_name": "Matt", "author_url": "", "post_date": "2020-07-27T19:25:19.257000", "content": "I don't think that alone was the reason YoloV5 excelled. They only did the 'semi-supervised' labeling on one of the sources from the test set (Tokyo_2 which makes up less than 10%) and it was YoloV3 (which has actually shown to have pretty poor performance) not V5.
Thank you @matthewmasters for the clarification. There must be some correlation though. Or let me rephrase it a correlation on labelling (Tokyo_2) with the output of yolov5 would explain a lot :). Don't get me wrong it is a great piece of software! But the difference with the competition is huge and with now parameter tweaking.
\n\nIn my opinion, the biggest issue of automatic annotation using a software, is the ratio of the bounding boxes. This has to do with the anchor boxes and the detection settings. Maybe the test set annotations are more \"yolov5\" compatible. Again this is a speculation :)
", "votes": 2, "replies": [] } ] }, { "id": 947096, "author_name": "Alin Cijov", "author_url": "", "post_date": "2020-07-27T04:36:51.283000", "content": "All this AI/ML and Kaggle has no tech in place to detect when rules are being broken ?\nAlso look at the submission of those unknown novice accounts ;) fishy business going on
", "votes": 2, "replies": [] }, { "id": 936943, "author_name": "Rahul D", "author_url": "", "post_date": "2020-07-20T16:06:23.593000", "content": "What exactly is the problem with yolov5? Sorry I just joined this competition?
", "votes": 2, "replies": [ { "id": 936959, "author_name": "Dezső Ribli", "author_url": "", "post_date": "2020-07-20T16:16:45.027000", "content": "The competition rules do not allow it because of its licence.
", "votes": 0, "replies": [] }, { "id": 948754, "author_name": "Alin Cijov", "author_url": "", "post_date": "2020-07-28T07:36:06.217000", "content": "license and regulations derp derp :(
", "votes": 0, "replies": [] } ] }, { "id": 948269, "author_name": "Sanjay K", "author_url": "", "post_date": "2020-07-27T19:11:31.870000", "content": "Yes, I agree!
", "votes": 0, "replies": [] }, { "id": 948464, "author_name": "momomo", "author_url": "", "post_date": "2020-07-28T01:21:32.987000", "content": "The reason for this post is obviously jealousy
", "votes": -21, "replies": [] }, { "id": 954634, "author_name": "StoneWST98", "author_url": "", "post_date": "2020-08-02T00:09:06.750000", "content": "How to remove the submitted one? I don't find it
", "votes": 0, "replies": [ { "id": 954728, "author_name": "Dapeng", "author_url": "", "post_date": "2020-08-02T03:35:08.610000", "content": "You can't remove it and don't need to remove it.Just don't select it as final submission.
", "votes": 3, "replies": [] } ] }, { "id": 939238, "author_name": "Qingyuan Wang", "author_url": "", "post_date": "2020-07-22T05:43:51.910000", "content": "Agree, now that hosts banned yolov5, they should have some methods to enforce this rule. Otherwise it will introduce new unfairness.
", "votes": 0, "replies": [] }, { "id": 937258, "author_name": "Jacob", "author_url": "", "post_date": "2020-07-20T21:30:38.837000", "content": "Without a good automatically removing, it's like a prisoner's dilemma. 🙂
", "votes": 0, "replies": [ { "id": 946285, "author_name": "Ahmmad Rashid", "author_url": "", "post_date": "2020-07-26T13:46:11.813000", "content": "As, have stop working with YOLOv5. I dont even know what is our position in leaderboard
", "votes": 1, "replies": [] }, { "id": 946719, "author_name": "Matt", "author_url": "", "post_date": "2020-07-26T19:06:16.797000", "content": "Maybe you'll find my thread (Leaderboard of Compliant Submissions) helpful.
", "votes": 0, "replies": [] } ] }, { "id": 937016, "author_name": "VinayVikram", "author_url": "", "post_date": "2020-07-20T17:03:43.480000", "content": "Completely agreed.. Instead of living in a delusion of if's and but's, better to drop all YOLO submissions..
\n\nOne approach could be- Download the .py version of all notebooks and as in yolov5, everyone is cloning a Github repo related to YOLO version model training. And using regular expression find all those solutions that posses the mentioned condition and drop them.
\n\nWhat you guys think about the above approach..
", "votes": 0, "replies": [ { "id": 943148, "author_name": "Robin Smits", "author_url": "", "post_date": "2020-07-24T07:01:28.120000", "content": "@vin1234 Yes agree. Not only check the top solutions for YoloV5 where there is price money involved..but check all selected final sumissions on YoloV5 to have a fair distribution of the leaderboard.
\n\nI'am not sure if YoloV5 submissions should be dropped if they are not selected for the final leaderboard. If someone trully wants to experiment with YoloV5 but does not use it for its final submissions .. then I guess that should be fine.
", "votes": 0, "replies": [] }, { "id": 945603, "author_name": "Stanley Zheng", "author_url": "", "post_date": "2020-07-26T02:15:23.660000", "content": "I think it's important to make a distinction between the versions of YOLO. YOLOV5 was made by a different person and licenced GPL3. The darknet implementation of YOLOV4 is public domain, and therefore allowed.
", "votes": 1, "replies": [ { "id": 946219, "author_name": "Dezső Ribli", "author_url": "", "post_date": "2020-07-26T12:50:05.907000", "content": "Don't worry, YOLOv5 has unique source code, which is super easy to tell apart from other YOLO versions, there is no way compliant YOLO(v3/v4) based submissions will be invalidated.
", "votes": 1, "replies": [] } ] } ] }, { "id": 939865, "author_name": "", "author_url": "", "post_date": "2020-07-22T14:34:30.750000", "content": "", "votes": 0, "replies": [ { "id": 939866, "author_name": "Matt", "author_url": "", "post_date": "2020-07-22T14:37:32.927000", "content": "The rules differ competition from competition. This one required an MIT license and therefor cannot use anything under GPLv3.
", "votes": 1, "replies": [] }, { "id": 940558, "author_name": "", "author_url": "", "post_date": "2020-07-23T02:46:22.103000", "content": "", "votes": 0, "replies": [] } ] }, { "id": 937677, "author_name": "", "author_url": "", "post_date": "2020-07-21T05:54:01.947000", "content": "", "votes": -3, "replies": [] }, { "id": 937333, "author_name": "", "author_url": "", "post_date": "2020-07-20T23:58:42.273000", "content": "", "votes": -2, "replies": [] } ], "raw_markdown_by_id": { "936502": "The organizers made it clear that they will check \"winning\" submissions for licence compliance, e.g.: probably the first 10 or 20, but It would be way too much work to check a few hundred more. \n\nWhile in a perfect world we could expect everyone to select only compliant final submissions (YOLOv5 free), unfortunately I think we all know this is not going to happen. The urge to reach nice shiny medals is too hard to resist for some.\n\nI feel like in the end we will have many many YOLOv5 based submissions around positions 20-500 or so. I think it would be quite disappointing for those who put in a lot of effort, using compliant code but still end up quite far behind on the leaderboard because of hundreds of YOLOv5 based submissions.\n\nWould it be possible to simply programatically scan the input folders of all solutions and invalidate those which use YOLOv5 source code? Of course If you try hard you could avoid being detected, but it could eliminate 95-99% percent of non-compliant solutions, and only people trying hard to willingly cheat would survive this cleaning. It may even require less work than manually checking tens of winning solutions, so Kaggle could save some effort too.\n\nI think it would make the competition much more motivating, knowing that our honest efforts will probably have results.\n\nPlease consider this solution,. thanks.\n\n---\nUdpate: I understand that the organizers did not comment on this issue during the competition, as it could have helped cheaters to avoid detection. However, now that submissions are over, I think it would be nice to react to a topic with ~ 70 upvotes. Thanks!", "936609": "Kaggle should make everything (including your nice suggestion) to detect as much as possible of non-compliant users. It is not about the prize, nor the gold medals, it's about supporting those who deserve to be on the top, those who followed the rules. ", "936562": "I cannot agree with you more.", "943122": "Maybe , Kaggle team has to finetune CodeBERT on YoloV5 source code, and make classification if the submitted source/input really similar to the originanl YoloV5 repo , or not ?\n\nHow about this @riblidezso ? 😁 ", "937209": "I agree completely with your viewpoint. At this point, we just have to put our trust into the Kaggle compliance team that they will do their best. Even if they are using such an automated system, I doubt they would want to come out and say that they are. Those who try to game the system and submit using YoloV5 will just try to use this information to their advantage and avoid detection by changing the inference code.", "937874": "This would trully be a nice idea for a next competition. Automatically detect/classify the frameworks and licenses used. Likely already done by someone on this planet ;-)", "973510": "@philculliton could you comment? Is there an automatic mechanism for removal, or are you going from the top line by line on the leaderboard? Thanks!", "948724": "I'm pretty confident in Kaggle's skills and determination in keeping the competitions fair. I believe that there is already a system in place to detect suspicious activities in general (multiple accounts, etc.) As for non-compliant solutions, it will become very obvious to which team is using Yolov5-based solutions when they open-source their code and the organizer is trying to replicate their results. They even banned a Grandmaster for cheating regarding a kernel competition few months back.", "945601": "I agree, but this would have to be extremely carefully done. I'm using the darknet implementation of YOLOV4 (public domain) for my submission and I'm sure many others are too. ", "944922": "I totally agree!", "944861": "I am new to kaggle and I worked very hard for a good prediction. And somehow I managed to drop 400 positions in a span of 2-3 days. Now I know why :(", "944800": "I agree.", "943273": "I totally agree. I spent many hours working on a compliant solution using other models, and probably there are 10's or 100's of participants above me using YOLO v5. This shoud be a fair competition and I think all medal positions should be checked", "939148": "100% agree kaggle admins should take some steps", "937969": "agree with request. If someone want to try yolov5 to expand knowledge, thats ok, but this should not be marked as final submission", "948278": "Personally I believe that the biggest issue with yolov5 is the possibility it was used as a generator of bounding boxes for some of the test cases. I saw some other post insinuating it and I believe that this explains the great advantage of this implementation. \n\nAlso the organizers mentioned that they changed the private test set in the sense that they corrected it... What if they just recalculated the bounding boxes for the \"yolov5 inferenced cases\" and now this big advantage is simply lost in the private leaderboard. I don't say that this is a solution,I don't even have a solid proof for this speculation. It is just an opinion...", "947096": "All this AI/ML and Kaggle has no tech in place to detect when rules are being broken ?\nAlso look at the submission of those unknown novice accounts ;) fishy business going on", "936943": "What exactly is the problem with yolov5? Sorry I just joined this competition?", "948269": "Yes, I agree!", "948464": "The reason for this post is obviously jealousy", "954634": "How to remove the submitted one? I don't find it", "939238": "Agree, now that hosts banned yolov5, they should have some methods to enforce this rule. Otherwise it will introduce new unfairness.", "937258": "Without a good automatically removing, it's like a prisoner's dilemma. 🙂 ", "937016": "Completely agreed.. Instead of living in a delusion of if's and but's, better to drop all YOLO submissions..\n\nOne approach could be- Download the .py version of all notebooks and as in yolov5, everyone is cloning a Github repo related to YOLO version model training. And using regular expression find all those solutions that posses the mentioned condition and drop them. \n\nWhat you guys think about the above approach..", "939865": "", "937677": "", "937333": "" } }