Contestants are encouraged to use the Competition Data as training data set(s). The use of a third-party data set, not provided on the Competition Website, is contestant’s choice; however, data sets are subject to open source or other licenses, with varying restrictions. It is the responsibility of each contestant to ensure that, given the possibility of remuneration, contestant understands any existing restrictions on the data set(s) employed. Benetech’s legal team has clarified that Benetech cannot, and will not, provide legal advice to contestants.
\nIt should be noted that submissions using datasets that are not compliant with the original Rule 8.C. (now, Section A.3), such as datasets licensed under CC BY-NC-SA 4.0, will not be accepted due to incompatibility with the Competition Rules, which clearly specify Benetech’s anticipation of commercial use.
\nWe are extending the deadline for this competition to June 19, 2023.
", "messages": [ { "id": 2289942, "postDate": "2023-06-06T13:01:03.360Z", "content": "You speak about the responsibility of contestants, but what about the responsibility of competition host? Precisely because of the possibility of a remuneration.
\nThat the data page recommended a dataset incompatible with the rules is irresponsible, to stay polite. Couldn't you do your homework correctly before competition launch?
\nI don't think you measure the amount of work that kagglers have spent on this (as in every competition) for free. Nullifying that work now is just unbelievable. I see two interesting possible outcomes here:
\nHi @benjiaa !
\nI have 2 questions :
\nThose are important questions that require answers !
\nTo be completely transparent : I'd rather decline the prize than spend the remaining of the competition re-doing what I did the past month. But I also don't want to get removed from LB as this will nullify our efforts.
", "rawMarkdown": "Hi @benjiaa !\n\nI have 2 questions : \n- Can we use data / models under the GPL 3 license be used ? Such license **allows** for commercial use but is more restrictive than MIT / Apache.\n- Will users in the prize area using datasets that make them ineligible for prizes be removed from the leaderboard ? Asked by @crodoc. \n\nThose are important questions that require answers !\n\nTo be completely transparent : I'd rather decline the prize than spend the remaining of the competition re-doing what I did the past month. But I also don't want to get removed from LB as this will nullify our efforts.", "votes": 20 }, { "id": 2290538, "postDate": "2023-06-06T23:19:35.503Z", "content": "@benjiaa
\nIn all likelihood, it is too late to decide to prohibit the use of the ICDAR dataset now, and the fairness of the competition will be compromised. As others have said, the use of the dataset should be allowed since there is an option to retrain after the end of the competition.
\nIn the first place, this dataset was initially introduced by you guys in the data tab with a recommendation for external use (although for some reason you sneakily removed all mention of it again). And, you also explicitly stated that you did not intend to prohibit its use. Furthermore, the reference to the license has been posted in the discussion for months, yet it is you organizers who have failed to take it seriously.
\nYou organizers certainly have the right to set the rules of the competition, but you have the responsibility to ensure that the competition proceeds smoothly and fairly. It is not right to disregard that responsibility, to change the rules as you please, and to drag the competition into a mess.
\nFor some reason, you seem to be referring to the responsibility of the participants, but it makes no sense to hold them responsible for your irresponsible behavior.
\nThis problem was caused by your failure to fulfill your responsibility to properly manage the competition, and you should take responsibility for it. Even if it is a solution that you do not want, you should accept it because you have left the problem to this point.
Also, realistically speaking, how are you guys going to make sure that the winners' solutions meet the requirements?
\nThe source code is easy to determine by reading it, but the dataset is very difficult to determine if the participant used it or not.
\nIf you were to do it, the only way to do it would be to receive the training scripts and see if you could regenerate the results, but are you really going to do that for all the winners? It takes time to receive the scripts and it also takes time to generate the results. (In extreme cases, if a team emerges claiming to have used a model that has been trained for 3 months, you would have to spend 3 months regenerating and confirming the results, wouldn't you?)
\nAnd if you find out where the cheating is, you have to do it for the teams that are further back in the standings.
\nAre you going to extend the LB finalization until this is completely done?
\nThis is impractical by any stretch of the imagination.
Again, I think it is best for both participants and management to allow the use of the ICDAR dataset since there is the option of retraining.
", "rawMarkdown": "@benjiaa \nIn all likelihood, it is too late to decide to prohibit the use of the ICDAR dataset now, and the fairness of the competition will be compromised. As others have said, the use of the dataset should be allowed since there is an option to retrain after the end of the competition.\nIn the first place, this dataset was initially introduced by you guys in the data tab with a recommendation for external use (although for some reason you sneakily removed all mention of it again). And, you also explicitly stated that you did not intend to prohibit its use. Furthermore, the reference to the license has been posted in the discussion for months, yet it is you organizers who have failed to take it seriously.\nYou organizers certainly have the right to set the rules of the competition, but you have the responsibility to ensure that the competition proceeds smoothly and fairly. It is not right to disregard that responsibility, to change the rules as you please, and to drag the competition into a mess.\nFor some reason, you seem to be referring to the responsibility of the participants, but it makes no sense to hold them responsible for your irresponsible behavior.\nThis problem was caused by your failure to fulfill your responsibility to properly manage the competition, and you should take responsibility for it. Even if it is a solution that you do not want, you should accept it because you have left the problem to this point.\n\n\nAlso, realistically speaking, how are you guys going to make sure that the winners' solutions meet the requirements?\nThe source code is easy to determine by reading it, but the dataset is very difficult to determine if the participant used it or not.\nIf you were to do it, the only way to do it would be to receive the training scripts and see if you could regenerate the results, but are you really going to do that for all the winners? It takes time to receive the scripts and it also takes time to generate the results. (In extreme cases, if a team emerges claiming to have used a model that has been trained for 3 months, you would have to spend 3 months regenerating and confirming the results, wouldn't you?)\nAnd if you find out where the cheating is, you have to do it for the teams that are further back in the standings.\nAre you going to extend the LB finalization until this is completely done?\nThis is impractical by any stretch of the imagination. \n\n\nAgain, I think it is best for both participants and management to allow the use of the ICDAR dataset since there is the option of retraining.", "votes": 15, "replies": [ { "id": 2290815, "postDate": "2023-06-07T06:01:32.100Z", "content": "I love this:
\n\n\nFor some reason, you seem to be referring to the responsibility of the participants, but it makes no sense to hold them responsible for your irresponsible behavior.
\n
It is what triggered my own response. Host doesn't take responsibility for the issue.
\nI guess it is because their lawyers told them to not leave open any legal action by participants. But the message to all those who worked hard here is devastating.
", "rawMarkdown": "I love this:\n> For some reason, you seem to be referring to the responsibility of the participants, but it makes no sense to hold them responsible for your irresponsible behavior.\n\nIt is what triggered my own response. Host doesn't take responsibility for the issue.\n\nI guess it is because their lawyers told them to not leave open any legal action by participants. But the message to all those who worked hard here is devastating.", "votes": 4 } ] }, { "id": 2290103, "postDate": "2023-06-06T14:49:37.367Z", "content": "Why was the competition extended?
\n\n\nWe are extending the deadline for this competition to June 19, 2023
\n
I know that this might help people to re run experiments without the ICDAR dataset but at least for me I have been in this competition for the past few months and I have other commitments. This means dedicating even more time to the competition and thus I will have to leave some personal matters aside.
", "rawMarkdown": "Why was the competition extended?\n> We are extending the deadline for this competition to June 19, 2023\n\nI know that this might help people to re run experiments without the ICDAR dataset but at least for me I have been in this competition for the past few months and I have other commitments. This means dedicating even more time to the competition and thus I will have to leave some personal matters aside.", "votes": 15 }, { "id": 2289175, "postDate": "2023-06-06T01:29:31.477Z", "content": "\nAre you sure you cannot allow external data? You will be able to re-train your models without the data you are unwilling to use, based on the top solutions submitted after the competition ended. I don't expect that extending the period by a week will significantly change our solution. Then I think it would be better to allow the data for the sake of fairness in the competition.
\nIf you prohibit its use, please do not remove participants from the leaderboard in exchange for not receiving prize. Many participants, including myself, have developed models based on this statement by you. Like @theoviel, I will continue to use external data if I can win a medal by declining the prize money.
", "rawMarkdown": "@benjiaa \n\nAre you sure you cannot allow external data? You will be able to re-train your models without the data you are unwilling to use, based on the top solutions submitted after the competition ended. I don't expect that extending the period by a week will significantly change our solution. Then I think it would be better to allow the data for the sake of fairness in the competition.\n\nIf you prohibit its use, please do not remove participants from the leaderboard in exchange for not receiving prize. Many participants, including myself, have developed models based on [this statement](https://www.kaggle.com/competitions/benetech-making-graphs-accessible/discussion/407485#2264697) by you. Like @theoviel, I will continue to use external data if I can win a medal by declining the prize money.\n\n\n", "votes": 15, "replies": [ { "id": 2289299, "postDate": "2023-06-06T04:21:42.107Z", "content": "@benjiaa
\nNot clarifying the process regarding illegal submissions breaks the competition and does no one any good, including the administration. Some participants will submit illegal answers with high scores in the hopes that they will not be out of the prize zone or not removed from LB. In that case, you will all lose the opportunity to get better solutions. Participants risk being removed from the LB, and honest teams may not receive a gold medal even though they are in the real top11 with legal submissions.
Again, my best suggestion is to allow external data. In that case, you will not get the weights of the commercially available model wegihts, but you can use the training code submitted after the competition and retrain with the available data, including 4000 samples of test data. You will get better solutions and we will be able to focus on the rest of the competition without doubting your or other participants.
", "rawMarkdown": "@benjiaa \nNot clarifying the process regarding illegal submissions breaks the competition and does no one any good, including the administration. Some participants will submit illegal answers with high scores in the hopes that they will not be out of the prize zone or not removed from LB. In that case, you will all lose the opportunity to get better solutions. Participants risk being removed from the LB, and honest teams may not receive a gold medal even though they are in the real top11 with legal submissions.\n\nAgain, my best suggestion is to allow external data. In that case, you will not get the weights of the commercially available model wegihts, but you can use the training code submitted after the competition and retrain with the available data, including 4000 samples of test data. You will get better solutions and we will be able to focus on the rest of the competition without doubting your or other participants.", "votes": 11 } ] }, { "id": 2289302, "postDate": "2023-06-06T04:26:04.917Z", "content": "I agree with penguin46's suggestion. After the competition, you have the option to retrain your model. This can be done by excluding unwanted data once you've received the top solutions. This method should solve any licensing problems. You've previously made it clear that the use of ICDAR data is permitted. If you are to override this, please clarify why the aforementioned solution wouldn't be applicable.
\nIf the use of ICDAR is to be prohibited, I believe that in order to maintain fairness, not just the submission code, but also the training data and code of everyone who earns a medal after the competition need to be checked. If only the winners of the prize money are checked, there might be individuals breaking the rules to win medals. This could potentially compromise Kaggle's value as a platform for data science competitions.
", "rawMarkdown": "I agree with penguin46's suggestion. After the competition, you have the option to retrain your model. This can be done by excluding unwanted data once you've received the top solutions. This method should solve any licensing problems. You've previously made it clear that the use of ICDAR data is permitted. If you are to override this, please clarify why the aforementioned solution wouldn't be applicable.\n\nIf the use of ICDAR is to be prohibited, I believe that in order to maintain fairness, not just the submission code, but also the training data and code of everyone who earns a medal after the competition need to be checked. If only the winners of the prize money are checked, there might be individuals breaking the rules to win medals. This could potentially compromise Kaggle's value as a platform for data science competitions.", "votes": 11 }, { "id": 2293908, "postDate": "2023-06-09T15:11:45.933Z", "content": "@benjiaa
\nI fully understand that you guys are irresponsible, but since this is at least a competition, can you please state the rules clearly?
\nI believe that many of the participants are currently contacting you guys with the following queries regarding the rules.
Perhaps you don't understand, but what you are organizing is a competition.
\nAs it is a competition, rules are absolutely necessary.
\nIf participants do not understand the rules, they will not know what is allowed and what is not allowed, and they will not be able to decide which tasks to devote their time to.
\nAnyway, please explain properly what the current rules are.
\nIt's a competition that you guys are organizing, so at the very least get that right.
@ryanholbrook @maggiemd
\nCan you guys please make some statement instead of remaining silent with the host.
\nThis host is doing whatever they want, wasting the participants' efforts and makes the participants look like fools.
\nI think they are hurting the value of kaggle greatly. What do you think of this situation?
The rudeness of shifting the blame for their mistakes onto the participants without even apologizing is astonishing, but what's more, they can't even answer a question they were asked a week ago about the rules of the competition they are organizing.
\nBy all accounts, Benetech has no integrity, no common sense, and no qualifications to be a host.
Anyway, this unbelievable host is causing confusion for all participants and making the competition impossible.
\nI think you are at a certain fault because you allowed the competition to be held by an organization with such a high level of problematic behavior.
\nPlease state what the rules are for this competition and how you are going to fix this situation.
\nIsn't this your duty?
For clarity the host did not change the rules. The rules have remained the same. The update made to the rules was to emphasize the point since there was confusion on the forums about what datasets were allowed. Winner's will have to produce information to support the license required. They extended the deadline so that those who may have used a dataset that does not comply can have time to train with new datasets.
\nThe goal of this competition is the same, they are trying to help children with learning disabilities benefit from charts that they cannot currently see. For context, those who decide to host a competition sometimes are learning about these legal issues for the first time. When there was confusion they took a lot of time to research this and spent time to clarify the rules. There was no malicious intent here.
", "rawMarkdown": "For clarity the host did not change the rules. The rules have remained the same. The update made to the rules was to emphasize the point since there was confusion on the forums about what datasets were allowed. Winner's will have to produce information to support the license required. They extended the deadline so that those who may have used a dataset that does not comply can have time to train with new datasets. \n\nThe goal of this competition is the same, they are trying to help children with learning disabilities benefit from charts that they cannot currently see. For context, those who decide to host a competition sometimes are learning about these legal issues for the first time. When there was confusion they took a lot of time to research this and spent time to clarify the rules. There was no malicious intent here. ", "votes": 5, "replies": [ { "id": 2294915, "postDate": "2023-06-10T12:56:47.907Z", "content": "If the rules remained the same then why was there a silent change in the data page of the competition? Other questions still remains unanswered:
\n@maggiemd thank you for responding. Let's all try and make the best out of this situation!
\nOne of the main questions people still have is what happens to submissions that are not compliant with the MIT License (both regarding models & datasets)? Will they be removed from the LB? Most competitors are motivated by ranking rather than the money. So people want to be sure if they make select a submission that is fully MIT License compliant, there won't be people in from of them on LB who did not do the same. Using the disallowed dataset or non-compliant models can give you a boost.
\nOur team is working hard to make the final submission compliant with the MIT License mentioned in the rules.
", "rawMarkdown": "@maggiemd thank you for responding. Let's all try and make the best out of this situation!\n\nOne of the main questions people still have is what happens to submissions that are not compliant with the MIT License (both regarding models & datasets)? Will they be removed from the LB? Most competitors are motivated by ranking rather than the money. So people want to be sure if they make select a submission that is fully MIT License compliant, there won't be people in from of them on LB who did not do the same. Using the disallowed dataset or non-compliant models can give you a boost.\n\nOur team is working hard to make the final submission compliant with the MIT License mentioned in the rules.", "votes": 2 }, { "id": 2294996, "postDate": "2023-06-10T13:55:04.140Z", "content": "@maggiemd Thank you for breaking the silence!
\nI am reconfiguring the pipeline to be MIT compliant and hope that the competition will end in a way that is acceptable to participants and hosts!
I would especially like to know if participants who do not meet the license will be removed from LB, as the two above have claimed.
\n", "rawMarkdown": "@maggiemd Thank you for breaking the silence!\nI am reconfiguring the pipeline to be MIT compliant and hope that the competition will end in a way that is acceptable to participants and hosts!\n\nI would especially like to know if participants who do not meet the license will be removed from LB, as the two above have claimed.\n\n~~Also, the host said a month ago \"Benetech has no intention of disallowing the use of CHART-Info / ICDAR datasets for this competition.\" Then, just before the end of the competition, they banned it. Isn't this a rule change?~~" }, { "id": 2295045, "postDate": "2023-06-10T14:47:32.373Z", "content": "Thank you for your statement.
\nIf the rules haven't changed, then that means that the host was introducing a data set that is not allowed to be used in the competition on the official page for uses that are not permitted, which is annoying and I think they should at least apologize, regardless of whether or not there is any ill intent, but that is probably not productive to argue about now, so I'll leave it at that for now.
\nMore importantly, as you all have said, I would like to know the details of the rules.
\nI will restate the specifics, but I would like an answer to the following question.
To put it more succinctly, the question is what happens if the MIT licensing requirements are not met, and how do you intend to make sure that they are?
\nI know that this type of problem is not limited to this competition. But the major difference is that in this competition, the hosts have long made known the prohibited data sets that they are prohibited from using in the first place. I am very concerned that I will not be disadvantaged by my honest adherence to the licensing requirements.
", "rawMarkdown": "Thank you for your statement.\n\nIf the rules haven't changed, then that means that the host was introducing a data set that is not allowed to be used in the competition on the official page for uses that are not permitted, which is annoying and I think they should at least apologize, regardless of whether or not there is any ill intent, but that is probably not productive to argue about now, so I'll leave it at that for now.\n\nMore importantly, as you all have said, I would like to know the details of the rules.\nI will restate the specifics, but I would like an answer to the following question.\n\n* Can a participant refuse to accept a prize?\n* If a participant can refuse to win a prize, will they keep their position on the leaderboard even if the solution they used does not meet the requirements for a prize?\n* If you intend to remove participants from the LB who cannot be denied the prize and do not meet the requirements, how do you determine if the solution used meets the prize requirements?\n(It is quite difficult to determine if a dataset was used or not, but do you have a way to ensure that you play the cheating participant without creating false accusations?)\n\nTo put it more succinctly, the question is what happens if the MIT licensing requirements are not met, and how do you intend to make sure that they are?\n\nI know that this type of problem is not limited to this competition. But the major difference is that in this competition, the hosts have long made known the prohibited data sets that they are prohibited from using in the first place. I am very concerned that I will not be disadvantaged by my honest adherence to the licensing requirements.", "votes": 4 }, { "id": 2295384, "postDate": "2023-06-10T21:46:28.103Z", "content": "@maggiemd Thanks for responding.
\nIndeed, the rules stated upfront that winners must provide open source license suited for commercial use, as in many other kaggle competitions (but not al).
\nWhat infuriates (rightly if you ask me) many participants is that the data page recommended a dataset incompatible with the rules, and that it took 5 or 6 weeks to answer the questions that participants raised about this.
\nAnd when the response comes, it is as if the mistake had been done by participants. Participants were labelled as irresponsible for using a non compliant dataset.
\nThe only mistake participants made was to have trusted the host. They assumed that the host recommended a dataset compliant with the rules of the competition.
\nSome apology instead of finger pointing in the original post would have been much appreciated I guess. It would not change the underlying issue, but it would show some consideration.
", "rawMarkdown": "@maggiemd Thanks for responding.\n\nIndeed, the rules stated upfront that winners must provide open source license suited for commercial use, as in many other kaggle competitions (but not al). \n\nWhat infuriates (rightly if you ask me) many participants is that the data page recommended a dataset incompatible with the rules, and that it took 5 or 6 weeks to answer the questions that participants raised about this.\n\nAnd when the response comes, it is as if the mistake had been done by participants. Participants were labelled as irresponsible for using a non compliant dataset.\n\nThe only mistake participants made was to have trusted the host. They assumed that the host recommended a dataset compliant with the rules of the competition. \n\nSome apology instead of finger pointing in the original post would have been much appreciated I guess. It would not change the underlying issue, but it would show some consideration.\n", "votes": 7 }, { "id": 2297276, "postDate": "2023-06-12T13:27:44.870Z", "rawMarkdown": "", "isDeleted": true } ] } ] }, { "id": 2292526, "postDate": "2023-06-08T12:16:38.847Z", "content": "It took about 3 months for them to figure out which license is allowed in this competition.
\nHow quickly would you expect them to answer all questions and figure out what to do next?!
If that's the case, I suggest a thorough investigation into whether the participants in the gold medal zone, and even the silver zone, have used external data which is mentioned above. It would be challenging to enforce each participant to voluntarily disclose their training code. Alternatively, it would be necessary to develop an algorithm that can automatically detect if participants have used the aforementioned database. Otherwise, it would be unfair to other participants.
", "rawMarkdown": "If that's the case, I suggest a thorough investigation into whether the participants in the gold medal zone, and even the silver zone, have used external data which is mentioned above. It would be challenging to enforce each participant to voluntarily disclose their training code. Alternatively, it would be necessary to develop an algorithm that can automatically detect if participants have used the aforementioned database. Otherwise, it would be unfair to other participants.", "votes": 10 }, { "id": 2292066, "postDate": "2023-06-08T03:52:50.250Z", "content": "IMO Kaggle's \"moat\" is its amazing community. Not difficult to imagine impact of acts like sneakily updating the data /rules page, ineffective communication, believing in 'Silence Is All You Need' etc…
", "rawMarkdown": "IMO Kaggle's \"moat\" is its amazing community. Not difficult to imagine impact of acts like sneakily updating the data /rules page, ineffective communication, believing in 'Silence Is All You Need' etc...", "votes": 8 }, { "id": 2291919, "postDate": "2023-06-08T00:13:42.673Z", "content": "\nCould you please provide a response to participants' inquiries as promptly as possible, regardless of whether you choose to continue prohibiting the use of ICDAR data or not? I would like to reiterate my concerns:
\nIf you are currently contemplating these issues, please make it known. Remaining silent only serves to sow confusion among the participants.
", "rawMarkdown": "@benjiaa \n\nCould you please provide a response to participants' inquiries as promptly as possible, regardless of whether you choose to continue prohibiting the use of ICDAR data or not? I would like to reiterate my concerns:\n\n* Would it not be acceptable to allow the use of ICDAR data and then have the host retrain the data after the competition's conclusion?\n* In the event that you do not allow this, how do you plan on verifying compliance with the rules?\n * As previously expressed by others, it does not seem feasible to reliably determine whether a submission violates these rules.\n * If the proper monitoring of submissions isn't possible, it is unreasonable to expect all participants to develop a rule-abiding solution.\n\nIf you are currently contemplating these issues, please make it known. Remaining silent only serves to sow confusion among the participants.", "votes": 7, "replies": [ { "id": 2292019, "postDate": "2023-06-08T03:08:14.153Z", "rawMarkdown": "", "isDeleted": true, "replies": [ { "id": 2292259, "postDate": "2023-06-08T06:52:51.600Z", "content": "I think realistically their lawyer has almost certainly told them not to say anything. It's a rubbish way for this competition to end tbh
", "rawMarkdown": "I think realistically their lawyer has almost certainly told them not to say anything. It's a rubbish way for this competition to end tbh", "votes": 3 } ] } ] }, { "id": 2291426, "postDate": "2023-06-07T14:45:06.790Z", "content": "I have not used the ICDAR dataset until now because I am wary of the licensing requirements. As such, I can offer an objective opinion.
\nOriginally, the host recommended the ICDAR dataset to the participants. As a result, all participants were divided into two groups: those who used the ICDAR dataset and those who did not. Neither group had any intention of violating the rules, of course.
\nOn the other hand, it was the host who recommended the ICDAR dataset to the participants, and it is also the host who says that the ICDAR dataset is prohibited. Regardless of these conflicting action and statement, all participants have made a good faith effort to comply with the rules.
\nIt is the host, and no one else, who has led the participants into unintentional violations of the rules. It is the natural \"responsibility\" as the host to fairly reward both those who used the ICDAR dataset and those who did not.
", "rawMarkdown": "I have not used the ICDAR dataset until now because I am wary of the licensing requirements. As such, I can offer an objective opinion.\n\nOriginally, the host recommended the ICDAR dataset to the participants. As a result, all participants were divided into two groups: those who used the ICDAR dataset and those who did not. Neither group had any intention of violating the rules, of course.\n\nOn the other hand, it was the host who recommended the ICDAR dataset to the participants, and it is also the host who says that the ICDAR dataset is prohibited. Regardless of these conflicting action and statement, all participants have made a good faith effort to comply with the rules.\n\nIt is the host, and no one else, who has led the participants into unintentional violations of the rules. It is the natural \"responsibility\" as the host to fairly reward both those who used the ICDAR dataset and those who did not.", "votes": 7 }, { "id": 2289555, "postDate": "2023-06-06T08:31:24.580Z", "content": "Hi @benjiaa ,
\nHow do you plan to distinguish the teams that are not in the prize zone but used an external dataset that does not comply with this rules?
", "rawMarkdown": "Hi @benjiaa ,\n\nHow do you plan to distinguish the teams that are not in the prize zone but used an external dataset that does not comply with this rules?", "votes": 7 }, { "id": 2292867, "postDate": "2023-06-08T17:34:02.267Z", "content": "A month ago, I had the option of using the dataset recommended in the dataset tab without telling you about the license. But in that case, you may not have been able to acquire the winner's solution for commercial use, and participants who did not question your recommendations may have been removed from LB. That is why I contacted you to ensure that the competition ended in a mutually satisfactory manner. I did not expect to be betrayed in this way.
", "rawMarkdown": "A month ago, I had the option of using the dataset recommended in the dataset tab without telling you about the license. But in that case, you may not have been able to acquire the winner's solution for commercial use, and participants who did not question your recommendations may have been removed from LB. That is why I contacted you to ensure that the competition ended in a mutually satisfactory manner. I did not expect to be betrayed in this way.", "votes": 8 }, { "id": 2292007, "postDate": "2023-06-08T03:01:50.920Z", "content": "@benjiaa
\nWhy not respond in some way first?
\nAs everyone has said, your decision to remain silent after so many inquiries is incredibly irresponsible.
I would also suggest another constructive option besides retraining after the competition is over.
\nSomething similar has already been said by some participants, but a way to add a rule that says, \"participants who are not able to fulfill this license requirement may retain their position on the leaderboard, but will not receive a prize\".
Such a rule was taken in IMC2022, so it should not be impossible. By the way, in IMC2022 competition we could not receive prize money for using source code that was not available for commercial use, but we did receive a medal and we provided all the source code from us. Also, if I remember correctly, the prize slots were moved to a lower level and the teams that met the requirements would also have provided source code to the host.
\nAlso, if you are concerned that allowing the use of the dataset will discourage anyone from submitting clean solutions, I think that is unfounded.
\nIn fact, there are participants like @takuji have achieved excellent scores without using this dataset, and I believe there was also a team at IMC2022 that won a prize in the Gold Zone.
This way, the participants would not have had their efforts wasted and you would have both a highly accurate solution using the ICDAR dataset and a clean solution not using the ICDAR dataset. Participants who did not use the ICDAR dataset would also get a quid pro quo in the form of prize money.
\nMoreover, you can check if the prize money requirements have been met at your leisure, independently of the finalization of the LB, so this will not be a major obstacle to the management of the competition.
Isn't this a win-win way for all?
", "rawMarkdown": "@benjiaa \nWhy not respond in some way first?\nAs everyone has said, your decision to remain silent after so many inquiries is incredibly irresponsible.\n\nI would also suggest another constructive option besides retraining after the competition is over.\nSomething similar has already been said by some participants, but a way to add a rule that says, \"participants who are not able to fulfill this license requirement may retain their position on the leaderboard, but will not receive a prize\".\n\nSuch a rule was taken in [IMC2022](https://www.kaggle.com/competitions/image-matching-challenge-2022/rules), so it should not be impossible. By the way, in IMC2022 competition we could not receive prize money for using source code that was not available for commercial use, but we did receive a medal and we provided all the source code from us. Also, if I remember correctly, the prize slots were moved to a lower level and the teams that met the requirements would also have provided source code to the host.\nAlso, if you are concerned that allowing the use of the dataset will discourage anyone from submitting clean solutions, I think that is unfounded.\nIn fact, there are participants like @takuji have achieved excellent scores without using this dataset, and I believe there was also a team at IMC2022 that won a prize in the Gold Zone.\n\nThis way, the participants would not have had their efforts wasted and you would have both a highly accurate solution using the ICDAR dataset and a clean solution not using the ICDAR dataset. Participants who did not use the ICDAR dataset would also get a quid pro quo in the form of prize money.\nMoreover, you can check if the prize money requirements have been met at your leisure, independently of the finalization of the LB, so this will not be a major obstacle to the management of the competition.\n\nIsn't this a win-win way for all?", "votes": 6 }, { "id": 2290470, "postDate": "2023-06-06T20:24:54.627Z", "content": "It is incredibly frustrating that, after a significant systemic change is implemented, now 24 hours later, we are still unable to obtain answers to basic questions in a competition that has been running for three months. And to do this right before the original deadline is ridiculous. Moreover, this entire situation could have been easily avoided if proper research had been conducted prior to the competition's start.
", "rawMarkdown": "It is incredibly frustrating that, after a significant systemic change is implemented, now 24 hours later, we are still unable to obtain answers to basic questions in a competition that has been running for three months. And to do this right before the original deadline is ridiculous. Moreover, this entire situation could have been easily avoided if proper research had been conducted prior to the competition's start.", "votes": 5 }, { "id": 2291014, "postDate": "2023-06-07T09:02:44.540Z", "content": "@benjiaa
\nDo you intend to respond in any way? Or will you remain silent?
I have another suggestion. In the past, a competition was ruined by incidents for which the participants were not responsible. In that case, the host (or kaggle team?) paid additional prize money to the team that should have won. If you want a clean model without re-training, you can do something similar.
\nIn other words, apart from the regular LB with external data, you can create another LB that is aggregated only with submissions that meet the license, and negotiate with the winning team to buy the model for the appropriate amount. The submissions to be used for aggregation should be declared in advance. It would cost you extra, but you would get a license compliant model.
", "rawMarkdown": "@benjiaa \nDo you intend to respond in any way? Or will you remain silent?\n\nI have another suggestion. In the past, [a competition](https://www.kaggle.com/competitions/foursquare-location-matching/discussion/338035) was ruined by incidents for which the participants were not responsible. In that case, the host (or kaggle team?) paid additional prize money to the team that should have won. If you want a clean model without re-training, you can do something similar.\n\nIn other words, apart from the regular LB with external data, you can create another LB that is aggregated only with submissions that meet the license, and negotiate with the winning team to buy the model for the appropriate amount. The submissions to be used for aggregation should be declared in advance. It would cost you extra, but you would get a license compliant model.", "votes": 6, "replies": [ { "id": 2291122, "postDate": "2023-06-07T10:41:56.677Z", "content": "Foursquare was different, the mistake was on Kaggle's side.
\n@ryotayoshinobu I think the best option for high-ranked participants is to rerun their pipeline without the rule-infringing components. That's what we've been doing since the announcement.
\nWe're observing performance drops but if everyone at the top abides by the same rules, the competition should still be fair.
Yes, certainly the situation may be different from foursquare.
\nAll I was trying to say here is that the host has an option to buy the winner's solution directly.
Thanks for the advice on the approach for the remaining period. I too think that is a wise decision, and I have already started creating a clean pipeline as you.
", "rawMarkdown": "Yes, certainly the situation may be different from foursquare.\nAll I was trying to say here is that the host has an option to buy the winner's solution directly.\n\nThanks for the advice on the approach for the remaining period. I too think that is a wise decision, and I have already started creating a clean pipeline as you.", "votes": 2 } ] }, { "id": 2291172, "postDate": "2023-06-07T11:28:36.313Z", "rawMarkdown": "", "votes": 1, "isDeleted": true } ] }, { "id": 2290234, "postDate": "2023-06-06T16:33:25.827Z", "content": "As others have said, disallowing the use of ICDAR this late makes it almost impossible to maintain the integrity and fairness of this competition. If the host doesn't want to deal with license issue for commercial use, they could retrain the winning solutions and replace ICDAR data with the 4000 extracted images they have in the test set. The result will be an even better model I imagine.
", "rawMarkdown": "As others have said, disallowing the use of ICDAR this late makes it almost impossible to maintain the integrity and fairness of this competition. If the host doesn't want to deal with license issue for commercial use, they could retrain the winning solutions and replace ICDAR data with the 4000 extracted images they have in the test set. The result will be an even better model I imagine.", "votes": 6 }, { "id": 2289100, "postDate": "2023-06-05T23:17:43.373Z", "content": "Hello, can we use the ICDAR dataset, if we do not want the money prize and just would like to win a medal?
", "rawMarkdown": "Hello, can we use the ICDAR dataset, if we do not want the money prize and just would like to win a medal?", "votes": 6 }, { "id": 2295619, "postDate": "2023-06-11T06:40:55.333Z", "content": "The entire model (or backbone) of many text detection/recognition model are trained on ImageNet or SynthText. However, ImageNet(https://www.image-net.org/) and SynthText (https://www.robots.ox.ac.uk/~vgg/terms/dataset-group-2-access.html) does not allow commercial use ❌.
\nDoes that mean I can not fine-tune any of these pre-trained OCR models or use the ResNet backbones in this competition?
", "rawMarkdown": "The entire model (or backbone) of many text detection/recognition model are trained on ImageNet or SynthText. However, ImageNet(https://www.image-net.org/) and SynthText (https://www.robots.ox.ac.uk/~vgg/terms/dataset-group-2-access.html) does not allow commercial use ❌. \n\nDoes that mean I can not fine-tune any of these pre-trained OCR models or use the ResNet backbones in this competition?", "votes": 4 }, { "id": 2289382, "postDate": "2023-06-06T05:47:56.967Z", "content": "Hi, BENJIAA, how about this dataset which was created by BARTLEY and published on Kaggle? https://www.kaggle.com/datasets/brendanartley/benetech-extra-generated-data
", "rawMarkdown": "Hi, BENJIAA, how about this dataset which was created by BARTLEY and published on Kaggle? [https://www.kaggle.com/datasets/brendanartley/benetech-extra-generated-data](url)", "votes": 1, "replies": [ { "id": 2290056, "postDate": "2023-06-06T14:15:04.513Z", "content": "I would like to make this dataset open for use in this competition. @benjiaa
\nI have no idea which license to use on the dataset (as I do not see MIT in the dropdown menu). Does anyone have any idea which license to use here?
", "rawMarkdown": "I would like to make this dataset open for use in this competition. @benjiaa\n\nI have no idea which license to use on the dataset (as I do not see MIT in the dropdown menu). Does anyone have any idea which license to use here?", "votes": 1, "replies": [ { "id": 2290086, "postDate": "2023-06-06T14:37:55.580Z", "content": "@brendanartley I think you can add it under the License Card and use CCO which I think it's the most open and allows commercial use. I am not a legal expert though but I believe that would be just fine.
Sorry Data Card -> License -> CCO
Thanks @alejopaullier. I will update the dataset with this for now.
\nCan the organizers confirm this please?
", "rawMarkdown": "Thanks @alejopaullier. I will update the dataset with this for now.\n\nCan the organizers confirm this please?" }, { "id": 2290387, "postDate": "2023-06-06T18:41:57.080Z", "content": "I don't think it is a problem. It is actually a dataset created by your notebook. Everyone can just create the data instead of download them if they feel somehow confused.
", "rawMarkdown": "I don't think it is a problem. It is actually a dataset created by your notebook. Everyone can just create the data instead of download them if they feel somehow confused.", "votes": 1 } ] } ] }, { "id": 2297193, "postDate": "2023-06-12T12:12:47.403Z", "content": "@benjiaa
\nWe have one week left in this competition and I was wondering if you could quickly answer some of the questions that have been submitted by the participants.
I am not asking you to change the rules, I am asking you to give me the details of the rules of the competition you are organizing, what could be so difficult that it would take a week?
\nAlso, since there is a new question about whether the pre-train model can be used, can I ask you to answer it properly?
\nIf you guys are really concerned about a clean license, then basically all pre-train models cannot be used. In that case, you guys would only be entitled to pay the prize money and receive solutions with a level of accuracy that you can never use in practical applications. If that is what you want, I will say nothing more and you can do whatever you want, but I am sure that this is the worst-case scenario that will benefit no one.
\nOn the other hand, if you allow pre-train models because you only want to refer to how to do the solution, etc., then I think you should allow the use of the ICDAR dataset because it is essentially the same thing.
\nAt least in this rule, I don't think there is any convenient way to allow only pre-train models and prohibit ICDAR datasets.
\nUnless, you are going to take the outlaw stance of only allowing the use of pre-train models, even though you know it is illegal to do so.
I don't think you would do such a thing against justice, but if you are going to make such a decision, you should explain well how you intend to use the solution and why you decided to allow only the pre-train model so that participants are not mistakenly at risk of being involved in a lawsuit.
\nAs for the license, you are asking us, the participants, to take responsibility, but there is no way for outsiders to know what is allowed and to what extent within the organizer, and it is impossible for us to guess at every single thing. It is too late now, but at the very least, as hosts, you should fulfill your responsibility to explain.
\nFrankly, I don't care what conclusion you guys come to, as I no longer expect things to get better, but at least the host should give a proper explanation.
\nWithout your proper explanation, I have no idea what to believe in this competition.
@ryanholbrook @maggiemd
\nAlso, to the Kaggle team, this issue is happening on your platform, so please work with the host to take appropriate action.
Contestants are encouraged to use the Competition Data as training data set(s). The use of a third-party data set, not provided on the Competition Website, is contestant’s choice; however, data sets are subject to open source or other licenses, with varying restrictions. It is the responsibility of each contestant to ensure that, given the possibility of remuneration, contestant understands any existing restrictions on the data set(s) employed. Benetech’s legal team has clarified that Benetech cannot, and will not, provide legal advice to contestants.
\nIt should be noted that submissions using datasets that are not compliant with the original Rule 8.C. (now, Section A.3), such as datasets licensed under CC BY-NC-SA 4.0, will not be accepted due to incompatibility with the Competition Rules, which clearly specify Benetech’s anticipation of commercial use.
\nWe are extending the deadline for this competition to June 19, 2023.
", "rawMarkdown": "Contestants are encouraged to use the Competition Data as training data set(s). The use of a third-party data set, not provided on the Competition Website, is contestant’s choice; however, data sets are subject to open source or other licenses, with varying restrictions. It is the responsibility of each contestant to ensure that, given the possibility of remuneration, contestant understands any existing restrictions on the data set(s) employed. Benetech’s legal team has clarified that Benetech cannot, and will not, provide legal advice to contestants.\n\nIt should be noted that submissions using datasets that are not compliant with the original Rule 8.C. (now, Section A.3), such as datasets licensed under CC BY-NC-SA 4.0, will not be accepted due to incompatibility with the Competition Rules, which clearly specify Benetech’s anticipation of commercial use.\n \nWe are extending the deadline for this competition to June 19, 2023.", "votes": -41 }, { "id": 2290797, "postDate": "2023-06-07T05:39:51.983Z", "content": "In my humble opinion, the focus on improving model performance may depend not only on the type of external data used, but on the quality of the external data closely aligned with the LB data. If this is true, then substituting a batch of re-simulated LB data can avoid the advantage of some specific external data.
", "rawMarkdown": "In my humble opinion, the focus on improving model performance may depend not only on the type of external data used, but on the quality of the external data closely aligned with the LB data. If this is true, then substituting a batch of re-simulated LB data can avoid the advantage of some specific external data." }, { "id": 2295606, "postDate": "2023-06-11T06:20:10.730Z", "rawMarkdown": "", "isDeleted": true }, { "id": 2293959, "postDate": "2023-06-09T15:54:38.263Z", "rawMarkdown": "", "isDeleted": true }, { "id": 2289504, "postDate": "2023-06-06T07:46:05.143Z", "rawMarkdown": "", "isDeleted": true }, { "id": 2289456, "postDate": "2023-06-06T07:05:43.503Z", "rawMarkdown": "", "isDeleted": true }, { "id": 2289208, "postDate": "2023-06-06T02:34:10.813Z", "rawMarkdown": "", "isDeleted": true } ], "comments": [ { "id": 2289942, "author_name": "CPMP", "author_url": "", "post_date": "2023-06-06T13:01:03.360000", "content": "You speak about the responsibility of contestants, but what about the responsibility of competition host? Precisely because of the possibility of a remuneration.
\nThat the data page recommended a dataset incompatible with the rules is irresponsible, to stay polite. Couldn't you do your homework correctly before competition launch?
\nI don't think you measure the amount of work that kagglers have spent on this (as in every competition) for free. Nullifying that work now is just unbelievable. I see two interesting possible outcomes here:
\nHi @benjiaa !
\nI have 2 questions :
\nThose are important questions that require answers !
\nTo be completely transparent : I'd rather decline the prize than spend the remaining of the competition re-doing what I did the past month. But I also don't want to get removed from LB as this will nullify our efforts.
", "votes": 20, "replies": [] }, { "id": 2290538, "author_name": "YumeNeko", "author_url": "", "post_date": "2023-06-06T23:19:35.503000", "content": "@benjiaa
\nIn all likelihood, it is too late to decide to prohibit the use of the ICDAR dataset now, and the fairness of the competition will be compromised. As others have said, the use of the dataset should be allowed since there is an option to retrain after the end of the competition.
\nIn the first place, this dataset was initially introduced by you guys in the data tab with a recommendation for external use (although for some reason you sneakily removed all mention of it again). And, you also explicitly stated that you did not intend to prohibit its use. Furthermore, the reference to the license has been posted in the discussion for months, yet it is you organizers who have failed to take it seriously.
\nYou organizers certainly have the right to set the rules of the competition, but you have the responsibility to ensure that the competition proceeds smoothly and fairly. It is not right to disregard that responsibility, to change the rules as you please, and to drag the competition into a mess.
\nFor some reason, you seem to be referring to the responsibility of the participants, but it makes no sense to hold them responsible for your irresponsible behavior.
\nThis problem was caused by your failure to fulfill your responsibility to properly manage the competition, and you should take responsibility for it. Even if it is a solution that you do not want, you should accept it because you have left the problem to this point.
Also, realistically speaking, how are you guys going to make sure that the winners' solutions meet the requirements?
\nThe source code is easy to determine by reading it, but the dataset is very difficult to determine if the participant used it or not.
\nIf you were to do it, the only way to do it would be to receive the training scripts and see if you could regenerate the results, but are you really going to do that for all the winners? It takes time to receive the scripts and it also takes time to generate the results. (In extreme cases, if a team emerges claiming to have used a model that has been trained for 3 months, you would have to spend 3 months regenerating and confirming the results, wouldn't you?)
\nAnd if you find out where the cheating is, you have to do it for the teams that are further back in the standings.
\nAre you going to extend the LB finalization until this is completely done?
\nThis is impractical by any stretch of the imagination.
Again, I think it is best for both participants and management to allow the use of the ICDAR dataset since there is the option of retraining.
", "votes": 15, "replies": [ { "id": 2290815, "author_name": "CPMP", "author_url": "", "post_date": "2023-06-07T06:01:32.100000", "content": "I love this:
\n\n\nFor some reason, you seem to be referring to the responsibility of the participants, but it makes no sense to hold them responsible for your irresponsible behavior.
\n
It is what triggered my own response. Host doesn't take responsibility for the issue.
\nI guess it is because their lawyers told them to not leave open any legal action by participants. But the message to all those who worked hard here is devastating.
", "votes": 4, "replies": [] } ] }, { "id": 2290103, "author_name": "moth", "author_url": "", "post_date": "2023-06-06T14:49:37.367000", "content": "Why was the competition extended?
\n\n\nWe are extending the deadline for this competition to June 19, 2023
\n
I know that this might help people to re run experiments without the ICDAR dataset but at least for me I have been in this competition for the past few months and I have other commitments. This means dedicating even more time to the competition and thus I will have to leave some personal matters aside.
", "votes": 15, "replies": [] }, { "id": 2289175, "author_name": "penguin46", "author_url": "", "post_date": "2023-06-06T01:29:31.477000", "content": "\nAre you sure you cannot allow external data? You will be able to re-train your models without the data you are unwilling to use, based on the top solutions submitted after the competition ended. I don't expect that extending the period by a week will significantly change our solution. Then I think it would be better to allow the data for the sake of fairness in the competition.
\nIf you prohibit its use, please do not remove participants from the leaderboard in exchange for not receiving prize. Many participants, including myself, have developed models based on this statement by you. Like @theoviel, I will continue to use external data if I can win a medal by declining the prize money.
", "votes": 15, "replies": [ { "id": 2289299, "author_name": "penguin46", "author_url": "", "post_date": "2023-06-06T04:21:42.107000", "content": "@benjiaa
\nNot clarifying the process regarding illegal submissions breaks the competition and does no one any good, including the administration. Some participants will submit illegal answers with high scores in the hopes that they will not be out of the prize zone or not removed from LB. In that case, you will all lose the opportunity to get better solutions. Participants risk being removed from the LB, and honest teams may not receive a gold medal even though they are in the real top11 with legal submissions.
Again, my best suggestion is to allow external data. In that case, you will not get the weights of the commercially available model wegihts, but you can use the training code submitted after the competition and retrain with the available data, including 4000 samples of test data. You will get better solutions and we will be able to focus on the rest of the competition without doubting your or other participants.
", "votes": 11, "replies": [] } ] }, { "id": 2289302, "author_name": "monnu", "author_url": "", "post_date": "2023-06-06T04:26:04.917000", "content": "I agree with penguin46's suggestion. After the competition, you have the option to retrain your model. This can be done by excluding unwanted data once you've received the top solutions. This method should solve any licensing problems. You've previously made it clear that the use of ICDAR data is permitted. If you are to override this, please clarify why the aforementioned solution wouldn't be applicable.
\nIf the use of ICDAR is to be prohibited, I believe that in order to maintain fairness, not just the submission code, but also the training data and code of everyone who earns a medal after the competition need to be checked. If only the winners of the prize money are checked, there might be individuals breaking the rules to win medals. This could potentially compromise Kaggle's value as a platform for data science competitions.
", "votes": 11, "replies": [] }, { "id": 2293908, "author_name": "YumeNeko", "author_url": "", "post_date": "2023-06-09T15:11:45.933000", "content": "@benjiaa
\nI fully understand that you guys are irresponsible, but since this is at least a competition, can you please state the rules clearly?
\nI believe that many of the participants are currently contacting you guys with the following queries regarding the rules.
Perhaps you don't understand, but what you are organizing is a competition.
\nAs it is a competition, rules are absolutely necessary.
\nIf participants do not understand the rules, they will not know what is allowed and what is not allowed, and they will not be able to decide which tasks to devote their time to.
\nAnyway, please explain properly what the current rules are.
\nIt's a competition that you guys are organizing, so at the very least get that right.
@ryanholbrook @maggiemd
\nCan you guys please make some statement instead of remaining silent with the host.
\nThis host is doing whatever they want, wasting the participants' efforts and makes the participants look like fools.
\nI think they are hurting the value of kaggle greatly. What do you think of this situation?
The rudeness of shifting the blame for their mistakes onto the participants without even apologizing is astonishing, but what's more, they can't even answer a question they were asked a week ago about the rules of the competition they are organizing.
\nBy all accounts, Benetech has no integrity, no common sense, and no qualifications to be a host.
Anyway, this unbelievable host is causing confusion for all participants and making the competition impossible.
\nI think you are at a certain fault because you allowed the competition to be held by an organization with such a high level of problematic behavior.
\nPlease state what the rules are for this competition and how you are going to fix this situation.
\nIsn't this your duty?
For clarity the host did not change the rules. The rules have remained the same. The update made to the rules was to emphasize the point since there was confusion on the forums about what datasets were allowed. Winner's will have to produce information to support the license required. They extended the deadline so that those who may have used a dataset that does not comply can have time to train with new datasets.
\nThe goal of this competition is the same, they are trying to help children with learning disabilities benefit from charts that they cannot currently see. For context, those who decide to host a competition sometimes are learning about these legal issues for the first time. When there was confusion they took a lot of time to research this and spent time to clarify the rules. There was no malicious intent here.
", "votes": 5, "replies": [ { "id": 2294915, "author_name": "Sinan Calisir", "author_url": "", "post_date": "2023-06-10T12:56:47.907000", "content": "If the rules remained the same then why was there a silent change in the data page of the competition? Other questions still remains unanswered:
\n@maggiemd thank you for responding. Let's all try and make the best out of this situation!
\nOne of the main questions people still have is what happens to submissions that are not compliant with the MIT License (both regarding models & datasets)? Will they be removed from the LB? Most competitors are motivated by ranking rather than the money. So people want to be sure if they make select a submission that is fully MIT License compliant, there won't be people in from of them on LB who did not do the same. Using the disallowed dataset or non-compliant models can give you a boost.
\nOur team is working hard to make the final submission compliant with the MIT License mentioned in the rules.
", "votes": 2, "replies": [] }, { "id": 2294996, "author_name": "penguin46", "author_url": "", "post_date": "2023-06-10T13:55:04.140000", "content": "@maggiemd Thank you for breaking the silence!
\nI am reconfiguring the pipeline to be MIT compliant and hope that the competition will end in a way that is acceptable to participants and hosts!
I would especially like to know if participants who do not meet the license will be removed from LB, as the two above have claimed.
\n", "votes": 0, "replies": [] }, { "id": 2295045, "author_name": "YumeNeko", "author_url": "", "post_date": "2023-06-10T14:47:32.373000", "content": "Thank you for your statement.
\nIf the rules haven't changed, then that means that the host was introducing a data set that is not allowed to be used in the competition on the official page for uses that are not permitted, which is annoying and I think they should at least apologize, regardless of whether or not there is any ill intent, but that is probably not productive to argue about now, so I'll leave it at that for now.
\nMore importantly, as you all have said, I would like to know the details of the rules.
\nI will restate the specifics, but I would like an answer to the following question.
To put it more succinctly, the question is what happens if the MIT licensing requirements are not met, and how do you intend to make sure that they are?
\nI know that this type of problem is not limited to this competition. But the major difference is that in this competition, the hosts have long made known the prohibited data sets that they are prohibited from using in the first place. I am very concerned that I will not be disadvantaged by my honest adherence to the licensing requirements.
", "votes": 4, "replies": [] }, { "id": 2295384, "author_name": "CPMP", "author_url": "", "post_date": "2023-06-10T21:46:28.103000", "content": "@maggiemd Thanks for responding.
\nIndeed, the rules stated upfront that winners must provide open source license suited for commercial use, as in many other kaggle competitions (but not al).
\nWhat infuriates (rightly if you ask me) many participants is that the data page recommended a dataset incompatible with the rules, and that it took 5 or 6 weeks to answer the questions that participants raised about this.
\nAnd when the response comes, it is as if the mistake had been done by participants. Participants were labelled as irresponsible for using a non compliant dataset.
\nThe only mistake participants made was to have trusted the host. They assumed that the host recommended a dataset compliant with the rules of the competition.
\nSome apology instead of finger pointing in the original post would have been much appreciated I guess. It would not change the underlying issue, but it would show some consideration.
", "votes": 7, "replies": [] }, { "id": 2297276, "author_name": "", "author_url": "", "post_date": "2023-06-12T13:27:44.870000", "content": "", "votes": 0, "replies": [] } ] } ] }, { "id": 2292526, "author_name": "june", "author_url": "", "post_date": "2023-06-08T12:16:38.847000", "content": "It took about 3 months for them to figure out which license is allowed in this competition.
\nHow quickly would you expect them to answer all questions and figure out what to do next?!
If that's the case, I suggest a thorough investigation into whether the participants in the gold medal zone, and even the silver zone, have used external data which is mentioned above. It would be challenging to enforce each participant to voluntarily disclose their training code. Alternatively, it would be necessary to develop an algorithm that can automatically detect if participants have used the aforementioned database. Otherwise, it would be unfair to other participants.
", "votes": 10, "replies": [] }, { "id": 2292066, "author_name": "Raja Biswas", "author_url": "", "post_date": "2023-06-08T03:52:50.250000", "content": "IMO Kaggle's \"moat\" is its amazing community. Not difficult to imagine impact of acts like sneakily updating the data /rules page, ineffective communication, believing in 'Silence Is All You Need' etc…
", "votes": 8, "replies": [] }, { "id": 2291919, "author_name": "monnu", "author_url": "", "post_date": "2023-06-08T00:13:42.673000", "content": "\nCould you please provide a response to participants' inquiries as promptly as possible, regardless of whether you choose to continue prohibiting the use of ICDAR data or not? I would like to reiterate my concerns:
\nIf you are currently contemplating these issues, please make it known. Remaining silent only serves to sow confusion among the participants.
", "votes": 7, "replies": [ { "id": 2292019, "author_name": "", "author_url": "", "post_date": "2023-06-08T03:08:14.153000", "content": "", "votes": 0, "replies": [ { "id": 2292259, "author_name": "Hamish", "author_url": "", "post_date": "2023-06-08T06:52:51.600000", "content": "I think realistically their lawyer has almost certainly told them not to say anything. It's a rubbish way for this competition to end tbh
", "votes": 3, "replies": [] } ] } ] }, { "id": 2291426, "author_name": "esprit", "author_url": "", "post_date": "2023-06-07T14:45:06.790000", "content": "I have not used the ICDAR dataset until now because I am wary of the licensing requirements. As such, I can offer an objective opinion.
\nOriginally, the host recommended the ICDAR dataset to the participants. As a result, all participants were divided into two groups: those who used the ICDAR dataset and those who did not. Neither group had any intention of violating the rules, of course.
\nOn the other hand, it was the host who recommended the ICDAR dataset to the participants, and it is also the host who says that the ICDAR dataset is prohibited. Regardless of these conflicting action and statement, all participants have made a good faith effort to comply with the rules.
\nIt is the host, and no one else, who has led the participants into unintentional violations of the rules. It is the natural \"responsibility\" as the host to fairly reward both those who used the ICDAR dataset and those who did not.
", "votes": 7, "replies": [] }, { "id": 2289555, "author_name": "Sinan Calisir", "author_url": "", "post_date": "2023-06-06T08:31:24.580000", "content": "Hi @benjiaa ,
\nHow do you plan to distinguish the teams that are not in the prize zone but used an external dataset that does not comply with this rules?
", "votes": 7, "replies": [] }, { "id": 2292867, "author_name": "penguin46", "author_url": "", "post_date": "2023-06-08T17:34:02.267000", "content": "A month ago, I had the option of using the dataset recommended in the dataset tab without telling you about the license. But in that case, you may not have been able to acquire the winner's solution for commercial use, and participants who did not question your recommendations may have been removed from LB. That is why I contacted you to ensure that the competition ended in a mutually satisfactory manner. I did not expect to be betrayed in this way.
", "votes": 8, "replies": [] }, { "id": 2292007, "author_name": "YumeNeko", "author_url": "", "post_date": "2023-06-08T03:01:50.920000", "content": "@benjiaa
\nWhy not respond in some way first?
\nAs everyone has said, your decision to remain silent after so many inquiries is incredibly irresponsible.
I would also suggest another constructive option besides retraining after the competition is over.
\nSomething similar has already been said by some participants, but a way to add a rule that says, \"participants who are not able to fulfill this license requirement may retain their position on the leaderboard, but will not receive a prize\".
Such a rule was taken in IMC2022, so it should not be impossible. By the way, in IMC2022 competition we could not receive prize money for using source code that was not available for commercial use, but we did receive a medal and we provided all the source code from us. Also, if I remember correctly, the prize slots were moved to a lower level and the teams that met the requirements would also have provided source code to the host.
\nAlso, if you are concerned that allowing the use of the dataset will discourage anyone from submitting clean solutions, I think that is unfounded.
\nIn fact, there are participants like @takuji have achieved excellent scores without using this dataset, and I believe there was also a team at IMC2022 that won a prize in the Gold Zone.
This way, the participants would not have had their efforts wasted and you would have both a highly accurate solution using the ICDAR dataset and a clean solution not using the ICDAR dataset. Participants who did not use the ICDAR dataset would also get a quid pro quo in the form of prize money.
\nMoreover, you can check if the prize money requirements have been met at your leisure, independently of the finalization of the LB, so this will not be a major obstacle to the management of the competition.
Isn't this a win-win way for all?
", "votes": 6, "replies": [] }, { "id": 2290470, "author_name": "Cody_Null", "author_url": "", "post_date": "2023-06-06T20:24:54.627000", "content": "It is incredibly frustrating that, after a significant systemic change is implemented, now 24 hours later, we are still unable to obtain answers to basic questions in a competition that has been running for three months. And to do this right before the original deadline is ridiculous. Moreover, this entire situation could have been easily avoided if proper research had been conducted prior to the competition's start.
", "votes": 5, "replies": [] }, { "id": 2291014, "author_name": "penguin46", "author_url": "", "post_date": "2023-06-07T09:02:44.540000", "content": "@benjiaa
\nDo you intend to respond in any way? Or will you remain silent?
I have another suggestion. In the past, a competition was ruined by incidents for which the participants were not responsible. In that case, the host (or kaggle team?) paid additional prize money to the team that should have won. If you want a clean model without re-training, you can do something similar.
\nIn other words, apart from the regular LB with external data, you can create another LB that is aggregated only with submissions that meet the license, and negotiate with the winning team to buy the model for the appropriate amount. The submissions to be used for aggregation should be declared in advance. It would cost you extra, but you would get a license compliant model.
", "votes": 6, "replies": [ { "id": 2291122, "author_name": "Theo Viel", "author_url": "", "post_date": "2023-06-07T10:41:56.677000", "content": "Foursquare was different, the mistake was on Kaggle's side.
\n@ryotayoshinobu I think the best option for high-ranked participants is to rerun their pipeline without the rule-infringing components. That's what we've been doing since the announcement.
\nWe're observing performance drops but if everyone at the top abides by the same rules, the competition should still be fair.
Yes, certainly the situation may be different from foursquare.
\nAll I was trying to say here is that the host has an option to buy the winner's solution directly.
Thanks for the advice on the approach for the remaining period. I too think that is a wise decision, and I have already started creating a clean pipeline as you.
", "votes": 2, "replies": [] } ] }, { "id": 2291172, "author_name": "", "author_url": "", "post_date": "2023-06-07T11:28:36.313000", "content": "", "votes": 1, "replies": [] } ] }, { "id": 2290234, "author_name": "qdv206", "author_url": "", "post_date": "2023-06-06T16:33:25.827000", "content": "As others have said, disallowing the use of ICDAR this late makes it almost impossible to maintain the integrity and fairness of this competition. If the host doesn't want to deal with license issue for commercial use, they could retrain the winning solutions and replace ICDAR data with the 4000 extracted images they have in the test set. The result will be an even better model I imagine.
", "votes": 6, "replies": [] }, { "id": 2289100, "author_name": "Man of the year", "author_url": "", "post_date": "2023-06-05T23:17:43.373000", "content": "Hello, can we use the ICDAR dataset, if we do not want the money prize and just would like to win a medal?
", "votes": 6, "replies": [] }, { "id": 2295619, "author_name": "Zifeng Li", "author_url": "", "post_date": "2023-06-11T06:40:55.333000", "content": "The entire model (or backbone) of many text detection/recognition model are trained on ImageNet or SynthText. However, ImageNet(https://www.image-net.org/) and SynthText (https://www.robots.ox.ac.uk/~vgg/terms/dataset-group-2-access.html) does not allow commercial use ❌.
\nDoes that mean I can not fine-tune any of these pre-trained OCR models or use the ResNet backbones in this competition?
", "votes": 4, "replies": [] }, { "id": 2289382, "author_name": "Mengfei Li", "author_url": "", "post_date": "2023-06-06T05:47:56.967000", "content": "Hi, BENJIAA, how about this dataset which was created by BARTLEY and published on Kaggle? https://www.kaggle.com/datasets/brendanartley/benetech-extra-generated-data
", "votes": 1, "replies": [ { "id": 2290056, "author_name": "Bartley", "author_url": "", "post_date": "2023-06-06T14:15:04.513000", "content": "I would like to make this dataset open for use in this competition. @benjiaa
\nI have no idea which license to use on the dataset (as I do not see MIT in the dropdown menu). Does anyone have any idea which license to use here?
", "votes": 1, "replies": [ { "id": 2290086, "author_name": "moth", "author_url": "", "post_date": "2023-06-06T14:37:55.580000", "content": "@brendanartley I think you can add it under the License Card and use CCO which I think it's the most open and allows commercial use. I am not a legal expert though but I believe that would be just fine.
Sorry Data Card -> License -> CCO
Thanks @alejopaullier. I will update the dataset with this for now.
\nCan the organizers confirm this please?
", "votes": 0, "replies": [] }, { "id": 2290387, "author_name": "Mengfei Li", "author_url": "", "post_date": "2023-06-06T18:41:57.080000", "content": "I don't think it is a problem. It is actually a dataset created by your notebook. Everyone can just create the data instead of download them if they feel somehow confused.
", "votes": 1, "replies": [] } ] } ] }, { "id": 2297193, "author_name": "YumeNeko", "author_url": "", "post_date": "2023-06-12T12:12:47.403000", "content": "@benjiaa
\nWe have one week left in this competition and I was wondering if you could quickly answer some of the questions that have been submitted by the participants.
I am not asking you to change the rules, I am asking you to give me the details of the rules of the competition you are organizing, what could be so difficult that it would take a week?
\nAlso, since there is a new question about whether the pre-train model can be used, can I ask you to answer it properly?
\nIf you guys are really concerned about a clean license, then basically all pre-train models cannot be used. In that case, you guys would only be entitled to pay the prize money and receive solutions with a level of accuracy that you can never use in practical applications. If that is what you want, I will say nothing more and you can do whatever you want, but I am sure that this is the worst-case scenario that will benefit no one.
\nOn the other hand, if you allow pre-train models because you only want to refer to how to do the solution, etc., then I think you should allow the use of the ICDAR dataset because it is essentially the same thing.
\nAt least in this rule, I don't think there is any convenient way to allow only pre-train models and prohibit ICDAR datasets.
\nUnless, you are going to take the outlaw stance of only allowing the use of pre-train models, even though you know it is illegal to do so.
I don't think you would do such a thing against justice, but if you are going to make such a decision, you should explain well how you intend to use the solution and why you decided to allow only the pre-train model so that participants are not mistakenly at risk of being involved in a lawsuit.
\nAs for the license, you are asking us, the participants, to take responsibility, but there is no way for outsiders to know what is allowed and to what extent within the organizer, and it is impossible for us to guess at every single thing. It is too late now, but at the very least, as hosts, you should fulfill your responsibility to explain.
\nFrankly, I don't care what conclusion you guys come to, as I no longer expect things to get better, but at least the host should give a proper explanation.
\nWithout your proper explanation, I have no idea what to believe in this competition.
@ryanholbrook @maggiemd
\nAlso, to the Kaggle team, this issue is happening on your platform, so please work with the host to take appropriate action.
In my humble opinion, the focus on improving model performance may depend not only on the type of external data used, but on the quality of the external data closely aligned with the LB data. If this is true, then substituting a batch of re-simulated LB data can avoid the advantage of some specific external data.
", "votes": 0, "replies": [] }, { "id": 2295606, "author_name": "", "author_url": "", "post_date": "2023-06-11T06:20:10.730000", "content": "", "votes": 0, "replies": [] }, { "id": 2293959, "author_name": "", "author_url": "", "post_date": "2023-06-09T15:54:38.263000", "content": "", "votes": 0, "replies": [] }, { "id": 2289504, "author_name": "", "author_url": "", "post_date": "2023-06-06T07:46:05.143000", "content": "", "votes": 0, "replies": [] }, { "id": 2289456, "author_name": "", "author_url": "", "post_date": "2023-06-06T07:05:43.503000", "content": "", "votes": 0, "replies": [] }, { "id": 2289208, "author_name": "", "author_url": "", "post_date": "2023-06-06T02:34:10.813000", "content": "", "votes": 0, "replies": [] } ], "raw_markdown_by_id": { "2289942": "You speak about the responsibility of contestants, but what about the responsibility of competition host? Precisely because of the possibility of a remuneration.\n\nThat the data page recommended a dataset incompatible with the rules is irresponsible, to stay polite. Couldn't you do your homework correctly before competition launch? \n\nI don't think you measure the amount of work that kagglers have spent on this (as in every competition) for free. Nullifying that work now is just unbelievable. I see two interesting possible outcomes here:\n\n- Maybe kagglers should boycott this competition and refuse to provide any winner material.\n- Alternatively, it would be fun if prize winners that used that dataset sue to get their money. What would bear more weight in court: rules or data pages?\n", "2289041": "Hi @benjiaa !\n\nI have 2 questions : \n- Can we use data / models under the GPL 3 license be used ? Such license **allows** for commercial use but is more restrictive than MIT / Apache.\n- Will users in the prize area using datasets that make them ineligible for prizes be removed from the leaderboard ? Asked by @crodoc. \n\nThose are important questions that require answers !\n\nTo be completely transparent : I'd rather decline the prize than spend the remaining of the competition re-doing what I did the past month. But I also don't want to get removed from LB as this will nullify our efforts.", "2290538": "@benjiaa \nIn all likelihood, it is too late to decide to prohibit the use of the ICDAR dataset now, and the fairness of the competition will be compromised. As others have said, the use of the dataset should be allowed since there is an option to retrain after the end of the competition.\nIn the first place, this dataset was initially introduced by you guys in the data tab with a recommendation for external use (although for some reason you sneakily removed all mention of it again). And, you also explicitly stated that you did not intend to prohibit its use. Furthermore, the reference to the license has been posted in the discussion for months, yet it is you organizers who have failed to take it seriously.\nYou organizers certainly have the right to set the rules of the competition, but you have the responsibility to ensure that the competition proceeds smoothly and fairly. It is not right to disregard that responsibility, to change the rules as you please, and to drag the competition into a mess.\nFor some reason, you seem to be referring to the responsibility of the participants, but it makes no sense to hold them responsible for your irresponsible behavior.\nThis problem was caused by your failure to fulfill your responsibility to properly manage the competition, and you should take responsibility for it. Even if it is a solution that you do not want, you should accept it because you have left the problem to this point.\n\n\nAlso, realistically speaking, how are you guys going to make sure that the winners' solutions meet the requirements?\nThe source code is easy to determine by reading it, but the dataset is very difficult to determine if the participant used it or not.\nIf you were to do it, the only way to do it would be to receive the training scripts and see if you could regenerate the results, but are you really going to do that for all the winners? It takes time to receive the scripts and it also takes time to generate the results. (In extreme cases, if a team emerges claiming to have used a model that has been trained for 3 months, you would have to spend 3 months regenerating and confirming the results, wouldn't you?)\nAnd if you find out where the cheating is, you have to do it for the teams that are further back in the standings.\nAre you going to extend the LB finalization until this is completely done?\nThis is impractical by any stretch of the imagination. \n\n\nAgain, I think it is best for both participants and management to allow the use of the ICDAR dataset since there is the option of retraining.", "2290103": "Why was the competition extended?\n> We are extending the deadline for this competition to June 19, 2023\n\nI know that this might help people to re run experiments without the ICDAR dataset but at least for me I have been in this competition for the past few months and I have other commitments. This means dedicating even more time to the competition and thus I will have to leave some personal matters aside.", "2289175": "@benjiaa \n\nAre you sure you cannot allow external data? You will be able to re-train your models without the data you are unwilling to use, based on the top solutions submitted after the competition ended. I don't expect that extending the period by a week will significantly change our solution. Then I think it would be better to allow the data for the sake of fairness in the competition.\n\nIf you prohibit its use, please do not remove participants from the leaderboard in exchange for not receiving prize. Many participants, including myself, have developed models based on [this statement](https://www.kaggle.com/competitions/benetech-making-graphs-accessible/discussion/407485#2264697) by you. Like @theoviel, I will continue to use external data if I can win a medal by declining the prize money.\n\n\n", "2289302": "I agree with penguin46's suggestion. After the competition, you have the option to retrain your model. This can be done by excluding unwanted data once you've received the top solutions. This method should solve any licensing problems. You've previously made it clear that the use of ICDAR data is permitted. If you are to override this, please clarify why the aforementioned solution wouldn't be applicable.\n\nIf the use of ICDAR is to be prohibited, I believe that in order to maintain fairness, not just the submission code, but also the training data and code of everyone who earns a medal after the competition need to be checked. If only the winners of the prize money are checked, there might be individuals breaking the rules to win medals. This could potentially compromise Kaggle's value as a platform for data science competitions.", "2293908": "@benjiaa \nI fully understand that you guys are irresponsible, but since this is at least a competition, can you please state the rules clearly?\nI believe that many of the participants are currently contacting you guys with the following queries regarding the rules.\n * Can a participant refuse to accept a prize?\n * If a participant can refuse to win a prize, will they keep their position on the leaderboard even if the solution they used does not meet the requirements for a prize?\n * If you intend to remove participants from the LB who cannot be denied the prize and do not meet the requirements, how do you determine if the solution used meets the prize requirements?\n(It is quite difficult to determine if a dataset was used or not, but do you have a way to ensure that you play the cheating participant without creating false accusations?)\n\nPerhaps you don't understand, but what you are organizing is a competition.\nAs it is a competition, rules are absolutely necessary.\nIf participants do not understand the rules, they will not know what is allowed and what is not allowed, and they will not be able to decide which tasks to devote their time to.\nAnyway, please explain properly what the current rules are.\nIt's a competition that you guys are organizing, so at the very least get that right.\n\n@ryanholbrook @maggiemd\nCan you guys please make some statement instead of remaining silent with the host.\nThis host is doing whatever they want, wasting the participants' efforts and makes the participants look like fools.\nI think they are hurting the value of kaggle greatly. What do you think of this situation?\n\nThe rudeness of shifting the blame for their mistakes onto the participants without even apologizing is astonishing, but what's more, they can't even answer a question they were asked a week ago about the rules of the competition they are organizing.\nBy all accounts, Benetech has no integrity, no common sense, and no qualifications to be a host.\n\nAnyway, this unbelievable host is causing confusion for all participants and making the competition impossible.\nI think you are at a certain fault because you allowed the competition to be held by an organization with such a high level of problematic behavior.\nPlease state what the rules are for this competition and how you are going to fix this situation.\nIsn't this your duty?", "2292526": "It took about 3 months for them to figure out which license is allowed in this competition.\nHow quickly would you expect them to answer all questions and figure out what to do next?!", "2289393": "If that's the case, I suggest a thorough investigation into whether the participants in the gold medal zone, and even the silver zone, have used external data which is mentioned above. It would be challenging to enforce each participant to voluntarily disclose their training code. Alternatively, it would be necessary to develop an algorithm that can automatically detect if participants have used the aforementioned database. Otherwise, it would be unfair to other participants.", "2292066": "IMO Kaggle's \"moat\" is its amazing community. Not difficult to imagine impact of acts like sneakily updating the data /rules page, ineffective communication, believing in 'Silence Is All You Need' etc...", "2291919": "@benjiaa \n\nCould you please provide a response to participants' inquiries as promptly as possible, regardless of whether you choose to continue prohibiting the use of ICDAR data or not? I would like to reiterate my concerns:\n\n* Would it not be acceptable to allow the use of ICDAR data and then have the host retrain the data after the competition's conclusion?\n* In the event that you do not allow this, how do you plan on verifying compliance with the rules?\n * As previously expressed by others, it does not seem feasible to reliably determine whether a submission violates these rules.\n * If the proper monitoring of submissions isn't possible, it is unreasonable to expect all participants to develop a rule-abiding solution.\n\nIf you are currently contemplating these issues, please make it known. Remaining silent only serves to sow confusion among the participants.", "2291426": "I have not used the ICDAR dataset until now because I am wary of the licensing requirements. As such, I can offer an objective opinion.\n\nOriginally, the host recommended the ICDAR dataset to the participants. As a result, all participants were divided into two groups: those who used the ICDAR dataset and those who did not. Neither group had any intention of violating the rules, of course.\n\nOn the other hand, it was the host who recommended the ICDAR dataset to the participants, and it is also the host who says that the ICDAR dataset is prohibited. Regardless of these conflicting action and statement, all participants have made a good faith effort to comply with the rules.\n\nIt is the host, and no one else, who has led the participants into unintentional violations of the rules. It is the natural \"responsibility\" as the host to fairly reward both those who used the ICDAR dataset and those who did not.", "2289555": "Hi @benjiaa ,\n\nHow do you plan to distinguish the teams that are not in the prize zone but used an external dataset that does not comply with this rules?", "2292867": "A month ago, I had the option of using the dataset recommended in the dataset tab without telling you about the license. But in that case, you may not have been able to acquire the winner's solution for commercial use, and participants who did not question your recommendations may have been removed from LB. That is why I contacted you to ensure that the competition ended in a mutually satisfactory manner. I did not expect to be betrayed in this way.", "2292007": "@benjiaa \nWhy not respond in some way first?\nAs everyone has said, your decision to remain silent after so many inquiries is incredibly irresponsible.\n\nI would also suggest another constructive option besides retraining after the competition is over.\nSomething similar has already been said by some participants, but a way to add a rule that says, \"participants who are not able to fulfill this license requirement may retain their position on the leaderboard, but will not receive a prize\".\n\nSuch a rule was taken in [IMC2022](https://www.kaggle.com/competitions/image-matching-challenge-2022/rules), so it should not be impossible. By the way, in IMC2022 competition we could not receive prize money for using source code that was not available for commercial use, but we did receive a medal and we provided all the source code from us. Also, if I remember correctly, the prize slots were moved to a lower level and the teams that met the requirements would also have provided source code to the host.\nAlso, if you are concerned that allowing the use of the dataset will discourage anyone from submitting clean solutions, I think that is unfounded.\nIn fact, there are participants like @takuji have achieved excellent scores without using this dataset, and I believe there was also a team at IMC2022 that won a prize in the Gold Zone.\n\nThis way, the participants would not have had their efforts wasted and you would have both a highly accurate solution using the ICDAR dataset and a clean solution not using the ICDAR dataset. Participants who did not use the ICDAR dataset would also get a quid pro quo in the form of prize money.\nMoreover, you can check if the prize money requirements have been met at your leisure, independently of the finalization of the LB, so this will not be a major obstacle to the management of the competition.\n\nIsn't this a win-win way for all?", "2290470": "It is incredibly frustrating that, after a significant systemic change is implemented, now 24 hours later, we are still unable to obtain answers to basic questions in a competition that has been running for three months. And to do this right before the original deadline is ridiculous. Moreover, this entire situation could have been easily avoided if proper research had been conducted prior to the competition's start.", "2291014": "@benjiaa \nDo you intend to respond in any way? Or will you remain silent?\n\nI have another suggestion. In the past, [a competition](https://www.kaggle.com/competitions/foursquare-location-matching/discussion/338035) was ruined by incidents for which the participants were not responsible. In that case, the host (or kaggle team?) paid additional prize money to the team that should have won. If you want a clean model without re-training, you can do something similar.\n\nIn other words, apart from the regular LB with external data, you can create another LB that is aggregated only with submissions that meet the license, and negotiate with the winning team to buy the model for the appropriate amount. The submissions to be used for aggregation should be declared in advance. It would cost you extra, but you would get a license compliant model.", "2290234": "As others have said, disallowing the use of ICDAR this late makes it almost impossible to maintain the integrity and fairness of this competition. If the host doesn't want to deal with license issue for commercial use, they could retrain the winning solutions and replace ICDAR data with the 4000 extracted images they have in the test set. The result will be an even better model I imagine.", "2289100": "Hello, can we use the ICDAR dataset, if we do not want the money prize and just would like to win a medal?", "2295619": "The entire model (or backbone) of many text detection/recognition model are trained on ImageNet or SynthText. However, ImageNet(https://www.image-net.org/) and SynthText (https://www.robots.ox.ac.uk/~vgg/terms/dataset-group-2-access.html) does not allow commercial use ❌. \n\nDoes that mean I can not fine-tune any of these pre-trained OCR models or use the ResNet backbones in this competition?", "2289382": "Hi, BENJIAA, how about this dataset which was created by BARTLEY and published on Kaggle? [https://www.kaggle.com/datasets/brendanartley/benetech-extra-generated-data](url)", "2297193": "@benjiaa \nWe have one week left in this competition and I was wondering if you could quickly answer some of the questions that have been submitted by the participants.\n\n* Can we refuse a prize?\n* If we cannot refuse a prize, how will submissions that do not meet the licensing requirements be handled? (Will the team be removed from the LB?).\n* How do you check to see if the licensing requirements have been met?\n\nI am not asking you to change the rules, I am asking you to give me the details of the rules of the competition you are organizing, what could be so difficult that it would take a week?\n\n\nAlso, since there is a new question about whether the pre-train model can be used, can I ask you to answer it properly?\n\nIf you guys are really concerned about a clean license, then basically all pre-train models cannot be used. In that case, you guys would only be entitled to pay the prize money and receive solutions with a level of accuracy that you can never use in practical applications. If that is what you want, I will say nothing more and you can do whatever you want, but I am sure that this is the worst-case scenario that will benefit no one.\n\nOn the other hand, if you allow pre-train models because you only want to refer to how to do the solution, etc., then I think you should allow the use of the ICDAR dataset because it is essentially the same thing.\nAt least in this rule, I don't think there is any convenient way to allow only pre-train models and prohibit ICDAR datasets.\nUnless, you are going to take the outlaw stance of only allowing the use of pre-train models, even though you know it is illegal to do so.\n\nI don't think you would do such a thing against justice, but if you are going to make such a decision, you should explain well how you intend to use the solution and why you decided to allow only the pre-train model so that participants are not mistakenly at risk of being involved in a lawsuit.\n\nAs for the license, you are asking us, the participants, to take responsibility, but there is no way for outsiders to know what is allowed and to what extent within the organizer, and it is impossible for us to guess at every single thing. It is too late now, but at the very least, as hosts, you should fulfill your responsibility to explain.\n\nFrankly, I don't care what conclusion you guys come to, as I no longer expect things to get better, but at least the host should give a proper explanation.\nWithout your proper explanation, I have no idea what to believe in this competition.\n\n@ryanholbrook @maggiemd\nAlso, to the Kaggle team, this issue is happening on your platform, so please work with the host to take appropriate action.", "2289025": "Contestants are encouraged to use the Competition Data as training data set(s). The use of a third-party data set, not provided on the Competition Website, is contestant’s choice; however, data sets are subject to open source or other licenses, with varying restrictions. It is the responsibility of each contestant to ensure that, given the possibility of remuneration, contestant understands any existing restrictions on the data set(s) employed. Benetech’s legal team has clarified that Benetech cannot, and will not, provide legal advice to contestants.\n\nIt should be noted that submissions using datasets that are not compliant with the original Rule 8.C. (now, Section A.3), such as datasets licensed under CC BY-NC-SA 4.0, will not be accepted due to incompatibility with the Competition Rules, which clearly specify Benetech’s anticipation of commercial use.\n \nWe are extending the deadline for this competition to June 19, 2023.", "2290797": "In my humble opinion, the focus on improving model performance may depend not only on the type of external data used, but on the quality of the external data closely aligned with the LB data. If this is true, then substituting a batch of re-simulated LB data can avoid the advantage of some specific external data.", "2295606": "", "2293959": "", "2289504": "", "2289456": "", "2289208": "" } }