Per some requests, the following should provide additional clarification about the rules of this competition.
\n\n\n\n\nThis competition is indeed designed for GANs. However, Kaggle doesn't want to limit the solution algorithms one can use to solve a problem. For example, VAE methods such as this paper seem very promising, and we would love to see if they perform better than GANs! Therefore, we didn't want to specifically limit participants to use GANs in our official rules. However, we do want to discourage the use of \"altering training images\" - see #2.
\n
\n\n\nAs far as we know, the research community doesn’t have one universal metric for evaluating generative images (see this survey paper); furthermore, even when Kaggle adds a memorization component to the existing metrics, it still can’t catch all the combinations of image processing one can apply to the training images. That being said, if an image isn't generated from scratch, we would consider your method an \"altered version\" of images. In particular, your trained generator model should be able to stand alone and produce images without the input of training images. For example, generating some weights and blending training images, or do some sort of filtering to the training images, are not allowed. Think of it as a student going into an exam that doesn't allow bringing anything with you - no books, no photos, no phones, etc. This is not an open-book exam, therefore if you submit anything that seems like a copy from your textbook, we would consider that cheating.
\n
\n", "messages": [ { "id": 566105, "postDate": "2019-07-01T19:50:38Z", "content": "We are trying something completely new here. It's not easy to balance between innovation and fairness. Therefore, we ask to reserve the right to change things up during the competition. In a way, it's just like GANs, the discriminator has to train itself in each round when the generator generated more images.
\n
Per some requests, the following should provide additional clarification about the rules of this competition.
\n\n\n\n\nThis competition is indeed designed for GANs. However, Kaggle doesn't want to limit the solution algorithms one can use to solve a problem. For example, VAE methods such as this paper seem very promising, and we would love to see if they perform better than GANs! Therefore, we didn't want to specifically limit participants to use GANs in our official rules. However, we do want to discourage the use of \"altering training images\" - see #2.
\n
\n\n\nAs far as we know, the research community doesn’t have one universal metric for evaluating generative images (see this survey paper); furthermore, even when Kaggle adds a memorization component to the existing metrics, it still can’t catch all the combinations of image processing one can apply to the training images. That being said, if an image isn't generated from scratch, we would consider your method an \"altered version\" of images. In particular, your trained generator model should be able to stand alone and produce images without the input of training images. For example, generating some weights and blending training images, or do some sort of filtering to the training images, are not allowed. Think of it as a student going into an exam that doesn't allow bringing anything with you - no books, no photos, no phones, etc. This is not an open-book exam, therefore if you submit anything that seems like a copy from your textbook, we would consider that cheating.
\n
\n", "rawMarkdown": "Per some requests, the following should provide additional clarification about the rules of this competition.\n\n- Why do you say \"You'll be using GANs\" on the front page, but the rules say \"only generative models are allowed\"? \n\n> This competition is indeed designed for GANs. However, Kaggle doesn't want to limit the solution algorithms one can use to solve a problem. For example, VAE methods such as [this paper](https://arxiv.org/pdf/1906.00446.pdf) seem very promising, and we would love to see if they perform better than GANs! Therefore, we didn't want to specifically limit participants to use GANs in our official rules. However, we do want to discourage the use of \"altering training images\" - see #2. \n\n- What are \"altered versions\" of images? In particular is the blending method used in [this Kernel](https://www.kaggle.com/jazivxt/imitation-game) allowed?\n\n> As far as we know, the research community doesn’t have one universal metric for evaluating generative images (see [this survey paper](https://arxiv.org/pdf/1906.00446.pdf)); furthermore, even when Kaggle adds a memorization component to the existing metrics, it still can’t catch all the combinations of image processing one can apply to the training images. That being said, if an image isn't generated *from scratch*, we would consider your method an \"altered version\" of images. In particular, your *trained generator* model should be able to stand alone and produce images *without* the input of training images. For example, generating some weights and blending training images, or do some sort of filtering to the training images, are **not** allowed. Think of it as a student going into an exam that doesn't allow bringing anything with you - no books, no photos, no phones, etc. This is **not an open-book exam**, therefore if you submit anything that seems like a copy from your textbook, we would consider that cheating. \n\n- Why are you so vague about the rules? \n\n> We are trying something completely new here. It's not easy to balance between innovation and fairness. Therefore, we ask to reserve the right to change things up during the competition. In a way, it's just like GANs, the discriminator has to train itself in each round when the generator generated more images. \n\n\n\n", "votes": 49 }, { "id": 571105, "postDate": "2019-07-09T07:02:37.183Z", "content": "We are trying something completely new here. It's not easy to balance between innovation and fairness. Therefore, we ask to reserve the right to change things up during the competition. In a way, it's just like GANs, the discriminator has to train itself in each round when the generator generated more images.
\n
Could you please clarify whether memorizer GAN like this from Chris is allowed in this competition?
", "rawMarkdown": "Could you please clarify whether memorizer GAN like [this from Chris](https://www.kaggle.com/cdeotte/dog-memorizer-gan) is allowed in this competition?", "votes": 10, "replies": [ { "id": 571434, "postDate": "2019-07-09T15:30:48.923Z", "content": "Yes, clarification about this would be helpful. To help illuminate the situation, I explain where a Memorization GAN fits into the world of GANs here. If Memorization GANs are disallowed, are CGANs with Feature Matching allowed if we restrict the number of classes, and match features of layers closer to the output layer?
", "rawMarkdown": "Yes, clarification about this would be helpful. To help illuminate the situation, I explain where a Memorization GAN fits into the world of GANs [here][1]. If Memorization GANs are disallowed, are CGANs with Feature Matching allowed if we restrict the number of classes, and match features of layers closer to the output layer?\n\n[1]: https://www.kaggle.com/c/generative-dog-images/discussion/99215", "votes": 6 }, { "id": 572299, "postDate": "2019-07-10T18:42:29.607Z", "content": "This is a gray area and I had to think about it for a while. EDITED 8/9/2019: Memorization GAN is not allowed. Please don't select them in your final submissions.
", "rawMarkdown": "This is a gray area and I had to think about it for a while. EDITED 8/9/2019: Memorization GAN is not allowed. Please don't select them in your final submissions. ", "votes": 8 }, { "id": 572384, "postDate": "2019-07-10T21:17:03.897Z", "content": "thank you so much @wendykan we needed that! :)
", "rawMarkdown": "thank you so much @wendykan we needed that! :) ", "votes": 1 }, { "id": 572476, "postDate": "2019-07-11T01:32:18.010Z", "content": "Btw, For now, I think they are allowed For now? Do you mean that you can say at any moment that those methods are no longer allowed? For example 1 week out?
@wendykan please, could you explain this points:\n1. \"by memorizing the training examples you actually might overfit to the training sample distribution\"\n2. \"the memorization cosine distance still applies in the private leaderboard\"
\n\nDo you mean, that on Private leaderboard training examples are not used for FID calculation, but used for calculating the memoization penalty?
", "rawMarkdown": "@wendykan please, could you explain this points:\n1. \"by memorizing the training examples you actually **might overfit to the training** sample distribution\"\n2. \"the **memorization** cosine distance still applies in the **private leaderboard**\"\n\nDo you mean, that on Private leaderboard training examples are not used for FID calculation, but used for calculating the memoization penalty?", "votes": 1 }, { "id": 572664, "postDate": "2019-07-11T08:34:35.963Z", "content": "I hope that we can receive a final decision regarding in this concern as soon as possible. We are a month to go before the final submission.
", "rawMarkdown": "I hope that we can receive a final decision regarding in this concern as soon as possible. We are a month to go before the final submission.", "votes": 3 }, { "id": 572813, "postDate": "2019-07-11T12:32:36.983Z", "content": "Correct me if I'm wrong but:
\n\nConsidering that the metric is made to evaluate how realistic our dogs look, I believe that memorized dogs will score okay whichever model or dog dataset you choose. By okay I mean better than most legit GANs. I'd love to be proven wrong though.
As far as I understood memorization GANs, they are useless for the task that is asked here. I mean what is the point of outputting a memorized image, just output the initial image.
Therefore, I find the decision to allow such models really absurd. To quote the competition description : \n> If successful, not only will you help advance the state of the art in generative image creation, but you’ll enable us to create more experiments across a variety of domains in the future.
\n\nMemorizers don't help image generation state of the art, what is the point of encouraging the use of such models? Given that medals & prize reward successful participants, I believe they should reward approaches that are not trying to trick the metric but trying to actually generate good images.
\n\nFurthermore, there are a lot of people that focused on using real GANs, invested a lot of time, and it is very disappointing to realize that people who forked @cdeotte's great work could in fact end up well above them in the final LB.
\n\nI mean, just don't allow memorizers GANs and if the problem is to detect a memorizer GAN, there are a couple of things that should work : \n- Make sure that the noise is sampled randomly. If using cgans, the number of classes should not be larger than the number of breeds. \n- Make sure feature matching is made on one of the last layers, not on the first/second one
", "rawMarkdown": "Correct me if I'm wrong but: \n\n- Considering that the metric is made to evaluate how realistic our dogs look, I believe that memorized dogs will score okay whichever model or dog dataset you choose. By okay I mean better than most legit GANs. I'd love to be proven wrong though.\n\n- As far as I understood memorization GANs, they are useless for the task that is asked here. I mean what is the point of outputting a memorized image, just output the initial image. \n\nTherefore, I find the decision to allow such models really absurd. To quote the competition description : \n> If successful, not only will you help advance the state of the art in generative image creation, but you’ll enable us to create more experiments across a variety of domains in the future.\n\nMemorizers don't help image generation state of the art, what is the point of encouraging the use of such models? Given that medals & prize reward successful participants, I believe they should reward approaches that are not trying to trick the metric but trying to actually generate good images.\n\nFurthermore, there are a lot of people that focused on using real GANs, invested a lot of time, and it is very disappointing to realize that people who forked @cdeotte's great work could in fact end up well above them in the final LB.\n\nI mean, just don't allow memorizers GANs and if the problem is to detect a memorizer GAN, there are a couple of things that should work : \n- Make sure that the noise is sampled randomly. If using cgans, the number of classes should not be larger than the number of breeds. \n- Make sure feature matching is made on one of the last layers, not on the first/second one", "votes": 22 }, { "id": 572816, "postDate": "2019-07-11T12:36:21.660Z", "content": "Thanks for this clarification @wendykan , it means a lot.
\n\nAlthough I do not agree and think that memorizers should not count as a valid submission.
\n\nIt feels like they do not answer the question, which is \"can we generate new images of dogs that are original and realistic to the humans eye ?\". We can clearly see that memorizing gans do not allow to walk from z1 to z2 in the latent space with a real \"transformation\" but just a fading transition between the two images, whereas generalizing GANs would output 10 original valid images if given 10 vectors between z1 and z2, each with visual proximity to its neighbor, without what would look like a fading transition.
\n\nI think that this is a bit disapointing... But that is just my humble opinion, I'll continue on working on better generalizing GANs with the huge risk of ranking way lower (=bad) because anyone copy/pasting the Memorizer will have instantly a better score.
", "rawMarkdown": "Thanks for this clarification @wendykan , it means a lot.\n\nAlthough I do not agree and think that memorizers should not count as a valid submission.\n\nIt feels like they do not answer the question, which is \"can we generate new images of dogs that are original and realistic to the humans eye ?\". We can clearly see that memorizing gans do not allow to walk from z1 to z2 in the latent space with a real \"transformation\" but just a fading transition between the two images, whereas generalizing GANs would output 10 original valid images if given 10 vectors between z1 and z2, each with visual proximity to its neighbor, without what would look like a fading transition.\n\nI think that this is a bit disapointing... But that is just my humble opinion, I'll continue on working on better generalizing GANs with the huge risk of ranking way lower (=bad) because anyone copy/pasting the Memorizer will have instantly a better score.", "votes": 12 }, { "id": 572872, "postDate": "2019-07-11T14:14:18.573Z", "content": "As the author of Memorization GANs, I would prefer that they are disallowed too. Optimizing a memorization GAN is boring and a waste of time. It involves finding the best set of crops (or other preprocessing) to memorize. The best set avoids the cosine distance penalty and provides a large variation to maximize FID.
\n\n(Any set that avoids the penalty on public LB will also avoid the penalty on private LB because it is the same calculation using the training images. Any set that maximizes public FID will also score very well on private LB because dog images are dog images).
\n\nThat being said, I don't know how we could clearly define the rules to disallow Memorizer GANs because you can modify them slightly to convert them into traditional GANs. And I don't know how to detect them efficiently. It would be tons of work to review all submission code and/or images. Current research has not solved the problem of detecting memorization yet.
", "rawMarkdown": "As the author of Memorization GANs, I would prefer that they are **disallowed** too. Optimizing a memorization GAN is boring and a waste of time. It involves finding the best set of crops (or other preprocessing) to memorize. The best set avoids the cosine distance penalty and provides a large variation to maximize FID. \n\n(Any set that avoids the penalty on public LB will also avoid the penalty on private LB because it is the same calculation using the training images. Any set that maximizes public FID will also score very well on private LB because dog images are dog images).\n\nThat being said, I don't know how we could clearly define the rules to disallow Memorizer GANs because you can modify them slightly to convert them into traditional GANs. And I don't know how to detect them efficiently. It would be tons of work to review all submission code and/or images. Current research has not solved the problem of detecting memorization yet.", "votes": 13 }, { "id": 572887, "postDate": "2019-07-11T14:27:59.903Z", "content": "@cdeotte But it is difficult to disallow some method which you can't distinguish from other methods actually?
\n\nI don't think that kaggle team will be able to review all kernels including bronze zone manually.
", "rawMarkdown": "@cdeotte But it is difficult to disallow some method which you can't distinguish from other methods actually?\n\nI don't think that kaggle team will be able to review all kernels including bronze zone manually." }, { "id": 572895, "postDate": "2019-07-11T14:39:00.413Z", "content": "True Anna. All GANs memorize and generalize to some extent. There is currently no way to quantify generalization versus memorization.
", "rawMarkdown": "True Anna. All GANs memorize and generalize to some extent. There is currently no way to quantify generalization versus memorization.", "votes": 1 }, { "id": 572915, "postDate": "2019-07-11T15:08:10.090Z", "content": "@cdeotte In the end, a way for Kaggle to help improve state-of-the-art in GAN would be to make a competition about distinguishing memorizers vs generalizers. Then this competition 😄
", "rawMarkdown": "@cdeotte In the end, a way for Kaggle to help improve state-of-the-art in GAN would be to make a competition about distinguishing memorizers vs generalizers. Then this competition 😄 ", "votes": 6 }, { "id": 572965, "postDate": "2019-07-11T15:48:01.297Z", "content": "“I believe that memorized dogs will score okay whichever model or dog dataset you choose. ” ... I tried the reverse: an external dataset against the current metric, scoring 20ish. I suspect similar to the reverse ie private LB effect.
\n\nI’m glad I didn’t invest much time in the last 2 weeks. I now must consider whether to just use all the other loopholes I thought about, instead of making a better GAN. Sigh.
", "rawMarkdown": "“I believe that memorized dogs will score okay whichever model or dog dataset you choose. ” ... I tried the reverse: an external dataset against the current metric, scoring 20ish. I suspect similar to the reverse ie private LB effect.\n\nI’m glad I didn’t invest much time in the last 2 weeks. I now must consider whether to just use all the other loopholes I thought about, instead of making a better GAN. Sigh. ", "votes": 3 }, { "id": 573053, "postDate": "2019-07-11T18:18:45.907Z", "content": "@achikin I think you're right when you asked
\n\n\n\n\nDo you mean, that on Private leaderboard training examples are not used for FID calculation, but used for calculating the memoization penalty?
\n
The evaluation is here and it's clear, I don't know why kaggle said:
\n\n\n\n\n\"by memorizing the training examples you actually might overfit to the training sample distribution\"\n \"the memorization cosine distance still applies in the private leaderboard\"
\n
.png =400x*)
Thank you so much @cdeotte for sharing your point of view :)
\n\nIMHO this is hard for kaggle, since they are not experts in GANs, they don't know how to distinguish the different GANs types or say if a method is right or wrong, so they specified as a main rule that images should be generated, no altered versions /blendings allowed.
\n\nI don't think the solution is to say: \"hey, these types are allowed: GANs, DCGAN, CGAN etc\" the competition would lose the emotion.
\n\npossible solutions: change the submission from 10.000 to 100.000 images, and also submit the models.
\n\nBut I wouldn't like if the last week they say: \"supervised gans are no longer allowed\"
", "rawMarkdown": "Thank you so much @cdeotte for sharing your point of view :)\n\nIMHO this is hard for kaggle, since they are not experts in GANs, they don't know how to distinguish the different GANs types or say if a method is right or wrong, so they specified as a main rule that images should be generated, no altered versions /blendings allowed.\n\nI don't think the solution is to say: \"hey, these types are allowed: GANs, DCGAN, CGAN etc\" the competition would lose the emotion.\n\npossible solutions: change the submission from 10.000 to 100.000 images, and also submit the models.\n\nBut I wouldn't like if the **last week** they say: \"supervised gans are no longer allowed\"", "votes": -1 }, { "id": 573181, "postDate": "2019-07-11T22:29:07.563Z", "content": "@jesucristo and @achikin: \nAs the evaluation figure illustrated, the private leaderboard uses a \"mysterious private dog images dataset\" and a \"mysterious pre-trained NN\" to evaluate your submissions. Therefore, by overfitting to the training images by memorization, you might be caught in the memorization cosine distance in the private leaderboard because the private pre-trained NN could have a different behavior than the public one, therefore you might get a bad (high) score because we'll be dividing the FID by the cosine distance. I would suggest not trying to memorize because it's really difficult for you to guess how that NN behaves.
\n\n@cdeotte et al: \nThat said, I agree with all the sentiment here that we don't want to allow memorization GANs. However, there are a few reasons that makes it hard to just ban them:\n1. There are different levels of memorization in a GAN. There's no way to quantify how much memorization there is inside a GAN model. \n2. It's extremely hard to check the code for every submission to see if there's memorization. We will try different ways to test for memorization, and we will remove people if we think there's intentional memorization (for example directly submit some blend of training images). But we can't guarantee that we'll find everything.
\n\nApologies that in this experimental competition, the rules are vague. As described in this post:
\n\n\n", "rawMarkdown": "@jesucristo and @achikin: \nAs the evaluation figure illustrated, the private leaderboard uses a \"mysterious private dog images dataset\" and a \"mysterious pre-trained NN\" to evaluate your submissions. Therefore, by overfitting to the training images by memorization, you might be caught in the memorization cosine distance in the private leaderboard because the private pre-trained NN could have a different behavior than the public one, therefore you might get a bad (high) score because we'll be dividing the FID by the cosine distance. I would suggest not trying to memorize because it's really difficult for you to guess how that NN behaves. \n\n@cdeotte et al: \nThat said, I agree with all the sentiment here that we don't **want** to allow memorization GANs. However, there are a few reasons that makes it hard to just ban them:\n1. There are different levels of memorization in a GAN. There's no way to quantify how much memorization there is inside a GAN model. \n2. It's extremely hard to check the code for every submission to see if there's memorization. We will try different ways to test for memorization, and we will remove people if we think there's intentional memorization (for example directly submit some blend of training images). But we can't guarantee that we'll find everything. \n\nApologies that in this experimental competition, the rules are vague. As described in this post: \n> We are trying something completely new here. It's not easy to balance between innovation and fairness. Therefore, we ask to reserve the right to change things up during the competition. In a way, it's just like GANs, the discriminator has to train itself in each round when the generator generated more images.", "votes": 10 }, { "id": 573431, "postDate": "2019-07-12T09:16:00.967Z", "content": "We are trying something completely new here. It's not easy to balance between innovation and fairness. Therefore, we ask to reserve the right to change things up during the competition. In a way, it's just like GANs, the discriminator has to train itself in each round when the generator generated more images.
\n
@wendykan As I said, there is a huge difference between a GAN that is made to memorize 10 000 dogs from the training dataset and a Conditional GAN that memorizes the 120 classes. \nA memorizer GAN such as the one shared by Chris has to be designed on purpose.
\n\nThe difference will be seen on the private LB as those kernel will be on top, whereas legit GANs will lag behind, there is no automatic way to check that, you have to go through the code.
\n\nIt was already mentionned that people will try to cheat to get medals, there is one solution: \nhttps://www.kaggle.com/c/generative-dog-images/discussion/98229#latest-568083\nWhich is great.
\n\nAnyways, hopefully this issue is solved quickly. Thanks to the kaggle team for the great work!
", "rawMarkdown": "@wendykan As I said, there is a huge difference between a GAN that is made to memorize 10 000 dogs from the training dataset and a Conditional GAN that memorizes the 120 classes. \nA memorizer GAN such as the one shared by Chris has to be designed on purpose. \n\nThe difference will be seen on the private LB as those kernel will be on top, whereas legit GANs will lag behind, there is no automatic way to check that, you have to go through the code.\n\nIt was already mentionned that people will try to cheat to get medals, there is one solution: \nhttps://www.kaggle.com/c/generative-dog-images/discussion/98229#latest-568083\nWhich is great.\n\nAnyways, hopefully this issue is solved quickly. Thanks to the kaggle team for the great work!", "votes": 3 }, { "id": 573536, "postDate": "2019-07-12T11:37:43.220Z", "content": "I am actually very sad about your answer @wendykan \nIn the wild, memorizer GANs are of no use at all to anyone. With those being counted as valid submissions I have lost a large portion of my motivation to continue creating \"real\" GANs. There is no chance of even getting a medal with a \"real\" GAN. At the same time, people will move their focus to exploit the scoring and useful discussions will become more sparse.
", "rawMarkdown": "I am actually very sad about your answer @wendykan \nIn the wild, memorizer GANs are of no use at all to anyone. With those being counted as valid submissions I have lost a large portion of my motivation to continue creating \"real\" GANs. There is no chance of even getting a medal with a \"real\" GAN. At the same time, people will move their focus to exploit the scoring and useful discussions will become more sparse.", "votes": 5 }, { "id": 573598, "postDate": "2019-07-12T13:23:34.560Z", "content": "Memorizer GAN allowed? I think now everybody will just fork the highest score Kernel and stop working on \"real\" GANs.
", "rawMarkdown": "Memorizer GAN allowed? I think now everybody will just fork the highest score Kernel and stop working on \"real\" GANs.", "votes": 2 }, { "id": 573933, "postDate": "2019-07-13T01:24:37.427Z", "content": "\n\nI agree there are different levels of memorization in a GAN and there's no way to quantify how much memorization there is inside a GAN model.
\n\nBut to fit the above words, the model should first of all be a GAN model, while I think \"memorized GAN\" is not a GAN model as the discriminator and generator are not trained against each other. And the concept of \"memorization\" in GAN is different from what \"memorized GAN\" tries to memorize by training the discriminator first.
\n\nTraining D([dog , onehotdogName]) supervised by training data would basically record almost all the information of the training images in the neural network, let's say it is D*. After fixing D*, training the G network supervised by D* seems the same to me as training directly towards the training data. As for the \"generalization\", I feel it is coming from the noise injected to G during training process (no matter whether it's tuning the hyper-parameter, multiplying a percentage value on the one-hot vector, or incorporating several dog's one-hot vector together).
", "rawMarkdown": "@wendykan \n\nI agree there are different levels of memorization in a GAN and there's no way to quantify how much memorization there is inside a GAN model.\n\nBut to fit the above words, the model should first of all be a GAN model, while I think \"memorized GAN\" is not a GAN model as the discriminator and generator are not trained against each other. And the concept of \"memorization\" in GAN is different from what \"memorized GAN\" tries to memorize by training the discriminator first.\n\nTraining D([dog , onehotdogName]) supervised by training data would basically record almost all the information of the training images in the neural network, let's say it is D\\*. After fixing D\\*, training the G network supervised by D\\* seems the same to me as training directly towards the training data. As for the \"generalization\", I feel it is coming from the noise injected to G during training process (no matter whether it's tuning the hyper-parameter, multiplying a percentage value on the one-hot vector, or incorporating several dog's one-hot vector together). " }, { "id": 573939, "postDate": "2019-07-13T01:50:48.103Z", "content": "Smilence, every semi-supervised Conditional GAN (CGAN) where the number of class labels equals the number of training images is a Memorization GAN. Original CGAN paper here. Kaggle discussion post explanation here.
", "rawMarkdown": "Smilence, every semi-supervised Conditional GAN (CGAN) where the number of class labels equals the number of training images is a Memorization GAN. Original CGAN paper [here][1]. Kaggle discussion post explanation [here][2].\n\n[1]: https://arxiv.org/abs/1411.1784\n[2]: https://www.kaggle.com/c/generative-dog-images/discussion/99215" }, { "id": 573944, "postDate": "2019-07-13T02:06:41.540Z", "content": "@cdeotte I have been through them before these comments. I understand CGAN and Feature Matching and I understand your definition of \"Memorization GAN\" but I disagree it is a GAN model, since this special case of feature matching + CGAN collapse to just mimicking the training data. Also, as I said, I think GAN needs 2 models to train against each other but here it is not.
\n\nI will try to do some experiments to disprove it, but maybe I'm wrong. Thanks again for your great kernels and discussions.
", "rawMarkdown": "@cdeotte I have been through them before these comments. I understand CGAN and Feature Matching and I understand your definition of \"Memorization GAN\" but I disagree it is a GAN model, since this special case of feature matching + CGAN collapse to just mimicking the training data. Also, as I said, I think GAN needs 2 models to train against each other but here it is not.\n\nI will try to do some experiments to disprove it, but maybe I'm wrong. Thanks again for your great kernels and discussions.", "votes": 1 }, { "id": 575386, "postDate": "2019-07-15T11:18:19.920Z", "content": "@wendykan As you point out this is an experimental competition, this is totally fine. But why not remove prize money, medals, and competition points?
", "rawMarkdown": "@wendykan As you point out this is an experimental competition, this is totally fine. But why not remove prize money, medals, and competition points?", "votes": 6 }, { "id": 595766, "postDate": "2019-08-09T16:27:48.280Z", "content": "Thanks for the clarification @wendykan
", "rawMarkdown": "Thanks for the clarification @wendykan " } ] }, { "id": 567541, "postDate": "2019-07-03T16:53:18.217Z", "content": "It seems to me that the best procedure is to clean the house frequently and not to let the garbage accumulate until it becomes impossible to enter. I suggest trying to clear LB from the lowest score, until you find a kernel that obeys the rules. Maintain this analysis whenever a new low score appears on LB. So we would have a clean house, at least in front of our eyes.
", "rawMarkdown": "It seems to me that the best procedure is to clean the house frequently and not to let the garbage accumulate until it becomes impossible to enter. I suggest trying to clear LB from the lowest score, until you find a kernel that obeys the rules. Maintain this analysis whenever a new low score appears on LB. So we would have a clean house, at least in front of our eyes.", "votes": 10, "replies": [ { "id": 567631, "postDate": "2019-07-03T19:11:03.123Z", "content": "I've enjoyed participating in this challenge over the last few days. But I have no idea if my solution is really 10th, 100th, or 30th on the LB. Which makes a mockery of bothering to submit to the LB. Further, it seems very possible to \"spice\" a GAN model with a power boost of imitation results and go fairly undetected to get a silver medal. So thats it for me for now, I wish authentic competitors the best of luck but I'm stepping away while the LB looks like a dartboard.
\n\nIt would be a shame to see research challenges disappear. And I feel they deserve ranks and points as much as the commercial sponsorship challenges do. A policy to make research challenge kernels public to score points seems like a good idea, but I can see finger pointing over grey area kernels taking place soon after. I hope the organisers can come to some equitable solution for the problem.
", "rawMarkdown": "I've enjoyed participating in this challenge over the last few days. But I have no idea if my solution is really 10th, 100th, or 30th on the LB. Which makes a mockery of bothering to submit to the LB. Further, it seems very possible to \"spice\" a GAN model with a power boost of imitation results and go fairly undetected to get a silver medal. So thats it for me for now, I wish authentic competitors the best of luck but I'm stepping away while the LB looks like a dartboard.\n\nIt would be a shame to see research challenges disappear. And I feel they deserve ranks and points as much as the commercial sponsorship challenges do. A policy to make research challenge kernels public to score points seems like a good idea, but I can see finger pointing over grey area kernels taking place soon after. I hope the organisers can come to some equitable solution for the problem.", "votes": 4 }, { "id": 567636, "postDate": "2019-07-03T19:16:26.380Z", "content": "I believe that organizers should evaluate the gray area every day, not at the end of the competition. So they can change the rules sooner. Leaving to evaluate this in the end will be unfair to those who have followed the rules.
", "rawMarkdown": "I believe that organizers should evaluate the gray area every day, not at the end of the competition. So they can change the rules sooner. Leaving to evaluate this in the end will be unfair to those who have followed the rules.", "votes": 3 }, { "id": 568526, "postDate": "2019-07-05T04:18:59.613Z", "content": "In the 2nd stage of the Gendered Pronoun Resolution competition, the LB was all 0s so only as people submitted they placed on the LB. It was much better, people focused on their models which is really what research competitions would/should look for. This is an interesting area and competition opportunity so hopefully competitors that are interested in the research will not be dismayed by what the LB looks like.
", "rawMarkdown": "In the 2nd stage of the Gendered Pronoun Resolution competition, the LB was all 0s so only as people submitted they placed on the LB. It was much better, people focused on their models which is really what research competitions would/should look for. This is an interesting area and competition opportunity so hopefully competitors that are interested in the research will not be dismayed by what the LB looks like. " } ] }, { "id": 573340, "postDate": "2019-07-12T06:20:17.987Z", "content": "I suggest that the number that can be submitted at the end of the competition should be from 2 to 1.\nIt makes it easier to check submissions and has the effect of deterring submissions in the gray zone.
", "rawMarkdown": "I suggest that the number that can be submitted at the end of the competition should be from 2 to 1.\nIt makes it easier to check submissions and has the effect of deterring submissions in the gray zone.", "votes": 2 }, { "id": 566139, "postDate": "2019-07-01T20:51:47.247Z", "content": "Highly appreciated. Thanks for the quick clarification.
", "rawMarkdown": "Highly appreciated. Thanks for the quick clarification. ", "votes": 1 }, { "id": 566110, "postDate": "2019-07-01T20:00:01.637Z", "content": "Thanks for this. This competition is a fascinating idea and hopefully it will generate some innovative approaches. Ongoing dialogue about the rules like this may be necessary and helpful.
", "rawMarkdown": "Thanks for this. This competition is a fascinating idea and hopefully it will generate some innovative approaches. Ongoing dialogue about the rules like this may be necessary and helpful.", "votes": 1 }, { "id": 568543, "postDate": "2019-07-05T04:56:37.360Z", "content": "We know that using training image as input is illegal recently, but I have a new question now.
\n\nUsing Autoencoder to memorize training images, then use encoded array like [0, 0, 0, 1, 0, 0, …., 0, 0] to ‘generate’ image. Is that allowed for this competition?
In my opinion all generative methods that memorize and regurgitate images (with slight perturbations) should not win prizes. Even GANs can be trained to do this. I posted a GAN here that memorizes images and accepts any random vector of length 100 as the generating seed. It is a difficult task to distinguish generative methods that generalize versus memorize. I posted some suggestions in the linked kernel.
", "rawMarkdown": "In my opinion all generative methods that memorize and regurgitate images (with slight perturbations) should not win prizes. Even GANs can be trained to do this. I posted a GAN [here][1] that memorizes images and accepts any random vector of length 100 as the generating seed. It is a difficult task to distinguish generative methods that generalize versus memorize. I posted some suggestions in the linked kernel.\n\n[1]: https://www.kaggle.com/cdeotte/dog-memorizer-gan", "votes": 9 }, { "id": 568847, "postDate": "2019-07-05T14:36:34.090Z", "content": "Your post always give me many insights, thank you very much!
", "rawMarkdown": "Your post always give me many insights, thank you very much!", "votes": 1 } ] }, { "id": 566162, "postDate": "2019-07-01T21:18:08.570Z", "content": "Thanks for the clarifications. So, is the LB gonna be \"cleaned\" by sumbissions that use non appropriate methods?\ni mean that there are probably many sumbissions using methods of \"altered images\" and it will not be easy to check them one by one
", "rawMarkdown": "Thanks for the clarifications. So, is the LB gonna be \"cleaned\" by sumbissions that use non appropriate methods?\ni mean that there are probably many sumbissions using methods of \"altered images\" and it will not be easy to check them one by one", "votes": 2 }, { "id": 566204, "postDate": "2019-07-01T22:28:20.790Z", "content": "Thank you so much! I needed this hahaha 👍 \nso... imitation game is allowed or not?
", "rawMarkdown": "Thank you so much! I needed this hahaha 👍 \nso... [imitation game](https://www.kaggle.com/jazivxt/imitation-game) is allowed or not?", "replies": [ { "id": 566221, "postDate": "2019-07-01T23:04:18.930Z", "content": "No.
", "rawMarkdown": "No. ", "votes": 12 }, { "id": 566247, "postDate": "2019-07-02T00:52:40.220Z", "content": "Ｈ𝐀𝑷𝑷𝓎 🇰𝗮𝘨𝘨🇱𝖎Ｎɢ 💯
", "rawMarkdown": "Ｈ𝐀𝑷𝑷𝓎 🇰𝗮𝘨𝘨🇱𝖎Ｎɢ 💯", "votes": 4 } ] }, { "id": 592824, "postDate": "2019-08-05T21:11:17.603Z", "content": "For all questions regarding memorization GANs and clarifications on requirements to share kernels by the end of the competition, refer to Wendy's update post here.
", "rawMarkdown": "For all questions regarding memorization GANs and clarifications on requirements to share kernels by the end of the competition, refer to Wendy's [update post here](https://www.kaggle.com/c/generative-dog-images/discussion/102701)." }, { "id": 583599, "postDate": "2019-07-24T17:20:57.947Z", "content": "Can we use an external dataset or it will be consedered as a leakage ?
", "rawMarkdown": "Can we use an external dataset or it will be consedered as a **leakage** ?", "replies": [ { "id": 583604, "postDate": "2019-07-24T17:34:31.190Z", "content": "In this competition, you are not allowed to use external datasets.
", "rawMarkdown": "In this competition, you are not allowed to use external datasets." } ] }, { "id": 569249, "postDate": "2019-07-06T10:20:55.710Z", "content": "I understood the experimental character of this competition. While admiring the great attempt a lot of anxiety.\nFor example, is there a limit on the number of training data? The upper limit obviously exists. But what is the lower limit?\nIf you intentionally reduce the amount of data to be trained and cause overfitting, the generated image will be almost identical to the training image. Therefore, a high score on the criteria whether they look like a dog. However, the originality is low.
", "rawMarkdown": "I understood the experimental character of this competition. While admiring the great attempt a lot of anxiety.\nFor example, is there a limit on the number of training data? The upper limit obviously exists. But what is the lower limit?\nIf you intentionally reduce the amount of data to be trained and cause overfitting, the generated image will be almost identical to the training image. Therefore, a high score on the criteria whether they look like a dog. However, the originality is low.", "replies": [ { "id": 569583, "postDate": "2019-07-07T01:32:59.373Z", "content": "One more question: Is it acceptable to use flips or rotation to increase submission images?
", "rawMarkdown": "One more question: Is it acceptable to use flips or rotation to increase submission images?\n" } ] }, { "id": 569028, "postDate": "2019-07-05T22:32:37.830Z", "content": "Should the memorization distance be estimated between the generated images and the training images on both the private and public leaderboards?
", "rawMarkdown": "Should the memorization distance be estimated between the generated images and the training images on both the private and public leaderboards?" }, { "id": 566403, "postDate": "2019-07-02T06:18:53.097Z", "content": "@wendykan up to what leaderboard position do you intend to check for rule compliance after the competition ends? Maybe stating you are going to check all the medal area since now will prevent any further opportunistic behavior. Anyway, now the LB is just a big smoke screen where you really don't understand what is happening. Are you going to reset it or leave as it is?
", "rawMarkdown": "@wendykan up to what leaderboard position do you intend to check for rule compliance after the competition ends? Maybe stating you are going to check all the medal area since now will prevent any further opportunistic behavior. Anyway, now the LB is just a big smoke screen where you really don't understand what is happening. Are you going to reset it or leave as it is?" }, { "id": 590232, "postDate": "2019-08-01T23:02:54.760Z", "rawMarkdown": "", "isDeleted": true }, { "id": 573596, "postDate": "2019-07-12T13:22:32.163Z", "rawMarkdown": "", "isDeleted": true }, { "id": 2143045, "postDate": "2023-02-14T01:57:03.987Z", "content": "nice and Awesome thanks for sharing
", "rawMarkdown": "nice and Awesome thanks for sharing" } ], "comments": [ { "id": 571105, "author_name": "Anna Novikova", "author_url": "", "post_date": "2019-07-09T07:02:37.183000", "content": "Could you please clarify whether memorizer GAN like this from Chris is allowed in this competition?
", "votes": 10, "replies": [ { "id": 571434, "author_name": "Chris Deotte", "author_url": "", "post_date": "2019-07-09T15:30:48.923000", "content": "Yes, clarification about this would be helpful. To help illuminate the situation, I explain where a Memorization GAN fits into the world of GANs here. If Memorization GANs are disallowed, are CGANs with Feature Matching allowed if we restrict the number of classes, and match features of layers closer to the output layer?
", "votes": 6, "replies": [] }, { "id": 572299, "author_name": "Wendy Kan", "author_url": "", "post_date": "2019-07-10T18:42:29.607000", "content": "This is a gray area and I had to think about it for a while. EDITED 8/9/2019: Memorization GAN is not allowed. Please don't select them in your final submissions.
", "votes": 8, "replies": [] }, { "id": 572384, "author_name": "Nanashi", "author_url": "", "post_date": "2019-07-10T21:17:03.897000", "content": "thank you so much @wendykan we needed that! :)
", "votes": 1, "replies": [] }, { "id": 572476, "author_name": "Nanashi", "author_url": "", "post_date": "2019-07-11T01:32:18.010000", "content": "Btw, For now, I think they are allowed For now? Do you mean that you can say at any moment that those methods are no longer allowed? For example 1 week out?
@wendykan please, could you explain this points:\n1. \"by memorizing the training examples you actually might overfit to the training sample distribution\"\n2. \"the memorization cosine distance still applies in the private leaderboard\"
\n\nDo you mean, that on Private leaderboard training examples are not used for FID calculation, but used for calculating the memoization penalty?
", "votes": 1, "replies": [] }, { "id": 572664, "author_name": "FGPC", "author_url": "", "post_date": "2019-07-11T08:34:35.963000", "content": "I hope that we can receive a final decision regarding in this concern as soon as possible. We are a month to go before the final submission.
", "votes": 3, "replies": [] }, { "id": 572813, "author_name": "Theo Viel", "author_url": "", "post_date": "2019-07-11T12:32:36.983000", "content": "Correct me if I'm wrong but:
\n\nConsidering that the metric is made to evaluate how realistic our dogs look, I believe that memorized dogs will score okay whichever model or dog dataset you choose. By okay I mean better than most legit GANs. I'd love to be proven wrong though.
As far as I understood memorization GANs, they are useless for the task that is asked here. I mean what is the point of outputting a memorized image, just output the initial image.
Therefore, I find the decision to allow such models really absurd. To quote the competition description : \n> If successful, not only will you help advance the state of the art in generative image creation, but you’ll enable us to create more experiments across a variety of domains in the future.
\n\nMemorizers don't help image generation state of the art, what is the point of encouraging the use of such models? Given that medals & prize reward successful participants, I believe they should reward approaches that are not trying to trick the metric but trying to actually generate good images.
\n\nFurthermore, there are a lot of people that focused on using real GANs, invested a lot of time, and it is very disappointing to realize that people who forked @cdeotte's great work could in fact end up well above them in the final LB.
\n\nI mean, just don't allow memorizers GANs and if the problem is to detect a memorizer GAN, there are a couple of things that should work : \n- Make sure that the noise is sampled randomly. If using cgans, the number of classes should not be larger than the number of breeds. \n- Make sure feature matching is made on one of the last layers, not on the first/second one
", "votes": 22, "replies": [] }, { "id": 572816, "author_name": "Guillaume Desforges", "author_url": "", "post_date": "2019-07-11T12:36:21.660000", "content": "Thanks for this clarification @wendykan , it means a lot.
\n\nAlthough I do not agree and think that memorizers should not count as a valid submission.
\n\nIt feels like they do not answer the question, which is \"can we generate new images of dogs that are original and realistic to the humans eye ?\". We can clearly see that memorizing gans do not allow to walk from z1 to z2 in the latent space with a real \"transformation\" but just a fading transition between the two images, whereas generalizing GANs would output 10 original valid images if given 10 vectors between z1 and z2, each with visual proximity to its neighbor, without what would look like a fading transition.
\n\nI think that this is a bit disapointing... But that is just my humble opinion, I'll continue on working on better generalizing GANs with the huge risk of ranking way lower (=bad) because anyone copy/pasting the Memorizer will have instantly a better score.
", "votes": 12, "replies": [] }, { "id": 572872, "author_name": "Chris Deotte", "author_url": "", "post_date": "2019-07-11T14:14:18.573000", "content": "As the author of Memorization GANs, I would prefer that they are disallowed too. Optimizing a memorization GAN is boring and a waste of time. It involves finding the best set of crops (or other preprocessing) to memorize. The best set avoids the cosine distance penalty and provides a large variation to maximize FID.
\n\n(Any set that avoids the penalty on public LB will also avoid the penalty on private LB because it is the same calculation using the training images. Any set that maximizes public FID will also score very well on private LB because dog images are dog images).
\n\nThat being said, I don't know how we could clearly define the rules to disallow Memorizer GANs because you can modify them slightly to convert them into traditional GANs. And I don't know how to detect them efficiently. It would be tons of work to review all submission code and/or images. Current research has not solved the problem of detecting memorization yet.
", "votes": 13, "replies": [] }, { "id": 572887, "author_name": "Anna Novikova", "author_url": "", "post_date": "2019-07-11T14:27:59.903000", "content": "@cdeotte But it is difficult to disallow some method which you can't distinguish from other methods actually?
\n\nI don't think that kaggle team will be able to review all kernels including bronze zone manually.
", "votes": 0, "replies": [] }, { "id": 572895, "author_name": "Chris Deotte", "author_url": "", "post_date": "2019-07-11T14:39:00.413000", "content": "True Anna. All GANs memorize and generalize to some extent. There is currently no way to quantify generalization versus memorization.
", "votes": 1, "replies": [] }, { "id": 572915, "author_name": "Guillaume Desforges", "author_url": "", "post_date": "2019-07-11T15:08:10.090000", "content": "@cdeotte In the end, a way for Kaggle to help improve state-of-the-art in GAN would be to make a competition about distinguishing memorizers vs generalizers. Then this competition 😄
", "votes": 6, "replies": [] }, { "id": 572965, "author_name": "robga", "author_url": "", "post_date": "2019-07-11T15:48:01.297000", "content": "“I believe that memorized dogs will score okay whichever model or dog dataset you choose. ” ... I tried the reverse: an external dataset against the current metric, scoring 20ish. I suspect similar to the reverse ie private LB effect.
\n\nI’m glad I didn’t invest much time in the last 2 weeks. I now must consider whether to just use all the other loopholes I thought about, instead of making a better GAN. Sigh.
", "votes": 3, "replies": [] }, { "id": 573053, "author_name": "Nanashi", "author_url": "", "post_date": "2019-07-11T18:18:45.907000", "content": "@achikin I think you're right when you asked
\n\n\n\n\nDo you mean, that on Private leaderboard training examples are not used for FID calculation, but used for calculating the memoization penalty?
\n
The evaluation is here and it's clear, I don't know why kaggle said:
\n\n\n\n\n\"by memorizing the training examples you actually might overfit to the training sample distribution\"\n \"the memorization cosine distance still applies in the private leaderboard\"
\n
.png =400x*)
Thank you so much @cdeotte for sharing your point of view :)
\n\nIMHO this is hard for kaggle, since they are not experts in GANs, they don't know how to distinguish the different GANs types or say if a method is right or wrong, so they specified as a main rule that images should be generated, no altered versions /blendings allowed.
\n\nI don't think the solution is to say: \"hey, these types are allowed: GANs, DCGAN, CGAN etc\" the competition would lose the emotion.
\n\npossible solutions: change the submission from 10.000 to 100.000 images, and also submit the models.
\n\nBut I wouldn't like if the last week they say: \"supervised gans are no longer allowed\"
", "votes": -1, "replies": [] }, { "id": 573181, "author_name": "Wendy Kan", "author_url": "", "post_date": "2019-07-11T22:29:07.563000", "content": "@jesucristo and @achikin: \nAs the evaluation figure illustrated, the private leaderboard uses a \"mysterious private dog images dataset\" and a \"mysterious pre-trained NN\" to evaluate your submissions. Therefore, by overfitting to the training images by memorization, you might be caught in the memorization cosine distance in the private leaderboard because the private pre-trained NN could have a different behavior than the public one, therefore you might get a bad (high) score because we'll be dividing the FID by the cosine distance. I would suggest not trying to memorize because it's really difficult for you to guess how that NN behaves.
\n\n@cdeotte et al: \nThat said, I agree with all the sentiment here that we don't want to allow memorization GANs. However, there are a few reasons that makes it hard to just ban them:\n1. There are different levels of memorization in a GAN. There's no way to quantify how much memorization there is inside a GAN model. \n2. It's extremely hard to check the code for every submission to see if there's memorization. We will try different ways to test for memorization, and we will remove people if we think there's intentional memorization (for example directly submit some blend of training images). But we can't guarantee that we'll find everything.
\n\nApologies that in this experimental competition, the rules are vague. As described in this post:
\n\n\n", "votes": 10, "replies": [] }, { "id": 573431, "author_name": "Theo Viel", "author_url": "", "post_date": "2019-07-12T09:16:00.967000", "content": "We are trying something completely new here. It's not easy to balance between innovation and fairness. Therefore, we ask to reserve the right to change things up during the competition. In a way, it's just like GANs, the discriminator has to train itself in each round when the generator generated more images.
\n
@wendykan As I said, there is a huge difference between a GAN that is made to memorize 10 000 dogs from the training dataset and a Conditional GAN that memorizes the 120 classes. \nA memorizer GAN such as the one shared by Chris has to be designed on purpose.
\n\nThe difference will be seen on the private LB as those kernel will be on top, whereas legit GANs will lag behind, there is no automatic way to check that, you have to go through the code.
\n\nIt was already mentionned that people will try to cheat to get medals, there is one solution: \nhttps://www.kaggle.com/c/generative-dog-images/discussion/98229#latest-568083\nWhich is great.
\n\nAnyways, hopefully this issue is solved quickly. Thanks to the kaggle team for the great work!
", "votes": 3, "replies": [] }, { "id": 573536, "author_name": "Pascal Pfeiffer", "author_url": "", "post_date": "2019-07-12T11:37:43.220000", "content": "I am actually very sad about your answer @wendykan \nIn the wild, memorizer GANs are of no use at all to anyone. With those being counted as valid submissions I have lost a large portion of my motivation to continue creating \"real\" GANs. There is no chance of even getting a medal with a \"real\" GAN. At the same time, people will move their focus to exploit the scoring and useful discussions will become more sparse.
", "votes": 5, "replies": [] }, { "id": 573598, "author_name": "Felipe Loque", "author_url": "", "post_date": "2019-07-12T13:23:34.560000", "content": "Memorizer GAN allowed? I think now everybody will just fork the highest score Kernel and stop working on \"real\" GANs.
", "votes": 2, "replies": [] }, { "id": 573933, "author_name": "Smilence", "author_url": "", "post_date": "2019-07-13T01:24:37.427000", "content": "\n\nI agree there are different levels of memorization in a GAN and there's no way to quantify how much memorization there is inside a GAN model.
\n\nBut to fit the above words, the model should first of all be a GAN model, while I think \"memorized GAN\" is not a GAN model as the discriminator and generator are not trained against each other. And the concept of \"memorization\" in GAN is different from what \"memorized GAN\" tries to memorize by training the discriminator first.
\n\nTraining D([dog , onehotdogName]) supervised by training data would basically record almost all the information of the training images in the neural network, let's say it is D*. After fixing D*, training the G network supervised by D* seems the same to me as training directly towards the training data. As for the \"generalization\", I feel it is coming from the noise injected to G during training process (no matter whether it's tuning the hyper-parameter, multiplying a percentage value on the one-hot vector, or incorporating several dog's one-hot vector together).
", "votes": 0, "replies": [] }, { "id": 573939, "author_name": "Chris Deotte", "author_url": "", "post_date": "2019-07-13T01:50:48.103000", "content": "Smilence, every semi-supervised Conditional GAN (CGAN) where the number of class labels equals the number of training images is a Memorization GAN. Original CGAN paper here. Kaggle discussion post explanation here.
", "votes": 0, "replies": [] }, { "id": 573944, "author_name": "Smilence", "author_url": "", "post_date": "2019-07-13T02:06:41.540000", "content": "@cdeotte I have been through them before these comments. I understand CGAN and Feature Matching and I understand your definition of \"Memorization GAN\" but I disagree it is a GAN model, since this special case of feature matching + CGAN collapse to just mimicking the training data. Also, as I said, I think GAN needs 2 models to train against each other but here it is not.
\n\nI will try to do some experiments to disprove it, but maybe I'm wrong. Thanks again for your great kernels and discussions.
", "votes": 1, "replies": [] }, { "id": 575386, "author_name": "Psi", "author_url": "", "post_date": "2019-07-15T11:18:19.920000", "content": "@wendykan As you point out this is an experimental competition, this is totally fine. But why not remove prize money, medals, and competition points?
", "votes": 6, "replies": [] }, { "id": 595766, "author_name": "Kranthi Kumar", "author_url": "", "post_date": "2019-08-09T16:27:48.280000", "content": "Thanks for the clarification @wendykan
", "votes": 0, "replies": [] } ] }, { "id": 567541, "author_name": "Paulo Pinto", "author_url": "", "post_date": "2019-07-03T16:53:18.217000", "content": "It seems to me that the best procedure is to clean the house frequently and not to let the garbage accumulate until it becomes impossible to enter. I suggest trying to clear LB from the lowest score, until you find a kernel that obeys the rules. Maintain this analysis whenever a new low score appears on LB. So we would have a clean house, at least in front of our eyes.
", "votes": 10, "replies": [ { "id": 567631, "author_name": "robga", "author_url": "", "post_date": "2019-07-03T19:11:03.123000", "content": "I've enjoyed participating in this challenge over the last few days. But I have no idea if my solution is really 10th, 100th, or 30th on the LB. Which makes a mockery of bothering to submit to the LB. Further, it seems very possible to \"spice\" a GAN model with a power boost of imitation results and go fairly undetected to get a silver medal. So thats it for me for now, I wish authentic competitors the best of luck but I'm stepping away while the LB looks like a dartboard.
\n\nIt would be a shame to see research challenges disappear. And I feel they deserve ranks and points as much as the commercial sponsorship challenges do. A policy to make research challenge kernels public to score points seems like a good idea, but I can see finger pointing over grey area kernels taking place soon after. I hope the organisers can come to some equitable solution for the problem.
", "votes": 4, "replies": [] }, { "id": 567636, "author_name": "Paulo Pinto", "author_url": "", "post_date": "2019-07-03T19:16:26.380000", "content": "I believe that organizers should evaluate the gray area every day, not at the end of the competition. So they can change the rules sooner. Leaving to evaluate this in the end will be unfair to those who have followed the rules.
", "votes": 3, "replies": [] }, { "id": 568526, "author_name": "something4kag", "author_url": "", "post_date": "2019-07-05T04:18:59.613000", "content": "In the 2nd stage of the Gendered Pronoun Resolution competition, the LB was all 0s so only as people submitted they placed on the LB. It was much better, people focused on their models which is really what research competitions would/should look for. This is an interesting area and competition opportunity so hopefully competitors that are interested in the research will not be dismayed by what the LB looks like.
", "votes": 0, "replies": [] } ] }, { "id": 573340, "author_name": "hirune924", "author_url": "", "post_date": "2019-07-12T06:20:17.987000", "content": "I suggest that the number that can be submitted at the end of the competition should be from 2 to 1.\nIt makes it easier to check submissions and has the effect of deterring submissions in the gray zone.
", "votes": 2, "replies": [] }, { "id": 566139, "author_name": "Pascal Pfeiffer", "author_url": "", "post_date": "2019-07-01T20:51:47.247000", "content": "Highly appreciated. Thanks for the quick clarification.
", "votes": 1, "replies": [] }, { "id": 566110, "author_name": "MichaelP", "author_url": "", "post_date": "2019-07-01T20:00:01.637000", "content": "Thanks for this. This competition is a fascinating idea and hopefully it will generate some innovative approaches. Ongoing dialogue about the rules like this may be necessary and helpful.
", "votes": 1, "replies": [] }, { "id": 568543, "author_name": "Qishen Ha", "author_url": "", "post_date": "2019-07-05T04:56:37.360000", "content": "We know that using training image as input is illegal recently, but I have a new question now.
\n\nUsing Autoencoder to memorize training images, then use encoded array like [0, 0, 0, 1, 0, 0, …., 0, 0] to ‘generate’ image. Is that allowed for this competition?
In my opinion all generative methods that memorize and regurgitate images (with slight perturbations) should not win prizes. Even GANs can be trained to do this. I posted a GAN here that memorizes images and accepts any random vector of length 100 as the generating seed. It is a difficult task to distinguish generative methods that generalize versus memorize. I posted some suggestions in the linked kernel.
", "votes": 9, "replies": [] }, { "id": 568847, "author_name": "Qishen Ha", "author_url": "", "post_date": "2019-07-05T14:36:34.090000", "content": "Your post always give me many insights, thank you very much!
", "votes": 1, "replies": [] } ] }, { "id": 566162, "author_name": "Dimosthenis Karaflos", "author_url": "", "post_date": "2019-07-01T21:18:08.570000", "content": "Thanks for the clarifications. So, is the LB gonna be \"cleaned\" by sumbissions that use non appropriate methods?\ni mean that there are probably many sumbissions using methods of \"altered images\" and it will not be easy to check them one by one
", "votes": 2, "replies": [] }, { "id": 566204, "author_name": "Nanashi", "author_url": "", "post_date": "2019-07-01T22:28:20.790000", "content": "Thank you so much! I needed this hahaha 👍 \nso... imitation game is allowed or not?
", "votes": 0, "replies": [ { "id": 566221, "author_name": "Wendy Kan", "author_url": "", "post_date": "2019-07-01T23:04:18.930000", "content": "No.
", "votes": 12, "replies": [] }, { "id": 566247, "author_name": "jazivxt", "author_url": "", "post_date": "2019-07-02T00:52:40.220000", "content": "Ｈ𝐀𝑷𝑷𝓎 🇰𝗮𝘨𝘨🇱𝖎Ｎɢ 💯
", "votes": 4, "replies": [] } ] }, { "id": 592824, "author_name": "Julia Elliott", "author_url": "", "post_date": "2019-08-05T21:11:17.603000", "content": "For all questions regarding memorization GANs and clarifications on requirements to share kernels by the end of the competition, refer to Wendy's update post here.
", "votes": 0, "replies": [] }, { "id": 583599, "author_name": "Meed Zaoui", "author_url": "", "post_date": "2019-07-24T17:20:57.947000", "content": "Can we use an external dataset or it will be consedered as a leakage ?
", "votes": 0, "replies": [ { "id": 583604, "author_name": "Chris Deotte", "author_url": "", "post_date": "2019-07-24T17:34:31.190000", "content": "In this competition, you are not allowed to use external datasets.
", "votes": 0, "replies": [] } ] }, { "id": 569249, "author_name": "Tanrei(nama)", "author_url": "", "post_date": "2019-07-06T10:20:55.710000", "content": "I understood the experimental character of this competition. While admiring the great attempt a lot of anxiety.\nFor example, is there a limit on the number of training data? The upper limit obviously exists. But what is the lower limit?\nIf you intentionally reduce the amount of data to be trained and cause overfitting, the generated image will be almost identical to the training image. Therefore, a high score on the criteria whether they look like a dog. However, the originality is low.
", "votes": 0, "replies": [ { "id": 569583, "author_name": "Tanrei(nama)", "author_url": "", "post_date": "2019-07-07T01:32:59.373000", "content": "One more question: Is it acceptable to use flips or rotation to increase submission images?
", "votes": 0, "replies": [] } ] }, { "id": 569028, "author_name": "A_Elsheikh", "author_url": "", "post_date": "2019-07-05T22:32:37.830000", "content": "Should the memorization distance be estimated between the generated images and the training images on both the private and public leaderboards?
", "votes": 0, "replies": [] }, { "id": 566403, "author_name": "Luca Massaron", "author_url": "", "post_date": "2019-07-02T06:18:53.097000", "content": "@wendykan up to what leaderboard position do you intend to check for rule compliance after the competition ends? Maybe stating you are going to check all the medal area since now will prevent any further opportunistic behavior. Anyway, now the LB is just a big smoke screen where you really don't understand what is happening. Are you going to reset it or leave as it is?
", "votes": 0, "replies": [] }, { "id": 590232, "author_name": "", "author_url": "", "post_date": "2019-08-01T23:02:54.760000", "content": "", "votes": 0, "replies": [] }, { "id": 573596, "author_name": "", "author_url": "", "post_date": "2019-07-12T13:22:32.163000", "content": "", "votes": 0, "replies": [] }, { "id": 2143045, "author_name": "Tamader.", "author_url": "", "post_date": "2023-02-14T01:57:03.987000", "content": "nice and Awesome thanks for sharing
", "votes": 0, "replies": [] } ], "raw_markdown_by_id": { "566105": "Per some requests, the following should provide additional clarification about the rules of this competition.\n\n- Why do you say \"You'll be using GANs\" on the front page, but the rules say \"only generative models are allowed\"? \n\n> This competition is indeed designed for GANs. However, Kaggle doesn't want to limit the solution algorithms one can use to solve a problem. For example, VAE methods such as [this paper](https://arxiv.org/pdf/1906.00446.pdf) seem very promising, and we would love to see if they perform better than GANs! Therefore, we didn't want to specifically limit participants to use GANs in our official rules. However, we do want to discourage the use of \"altering training images\" - see #2. \n\n- What are \"altered versions\" of images? In particular is the blending method used in [this Kernel](https://www.kaggle.com/jazivxt/imitation-game) allowed?\n\n> As far as we know, the research community doesn’t have one universal metric for evaluating generative images (see [this survey paper](https://arxiv.org/pdf/1906.00446.pdf)); furthermore, even when Kaggle adds a memorization component to the existing metrics, it still can’t catch all the combinations of image processing one can apply to the training images. That being said, if an image isn't generated *from scratch*, we would consider your method an \"altered version\" of images. In particular, your *trained generator* model should be able to stand alone and produce images *without* the input of training images. For example, generating some weights and blending training images, or do some sort of filtering to the training images, are **not** allowed. Think of it as a student going into an exam that doesn't allow bringing anything with you - no books, no photos, no phones, etc. This is **not an open-book exam**, therefore if you submit anything that seems like a copy from your textbook, we would consider that cheating. \n\n- Why are you so vague about the rules? \n\n> We are trying something completely new here. It's not easy to balance between innovation and fairness. Therefore, we ask to reserve the right to change things up during the competition. In a way, it's just like GANs, the discriminator has to train itself in each round when the generator generated more images. \n\n\n\n", "571105": "Could you please clarify whether memorizer GAN like [this from Chris](https://www.kaggle.com/cdeotte/dog-memorizer-gan) is allowed in this competition?", "567541": "It seems to me that the best procedure is to clean the house frequently and not to let the garbage accumulate until it becomes impossible to enter. I suggest trying to clear LB from the lowest score, until you find a kernel that obeys the rules. Maintain this analysis whenever a new low score appears on LB. So we would have a clean house, at least in front of our eyes.", "573340": "I suggest that the number that can be submitted at the end of the competition should be from 2 to 1.\nIt makes it easier to check submissions and has the effect of deterring submissions in the gray zone.", "566139": "Highly appreciated. Thanks for the quick clarification. ", "566110": "Thanks for this. This competition is a fascinating idea and hopefully it will generate some innovative approaches. Ongoing dialogue about the rules like this may be necessary and helpful.", "568543": "We know that using training image as input is illegal recently, but I have a new question now.\n\nUsing Autoencoder to memorize training images, then use encoded array like ` [0, 0, 0, 1, 0, 0, …., 0, 0] ` to ‘generate’ image. Is that allowed for this competition?", "566162": "Thanks for the clarifications. So, is the LB gonna be \"cleaned\" by sumbissions that use non appropriate methods?\ni mean that there are probably many sumbissions using methods of \"altered images\" and it will not be easy to check them one by one", "566204": "Thank you so much! I needed this hahaha 👍 \nso... [imitation game](https://www.kaggle.com/jazivxt/imitation-game) is allowed or not?", "592824": "For all questions regarding memorization GANs and clarifications on requirements to share kernels by the end of the competition, refer to Wendy's [update post here](https://www.kaggle.com/c/generative-dog-images/discussion/102701).", "583599": "Can we use an external dataset or it will be consedered as a **leakage** ?", "569249": "I understood the experimental character of this competition. While admiring the great attempt a lot of anxiety.\nFor example, is there a limit on the number of training data? The upper limit obviously exists. But what is the lower limit?\nIf you intentionally reduce the amount of data to be trained and cause overfitting, the generated image will be almost identical to the training image. Therefore, a high score on the criteria whether they look like a dog. However, the originality is low.", "569028": "Should the memorization distance be estimated between the generated images and the training images on both the private and public leaderboards?", "566403": "@wendykan up to what leaderboard position do you intend to check for rule compliance after the competition ends? Maybe stating you are going to check all the medal area since now will prevent any further opportunistic behavior. Anyway, now the LB is just a big smoke screen where you really don't understand what is happening. Are you going to reset it or leave as it is?", "590232": "", "573596": "", "2143045": "nice and Awesome thanks for sharing" } }