![\"happy\"](\"https://media1.tenor.com/images/88848d2067d622de8e4f314e28dc431a/tenor.gif\")
We would like to announce a few decisions on the competition logistics before the end of competition.
\n\nDue to the error reports, we would like to extend the deadline of the competition by 5 days. The new deadline is August 14. You have 11 more days before the deadline of the competition.
We will review the final selected submission Kernels for all teams earning medals. Therefore, you are required to either publicly or privately share your final Kernels with the admins: Wendy Kan (@wendykan), Ching-yuan Bai (@andrewcybai), and Julia Elliott (@juliaelliott). If a team fails to share their Kernels with us, we have the right to remove them from the leaderboard.
Memorization GANs not allowed. Please don't select them as your final submissions.
We will not be revealing the private leaderboard immediately at the end of the competition. We will perform the code inspection and wait to release the private leaderboard when it's final.
Thank you so much for participating in this competition! We look forward to seeing the results.
", "messages": [ { "id": 591652, "postDate": "2019-08-04T04:35:17.460Z", "content": "We would like to announce a few decisions on the competition logistics before the end of competition.
\n\nDue to the error reports, we would like to extend the deadline of the competition by 5 days. The new deadline is August 14. You have 11 more days before the deadline of the competition.
We will review the final selected submission Kernels for all teams earning medals. Therefore, you are required to either publicly or privately share your final Kernels with the admins: Wendy Kan (@wendykan), Ching-yuan Bai (@andrewcybai), and Julia Elliott (@juliaelliott). If a team fails to share their Kernels with us, we have the right to remove them from the leaderboard.
Memorization GANs not allowed. Please don't select them as your final submissions.
We will not be revealing the private leaderboard immediately at the end of the competition. We will perform the code inspection and wait to release the private leaderboard when it's final.
Thank you so much for participating in this competition! We look forward to seeing the results.
", "rawMarkdown": "We would like to announce a few decisions on the competition logistics before the end of competition. \n\n1. Due to the error reports, we would like to extend the deadline of the competition by 5 days. The new deadline is August 14. You have 11 more days before the deadline of the competition. \n\n2. We will review the final selected submission Kernels for all teams earning medals. Therefore, you are required to either publicly or privately share your final Kernels with the admins: Wendy Kan (@wendykan), Ching-yuan Bai (@andrewcybai), and Julia Elliott (@juliaelliott). If a team fails to share their Kernels with us, we have the right to remove them from the leaderboard. \n\n3. Memorization GANs not allowed. Please don't select them as your final submissions. \n\n4. We will not be revealing the private leaderboard immediately at the end of the competition. We will perform the code inspection and wait to release the private leaderboard when it's final. \n\nThank you so much for participating in this competition! We look forward to seeing the results. ", "votes": 75 }, { "id": 598980, "postDate": "2019-08-14T10:20:51.790Z", "content": "Hi @juliaelliott, @wendykan, @andrewcybai,
\n\nThank for your announcement, it’s a great step to a successful competition, I seldom talk about rules and setting of a competition, I respect all the hard work of you have done and understand how difficult to host a competition, but I have been worrying about the evaluation metric since last week.
\n\nWhat I worried is the distance threshold of MiFID, I understand you want to is to prevent invalid GAN which just make modification to the train images or mem-GAN. As we know, you defined a memorization distance threshold, but the memorization distance is sensitive to feature extraction networks, I tested several networks and I found 0.1 is a dangerous threshold on InceptionResnetV2 and Xception.
\n\nI downloaded some other datasets containing dogs from kaggle, and calculated the memorization distance to all-dogs, they are all near 0.1. it means that even if our models can generate almost real dogs, they may be severely punished by more than 10x. On the contrary, the distance is about 0.15 on inception v3 and v4, it’s quite safe.
\n\nI also tested more than 30 models of mine scoring range from 31.x to 12.x, using these two networks to extract features , I am sure my GAN models are 100% valid GANs if StyleGAN,BigGAN,SA-GAN,AC-GAN are valid GANs, if the public scores are better than 30, the memorization distances of half of them were under 0.1, if we choose 0.1 as threshold, it will be horrible to some of the competitors whose scores are less than 30. As far as I know, there are many kagglers scores are below 30, for example, @dvorobiev @yukia18, we may score >100 and will not have chance be viewed by you.
\n\nSince you will check all the codes, the memorization distance is not so important now, IS/FID and other metrics commonly used on papers are good choices. If you want to use MiFID, I suggest you use BIGGAN to generate dogs and calculate the distance as a reference. https://tfhub.dev/deepmind/biggan-128/2
\n\nIf you want to verify my code, I have shared a kernel with public score 13.4 to you, if you have any question about it, please contact me, perhaps it’s too complex with more than 3000 lines in a kernel.
\n\nPerhaps it’s not convenient for you to answer me now, but please take consideration of my post here. If the problem is happen to me only, I don't expect I might benefit from this post. We all want to have a successful competition, especially such a creative one and encourage more kagglers to participate similar one in the future, I am waiting for a reinforcement learning one.
\n\nThank you for your consideration.
", "rawMarkdown": "Hi @juliaelliott, @wendykan, @andrewcybai,\n\nThank for your announcement, it’s a great step to a successful competition, I seldom talk about rules and setting of a competition, I respect all the hard work of you have done and understand how difficult to host a competition, but I have been worrying about the evaluation metric since last week.\n\nWhat I worried is the distance threshold of MiFID, I understand you want to is to prevent invalid GAN which just make modification to the train images or mem-GAN. As we know, you defined a memorization distance threshold, but the memorization distance is sensitive to feature extraction networks, I tested several networks and I found 0.1 is a dangerous threshold on InceptionResnetV2 and Xception. \n\nI downloaded some other datasets containing dogs from kaggle, and calculated the memorization distance to all-dogs, they are all near 0.1. it means that even if our models can generate almost real dogs, they may be severely punished by more than 10x. On the contrary, the distance is about 0.15 on inception v3 and v4, it’s quite safe. \n\nI also tested more than 30 models of mine scoring range from 31.x to 12.x, using these two networks to extract features , I am sure my GAN models are 100% valid GANs if StyleGAN,BigGAN,SA-GAN,AC-GAN are valid GANs, if the public scores are better than 30, the memorization distances of half of them were under 0.1, if we choose 0.1 as threshold, it will be horrible to some of the competitors whose scores are less than 30. As far as I know, there are many kagglers scores are below 30, for example, @dvorobiev @yukia18, we may score >100 and will not have chance be viewed by you.\n\nSince you will check all the codes, the memorization distance is not so important now, IS/FID and other metrics commonly used on papers are good choices. If you want to use MiFID, I suggest you use BIGGAN to generate dogs and calculate the distance as a reference. https://tfhub.dev/deepmind/biggan-128/2\n\nIf you want to verify my code, I have shared a kernel with public score 13.4 to you, if you have any question about it, please contact me, perhaps it’s too complex with more than 3000 lines in a kernel.\n\nPerhaps it’s not convenient for you to answer me now, but please take consideration of my post here. If the problem is happen to me only, I don't expect I might benefit from this post. We all want to have a successful competition, especially such a creative one and encourage more kagglers to participate similar one in the future, I am waiting for a reinforcement learning one. \n\nThank you for your consideration.\n\n", "votes": 29, "replies": [ { "id": 599007, "postDate": "2019-08-14T11:04:36.760Z", "content": "Good point. Before you posted your findings, I thought my pictures are crappy enough so I don't need to worry about memorization :) But I never tried to test it with other architectures... Thanks for your efforts!
", "rawMarkdown": "Good point. Before you posted your findings, I thought my pictures are crappy enough so I don't need to worry about memorization :) But I never tried to test it with other architectures... Thanks for your efforts!", "votes": 1 }, { "id": 599245, "postDate": "2019-08-14T18:05:14.310Z", "content": "I have just generated the dog images by using pretrained BigGan models from https://tfhub.dev/deepmind/biggan-128/2 and https://tfhub.dev/deepmind/biggan-256/2:
\nthe distances to all-dogs and scores of pretrained BigGan 128 model on\nInceptionV3: distance:0.130 FID: 21.9\nInceptionV4: distance:0.109 FID: 25.0\nInceptionResnetV2: distance:0.064 FID: 19.9 MFID:309.1\nXception: distance:0.096 FID: 0.097 FID:73.7 MFID:760.1\n
\nthe distances to all-dogs and scores of pretrained BigGan 256 model on\nInceptionV3: distance:0.128 FID: 24.0 \nInceptionV4: distance:0.105 FID: 27.5\nInceptionResnetV2: distance: 0.060 FID: 21.8 MFID:358.4\nXception: distance:0.097 FID: 30.0 MFID:310.0\n
\nthe distances to all-dogs and scores of Memorization-GAN on\nInceptionV3: distance:0.103 FID: 7.2\nInceptionV4: distance:0.097 FID: 6.6 MFID:68.1\nInceptionResnetV2: distance:0.059 FID: 4.3 MFID:74.1\nXception: distance:0.080 FID: 11.5 MFID:144.4
@bestfitting Thank you for your feedback, and we hear your worry. In an attempt to address your question the best we can: We will penalize based on the private evaluation metric, which includes a distance threshold that we've evaluated to be indicative of using invalid methods (i.e. direct image modifications that are not generative). The specifics of that private evaluation and the distance is, for obvious reasons, not going to be revealed. Because this is a highly experimental competition format, we will also be accompanying the objective metric score with manual code inspection, to also identify any prohibited methods that may not have been captured by objective means. We are confident these measures will support a fair evaluation of the final leaderboard. For those who are not attempting prohibited, non-generative methods, you should have nothing to fear.
", "rawMarkdown": "@bestfitting Thank you for your feedback, and we hear your worry. In an attempt to address your question the best we can: We will penalize based on the private evaluation metric, which includes a distance threshold that we've evaluated to be indicative of using invalid methods (i.e. direct image modifications that are not generative). The specifics of that private evaluation and the distance is, for obvious reasons, not going to be revealed. Because this is a highly experimental competition format, we will also be accompanying the objective metric score with manual code inspection, to also identify any prohibited methods that may not have been captured by objective means. We are confident these measures will support a fair evaluation of the final leaderboard. For those who are not attempting prohibited, non-generative methods, you should have nothing to fear.", "votes": 8 }, { "id": 599422, "postDate": "2019-08-15T00:07:59.013Z", "content": "Wow, Big GAN makes amazing images and awesome latent walks. I never played with it until today. I posted it in a Kaggle kernel here. With truncation=1.0, it scores LB 15. Impressive.
Congratulations on another gold medal，maybe the top one。my god 😉
", "rawMarkdown": "Congratulations on another gold medal，maybe the top one。my god 😉 ", "votes": 1 } ] }, { "id": 594873, "postDate": "2019-08-08T15:01:08.340Z", "content": "i agree👍
", "rawMarkdown": "i agree👍 ", "votes": 8 }, { "id": 591700, "postDate": "2019-08-04T06:16:06.823Z", "content": "Great decisions!
", "rawMarkdown": "Great decisions!", "votes": 8 }, { "id": 592527, "postDate": "2019-08-05T12:34:16.583Z", "content": "At the end of the competition how are you going to determine Memorization GAN and Ordinary GAN?\nYou should see and answer this question now. We don't have time to wonder how to get good results. \nhttps://www.kaggle.com/c/generative-dog-images/discussion/99215#latest-592613
", "rawMarkdown": "At the end of the competition how are you going to determine Memorization GAN and Ordinary GAN?\nYou should see and answer this question now. We don't have time to wonder how to get good results. \nhttps://www.kaggle.com/c/generative-dog-images/discussion/99215#latest-592613", "votes": 5 }, { "id": 599539, "postDate": "2019-08-15T05:03:42.677Z", "content": "The competition details say '17 days to go'. Is that the time required to compile the final result or is it just a bug?
", "rawMarkdown": "The competition details say '17 days to go'. Is that the time required to compile the final result or is it just a bug?", "votes": 6, "replies": [ { "id": 599553, "postDate": "2019-08-15T05:43:32.873Z", "content": "It is time to verify the solutions, I believe, as far as the kernels are most probably inspected manually.
", "rawMarkdown": "It is time to verify the solutions, I believe, as far as the kernels are most probably inspected manually." } ] }, { "id": 595915, "postDate": "2019-08-09T20:47:55.897Z", "content": "I wonder if there is something in the recent Kaggle competitions that prompted these very interesting new rules. 🤔
", "rawMarkdown": "I wonder if there is something in the recent Kaggle competitions that prompted these very interesting new rules. 🤔 ", "votes": 6 }, { "id": 599541, "postDate": "2019-08-15T05:11:10.763Z", "content": "17 days to go???😨
", "rawMarkdown": "17 days to go???😨 ", "votes": 3 }, { "id": 592009, "postDate": "2019-08-04T16:07:09.523Z", "content": "@wendy: since we have changes in competition logistics, is it possible for your team to consider resetting of the leaderboard as part of changes? many thanks! :-)
", "rawMarkdown": "@wendy: since we have changes in competition logistics, is it possible for your team to consider resetting of the leaderboard as part of changes? many thanks! :-)", "votes": 3, "replies": [ { "id": 592816, "postDate": "2019-08-05T21:03:06.567Z", "content": "We will not be resetting the leaderboard.
", "rawMarkdown": "We will not be resetting the leaderboard." } ] }, { "id": 591659, "postDate": "2019-08-04T04:52:45.090Z", "content": "Thats cool. It would give motivation for many people.
", "rawMarkdown": "Thats cool. It would give motivation for many people. ", "votes": 3 }, { "id": 591691, "postDate": "2019-08-04T06:05:50.553Z", "content": "\n\n\nMemorization GANs not allowed. Please don't select them as your final submissions.
\n
Great! This has got a lot more interesting now. Especially because the public LB is completely meaningless and we have no idea where we stand.
", "rawMarkdown": ">Memorization GANs not allowed. Please don't select them as your final submissions. \n\nGreat! This has got a lot more interesting now. Especially because the public LB is completely meaningless and we have no idea where we stand. ", "votes": 4, "replies": [ { "id": 592088, "postDate": "2019-08-04T19:29:27.643Z", "content": "I agree with you. It's becoming nothing short of a wild west. I didn't expect so much drama for my first Kaggle competition.😏 😲 This will definitely motivate people for the last stretch of the competition 👍
", "rawMarkdown": "I agree with you. It's becoming nothing short of a wild west. I didn't expect so much drama for my first Kaggle competition.😏 😲 This will definitely motivate people for the last stretch of the competition 👍 ", "votes": 1 } ] }, { "id": 600009, "postDate": "2019-08-15T14:30:24.080Z", "content": "The competition closed for submissions on August 14th. However, the deadline has been set to August 31st, so we can complete the scoring and review of all submissions, per Wendy’s 4th bullet of this post. No further submissions will be accepted.
", "rawMarkdown": "The competition closed for submissions on August 14th. However, the deadline has been set to August 31st, so we can complete the scoring and review of all submissions, per Wendy’s 4th bullet of this post. No further submissions will be accepted.", "votes": 1, "replies": [ { "id": 600263, "postDate": "2019-08-15T21:23:29.160Z", "content": "Is submission going to be opened after August 31?
", "rawMarkdown": "Is submission going to be opened after August 31? " } ] }, { "id": 591900, "postDate": "2019-08-04T12:45:53.547Z", "content": "Great!
", "rawMarkdown": "Great!", "votes": 1 }, { "id": 591721, "postDate": "2019-08-04T06:37:52.330Z", "content": "It's great !!\n@wendykan When is the deadline for sharing our final Kernels?
", "rawMarkdown": "It's great !!\n@wendykan When is the deadline for sharing our final Kernels?", "votes": 1, "replies": [ { "id": 592819, "postDate": "2019-08-05T21:04:41.210Z", "content": "Kernels must be shared (privately or publicly) by the updated competition deadline: August 14, 2019 11:59 PM UTC
", "rawMarkdown": "Kernels must be shared (privately or publicly) by the updated competition deadline: **August 14, 2019 11:59 PM UTC**", "votes": 5 }, { "id": 592867, "postDate": "2019-08-05T23:18:32.443Z", "content": "@juliaelliott Thanks for clarification !
", "rawMarkdown": "@juliaelliott Thanks for clarification !", "votes": 1 }, { "id": 593090, "postDate": "2019-08-06T07:20:00.537Z", "content": "Is this means that we should share our code (privately or publicly) after the private leaderboard been displayed.
", "rawMarkdown": "Is this means that we should share our code (privately or publicly) after the private leaderboard been displayed." }, { "id": 593530, "postDate": "2019-08-06T18:26:13.047Z", "content": "@adiachen - the deadline to share your code is the deadline of the competition. So you should be sure to share it at the same time the competition ends.
", "rawMarkdown": "@adiachen - the deadline to share your code is the deadline of the competition. So you should be sure to share it at the same time the competition ends.", "votes": 2 }, { "id": 593809, "postDate": "2019-08-07T06:31:43.173Z", "content": "@juliaelliott - so sharing the kernels publicly shouldn't be allowed, right? Someone else could just resubmit it?
", "rawMarkdown": "@juliaelliott - so sharing the kernels publicly shouldn't be allowed, right? Someone else could just resubmit it?" }, { "id": 595082, "postDate": "2019-08-08T20:21:43.533Z", "content": "We will begin reviewing code 24 hours following the competition's deadline. Therefore, you must share your kernels by then. If your selected kernels have not been shared, you risk those scores being disqualified.
", "rawMarkdown": "We will begin reviewing code 24 hours following the competition's deadline. Therefore, you must share your kernels by then. If your selected kernels have not been shared, you risk those scores being disqualified." } ] }, { "id": 595097, "postDate": "2019-08-08T20:40:04.757Z", "content": "
Very Nice!
", "rawMarkdown": "Very Nice!", "votes": 2 }, { "id": 591917, "postDate": "2019-08-04T13:22:48.637Z", "content": "How to share final kernels to admin privately or publicly?
", "rawMarkdown": "How to share final kernels to admin privately or publicly?", "votes": 2, "replies": [ { "id": 592266, "postDate": "2019-08-05T05:02:57.770Z", "content": "Open the sidebar visibility, the ->| arrow icon top right\nFor publicly - in your kernel settings sharing click on private change from private to public.\nFor privately - in your kernel settings sharing click on private add the names designated as collaborators on your kernel.
", "rawMarkdown": "Open the sidebar visibility, the ->| arrow icon top right\nFor publicly - in your kernel settings sharing click on private change from private to public.\nFor privately - in your kernel settings sharing click on private add the names designated as collaborators on your kernel.", "votes": 1 }, { "id": 592516, "postDate": "2019-08-05T12:11:40.610Z", "content": "Thank you!
", "rawMarkdown": "Thank you!", "votes": 1 } ] }, { "id": 591682, "postDate": "2019-08-04T05:42:48.100Z", "content": "Sounds like the game is going to be interesting again
", "rawMarkdown": "Sounds like the game is going to be interesting again", "votes": 2 }, { "id": 600037, "postDate": "2019-08-15T15:01:32.563Z", "content": "I've selected a wrong kernal as final result, how to change it now?🤕
", "rawMarkdown": "I've selected a wrong kernal as final result, how to change it now?🤕 ", "replies": [ { "id": 600103, "postDate": "2019-08-15T16:42:11.050Z", "content": "Unfortunately, it is no longer possible to change your selections since the competition’s submission deadline has passed.
", "rawMarkdown": "Unfortunately, it is no longer possible to change your selections since the competition’s submission deadline has passed." } ] }, { "id": 599365, "postDate": "2019-08-14T21:33:39.880Z", "content": "@juliaelliott can't we submit with your metric dataset uploaded? T___T just noticed it
", "rawMarkdown": " @juliaelliott can't we submit with your metric dataset uploaded? T___T just noticed it" }, { "id": 599073, "postDate": "2019-08-14T13:09:40.850Z", "content": "Very Nice!
", "rawMarkdown": "Very Nice!" }, { "id": 598959, "postDate": "2019-08-14T09:37:50.640Z", "content": "cool
", "rawMarkdown": "cool" }, { "id": 598556, "postDate": "2019-08-13T17:56:37.420Z", "content": "Hi @juliaelliott, @wendykan , I saw this description in the evaluation page:\n> You are going to generate 10,000 images that are in PNG format. Their sizes should be 64x64x3 (RGB). Then you need to zip those images and your output from your Kernel should only have ONE output file named images.zip.
\n\nWhat if we have more files in the output of Kernel? I think it is common for kagglers to also output a model file for local evaluations together with images.zip. If this is not allowed, many people might need to rerun their submissions with very limited quota left. Or they would fail to be listed in the final private LB.
\n\nCould you clarify it? Thanks!
", "rawMarkdown": "Hi @juliaelliott, @wendykan , I saw this description in the evaluation page:\n> You are going to generate 10,000 images that are in PNG format. Their sizes should be 64x64x3 (RGB). Then you need to zip those images and your output from your Kernel should only have ONE output file named images.zip.\n\nWhat if we have more files in the output of Kernel? I think it is common for kagglers to also output a model file for local evaluations together with images.zip. If this is not allowed, many people might need to rerun their submissions with very limited quota left. Or they would fail to be listed in the final private LB.\n\nCould you clarify it? Thanks!", "replies": [ { "id": 598577, "postDate": "2019-08-13T18:36:11.960Z", "content": "Having output files in addition to images.zip should submit without an error, unless it exceeds a constraint (i.e. compute time) already defined.
I think it just means that we need to zip 10,000 images.
", "rawMarkdown": "I think it just means that we need to zip 10,000 images." }, { "id": 598709, "postDate": "2019-08-13T23:20:01.050Z", "content": "@juliaelliott Thanks, it's crystal clear now.
", "rawMarkdown": "@juliaelliott Thanks, it's crystal clear now." } ] }, { "id": 598361, "postDate": "2019-08-13T14:09:23.180Z", "content": "I am sorry , if it is already answered . But soon after the competition ends , can I just make my kernel public , rather than sharing it with the aforementioned team alone privately . One of my best score Kernel is already public for long time and I can make another one public too . If I do that , \na) People who want to just have a look and learn something , they don't need to wait for competition results .\nb) I am confident that I am not using any memorization or any tricky thing whatsoever . Whatever I tried is already public knowledge .
", "rawMarkdown": "I am sorry , if it is already answered . But soon after the competition ends , can I just make my kernel public , rather than sharing it with the aforementioned team alone privately . One of my best score Kernel is already public for long time and I can make another one public too . If I do that , \na) People who want to just have a look and learn something , they don't need to wait for competition results .\nb) I am confident that I am not using any memorization or any tricky thing whatsoever . Whatever I tried is already public knowledge . ", "replies": [ { "id": 598375, "postDate": "2019-08-13T14:27:08.753Z", "content": "Yes, this is fine. We will start reviewing the Kernels 24 hours following the competition’s close. So those who wish to make their Kernels public can wait until the competition closes, as long as it is done within 24 hours following the competition’s close.
", "rawMarkdown": "Yes, this is fine. We will start reviewing the Kernels 24 hours following the competition’s close. So those who wish to make their Kernels public can wait until the competition closes, as long as it is done within 24 hours following the competition’s close.", "votes": 5 } ] }, { "id": 598307, "postDate": "2019-08-13T12:51:56.507Z", "content": "good!
", "rawMarkdown": "good!" }, { "id": 598224, "postDate": "2019-08-13T09:43:51.480Z", "content": "💯
", "rawMarkdown": "💯 " }, { "id": 596991, "postDate": "2019-08-11T16:11:02.627Z", "content": "@wendykan @juliaelliott @andrewcybai \nwhen I share two submissions with admins, Is it OK to set 'Can view' instead of 'Can edit'?
", "rawMarkdown": "@wendykan @juliaelliott @andrewcybai \nwhen I share two submissions with admins, Is it OK to set 'Can view' instead of 'Can edit'?", "replies": [ { "id": 597890, "postDate": "2019-08-12T21:34:16.887Z", "content": "Yes, View access is fine.
", "rawMarkdown": "Yes, View access is fine." } ] }, { "id": 595622, "postDate": "2019-08-09T13:28:14.643Z", "content": "@wendykan @juliaelliott @andrewcybai \n Hello, do we have to share the kernels with all of you or only one of you ? :)\nThank you !
", "rawMarkdown": "@wendykan @juliaelliott @andrewcybai \n Hello, do we have to share the kernels with all of you or only one of you ? :)\nThank you !", "replies": [ { "id": 595689, "postDate": "2019-08-09T14:49:53.210Z", "content": "All of us, please.
", "rawMarkdown": "All of us, please.", "votes": 2 }, { "id": 595897, "postDate": "2019-08-09T20:10:15.463Z", "content": "Thank you for your answer @juliaelliott 👍
", "rawMarkdown": "Thank you for your answer @juliaelliott 👍 " } ] }, { "id": 592762, "postDate": "2019-08-05T19:36:30.280Z", "content": "@wendykan: You said here that Kernels like this from Chris are valid solutions for this competition. Is this still the case?
", "rawMarkdown": "@wendykan: You said [here](https://www.kaggle.com/c/generative-dog-images/discussion/98183) that Kernels like [this from Chris](https://www.kaggle.com/cdeotte/dog-memorizer-gan) are valid solutions for this competition. Is this still the case?", "replies": [ { "id": 592771, "postDate": "2019-08-05T19:48:58.007Z", "content": "It's not as memorized kernel are not allowed !!!!
", "rawMarkdown": "It's not as memorized kernel are not allowed !!!!" }, { "id": 592821, "postDate": "2019-08-05T21:08:43.373Z", "content": "This post from Wendy is making the updated clarification that memorization GANs are not permitted.
", "rawMarkdown": "This post from Wendy is making the updated clarification that **memorization GANs are not permitted.** ", "votes": 6 } ] }, { "id": 592729, "postDate": "2019-08-05T18:29:33.340Z", "content": "Is submission locked? I can't submit output archive from my kernel :(
", "rawMarkdown": "Is submission locked? I can't submit output archive from my kernel :(" }, { "id": 592087, "postDate": "2019-08-04T19:26:35.600Z", "content": "Thank you for the clarification, much appreciated!
", "rawMarkdown": "Thank you for the clarification, much appreciated! " }, { "id": 591944, "postDate": "2019-08-04T14:13:17.487Z", "content": "Courageous decisions indeed, Wendy !
", "rawMarkdown": "Courageous decisions indeed, Wendy !" }, { "id": 591942, "postDate": "2019-08-04T14:06:59.780Z", "content": "Great!
", "rawMarkdown": "Great!" }, { "id": 591786, "postDate": "2019-08-04T08:46:06.557Z", "content": "Public LB is meaningless and we don't know where we rank anymore
", "rawMarkdown": "Public LB is meaningless and we don't know where we rank anymore" }, { "id": 591715, "postDate": "2019-08-04T06:29:15.433Z", "content": "shake-ups ahead.
", "rawMarkdown": "shake-ups ahead." }, { "id": 591667, "postDate": "2019-08-04T05:07:54.497Z", "content": "This is great! :-)
", "rawMarkdown": "This is great! :-)" }, { "id": 591658, "postDate": "2019-08-04T04:48:39.067Z", "content": "Thanks for clarifying, sounds good to me, thanks.
", "rawMarkdown": "Thanks for clarifying, sounds good to me, thanks." }, { "id": 599557, "postDate": "2019-08-15T05:51:26.260Z", "rawMarkdown": "", "votes": -1, "isDeleted": true }, { "id": 597103, "postDate": "2019-08-11T20:31:03.083Z", "rawMarkdown": "", "isDeleted": true }, { "id": 595592, "postDate": "2019-08-09T12:29:37.793Z", "rawMarkdown": "", "isDeleted": true }, { "id": 591653, "postDate": "2019-08-04T04:36:48.563Z", "rawMarkdown": "", "isDeleted": true }, { "id": 601671, "postDate": "2019-08-18T02:31:03.750Z", "content": "thanks
", "rawMarkdown": "thanks" }, { "id": 593074, "postDate": "2019-08-06T06:55:29.643Z", "content": "Thank you.
", "rawMarkdown": "Thank you." }, { "id": 591830, "postDate": "2019-08-04T10:12:29.723Z", "content": "Thank you for the great decisions !!!!
", "rawMarkdown": "Thank you for the great decisions !!!!" }, { "id": 591666, "postDate": "2019-08-04T05:04:03.230Z", "content": "Thank you for sharing it！
", "rawMarkdown": "Thank you for sharing it！" } ], "comments": [ { "id": 598980, "author_name": "bestfitting", "author_url": "", "post_date": "2019-08-14T10:20:51.790000", "content": "Hi @juliaelliott, @wendykan, @andrewcybai,
\n\nThank for your announcement, it’s a great step to a successful competition, I seldom talk about rules and setting of a competition, I respect all the hard work of you have done and understand how difficult to host a competition, but I have been worrying about the evaluation metric since last week.
\n\nWhat I worried is the distance threshold of MiFID, I understand you want to is to prevent invalid GAN which just make modification to the train images or mem-GAN. As we know, you defined a memorization distance threshold, but the memorization distance is sensitive to feature extraction networks, I tested several networks and I found 0.1 is a dangerous threshold on InceptionResnetV2 and Xception.
\n\nI downloaded some other datasets containing dogs from kaggle, and calculated the memorization distance to all-dogs, they are all near 0.1. it means that even if our models can generate almost real dogs, they may be severely punished by more than 10x. On the contrary, the distance is about 0.15 on inception v3 and v4, it’s quite safe.
\n\nI also tested more than 30 models of mine scoring range from 31.x to 12.x, using these two networks to extract features , I am sure my GAN models are 100% valid GANs if StyleGAN,BigGAN,SA-GAN,AC-GAN are valid GANs, if the public scores are better than 30, the memorization distances of half of them were under 0.1, if we choose 0.1 as threshold, it will be horrible to some of the competitors whose scores are less than 30. As far as I know, there are many kagglers scores are below 30, for example, @dvorobiev @yukia18, we may score >100 and will not have chance be viewed by you.
\n\nSince you will check all the codes, the memorization distance is not so important now, IS/FID and other metrics commonly used on papers are good choices. If you want to use MiFID, I suggest you use BIGGAN to generate dogs and calculate the distance as a reference. https://tfhub.dev/deepmind/biggan-128/2
\n\nIf you want to verify my code, I have shared a kernel with public score 13.4 to you, if you have any question about it, please contact me, perhaps it’s too complex with more than 3000 lines in a kernel.
\n\nPerhaps it’s not convenient for you to answer me now, but please take consideration of my post here. If the problem is happen to me only, I don't expect I might benefit from this post. We all want to have a successful competition, especially such a creative one and encourage more kagglers to participate similar one in the future, I am waiting for a reinforcement learning one.
\n\nThank you for your consideration.
", "votes": 29, "replies": [ { "id": 599007, "author_name": "Dmitry Vorobiev", "author_url": "", "post_date": "2019-08-14T11:04:36.760000", "content": "Good point. Before you posted your findings, I thought my pictures are crappy enough so I don't need to worry about memorization :) But I never tried to test it with other architectures... Thanks for your efforts!
", "votes": 1, "replies": [] }, { "id": 599245, "author_name": "bestfitting", "author_url": "", "post_date": "2019-08-14T18:05:14.310000", "content": "I have just generated the dog images by using pretrained BigGan models from https://tfhub.dev/deepmind/biggan-128/2 and https://tfhub.dev/deepmind/biggan-256/2:
\nthe distances to all-dogs and scores of pretrained BigGan 128 model on\nInceptionV3: distance:0.130 FID: 21.9\nInceptionV4: distance:0.109 FID: 25.0\nInceptionResnetV2: distance:0.064 FID: 19.9 MFID:309.1\nXception: distance:0.096 FID: 0.097 FID:73.7 MFID:760.1\n
\nthe distances to all-dogs and scores of pretrained BigGan 256 model on\nInceptionV3: distance:0.128 FID: 24.0 \nInceptionV4: distance:0.105 FID: 27.5\nInceptionResnetV2: distance: 0.060 FID: 21.8 MFID:358.4\nXception: distance:0.097 FID: 30.0 MFID:310.0\n
\nthe distances to all-dogs and scores of Memorization-GAN on\nInceptionV3: distance:0.103 FID: 7.2\nInceptionV4: distance:0.097 FID: 6.6 MFID:68.1\nInceptionResnetV2: distance:0.059 FID: 4.3 MFID:74.1\nXception: distance:0.080 FID: 11.5 MFID:144.4
@bestfitting Thank you for your feedback, and we hear your worry. In an attempt to address your question the best we can: We will penalize based on the private evaluation metric, which includes a distance threshold that we've evaluated to be indicative of using invalid methods (i.e. direct image modifications that are not generative). The specifics of that private evaluation and the distance is, for obvious reasons, not going to be revealed. Because this is a highly experimental competition format, we will also be accompanying the objective metric score with manual code inspection, to also identify any prohibited methods that may not have been captured by objective means. We are confident these measures will support a fair evaluation of the final leaderboard. For those who are not attempting prohibited, non-generative methods, you should have nothing to fear.
", "votes": 8, "replies": [] }, { "id": 599422, "author_name": "Chris Deotte", "author_url": "", "post_date": "2019-08-15T00:07:59.013000", "content": "Wow, Big GAN makes amazing images and awesome latent walks. I never played with it until today. I posted it in a Kaggle kernel here. With truncation=1.0, it scores LB 15. Impressive.
Congratulations on another gold medal，maybe the top one。my god 😉
", "votes": 1, "replies": [] } ] }, { "id": 594873, "author_name": "", "author_url": "", "post_date": "2019-08-08T15:01:08.340000", "content": "i agree👍
", "votes": 8, "replies": [] }, { "id": 591700, "author_name": "yabea", "author_url": "", "post_date": "2019-08-04T06:16:06.823000", "content": "Great decisions!
", "votes": 8, "replies": [] }, { "id": 592527, "author_name": "hirune924", "author_url": "", "post_date": "2019-08-05T12:34:16.583000", "content": "At the end of the competition how are you going to determine Memorization GAN and Ordinary GAN?\nYou should see and answer this question now. We don't have time to wonder how to get good results. \nhttps://www.kaggle.com/c/generative-dog-images/discussion/99215#latest-592613
", "votes": 5, "replies": [] }, { "id": 599539, "author_name": "timetraveller", "author_url": "", "post_date": "2019-08-15T05:03:42.677000", "content": "The competition details say '17 days to go'. Is that the time required to compile the final result or is it just a bug?
", "votes": 6, "replies": [ { "id": 599553, "author_name": "Ilia Zaitsev", "author_url": "", "post_date": "2019-08-15T05:43:32.873000", "content": "It is time to verify the solutions, I believe, as far as the kernels are most probably inspected manually.
", "votes": 0, "replies": [] } ] }, { "id": 595915, "author_name": "Bojan Tunguz", "author_url": "", "post_date": "2019-08-09T20:47:55.897000", "content": "I wonder if there is something in the recent Kaggle competitions that prompted these very interesting new rules. 🤔
", "votes": 6, "replies": [] }, { "id": 599541, "author_name": "baigou", "author_url": "", "post_date": "2019-08-15T05:11:10.763000", "content": "17 days to go???😨
", "votes": 3, "replies": [] }, { "id": 592009, "author_name": "FGPC", "author_url": "", "post_date": "2019-08-04T16:07:09.523000", "content": "@wendy: since we have changes in competition logistics, is it possible for your team to consider resetting of the leaderboard as part of changes? many thanks! :-)
", "votes": 3, "replies": [ { "id": 592816, "author_name": "Julia Elliott", "author_url": "", "post_date": "2019-08-05T21:03:06.567000", "content": "We will not be resetting the leaderboard.
", "votes": 0, "replies": [] } ] }, { "id": 591659, "author_name": "Nirjhar Roy", "author_url": "", "post_date": "2019-08-04T04:52:45.090000", "content": "Thats cool. It would give motivation for many people.
", "votes": 3, "replies": [] }, { "id": 591691, "author_name": "timetraveller", "author_url": "", "post_date": "2019-08-04T06:05:50.553000", "content": "\n\n\nMemorization GANs not allowed. Please don't select them as your final submissions.
\n
Great! This has got a lot more interesting now. Especially because the public LB is completely meaningless and we have no idea where we stand.
", "votes": 4, "replies": [ { "id": 592088, "author_name": "Geosphere", "author_url": "", "post_date": "2019-08-04T19:29:27.643000", "content": "I agree with you. It's becoming nothing short of a wild west. I didn't expect so much drama for my first Kaggle competition.😏 😲 This will definitely motivate people for the last stretch of the competition 👍
", "votes": 1, "replies": [] } ] }, { "id": 600009, "author_name": "Julia Elliott", "author_url": "", "post_date": "2019-08-15T14:30:24.080000", "content": "The competition closed for submissions on August 14th. However, the deadline has been set to August 31st, so we can complete the scoring and review of all submissions, per Wendy’s 4th bullet of this post. No further submissions will be accepted.
", "votes": 1, "replies": [ { "id": 600263, "author_name": "Sara", "author_url": "", "post_date": "2019-08-15T21:23:29.160000", "content": "Is submission going to be opened after August 31?
", "votes": 0, "replies": [] } ] }, { "id": 591900, "author_name": "Felipe Loque", "author_url": "", "post_date": "2019-08-04T12:45:53.547000", "content": "Great!
", "votes": 1, "replies": [] }, { "id": 591721, "author_name": "Tom", "author_url": "", "post_date": "2019-08-04T06:37:52.330000", "content": "It's great !!\n@wendykan When is the deadline for sharing our final Kernels?
", "votes": 1, "replies": [ { "id": 592819, "author_name": "Julia Elliott", "author_url": "", "post_date": "2019-08-05T21:04:41.210000", "content": "Kernels must be shared (privately or publicly) by the updated competition deadline: August 14, 2019 11:59 PM UTC
", "votes": 5, "replies": [] }, { "id": 592867, "author_name": "Tom", "author_url": "", "post_date": "2019-08-05T23:18:32.443000", "content": "@juliaelliott Thanks for clarification !
", "votes": 1, "replies": [] }, { "id": 593090, "author_name": "Adiachen", "author_url": "", "post_date": "2019-08-06T07:20:00.537000", "content": "Is this means that we should share our code (privately or publicly) after the private leaderboard been displayed.
", "votes": 0, "replies": [] }, { "id": 593530, "author_name": "Julia Elliott", "author_url": "", "post_date": "2019-08-06T18:26:13.047000", "content": "@adiachen - the deadline to share your code is the deadline of the competition. So you should be sure to share it at the same time the competition ends.
", "votes": 2, "replies": [] }, { "id": 593809, "author_name": "quantumgeek", "author_url": "", "post_date": "2019-08-07T06:31:43.173000", "content": "@juliaelliott - so sharing the kernels publicly shouldn't be allowed, right? Someone else could just resubmit it?
", "votes": 0, "replies": [] }, { "id": 595082, "author_name": "Julia Elliott", "author_url": "", "post_date": "2019-08-08T20:21:43.533000", "content": "We will begin reviewing code 24 hours following the competition's deadline. Therefore, you must share your kernels by then. If your selected kernels have not been shared, you risk those scores being disqualified.
", "votes": 0, "replies": [] } ] }, { "id": 595097, "author_name": "Bojan Tunguz", "author_url": "", "post_date": "2019-08-08T20:40:04.757000", "content": "
Very Nice!
", "votes": 2, "replies": [] }, { "id": 591917, "author_name": "Kang", "author_url": "", "post_date": "2019-08-04T13:22:48.637000", "content": "How to share final kernels to admin privately or publicly?
", "votes": 2, "replies": [ { "id": 592266, "author_name": "something4kag", "author_url": "", "post_date": "2019-08-05T05:02:57.770000", "content": "Open the sidebar visibility, the ->| arrow icon top right\nFor publicly - in your kernel settings sharing click on private change from private to public.\nFor privately - in your kernel settings sharing click on private add the names designated as collaborators on your kernel.
", "votes": 1, "replies": [] }, { "id": 592516, "author_name": "Kang", "author_url": "", "post_date": "2019-08-05T12:11:40.610000", "content": "Thank you!
", "votes": 1, "replies": [] } ] }, { "id": 591682, "author_name": "MeisterMorxrc", "author_url": "", "post_date": "2019-08-04T05:42:48.100000", "content": "Sounds like the game is going to be interesting again
", "votes": 2, "replies": [] }, { "id": 600037, "author_name": "conformal", "author_url": "", "post_date": "2019-08-15T15:01:32.563000", "content": "I've selected a wrong kernal as final result, how to change it now?🤕
", "votes": 0, "replies": [ { "id": 600103, "author_name": "Julia Elliott", "author_url": "", "post_date": "2019-08-15T16:42:11.050000", "content": "Unfortunately, it is no longer possible to change your selections since the competition’s submission deadline has passed.
", "votes": 0, "replies": [] } ] }, { "id": 599365, "author_name": "empty", "author_url": "", "post_date": "2019-08-14T21:33:39.880000", "content": "@juliaelliott can't we submit with your metric dataset uploaded? T___T just noticed it
", "votes": 0, "replies": [] }, { "id": 599073, "author_name": "fajsoe", "author_url": "", "post_date": "2019-08-14T13:09:40.850000", "content": "Very Nice!
", "votes": 0, "replies": [] }, { "id": 598959, "author_name": "soda water", "author_url": "", "post_date": "2019-08-14T09:37:50.640000", "content": "cool
", "votes": 0, "replies": [] }, { "id": 598556, "author_name": "Mark Peng", "author_url": "", "post_date": "2019-08-13T17:56:37.420000", "content": "Hi @juliaelliott, @wendykan , I saw this description in the evaluation page:\n> You are going to generate 10,000 images that are in PNG format. Their sizes should be 64x64x3 (RGB). Then you need to zip those images and your output from your Kernel should only have ONE output file named images.zip.
\n\nWhat if we have more files in the output of Kernel? I think it is common for kagglers to also output a model file for local evaluations together with images.zip. If this is not allowed, many people might need to rerun their submissions with very limited quota left. Or they would fail to be listed in the final private LB.
\n\nCould you clarify it? Thanks!
", "votes": 0, "replies": [ { "id": 598577, "author_name": "Julia Elliott", "author_url": "", "post_date": "2019-08-13T18:36:11.960000", "content": "Having output files in addition to images.zip should submit without an error, unless it exceeds a constraint (i.e. compute time) already defined.
I think it just means that we need to zip 10,000 images.
", "votes": 0, "replies": [] }, { "id": 598709, "author_name": "Mark Peng", "author_url": "", "post_date": "2019-08-13T23:20:01.050000", "content": "@juliaelliott Thanks, it's crystal clear now.
", "votes": 0, "replies": [] } ] }, { "id": 598361, "author_name": "Nirjhar Roy", "author_url": "", "post_date": "2019-08-13T14:09:23.180000", "content": "I am sorry , if it is already answered . But soon after the competition ends , can I just make my kernel public , rather than sharing it with the aforementioned team alone privately . One of my best score Kernel is already public for long time and I can make another one public too . If I do that , \na) People who want to just have a look and learn something , they don't need to wait for competition results .\nb) I am confident that I am not using any memorization or any tricky thing whatsoever . Whatever I tried is already public knowledge .
", "votes": 0, "replies": [ { "id": 598375, "author_name": "Julia Elliott", "author_url": "", "post_date": "2019-08-13T14:27:08.753000", "content": "Yes, this is fine. We will start reviewing the Kernels 24 hours following the competition’s close. So those who wish to make their Kernels public can wait until the competition closes, as long as it is done within 24 hours following the competition’s close.
", "votes": 5, "replies": [] } ] }, { "id": 598307, "author_name": "Seong yun", "author_url": "", "post_date": "2019-08-13T12:51:56.507000", "content": "good!
", "votes": 0, "replies": [] }, { "id": 598224, "author_name": "Scofield", "author_url": "", "post_date": "2019-08-13T09:43:51.480000", "content": "💯
", "votes": 0, "replies": [] }, { "id": 596991, "author_name": "hirune924", "author_url": "", "post_date": "2019-08-11T16:11:02.627000", "content": "@wendykan @juliaelliott @andrewcybai \nwhen I share two submissions with admins, Is it OK to set 'Can view' instead of 'Can edit'?
", "votes": 0, "replies": [ { "id": 597890, "author_name": "Julia Elliott", "author_url": "", "post_date": "2019-08-12T21:34:16.887000", "content": "Yes, View access is fine.
", "votes": 0, "replies": [] } ] }, { "id": 595622, "author_name": "Arthur Llau", "author_url": "", "post_date": "2019-08-09T13:28:14.643000", "content": "@wendykan @juliaelliott @andrewcybai \n Hello, do we have to share the kernels with all of you or only one of you ? :)\nThank you !
", "votes": 0, "replies": [ { "id": 595689, "author_name": "Julia Elliott", "author_url": "", "post_date": "2019-08-09T14:49:53.210000", "content": "All of us, please.
", "votes": 2, "replies": [] }, { "id": 595897, "author_name": "Arthur Llau", "author_url": "", "post_date": "2019-08-09T20:10:15.463000", "content": "Thank you for your answer @juliaelliott 👍
", "votes": 0, "replies": [] } ] }, { "id": 592762, "author_name": "Karsten Tiemann", "author_url": "", "post_date": "2019-08-05T19:36:30.280000", "content": "@wendykan: You said here that Kernels like this from Chris are valid solutions for this competition. Is this still the case?
", "votes": 0, "replies": [ { "id": 592771, "author_name": "François", "author_url": "", "post_date": "2019-08-05T19:48:58.007000", "content": "It's not as memorized kernel are not allowed !!!!
", "votes": 0, "replies": [] }, { "id": 592821, "author_name": "Julia Elliott", "author_url": "", "post_date": "2019-08-05T21:08:43.373000", "content": "This post from Wendy is making the updated clarification that memorization GANs are not permitted.
", "votes": 6, "replies": [] } ] }, { "id": 592729, "author_name": "Ivan Balagurov", "author_url": "", "post_date": "2019-08-05T18:29:33.340000", "content": "Is submission locked? I can't submit output archive from my kernel :(
", "votes": 0, "replies": [] }, { "id": 592087, "author_name": "", "author_url": "", "post_date": "2019-08-04T19:26:35.600000", "content": "", "votes": 0, "replies": [] }, { "id": 591944, "author_name": "", "author_url": "", "post_date": "2019-08-04T14:13:17.487000", "content": "", "votes": 0, "replies": [] }, { "id": 591942, "author_name": "", "author_url": "", "post_date": "2019-08-04T14:06:59.780000", "content": "", "votes": 0, "replies": [] }, { "id": 591786, "author_name": "", "author_url": "", "post_date": "2019-08-04T08:46:06.557000", "content": "", "votes": 0, "replies": [] }, { "id": 591715, "author_name": "", "author_url": "", "post_date": "2019-08-04T06:29:15.433000", "content": "", "votes": 0, "replies": [] }, { "id": 591667, "author_name": "", "author_url": "", "post_date": "2019-08-04T05:07:54.497000", "content": "", "votes": 0, "replies": [] }, { "id": 591658, "author_name": "", "author_url": "", "post_date": "2019-08-04T04:48:39.067000", "content": "", "votes": 0, "replies": [] }, { "id": 599557, "author_name": "", "author_url": "", "post_date": "2019-08-15T05:51:26.260000", "content": "", "votes": -1, "replies": [] }, { "id": 597103, "author_name": "", "author_url": "", "post_date": "2019-08-11T20:31:03.083000", "content": "", "votes": 0, "replies": [] }, { "id": 595592, "author_name": "", "author_url": "", "post_date": "2019-08-09T12:29:37.793000", "content": "", "votes": 0, "replies": [] }, { "id": 591653, "author_name": "", "author_url": "", "post_date": "2019-08-04T04:36:48.563000", "content": "", "votes": 0, "replies": [] }, { "id": 601671, "author_name": "", "author_url": "", "post_date": "2019-08-18T02:31:03.750000", "content": "", "votes": 0, "replies": [] }, { "id": 593074, "author_name": "", "author_url": "", "post_date": "2019-08-06T06:55:29.643000", "content": "", "votes": 0, "replies": [] }, { "id": 591830, "author_name": "", "author_url": "", "post_date": "2019-08-04T10:12:29.723000", "content": "", "votes": 0, "replies": [] }, { "id": 591666, "author_name": "", "author_url": "", "post_date": "2019-08-04T05:04:03.230000", "content": "", "votes": 0, "replies": [] } ], "raw_markdown_by_id": { "591652": "We would like to announce a few decisions on the competition logistics before the end of competition. \n\n1. Due to the error reports, we would like to extend the deadline of the competition by 5 days. The new deadline is August 14. You have 11 more days before the deadline of the competition. \n\n2. We will review the final selected submission Kernels for all teams earning medals. Therefore, you are required to either publicly or privately share your final Kernels with the admins: Wendy Kan (@wendykan), Ching-yuan Bai (@andrewcybai), and Julia Elliott (@juliaelliott). If a team fails to share their Kernels with us, we have the right to remove them from the leaderboard. \n\n3. Memorization GANs not allowed. Please don't select them as your final submissions. \n\n4. We will not be revealing the private leaderboard immediately at the end of the competition. We will perform the code inspection and wait to release the private leaderboard when it's final. \n\nThank you so much for participating in this competition! We look forward to seeing the results. ", "598980": "Hi @juliaelliott, @wendykan, @andrewcybai,\n\nThank for your announcement, it’s a great step to a successful competition, I seldom talk about rules and setting of a competition, I respect all the hard work of you have done and understand how difficult to host a competition, but I have been worrying about the evaluation metric since last week.\n\nWhat I worried is the distance threshold of MiFID, I understand you want to is to prevent invalid GAN which just make modification to the train images or mem-GAN. As we know, you defined a memorization distance threshold, but the memorization distance is sensitive to feature extraction networks, I tested several networks and I found 0.1 is a dangerous threshold on InceptionResnetV2 and Xception. \n\nI downloaded some other datasets containing dogs from kaggle, and calculated the memorization distance to all-dogs, they are all near 0.1. it means that even if our models can generate almost real dogs, they may be severely punished by more than 10x. On the contrary, the distance is about 0.15 on inception v3 and v4, it’s quite safe. \n\nI also tested more than 30 models of mine scoring range from 31.x to 12.x, using these two networks to extract features , I am sure my GAN models are 100% valid GANs if StyleGAN,BigGAN,SA-GAN,AC-GAN are valid GANs, if the public scores are better than 30, the memorization distances of half of them were under 0.1, if we choose 0.1 as threshold, it will be horrible to some of the competitors whose scores are less than 30. As far as I know, there are many kagglers scores are below 30, for example, @dvorobiev @yukia18, we may score >100 and will not have chance be viewed by you.\n\nSince you will check all the codes, the memorization distance is not so important now, IS/FID and other metrics commonly used on papers are good choices. If you want to use MiFID, I suggest you use BIGGAN to generate dogs and calculate the distance as a reference. https://tfhub.dev/deepmind/biggan-128/2\n\nIf you want to verify my code, I have shared a kernel with public score 13.4 to you, if you have any question about it, please contact me, perhaps it’s too complex with more than 3000 lines in a kernel.\n\nPerhaps it’s not convenient for you to answer me now, but please take consideration of my post here. If the problem is happen to me only, I don't expect I might benefit from this post. We all want to have a successful competition, especially such a creative one and encourage more kagglers to participate similar one in the future, I am waiting for a reinforcement learning one. \n\nThank you for your consideration.\n\n", "594873": "i agree👍 ", "591700": "Great decisions!", "592527": "At the end of the competition how are you going to determine Memorization GAN and Ordinary GAN?\nYou should see and answer this question now. We don't have time to wonder how to get good results. \nhttps://www.kaggle.com/c/generative-dog-images/discussion/99215#latest-592613", "599539": "The competition details say '17 days to go'. Is that the time required to compile the final result or is it just a bug?", "595915": "I wonder if there is something in the recent Kaggle competitions that prompted these very interesting new rules. 🤔 ", "599541": "17 days to go???😨 ", "592009": "@wendy: since we have changes in competition logistics, is it possible for your team to consider resetting of the leaderboard as part of changes? many thanks! :-)", "591659": "Thats cool. It would give motivation for many people. ", "591691": ">Memorization GANs not allowed. Please don't select them as your final submissions. \n\nGreat! This has got a lot more interesting now. Especially because the public LB is completely meaningless and we have no idea where we stand. ", "600009": "The competition closed for submissions on August 14th. However, the deadline has been set to August 31st, so we can complete the scoring and review of all submissions, per Wendy’s 4th bullet of this post. No further submissions will be accepted.", "591900": "Great!", "591721": "It's great !!\n@wendykan When is the deadline for sharing our final Kernels?", "595097": "", "595089": "Very Nice!", "591917": "How to share final kernels to admin privately or publicly?", "591682": "Sounds like the game is going to be interesting again", "600037": "I've selected a wrong kernal as final result, how to change it now?🤕 ", "599365": " @juliaelliott can't we submit with your metric dataset uploaded? T___T just noticed it", "599073": "Very Nice!", "598959": "cool", "598556": "Hi @juliaelliott, @wendykan , I saw this description in the evaluation page:\n> You are going to generate 10,000 images that are in PNG format. Their sizes should be 64x64x3 (RGB). Then you need to zip those images and your output from your Kernel should only have ONE output file named images.zip.\n\nWhat if we have more files in the output of Kernel? I think it is common for kagglers to also output a model file for local evaluations together with images.zip. If this is not allowed, many people might need to rerun their submissions with very limited quota left. Or they would fail to be listed in the final private LB.\n\nCould you clarify it? Thanks!", "598361": "I am sorry , if it is already answered . But soon after the competition ends , can I just make my kernel public , rather than sharing it with the aforementioned team alone privately . One of my best score Kernel is already public for long time and I can make another one public too . If I do that , \na) People who want to just have a look and learn something , they don't need to wait for competition results .\nb) I am confident that I am not using any memorization or any tricky thing whatsoever . Whatever I tried is already public knowledge . ", "598307": "good!", "598224": "💯 ", "596991": "@wendykan @juliaelliott @andrewcybai \nwhen I share two submissions with admins, Is it OK to set 'Can view' instead of 'Can edit'?", "595622": "@wendykan @juliaelliott @andrewcybai \n Hello, do we have to share the kernels with all of you or only one of you ? :)\nThank you !", "592762": "@wendykan: You said [here](https://www.kaggle.com/c/generative-dog-images/discussion/98183) that Kernels like [this from Chris](https://www.kaggle.com/cdeotte/dog-memorizer-gan) are valid solutions for this competition. Is this still the case?", "592729": "Is submission locked? I can't submit output archive from my kernel :(", "592087": "Thank you for the clarification, much appreciated! ", "591944": "Courageous decisions indeed, Wendy !", "591942": "Great!", "591786": "Public LB is meaningless and we don't know where we rank anymore", "591715": "shake-ups ahead.", "591667": "This is great! :-)", "591658": "Thanks for clarifying, sounds good to me, thanks.", "599557": "", "597103": "", "595592": "", "591653": "", "601671": "thanks", "593074": "Thank you.", "591830": "Thank you for the great decisions !!!!", "591666": "Thank you for sharing it！" } }