{
  "id": 98912,
  "title": "Maybe it is a good idea to restart competition with the new score?",
  "url": "/competitions/generative-dog-images/discussion/98912",
  "author_name": "Vitaly Bondar",
  "post_date": "2019-07-07T14:31:12.571000",
  "votes": 19,
  "comment_count": 0,
  "views": 0,
  "content": "<p>Hello,</p>\n\n<p>That is great that Kaggle tries to experiment with the competitions for generating content. But @cdeotte brilliantly shown that you can just memorize images and got a good score. So it stops people form competing and stoping from the search of real generative models.</p>\n\n<p>I want to ask admins to pay attention to a very interesting paper of NVidia researches: Improved Precision and Recall Metric for Assessing Generative Models (<a href=\"https://arxiv.org/abs/1904.06991\">https://arxiv.org/abs/1904.06991</a>).\nThey show that low FID can be achieved by good precision (memorization do this) or by the good recall.</p>\n\n<p>So what if to use their method to calculate distributions precision and recalls and combine them to some distribution F1-score or whatever (with the unknown dog's dataset as reference). I believe that will help a lot because memorization only will not really help.</p>\n\n<p>I strongly belive that different score is much better than trying to stop something like memorization with 100 points in the rule and manual verification.\nI start this thread more for the discussion, probably community have other tips how to avoid memorization. </p>",
  "messages": [
    {
      "id": 569901,
      "postDate": "2019-07-07T14:31:12.573Z",
      "content": "<p>Hello,</p>\n\n<p>That is great that Kaggle tries to experiment with the competitions for generating content. But @cdeotte brilliantly shown that you can just memorize images and got a good score. So it stops people form competing and stoping from the search of real generative models.</p>\n\n<p>I want to ask admins to pay attention to a very interesting paper of NVidia researches: Improved Precision and Recall Metric for Assessing Generative Models (<a href=\"https://arxiv.org/abs/1904.06991\">https://arxiv.org/abs/1904.06991</a>).\nThey show that low FID can be achieved by good precision (memorization do this) or by the good recall.</p>\n\n<p>So what if to use their method to calculate distributions precision and recalls and combine them to some distribution F1-score or whatever (with the unknown dog's dataset as reference). I believe that will help a lot because memorization only will not really help.</p>\n\n<p>I strongly belive that different score is much better than trying to stop something like memorization with 100 points in the rule and manual verification.\nI start this thread more for the discussion, probably community have other tips how to avoid memorization. </p>",
      "rawMarkdown": "Hello,\n\nThat is great that Kaggle tries to experiment with the competitions for generating content. But @cdeotte brilliantly shown that you can just memorize images and got a good score. So it stops people form competing and stoping from the search of real generative models.\n\nI want to ask admins to pay attention to a very interesting paper of NVidia researches: Improved Precision and Recall Metric for Assessing Generative Models (https://arxiv.org/abs/1904.06991).\nThey show that low FID can be achieved by good precision (memorization do this) or by the good recall.\n\nSo what if to use their method to calculate distributions precision and recalls and combine them to some distribution F1-score or whatever (with the unknown dog's dataset as reference). I believe that will help a lot because memorization only will not really help.\n\nI strongly belive that different score is much better than trying to stop something like memorization with 100 points in the rule and manual verification.\nI start this thread more for the discussion, probably community have other tips how to avoid memorization. \n",
      "votes": 19
    }
  ],
  "comments": [],
  "raw_markdown_by_id": {
    "569901": "Hello,\n\nThat is great that Kaggle tries to experiment with the competitions for generating content. But @cdeotte brilliantly shown that you can just memorize images and got a good score. So it stops people form competing and stoping from the search of real generative models.\n\nI want to ask admins to pay attention to a very interesting paper of NVidia researches: Improved Precision and Recall Metric for Assessing Generative Models (https://arxiv.org/abs/1904.06991).\nThey show that low FID can be achieved by good precision (memorization do this) or by the good recall.\n\nSo what if to use their method to calculate distributions precision and recalls and combine them to some distribution F1-score or whatever (with the unknown dog's dataset as reference). I believe that will help a lot because memorization only will not really help.\n\nI strongly belive that different score is much better than trying to stop something like memorization with 100 points in the rule and manual verification.\nI start this thread more for the discussion, probably community have other tips how to avoid memorization. \n"
  }
}