{
  "id": 97780,
  "title": "How to valid your models inside a kernel if no external sources?",
  "url": "/competitions/generative-dog-images/discussion/97780",
  "author_name": "",
  "post_date": "2019-06-28T20:34:33.889843300Z",
  "votes": 9,
  "comment_count": 3,
  "views": 0,
  "content": "<p>I am super excited on the first Generative model competition, however, I do think there are some problems need to be handled. </p>\n\n<p>For instance, since there is no straightforward way to validate the models purely from datasets. Since the public LB is using Inception and Imagenet to calculate the evaluation metric, but we don't have the access to those resources in the kernel.</p>\n\n<p>Does that mean we could only test the model outside the kernel for submission?</p>",
  "messages": [
    {
      "id": "563965",
      "postDate": "06/28/2019 20:34:33",
      "content": "<p>I am super excited on the first Generative model competition, however, I do think there are some problems need to be handled. </p>\n\n<p>For instance, since there is no straightforward way to validate the models purely from datasets. Since the public LB is using Inception and Imagenet to calculate the evaluation metric, but we don't have the access to those resources in the kernel.</p>\n\n<p>Does that mean we could only test the model outside the kernel for submission?</p>",
      "rawMarkdown": "I am super excited on the first Generative model competition, however, I do think there are some problems need to be handled. \n\nFor instance, since there is no straightforward way to validate the models purely from datasets. Since the public LB is using Inception and Imagenet to calculate the evaluation metric, but we don't have the access to those resources in the kernel.\n\nDoes that mean we could only test the model outside the kernel for submission?",
      "votes": null
    },
    {
      "id": "563992",
      "postDate": "06/28/2019 21:16:14",
      "content": "<p>You can fork <a href=\"https://www.kaggle.com/wendykan/demo-mifid-metric-for-dog-image-generation-comp\">this</a> evaluation Kernel, and while you're at the editor, hit \"Add Dataset\" and add your own Kernel output as a data source. Then you can get your public score. You're of course welcome to run it on your local computer. </p>",
      "rawMarkdown": "You can fork [this](https://www.kaggle.com/wendykan/demo-mifid-metric-for-dog-image-generation-comp) evaluation Kernel, and while you're at the editor, hit \"Add Dataset\" and add your own Kernel output as a data source. Then you can get your public score. You're of course welcome to run it on your local computer.",
      "votes": null
    },
    {
      "id": "564121",
      "postDate": "06/29/2019 02:17:58",
      "content": "<p>It is really important to validate my models while training to select best ones. GANs can be easily collapsed, and valilla GANs objective doesn't represent generated image's quality.\nI think there are several options,\n- Use WGAN families,  and pick up the model which has lowest approximated Wasserstein distance. It doesn't reflect best FID but seems to reflect generated image's quality.\n- Experiment locally using pretrained network to calculate FID (Don't forget to validate its reproducibility between local and kernel), and determine best epoch and rerun on kernel.</p>\n\n<p>Other options? Or there are some unsuitable points above options?</p>",
      "rawMarkdown": "It is really important to validate my models while training to select best ones. GANs can be easily collapsed, and valilla GANs objective doesn't represent generated image's quality.\nI think there are several options,\n- Use WGAN families,  and pick up the model which has lowest approximated Wasserstein distance. It doesn't reflect best FID but seems to reflect generated image's quality.\n- Experiment locally using pretrained network to calculate FID (Don't forget to validate its reproducibility between local and kernel), and determine best epoch and rerun on kernel.\n\nOther options? Or there are some unsuitable points above options?",
      "votes": null
    },
    {
      "id": "564399",
      "postDate": "06/29/2019 11:15:40",
      "content": "<p>Great, thanks, this is helpful</p>",
      "rawMarkdown": "Great, thanks, this is helpful",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 563992,
      "author_name": "wendykan",
      "author_url": "",
      "post_date": "06/28/2019 21:16:14",
      "content": "<p>You can fork <a href=\"https://www.kaggle.com/wendykan/demo-mifid-metric-for-dog-image-generation-comp\">this</a> evaluation Kernel, and while you're at the editor, hit \"Add Dataset\" and add your own Kernel output as a data source. Then you can get your public score. You're of course welcome to run it on your local computer. </p>",
      "votes": null,
      "replies": [
        {
          "id": 564399,
          "author_name": "cdeotte",
          "author_url": "",
          "post_date": "06/29/2019 11:15:40",
          "content": "<p>Great, thanks, this is helpful</p>",
          "votes": null,
          "replies": []
        }
      ]
    },
    {
      "id": 564121,
      "author_name": "yukia18",
      "author_url": "",
      "post_date": "06/29/2019 02:17:58",
      "content": "<p>It is really important to validate my models while training to select best ones. GANs can be easily collapsed, and valilla GANs objective doesn't represent generated image's quality.\nI think there are several options,\n- Use WGAN families,  and pick up the model which has lowest approximated Wasserstein distance. It doesn't reflect best FID but seems to reflect generated image's quality.\n- Experiment locally using pretrained network to calculate FID (Don't forget to validate its reproducibility between local and kernel), and determine best epoch and rerun on kernel.</p>\n\n<p>Other options? Or there are some unsuitable points above options?</p>",
      "votes": null,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "563965": "I am super excited on the first Generative model competition, however, I do think there are some problems need to be handled. \n\nFor instance, since there is no straightforward way to validate the models purely from datasets. Since the public LB is using Inception and Imagenet to calculate the evaluation metric, but we don't have the access to those resources in the kernel.\n\nDoes that mean we could only test the model outside the kernel for submission?",
    "563992": "You can fork [this](https://www.kaggle.com/wendykan/demo-mifid-metric-for-dog-image-generation-comp) evaluation Kernel, and while you're at the editor, hit \"Add Dataset\" and add your own Kernel output as a data source. Then you can get your public score. You're of course welcome to run it on your local computer.",
    "564121": "It is really important to validate my models while training to select best ones. GANs can be easily collapsed, and valilla GANs objective doesn't represent generated image's quality.\nI think there are several options,\n- Use WGAN families,  and pick up the model which has lowest approximated Wasserstein distance. It doesn't reflect best FID but seems to reflect generated image's quality.\n- Experiment locally using pretrained network to calculate FID (Don't forget to validate its reproducibility between local and kernel), and determine best epoch and rerun on kernel.\n\nOther options? Or there are some unsuitable points above options?",
    "564399": "Great, thanks, this is helpful"
  },
  "source": "meta"
}