{
  "id": 50023,
  "title": "Problem of re-annotating kaggle dataset and external dataset",
  "url": "/competitions/data-science-bowl-2018/discussion/50023",
  "author_name": "",
  "post_date": "2018-02-18T14:38:39.984018400Z",
  "votes": 7,
  "comment_count": 1,
  "views": 0,
  "content": "<p>there is a big problem here. The way we annotate will not be the same as the way kaggle has annotated that data. Assume we are only using the kaggle dataset and we carried out the experiment below:</p>\n\n<p>experiment A:</p>\n\n<ul>\n<li><p>train_set = kaggle annotation, validation_set = kaggle annotation, </p></li>\n<li><p>results e.g. validation = 0.40, LB =0.30?</p></li>\n</ul>\n\n<p>experiment B: </p>\n\n<ul>\n<li><p>train_set = user annotation, validation_set = user annotation, </p></li>\n<li><p>e.g. validation = 0.40, LB =0.25?</p></li>\n</ul>\n\n<p>experiment C:</p>\n\n<ul>\n<li><p>train_set = kaggle annotation, validation_set = user annotation, </p></li>\n<li><p>e.g. validation = 0.30,  LB =0.30?</p></li>\n</ul>\n\n<p>experiment D: </p>\n\n<ul>\n<li><p>train_set = user annotation, validation_set = kaggle annotation, </p></li>\n<li><p>e.g. validation = 0.30, LB =0.25?</p></li>\n</ul>\n\n<p>the metric scores are just for illustrative purposes. you should do your experiment to verify.</p>\n\n<hr>\n\n<p>why?</p>\n\n<p>the LB metric is average precision. Different annotation affects precision at high IOU (e.g. threshold 0.8, 0.9). If the LB metric is only precision at IOU=0.5, maybe the user annotation will not be an issue</p>\n\n<hr>\n\n<p>what can you do?</p>\n\n<p>this challenge is about generalization. using external data will certainly help. You can break detection and segmentation into two separate problem. Detection is not so sensitive to user annotation.</p>\n\n<p>For segmentation, maybe you can only use kaggle annotation for training or you need to find away to \"transfer\" from user to kaggle annotation.</p>\n\n<p>Other solution includes, using trained model (trained on kaggle annotation) to create/modify mask ground truth of new dataset instead of just using user to create mask ground truth. </p>\n\n<p>Finally, remember that you need to predict kaggle annotation to pixel level, not just any annotation.</p>",
  "messages": [
    {
      "id": "284836",
      "postDate": "02/18/2018 14:38:39",
      "content": "<p>there is a big problem here. The way we annotate will not be the same as the way kaggle has annotated that data. Assume we are only using the kaggle dataset and we carried out the experiment below:</p>\n\n<p>experiment A:</p>\n\n<ul>\n<li><p>train_set = kaggle annotation, validation_set = kaggle annotation, </p></li>\n<li><p>results e.g. validation = 0.40, LB =0.30?</p></li>\n</ul>\n\n<p>experiment B: </p>\n\n<ul>\n<li><p>train_set = user annotation, validation_set = user annotation, </p></li>\n<li><p>e.g. validation = 0.40, LB =0.25?</p></li>\n</ul>\n\n<p>experiment C:</p>\n\n<ul>\n<li><p>train_set = kaggle annotation, validation_set = user annotation, </p></li>\n<li><p>e.g. validation = 0.30,  LB =0.30?</p></li>\n</ul>\n\n<p>experiment D: </p>\n\n<ul>\n<li><p>train_set = user annotation, validation_set = kaggle annotation, </p></li>\n<li><p>e.g. validation = 0.30, LB =0.25?</p></li>\n</ul>\n\n<p>the metric scores are just for illustrative purposes. you should do your experiment to verify.</p>\n\n<hr>\n\n<p>why?</p>\n\n<p>the LB metric is average precision. Different annotation affects precision at high IOU (e.g. threshold 0.8, 0.9). If the LB metric is only precision at IOU=0.5, maybe the user annotation will not be an issue</p>\n\n<hr>\n\n<p>what can you do?</p>\n\n<p>this challenge is about generalization. using external data will certainly help. You can break detection and segmentation into two separate problem. Detection is not so sensitive to user annotation.</p>\n\n<p>For segmentation, maybe you can only use kaggle annotation for training or you need to find away to \"transfer\" from user to kaggle annotation.</p>\n\n<p>Other solution includes, using trained model (trained on kaggle annotation) to create/modify mask ground truth of new dataset instead of just using user to create mask ground truth. </p>\n\n<p>Finally, remember that you need to predict kaggle annotation to pixel level, not just any annotation.</p>",
      "rawMarkdown": "there is a big problem here. The way we annotate will not be the same as the way kaggle has annotated that data. Assume we are only using the kaggle dataset and we carried out the experiment below:\n\nexperiment A:\n\n- train_set = kaggle annotation, validation_set = kaggle annotation, \n\n- results e.g. validation = 0.40, LB =0.30?\n\n\nexperiment B: \n\n- train_set = user annotation, validation_set = user annotation, \n\n- e.g. validation = 0.40, LB =0.25?\n\n\nexperiment C:\n\n- train_set = kaggle annotation, validation_set = user annotation, \n\n- e.g. validation = 0.30,  LB =0.30?\n\n\nexperiment D: \n\n- train_set = user annotation, validation_set = kaggle annotation, \n\n- e.g. validation = 0.30, LB =0.25?\n\n\nthe metric scores are just for illustrative purposes. you should do your experiment to verify.\n\n---\n\nwhy?\n\nthe LB metric is average precision. Different annotation affects precision at high IOU (e.g. threshold 0.8, 0.9). If the LB metric is only precision at IOU=0.5, maybe the user annotation will not be an issue\n\n---\n what can you do?\n\nthis challenge is about generalization. using external data will certainly help. You can break detection and segmentation into two separate problem. Detection is not so sensitive to user annotation.\n\nFor segmentation, maybe you can only use kaggle annotation for training or you need to find away to \"transfer\" from user to kaggle annotation.\n\nOther solution includes, using trained model (trained on kaggle annotation) to create/modify mask ground truth of new dataset instead of just using user to create mask ground truth. \n\nFinally, remember that you need to predict kaggle annotation to pixel level, not just any annotation.",
      "votes": null
    },
    {
      "id": "285303",
      "postDate": "02/19/2018 21:34:13",
      "content": "<p>The validation set must represent the test set so the only relevant experiments are A and D (do user annotations improve performance on a validation set that does not use them as the test set?).</p>",
      "rawMarkdown": "The validation set must represent the test set so the only relevant experiments are A and D (do user annotations improve performance on a validation set that does not use them as the test set?).",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 285303,
      "author_name": "brasnold",
      "author_url": "",
      "post_date": "02/19/2018 21:34:13",
      "content": "<p>The validation set must represent the test set so the only relevant experiments are A and D (do user annotations improve performance on a validation set that does not use them as the test set?).</p>",
      "votes": null,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "284836": "there is a big problem here. The way we annotate will not be the same as the way kaggle has annotated that data. Assume we are only using the kaggle dataset and we carried out the experiment below:\n\nexperiment A:\n\n- train_set = kaggle annotation, validation_set = kaggle annotation, \n\n- results e.g. validation = 0.40, LB =0.30?\n\n\nexperiment B: \n\n- train_set = user annotation, validation_set = user annotation, \n\n- e.g. validation = 0.40, LB =0.25?\n\n\nexperiment C:\n\n- train_set = kaggle annotation, validation_set = user annotation, \n\n- e.g. validation = 0.30,  LB =0.30?\n\n\nexperiment D: \n\n- train_set = user annotation, validation_set = kaggle annotation, \n\n- e.g. validation = 0.30, LB =0.25?\n\n\nthe metric scores are just for illustrative purposes. you should do your experiment to verify.\n\n---\n\nwhy?\n\nthe LB metric is average precision. Different annotation affects precision at high IOU (e.g. threshold 0.8, 0.9). If the LB metric is only precision at IOU=0.5, maybe the user annotation will not be an issue\n\n---\n what can you do?\n\nthis challenge is about generalization. using external data will certainly help. You can break detection and segmentation into two separate problem. Detection is not so sensitive to user annotation.\n\nFor segmentation, maybe you can only use kaggle annotation for training or you need to find away to \"transfer\" from user to kaggle annotation.\n\nOther solution includes, using trained model (trained on kaggle annotation) to create/modify mask ground truth of new dataset instead of just using user to create mask ground truth. \n\nFinally, remember that you need to predict kaggle annotation to pixel level, not just any annotation.",
    "285303": "The validation set must represent the test set so the only relevant experiments are A and D (do user annotations improve performance on a validation set that does not use them as the test set?)."
  },
  "source": "meta"
}