{
  "id": 49995,
  "title": "Discrepancy between training score and Kaggle submission score.",
  "url": "/competitions/data-science-bowl-2018/discussion/49995",
  "author_name": "",
  "post_date": "2018-02-17T20:53:13.625139900Z",
  "votes": null,
  "comment_count": 4,
  "views": 0,
  "content": "<p>Hi everyone, </p>\n\n<p>Over the past week or so we began our attempt at this challenge. This first steps for us have been to establish the process of running a model, evaluating the results on the training data set, and submitting it to kaggle. Our idea was to use a very simple segmentation method and submit this as an example. We used the evaluation method that William Cukierski posted. This uses np.histogram2D to calcualte the intersection between the two images. We were surprised to see that we achieved an mean average precision (over all training images) of 43%. We were expecting much lower. When submitting a Kaggle we received a score of .26, which is more along the lines of what we were expecting. After dissecting the evaluation method and comparing a few annotated ground truth images with the corresponding annotated predicted images we have been unable to identify an error. The assigned scores appear to be correct.</p>\n\n<p>Has anyone experienced this?  A score of .43 is much to high and the disparity between the kaggle score and our score on the training data seems imply that as well. Could it be that the test images are much more difficult to segment?</p>",
  "messages": [
    {
      "id": "284647",
      "postDate": "02/17/2018 20:53:13",
      "content": "<p>Hi everyone, </p>\n\n<p>Over the past week or so we began our attempt at this challenge. This first steps for us have been to establish the process of running a model, evaluating the results on the training data set, and submitting it to kaggle. Our idea was to use a very simple segmentation method and submit this as an example. We used the evaluation method that William Cukierski posted. This uses np.histogram2D to calcualte the intersection between the two images. We were surprised to see that we achieved an mean average precision (over all training images) of 43%. We were expecting much lower. When submitting a Kaggle we received a score of .26, which is more along the lines of what we were expecting. After dissecting the evaluation method and comparing a few annotated ground truth images with the corresponding annotated predicted images we have been unable to identify an error. The assigned scores appear to be correct.</p>\n\n<p>Has anyone experienced this?  A score of .43 is much to high and the disparity between the kaggle score and our score on the training data seems imply that as well. Could it be that the test images are much more difficult to segment?</p>",
      "rawMarkdown": "Hi everyone, \n\nOver the past week or so we began our attempt at this challenge. This first steps for us have been to establish the process of running a model, evaluating the results on the training data set, and submitting it to kaggle. Our idea was to use a very simple segmentation method and submit this as an example. We used the evaluation method that William Cukierski posted. This uses np.histogram2D to calcualte the intersection between the two images. We were surprised to see that we achieved an mean average precision (over all training images) of 43%. We were expecting much lower. When submitting a Kaggle we received a score of .26, which is more along the lines of what we were expecting. After dissecting the evaluation method and comparing a few annotated ground truth images with the corresponding annotated predicted images we have been unable to identify an error. The assigned scores appear to be correct.\n\nHas anyone experienced this?  A score of .43 is much to high and the disparity between the kaggle score and our score on the training data seems imply that as well. Could it be that the test images are much more difficult to segment?",
      "votes": null
    },
    {
      "id": "284827",
      "postDate": "02/18/2018 14:01:05",
      "content": "<p>If you look at test images closely, you will find out that there are 3 different types of them, and 2 types are not present in train set, so you simply overfitted. </p>",
      "rawMarkdown": "If you look at test images closely, you will find out that there are 3 different types of them, and 2 types are not present in train set, so you simply overfitted.",
      "votes": null
    },
    {
      "id": "285604",
      "postDate": "02/20/2018 17:34:04",
      "content": "<p>Hi Emil,</p>\n\n<p>Thanks for your reply.</p>\n\n<p>The segmentation approach uses a simple Otsu thresholding, so I don't think it can be overfitting on the training data.</p>\n\n<p>Also, I have created some montage images to eyeball whether the train and test image sets look significantly different, as far as I can tell they do not. Please could you elaborate on which images types are missing in the train?</p>\n\n<h2>Train images</h2>\n\n<p><img src=\"https://imgur.com/a/bcg8D\" alt=\"train images montage\"></p>\n\n<h2>Test images</h2>\n\n<p><img src=\"https://imgur.com/a/qwYmL\" alt=\"test images montage\"></p>",
      "rawMarkdown": "Hi Emil,\n\nThanks for your reply.\n\nThe segmentation approach uses a simple Otsu thresholding, so I don't think it can be overfitting on the training data.\n\nAlso, I have created some montage images to eyeball whether the train and test image sets look significantly different, as far as I can tell they do not. Please could you elaborate on which images types are missing in the train?\n\n## Train images\n![train images montage][1]\n\n## Test images\n![test images montage][2]\n\n\n  [1]: https://imgur.com/a/bcg8D\n  [2]: https://imgur.com/a/qwYmL",
      "votes": null
    },
    {
      "id": "285618",
      "postDate": "02/20/2018 18:21:55",
      "content": "<p>Look in \"3 different modalities. Why?\" discussion.  There I posted all types of images in train &amp; test sets</p>",
      "rawMarkdown": "Look in \"3 different modalities. Why?\" discussion.  There I posted all types of images in train &amp; test sets",
      "votes": null
    },
    {
      "id": "285634",
      "postDate": "02/20/2018 19:09:18",
      "content": "<p><a href=\"https://www.kaggle.com/c/data-science-bowl-2018/discussion/47640\">3 different modalities. Why?</a></p>\n\n<p>Great thanks!</p>\n\n<p>By the way I think the images I tried to post did not work how/where can I host images in kaggle forums? I tried imgur but looks lik it did not work</p>",
      "rawMarkdown": "[3 different modalities. Why?][1]\n\nGreat thanks!\n\nBy the way I think the images I tried to post did not work how/where can I host images in kaggle forums? I tried imgur but looks lik it did not work\n\n  [1]: https://www.kaggle.com/c/data-science-bowl-2018/discussion/47640",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 284827,
      "author_name": "bonlime",
      "author_url": "",
      "post_date": "02/18/2018 14:01:05",
      "content": "<p>If you look at test images closely, you will find out that there are 3 different types of them, and 2 types are not present in train set, so you simply overfitted. </p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 285604,
      "author_name": "williamgrimes",
      "author_url": "",
      "post_date": "02/20/2018 17:34:04",
      "content": "<p>Hi Emil,</p>\n\n<p>Thanks for your reply.</p>\n\n<p>The segmentation approach uses a simple Otsu thresholding, so I don't think it can be overfitting on the training data.</p>\n\n<p>Also, I have created some montage images to eyeball whether the train and test image sets look significantly different, as far as I can tell they do not. Please could you elaborate on which images types are missing in the train?</p>\n\n<h2>Train images</h2>\n\n<p><img src=\"https://imgur.com/a/bcg8D\" alt=\"train images montage\"></p>\n\n<h2>Test images</h2>\n\n<p><img src=\"https://imgur.com/a/qwYmL\" alt=\"test images montage\"></p>",
      "votes": null,
      "replies": [
        {
          "id": 285618,
          "author_name": "bonlime",
          "author_url": "",
          "post_date": "02/20/2018 18:21:55",
          "content": "<p>Look in \"3 different modalities. Why?\" discussion.  There I posted all types of images in train &amp; test sets</p>",
          "votes": null,
          "replies": []
        }
      ]
    },
    {
      "id": 285634,
      "author_name": "williamgrimes",
      "author_url": "",
      "post_date": "02/20/2018 19:09:18",
      "content": "<p><a href=\"https://www.kaggle.com/c/data-science-bowl-2018/discussion/47640\">3 different modalities. Why?</a></p>\n\n<p>Great thanks!</p>\n\n<p>By the way I think the images I tried to post did not work how/where can I host images in kaggle forums? I tried imgur but looks lik it did not work</p>",
      "votes": null,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "284647": "Hi everyone, \n\nOver the past week or so we began our attempt at this challenge. This first steps for us have been to establish the process of running a model, evaluating the results on the training data set, and submitting it to kaggle. Our idea was to use a very simple segmentation method and submit this as an example. We used the evaluation method that William Cukierski posted. This uses np.histogram2D to calcualte the intersection between the two images. We were surprised to see that we achieved an mean average precision (over all training images) of 43%. We were expecting much lower. When submitting a Kaggle we received a score of .26, which is more along the lines of what we were expecting. After dissecting the evaluation method and comparing a few annotated ground truth images with the corresponding annotated predicted images we have been unable to identify an error. The assigned scores appear to be correct.\n\nHas anyone experienced this?  A score of .43 is much to high and the disparity between the kaggle score and our score on the training data seems imply that as well. Could it be that the test images are much more difficult to segment?",
    "284827": "If you look at test images closely, you will find out that there are 3 different types of them, and 2 types are not present in train set, so you simply overfitted.",
    "285604": "Hi Emil,\n\nThanks for your reply.\n\nThe segmentation approach uses a simple Otsu thresholding, so I don't think it can be overfitting on the training data.\n\nAlso, I have created some montage images to eyeball whether the train and test image sets look significantly different, as far as I can tell they do not. Please could you elaborate on which images types are missing in the train?\n\n## Train images\n![train images montage][1]\n\n## Test images\n![test images montage][2]\n\n\n  [1]: https://imgur.com/a/bcg8D\n  [2]: https://imgur.com/a/qwYmL",
    "285618": "Look in \"3 different modalities. Why?\" discussion.  There I posted all types of images in train &amp; test sets",
    "285634": "[3 different modalities. Why?][1]\n\nGreat thanks!\n\nBy the way I think the images I tried to post did not work how/where can I host images in kaggle forums? I tried imgur but looks lik it did not work\n\n  [1]: https://www.kaggle.com/c/data-science-bowl-2018/discussion/47640"
  },
  "source": "meta"
}