{
  "id": 12486,
  "title": "A couple of questions about the test set",
  "url": "/competitions/datasciencebowl/discussion/12486",
  "author_name": "",
  "post_date": "2015-02-11T18:55:26.387Z",
  "votes": 1,
  "comment_count": 5,
  "views": 1604,
  "content": "<p>Firstly, sorry if these have been answered before. I've skimmed the relevant forum threads, but maybe I missed a decisive post.</p>\n<p>The number of samples per class in the train set is quite varied, and the test set is known to be injected with extra samples that are not scored.&nbsp;This makes me wonder about a couple aspects of the test set:</p>\n<p>1. Does the true test set (excluding the injected samples) have a <strong>roughly</strong> similar class distribution to the train set? I believe this is important for setting up a local cross-validation scheme.</p>\n<p>2. Are the injected (non-scored) samples drawn from a larger population as is, or are they somehow fabricated? The test set is 4 times the size of the train set, so it has the major weight with respect to unsupervised training. If the injected samples are superficial, I suspect they could cause trouble with unsupervised attempts.</p>\n<p>I understand Kaggle and the sponsors may not be willing to answer these, with&nbsp;rightful concerns about cheating, or favoring a robust solution. I still wanted to take my chance and ask, just in case they are fine with sharing this information.</p>\n<p>Thanks!</p>",
  "messages": [
    {
      "id": "64054",
      "postDate": "02/11/2015 18:55:26",
      "content": "<p>Firstly, sorry if these have been answered before. I've skimmed the relevant forum threads, but maybe I missed a decisive post.</p>\n<p>The number of samples per class in the train set is quite varied, and the test set is known to be injected with extra samples that are not scored.&nbsp;This makes me wonder about a couple aspects of the test set:</p>\n<p>1. Does the true test set (excluding the injected samples) have a <strong>roughly</strong> similar class distribution to the train set? I believe this is important for setting up a local cross-validation scheme.</p>\n<p>2. Are the injected (non-scored) samples drawn from a larger population as is, or are they somehow fabricated? The test set is 4 times the size of the train set, so it has the major weight with respect to unsupervised training. If the injected samples are superficial, I suspect they could cause trouble with unsupervised attempts.</p>\n<p>I understand Kaggle and the sponsors may not be willing to answer these, with&nbsp;rightful concerns about cheating, or favoring a robust solution. I still wanted to take my chance and ask, just in case they are fine with sharing this information.</p>\n<p>Thanks!</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "64057",
      "postDate": "02/11/2015 19:39:59",
      "content": "<ol>\n<li><span style=\"line-height: 1.4\">The splitting was done uniformly randomly (without regard to class representation)</span></li>\n<li><span style=\"line-height: 1.4\">They are fabricated</span></li>\n</ol>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "64059",
      "postDate": "02/11/2015 19:49:40",
      "content": "<p>Great, thank you!</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "64060",
      "postDate": "02/11/2015 19:53:12",
      "content": "<p>[quote=barisumog;64054]</p>\n<p>2. Are the injected (non-scored) samples drawn from a larger population as is, or are they somehow fabricated? The test set is 4 times the size of the train set, so it has the major weight with respect to unsupervised training. If the injected samples are superficial, I suspect they could cause trouble with unsupervised attempts.</p>\n<p>[/quote]</p>\n<p>I have examined this question by attempting to train classifier to distinguish between train and test sets - it fails. So looks like additional images&nbsp;would not cause problems.</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "64062",
      "postDate": "02/11/2015 20:01:58",
      "content": "<p>[quote=old-ufo;64060]</p>\n<p>I have examined this question by attempting to train classifier to distinguish between train and test sets - it fails. So looks like additional images&nbsp;would not cause problems.</p>\n<p>[/quote]</p>\n<p>I've just recently started looking at this data set, so it's probable my assumption was wrong. Thank you for sharing your experience!</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "64500",
      "postDate": "02/18/2015 15:23:26",
      "content": "<p>Hi Baris,</p>\n\n<p>How can i contact you? What's your email?</p>\n\n<p>Thanks,</p>",
      "rawMarkdown": "",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 64057,
      "author_name": "wcukierski",
      "author_url": "",
      "post_date": "02/11/2015 19:39:59",
      "content": "<ol>\n<li><span style=\"line-height: 1.4\">The splitting was done uniformly randomly (without regard to class representation)</span></li>\n<li><span style=\"line-height: 1.4\">They are fabricated</span></li>\n</ol>",
      "votes": null,
      "replies": []
    },
    {
      "id": 64059,
      "author_name": "barisumog",
      "author_url": "",
      "post_date": "02/11/2015 19:49:40",
      "content": "<p>Great, thank you!</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 64060,
      "author_name": "oldufo",
      "author_url": "",
      "post_date": "02/11/2015 19:53:12",
      "content": "<p>[quote=barisumog;64054]</p>\n<p>2. Are the injected (non-scored) samples drawn from a larger population as is, or are they somehow fabricated? The test set is 4 times the size of the train set, so it has the major weight with respect to unsupervised training. If the injected samples are superficial, I suspect they could cause trouble with unsupervised attempts.</p>\n<p>[/quote]</p>\n<p>I have examined this question by attempting to train classifier to distinguish between train and test sets - it fails. So looks like additional images&nbsp;would not cause problems.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 64062,
      "author_name": "barisumog",
      "author_url": "",
      "post_date": "02/11/2015 20:01:58",
      "content": "<p>[quote=old-ufo;64060]</p>\n<p>I have examined this question by attempting to train classifier to distinguish between train and test sets - it fails. So looks like additional images&nbsp;would not cause problems.</p>\n<p>[/quote]</p>\n<p>I've just recently started looking at this data set, so it's probable my assumption was wrong. Thank you for sharing your experience!</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 64500,
      "author_name": "semanticperson",
      "author_url": "",
      "post_date": "02/18/2015 15:23:26",
      "content": "<p>Hi Baris,</p>\n\n<p>How can i contact you? What's your email?</p>\n\n<p>Thanks,</p>",
      "votes": null,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "64054": "",
    "64057": "",
    "64059": "",
    "64060": "",
    "64062": "",
    "64500": ""
  },
  "source": "meta"
}