{
  "id": 2546,
  "title": "Need some clarity on Training set and Data set",
  "url": "/competitions/MerckActivity/discussion/2546",
  "author_name": "",
  "post_date": "2012-09-02T11:32:27.563Z",
  "votes": null,
  "comment_count": 4,
  "views": 3004,
  "content": "<p>hi everybody,</p>\r\n<p>The number of entries in Molecule ID colum in ACT2_training(8717) and ACT2_test(2908) are different. The ACT2_training also does not contain any duplicate entry for Molecule ID. Is it supposed to be like this?</p>\r\n<p>Because as per the Rsquare.R script, the input and output variable should be having the same length. Pllease let me know if I am wrong.</p>",
  "messages": [
    {
      "id": "13795",
      "postDate": "09/02/2012 11:32:27",
      "content": "<p>hi everybody,</p>\r\n<p>The number of entries in Molecule ID colum in ACT2_training(8717) and ACT2_test(2908) are different. The ACT2_training also does not contain any duplicate entry for Molecule ID. Is it supposed to be like this?</p>\r\n<p>Because as per the Rsquare.R script, the input and output variable should be having the same length. Pllease let me know if I am wrong.</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "13800",
      "postDate": "09/02/2012 14:44:22",
      "content": "<p>The training set for ACT2 provides examples with correct answers for you to apply a learning algorithm to. The test set is a smaller set of observations for which Kaggle withholds the answers.</p>\r\n<p>The example Rsquare.R script can be used to self evaluate your own cross-validation predictions on the training set. Although, due to the date split, a standard cross-validation procedure would be inappropriate for this contest.</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "13835",
      "postDate": "09/03/2012 13:19:49",
      "content": "<p>Sorry for my poor understanding....</p>\r\n<p>But still need some more confirmation....</p>\r\n<p>How can I verify my model?</p>\r\n<p>1. Take ACTX_training data and come up with a model.</p>\r\n<p>2. Apply that model to predict the activity outcomes for corresponding ACTX_test data</p>\r\n<p>3. Check with RSquare script if my training activities are corelated to my corresponding predicted activities for test data</p>\r\n<p>4. Do the above 3 steps for all 15 training and test sets and get 15 RSquare numbers.</p>\r\n<p>5. Then take average to get the final score.</p>\r\n<p>Is the logical algo to be followed?&nbsp;</p>\r\n<p>If yes to calculate Rsquare for a training and test test the number of entries should match.</p>\r\n<p>If no... please let me know if I am thinking something wrong.</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "13842",
      "postDate": "09/03/2012 18:21:02",
      "content": "<p>On most contests, you would save cross-validated predictions on your training set and feed those plus the truths into the R2 formula.</p>\r\n<p>Since in this contest the test set is from a different distribution, your cross-validated error metric will not match the leaderboard.</p>\r\n<p>Specifically responding to your steps: #3 doesn't really make any sense. There is no real reason to believe the test set activity scores should have the same distribution as the training set activity scores. Especially when they come from different areas\r\n of feature space.</p>\r\n<p>As for what you should do?&nbsp; I'd consider trying the handwriting practice contest because I (or someone else) would be more willing to give you explicit answers there.</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "14234",
      "postDate": "09/11/2012 19:45:38",
      "content": "<p>Agree with Shea here.</p>\r\n<p>It is like a blind man when you cannot trust your CV score. Infact you cannot even trust the leaderboard score - unless it is a sufficiently random sample of the test data - because the final standings might be completely different</p>\r\n<p>&nbsp;</p>\r\n<p>Reminds me of the personality prediction challenge on kaggle</p>",
      "rawMarkdown": "",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 13800,
      "author_name": "biasvariance",
      "author_url": "",
      "post_date": "09/02/2012 14:44:22",
      "content": "<p>The training set for ACT2 provides examples with correct answers for you to apply a learning algorithm to. The test set is a smaller set of observations for which Kaggle withholds the answers.</p>\r\n<p>The example Rsquare.R script can be used to self evaluate your own cross-validation predictions on the training set. Although, due to the date split, a standard cross-validation procedure would be inappropriate for this contest.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 13835,
      "author_name": "surjya",
      "author_url": "",
      "post_date": "09/03/2012 13:19:49",
      "content": "<p>Sorry for my poor understanding....</p>\r\n<p>But still need some more confirmation....</p>\r\n<p>How can I verify my model?</p>\r\n<p>1. Take ACTX_training data and come up with a model.</p>\r\n<p>2. Apply that model to predict the activity outcomes for corresponding ACTX_test data</p>\r\n<p>3. Check with RSquare script if my training activities are corelated to my corresponding predicted activities for test data</p>\r\n<p>4. Do the above 3 steps for all 15 training and test sets and get 15 RSquare numbers.</p>\r\n<p>5. Then take average to get the final score.</p>\r\n<p>Is the logical algo to be followed?&nbsp;</p>\r\n<p>If yes to calculate Rsquare for a training and test test the number of entries should match.</p>\r\n<p>If no... please let me know if I am thinking something wrong.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 13842,
      "author_name": "biasvariance",
      "author_url": "",
      "post_date": "09/03/2012 18:21:02",
      "content": "<p>On most contests, you would save cross-validated predictions on your training set and feed those plus the truths into the R2 formula.</p>\r\n<p>Since in this contest the test set is from a different distribution, your cross-validated error metric will not match the leaderboard.</p>\r\n<p>Specifically responding to your steps: #3 doesn't really make any sense. There is no real reason to believe the test set activity scores should have the same distribution as the training set activity scores. Especially when they come from different areas\r\n of feature space.</p>\r\n<p>As for what you should do?&nbsp; I'd consider trying the handwriting practice contest because I (or someone else) would be more willing to give you explicit answers there.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 14234,
      "author_name": "rkirana",
      "author_url": "",
      "post_date": "09/11/2012 19:45:38",
      "content": "<p>Agree with Shea here.</p>\r\n<p>It is like a blind man when you cannot trust your CV score. Infact you cannot even trust the leaderboard score - unless it is a sufficiently random sample of the test data - because the final standings might be completely different</p>\r\n<p>&nbsp;</p>\r\n<p>Reminds me of the personality prediction challenge on kaggle</p>",
      "votes": null,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "13795": "",
    "13800": "",
    "13835": "",
    "13842": "",
    "14234": ""
  },
  "source": "meta"
}