{
  "id": 285705,
  "title": "Sequence & Overlapping train - test ",
  "url": "/competitions/g-research-crypto-forecasting/discussion/285705",
  "author_name": "",
  "post_date": "2021-11-05T21:11:10.483330600Z",
  "votes": null,
  "comment_count": 2,
  "views": 0,
  "content": "<p>Hello,</p>\n<p>I have seen that there is an overlap between train and test.</p>\n<p>Details here:<br>\n<a href=\"https://www.kaggle.com/dariotorregrossa/important-time-sequence-train-vs-test\" target=\"_blank\">https://www.kaggle.com/dariotorregrossa/important-time-sequence-train-vs-test</a></p>\n<p>Is this overlap constant?<br>\nIt will be the same for validation?</p>\n<p>Thanks</p>",
  "messages": [
    {
      "id": "1572674",
      "postDate": "11/05/2021 21:11:10",
      "content": "<p>Hello,</p>\n<p>I have seen that there is an overlap between train and test.</p>\n<p>Details here:<br>\n<a href=\"https://www.kaggle.com/dariotorregrossa/important-time-sequence-train-vs-test\" target=\"_blank\">https://www.kaggle.com/dariotorregrossa/important-time-sequence-train-vs-test</a></p>\n<p>Is this overlap constant?<br>\nIt will be the same for validation?</p>\n<p>Thanks</p>",
      "rawMarkdown": "Hello,\n\nI have seen that there is an overlap between train and test.\n\nDetails here:\nhttps://www.kaggle.com/dariotorregrossa/important-time-sequence-train-vs-test\n\nIs this overlap constant?\nIt will be the same for validation?\n\nThanks",
      "votes": null
    },
    {
      "id": "1576962",
      "postDate": "11/09/2021 15:53:46",
      "content": "<p>The train and test data has overlapping values as pre-warned on the data tab (unless you didn't read the preface). Some on the LB already reached 0.999 and 1.0 accuracy so you can say that this confirms your question. As for the private LB data, <strong>\"The final private leaderboard will be determined using real market data gathered after the submission period closes.\"</strong> so there's no problem of overlapping here.</p>\n<p>P.S.<br>\nThe overlapping data is approximately 8% of the train data (from the start of the test's lowest timestamp to the end of train timestamp). I suggest you use the tail of the train data for validation purposes instead of training instead.</p>",
      "rawMarkdown": "The train and test data has overlapping values as pre-warned on the data tab (unless you didn't read the preface). Some on the LB already reached 0.999 and 1.0 accuracy so you can say that this confirms your question. As for the private LB data, **\"The final private leaderboard will be determined using real market data gathered after the submission period closes.\"** so there's no problem of overlapping here.\n\nP.S.\nThe overlapping data is approximately 8% of the train data (from the start of the test's lowest timestamp to the end of train timestamp). I suggest you use the tail of the train data for validation purposes instead of training instead.",
      "votes": null
    },
    {
      "id": "1577001",
      "postDate": "11/09/2021 16:51:07",
      "content": "<p><a href=\"https://www.kaggle.com/seraphwedd18\" target=\"_blank\">@seraphwedd18</a> is correct.</p>",
      "rawMarkdown": "seraphwedd18 is correct.",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 1576962,
      "author_name": "seraphwedd18",
      "author_url": "",
      "post_date": "11/09/2021 15:53:46",
      "content": "<p>The train and test data has overlapping values as pre-warned on the data tab (unless you didn't read the preface). Some on the LB already reached 0.999 and 1.0 accuracy so you can say that this confirms your question. As for the private LB data, <strong>\"The final private leaderboard will be determined using real market data gathered after the submission period closes.\"</strong> so there's no problem of overlapping here.</p>\n<p>P.S.<br>\nThe overlapping data is approximately 8% of the train data (from the start of the test's lowest timestamp to the end of train timestamp). I suggest you use the tail of the train data for validation purposes instead of training instead.</p>",
      "votes": null,
      "replies": [
        {
          "id": 1577001,
          "author_name": "sohier",
          "author_url": "",
          "post_date": "11/09/2021 16:51:07",
          "content": "<p><a href=\"https://www.kaggle.com/seraphwedd18\" target=\"_blank\">@seraphwedd18</a> is correct.</p>",
          "votes": null,
          "replies": []
        }
      ]
    }
  ],
  "raw_markdown_by_id": {
    "1572674": "Hello,\n\nI have seen that there is an overlap between train and test.\n\nDetails here:\nhttps://www.kaggle.com/dariotorregrossa/important-time-sequence-train-vs-test\n\nIs this overlap constant?\nIt will be the same for validation?\n\nThanks",
    "1576962": "The train and test data has overlapping values as pre-warned on the data tab (unless you didn't read the preface). Some on the LB already reached 0.999 and 1.0 accuracy so you can say that this confirms your question. As for the private LB data, **\"The final private leaderboard will be determined using real market data gathered after the submission period closes.\"** so there's no problem of overlapping here.\n\nP.S.\nThe overlapping data is approximately 8% of the train data (from the start of the test's lowest timestamp to the end of train timestamp). I suggest you use the tail of the train data for validation purposes instead of training instead.",
    "1577001": "seraphwedd18 is correct."
  },
  "source": "meta"
}