{
  "id": 287777,
  "title": "Public leaderboard scores",
  "url": "/competitions/g-research-crypto-forecasting/discussion/287777",
  "author_name": "Mani Sarkar",
  "post_date": "2021-11-15T14:16:39.262000",
  "votes": 10,
  "comment_count": 5,
  "views": 0,
  "content": "<p>I just started looking at this competition and I find that many submissions are near <em>0.999</em> with one of them with a <em>1.000</em> score. </p>\n<p>Of course, the private leaderboard may reflect differently but is this desirable? Are we to conclude overfitting or data leakage from such an occurrence?</p>\n<p>Any thoughts?</p>",
  "messages": [
    {
      "id": 1583082,
      "postDate": "2021-11-15T14:16:39.263Z",
      "content": "<p>I just started looking at this competition and I find that many submissions are near <em>0.999</em> with one of them with a <em>1.000</em> score. </p>\n<p>Of course, the private leaderboard may reflect differently but is this desirable? Are we to conclude overfitting or data leakage from such an occurrence?</p>\n<p>Any thoughts?</p>",
      "rawMarkdown": "I just started looking at this competition and I find that many submissions are near *0.999* with one of them with a *1.000* score. \n\nOf course, the private leaderboard may reflect differently but is this desirable? Are we to conclude overfitting or data leakage from such an occurrence?\n\nAny thoughts?",
      "votes": 10
    },
    {
      "id": 1584375,
      "postDate": "2021-11-16T13:46:12.177Z",
      "content": "<p>The leak is a deliberately introduced feature of the dataset and is well-known. Under the Data tab of the competition home page, we read: </p>\n<p>\"This dataset contains information on historic trades for several cryptoassets, such as Bitcoin and Ethereum. Your challenge is to predict their future returns.</p>\n<p>As historic cryptocurrency prices are not confidential this will be a forecasting competition using the time series API. Furthermore the public leaderboard targets are publicly available and are provided as part of the competition dataset. Expect to see many people submitting perfect submissions for fun. Accordingly, THE PUBLIC LEADERBOARD FOR THIS COMPETITION IS NOT MEANINGFUL and is only provided as a convenience for anyone who wants to test their code. The final private leaderboard will be determined using real market data gathered after the submission period closes.\"</p>\n<p>That is, there was not intended to be a meaningful public leaderboard for this competition. Whether that's a good or bad thing is up for debate. However, it is possible to exclude the test data from your training set should you choose to do so, and there are a number of examples of how to do this among the public notebooks.</p>",
      "rawMarkdown": "The leak is a deliberately introduced feature of the dataset and is well-known. Under the Data tab of the competition home page, we read: \n\n\"This dataset contains information on historic trades for several cryptoassets, such as Bitcoin and Ethereum. Your challenge is to predict their future returns.\n\nAs historic cryptocurrency prices are not confidential this will be a forecasting competition using the time series API. Furthermore the public leaderboard targets are publicly available and are provided as part of the competition dataset. Expect to see many people submitting perfect submissions for fun. Accordingly, THE PUBLIC LEADERBOARD FOR THIS COMPETITION IS NOT MEANINGFUL and is only provided as a convenience for anyone who wants to test their code. The final private leaderboard will be determined using real market data gathered after the submission period closes.\"\n\nThat is, there was not intended to be a meaningful public leaderboard for this competition. Whether that's a good or bad thing is up for debate. However, it is possible to exclude the test data from your training set should you choose to do so, and there are a number of examples of how to do this among the public notebooks.",
      "votes": 6,
      "replies": [
        {
          "id": 1584658,
          "postDate": "2021-11-16T17:03:35.577Z",
          "content": "<p>I can see this being useful. helps test our code better for the private leaderboard</p>",
          "rawMarkdown": "I can see this being useful. helps test our code better for the private leaderboard"
        },
        {
          "id": 1584990,
          "postDate": "2021-11-17T00:35:08.253Z",
          "content": "<p>thx your reply</p>",
          "rawMarkdown": "thx your reply",
          "votes": 1
        },
        {
          "id": 1585907,
          "postDate": "2021-11-17T16:32:47.330Z",
          "content": "<p>Thanks John indeed, I guess I spoke a bit early then.</p>",
          "rawMarkdown": "Thanks John indeed, I guess I spoke a bit early then.",
          "votes": 1
        }
      ]
    },
    {
      "id": 1584331,
      "postDate": "2021-11-16T13:24:26.093Z",
      "rawMarkdown": "",
      "isDeleted": true
    }
  ],
  "comments": [
    {
      "id": 1584375,
      "author_name": "John Mitchell",
      "author_url": "",
      "post_date": "2021-11-16T13:46:12.177000",
      "content": "<p>The leak is a deliberately introduced feature of the dataset and is well-known. Under the Data tab of the competition home page, we read: </p>\n<p>\"This dataset contains information on historic trades for several cryptoassets, such as Bitcoin and Ethereum. Your challenge is to predict their future returns.</p>\n<p>As historic cryptocurrency prices are not confidential this will be a forecasting competition using the time series API. Furthermore the public leaderboard targets are publicly available and are provided as part of the competition dataset. Expect to see many people submitting perfect submissions for fun. Accordingly, THE PUBLIC LEADERBOARD FOR THIS COMPETITION IS NOT MEANINGFUL and is only provided as a convenience for anyone who wants to test their code. The final private leaderboard will be determined using real market data gathered after the submission period closes.\"</p>\n<p>That is, there was not intended to be a meaningful public leaderboard for this competition. Whether that's a good or bad thing is up for debate. However, it is possible to exclude the test data from your training set should you choose to do so, and there are a number of examples of how to do this among the public notebooks.</p>",
      "votes": 6,
      "replies": [
        {
          "id": 1584658,
          "author_name": "Mani Sarkar",
          "author_url": "",
          "post_date": "2021-11-16T17:03:35.577000",
          "content": "<p>I can see this being useful. helps test our code better for the private leaderboard</p>",
          "votes": 0,
          "replies": []
        },
        {
          "id": 1584990,
          "author_name": "Cicilia",
          "author_url": "",
          "post_date": "2021-11-17T00:35:08.253000",
          "content": "<p>thx your reply</p>",
          "votes": 1,
          "replies": []
        },
        {
          "id": 1585907,
          "author_name": "Mani Sarkar",
          "author_url": "",
          "post_date": "2021-11-17T16:32:47.330000",
          "content": "<p>Thanks John indeed, I guess I spoke a bit early then.</p>",
          "votes": 1,
          "replies": []
        }
      ]
    },
    {
      "id": 1584331,
      "author_name": "",
      "author_url": "",
      "post_date": "2021-11-16T13:24:26.093000",
      "content": "",
      "votes": 0,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "1583082": "I just started looking at this competition and I find that many submissions are near *0.999* with one of them with a *1.000* score. \n\nOf course, the private leaderboard may reflect differently but is this desirable? Are we to conclude overfitting or data leakage from such an occurrence?\n\nAny thoughts?",
    "1584375": "The leak is a deliberately introduced feature of the dataset and is well-known. Under the Data tab of the competition home page, we read: \n\n\"This dataset contains information on historic trades for several cryptoassets, such as Bitcoin and Ethereum. Your challenge is to predict their future returns.\n\nAs historic cryptocurrency prices are not confidential this will be a forecasting competition using the time series API. Furthermore the public leaderboard targets are publicly available and are provided as part of the competition dataset. Expect to see many people submitting perfect submissions for fun. Accordingly, THE PUBLIC LEADERBOARD FOR THIS COMPETITION IS NOT MEANINGFUL and is only provided as a convenience for anyone who wants to test their code. The final private leaderboard will be determined using real market data gathered after the submission period closes.\"\n\nThat is, there was not intended to be a meaningful public leaderboard for this competition. Whether that's a good or bad thing is up for debate. However, it is possible to exclude the test data from your training set should you choose to do so, and there are a number of examples of how to do this among the public notebooks.",
    "1584331": ""
  }
}