{
  "id": 300416,
  "title": "missing timestamps",
  "url": "/competitions/g-research-crypto-forecasting/discussion/300416",
  "author_name": "",
  "post_date": "2022-01-12T14:39:46.928472200Z",
  "votes": 1,
  "comment_count": 2,
  "views": 0,
  "content": "<p>I know this issue has been discussed, but not sure if there was a concrete-final decision or way-to-go:<br>\nI am within the env.iter_test() environment.<br>\nNext data arrives, and its timestamp is not +60seconds. Assume it is +120seconds. So for example the timestamp in the last iteration was: 1622505540 and now comes 1622505660, instead of the expected 1622505600.</p>\n<p>Now I predict the target: R(t) and calculate m(t). What about the beta calculation. Should I pretend as if there is no missing data and just roll over the latest 3750 for the rolling stats required to calculate beta? </p>\n<p>Here is the catch: each fully missing timestamp will cause 16 R(t+16) calculations to go wrong. By 'wrong' I mean: those affected existing 16 timestamps will have R(t+17) instead of R(t+16) because the simple 16period shift will actually mean a 17minutes shift. Is that, what is wanted?</p>\n<p>Why it bothers me: I am not getting high correlation scores, and dont want it to go lower :-)</p>",
  "messages": [
    {
      "id": "1647378",
      "postDate": "01/12/2022 14:39:46",
      "content": "<p>I know this issue has been discussed, but not sure if there was a concrete-final decision or way-to-go:<br>\nI am within the env.iter_test() environment.<br>\nNext data arrives, and its timestamp is not +60seconds. Assume it is +120seconds. So for example the timestamp in the last iteration was: 1622505540 and now comes 1622505660, instead of the expected 1622505600.</p>\n<p>Now I predict the target: R(t) and calculate m(t). What about the beta calculation. Should I pretend as if there is no missing data and just roll over the latest 3750 for the rolling stats required to calculate beta? </p>\n<p>Here is the catch: each fully missing timestamp will cause 16 R(t+16) calculations to go wrong. By 'wrong' I mean: those affected existing 16 timestamps will have R(t+17) instead of R(t+16) because the simple 16period shift will actually mean a 17minutes shift. Is that, what is wanted?</p>\n<p>Why it bothers me: I am not getting high correlation scores, and dont want it to go lower :-)</p>",
      "rawMarkdown": "I know this issue has been discussed, but not sure if there was a concrete-final decision or way-to-go:\nI am within the env.iter_test() environment.\nNext data arrives, and its timestamp is not +60seconds. Assume it is +120seconds. So for example the timestamp in the last iteration was: 1622505540 and now comes 1622505660, instead of the expected 1622505600.\n\nNow I predict the target: R(t) and calculate m(t). What about the beta calculation. Should I pretend as if there is no missing data and just roll over the latest 3750 for the rolling stats required to calculate beta? \n\nHere is the catch: each fully missing timestamp will cause 16 R(t+16) calculations to go wrong. By 'wrong' I mean: those affected existing 16 timestamps will have R(t+17) instead of R(t+16) because the simple 16period shift will actually mean a 17minutes shift. Is that, what is wanted?\n\nWhy it bothers me: I am not getting high correlation scores, and dont want it to go lower :-)",
      "votes": null
    },
    {
      "id": "1651221",
      "postDate": "01/15/2022 16:12:34",
      "content": "<p>I'm wondering if the test data's timestamp will be real timestamp, or they make up some fake timestamps?</p>",
      "rawMarkdown": "I'm wondering if the test data's timestamp will be real timestamp, or they make up some fake timestamps?",
      "votes": null
    },
    {
      "id": "1651301",
      "postDate": "01/15/2022 17:05:42",
      "content": "<p>They be real.</p>",
      "rawMarkdown": "They be real.",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 1651221,
      "author_name": "yuanzhezhou",
      "author_url": "",
      "post_date": "01/15/2022 16:12:34",
      "content": "<p>I'm wondering if the test data's timestamp will be real timestamp, or they make up some fake timestamps?</p>",
      "votes": null,
      "replies": [
        {
          "id": 1651301,
          "author_name": "abdulkadirguner",
          "author_url": "",
          "post_date": "01/15/2022 17:05:42",
          "content": "<p>They be real.</p>",
          "votes": null,
          "replies": []
        }
      ]
    }
  ],
  "raw_markdown_by_id": {
    "1647378": "I know this issue has been discussed, but not sure if there was a concrete-final decision or way-to-go:\nI am within the env.iter_test() environment.\nNext data arrives, and its timestamp is not +60seconds. Assume it is +120seconds. So for example the timestamp in the last iteration was: 1622505540 and now comes 1622505660, instead of the expected 1622505600.\n\nNow I predict the target: R(t) and calculate m(t). What about the beta calculation. Should I pretend as if there is no missing data and just roll over the latest 3750 for the rolling stats required to calculate beta? \n\nHere is the catch: each fully missing timestamp will cause 16 R(t+16) calculations to go wrong. By 'wrong' I mean: those affected existing 16 timestamps will have R(t+17) instead of R(t+16) because the simple 16period shift will actually mean a 17minutes shift. Is that, what is wanted?\n\nWhy it bothers me: I am not getting high correlation scores, and dont want it to go lower :-)",
    "1651221": "I'm wondering if the test data's timestamp will be real timestamp, or they make up some fake timestamps?",
    "1651301": "They be real."
  },
  "source": "meta"
}