{
  "id": 302507,
  "title": "Question about Hist data in private scoring",
  "url": "/competitions/g-research-crypto-forecasting/discussion/302507",
  "author_name": "",
  "post_date": "2022-01-22T20:17:56.711450100Z",
  "votes": 2,
  "comment_count": 5,
  "views": 0,
  "content": "<p>Hi, I'm new to competitions so sorry if this is a dumb question. I see that the dataset used in the private scoring manner is hidden, and the public test data overlaps with the train data.</p>\n<p>My question is can I use a reccurrent approach like an LSTM and have it look at say the previous 30 samples? I know this is possible with the public test data, but is this even acheivable with the private testing data? </p>\n<p>Another example of this is using historical information for feature engineering. I have this notebook which ran successfully on the public test data, but failed on the private test data. Is this becuase the private test data is different and it it not possible to access historical information or prior steps?</p>\n<p>Thank you, and again sorry if this is an obvious question.</p>",
  "messages": [
    {
      "id": "1660640",
      "postDate": "01/22/2022 20:17:56",
      "content": "<p>Hi, I'm new to competitions so sorry if this is a dumb question. I see that the dataset used in the private scoring manner is hidden, and the public test data overlaps with the train data.</p>\n<p>My question is can I use a reccurrent approach like an LSTM and have it look at say the previous 30 samples? I know this is possible with the public test data, but is this even acheivable with the private testing data? </p>\n<p>Another example of this is using historical information for feature engineering. I have this notebook which ran successfully on the public test data, but failed on the private test data. Is this becuase the private test data is different and it it not possible to access historical information or prior steps?</p>\n<p>Thank you, and again sorry if this is an obvious question.</p>",
      "rawMarkdown": "Hi, I'm new to competitions so sorry if this is a dumb question. I see that the dataset used in the private scoring manner is hidden, and the public test data overlaps with the train data.\n\nMy question is can I use a reccurrent approach like an LSTM and have it look at say the previous 30 samples? I know this is possible with the public test data, but is this even acheivable with the private testing data? \n\nAnother example of this is using historical information for feature engineering. I have this notebook which ran successfully on the public test data, but failed on the private test data. Is this becuase the private test data is different and it it not possible to access historical information or prior steps?\n\nThank you, and again sorry if this is an obvious question.",
      "votes": null
    },
    {
      "id": "1660643",
      "postDate": "01/22/2022 20:19:40",
      "content": "<p>for reference I dont know how to share notebook, but I made this notebook public. </p>\n<p>Fork of xgboost on all returns </p>\n<p>it demonstrates a successful run with historical data, but failed when running on submission</p>",
      "rawMarkdown": "for reference I dont know how to share notebook, but I made this notebook public. \n\nFork of xgboost on all returns \n\nit demonstrates a successful run with historical data, but failed when running on submission",
      "votes": null
    },
    {
      "id": "1660648",
      "postDate": "01/22/2022 20:31:22",
      "content": "<p>Hello, for the testing part you only have access to minute by minute data. If you want to make some features using the previous 30 minutes you should build a data handler. You will first have to initialize it using the data from the supplemental_train.csv for instance, then just append the data from the env() function. Hope it will help.</p>",
      "rawMarkdown": "Hello, for the testing part you only have access to minute by minute data. If you want to make some features using the previous 30 minutes you should build a data handler. You will first have to initialize it using the data from the supplemental_train.csv for instance, then just append the data from the env() function. Hope it will help.",
      "votes": null
    },
    {
      "id": "1660659",
      "postDate": "01/22/2022 20:42:38",
      "content": "<p>Thank you! like I would be able to use the .get_loc() pandas function to get the index and then select the previous 30 minutes of data on the public training data, but do you know if this pipeline would work on the private data?</p>\n<p>I can't remember the exact dates but ex.<br>\nTraining Data: Jan 1st to Dec 20th<br>\nPublic Test Data: Dec 1st to Dec 20th<br>\nPrivate Test Data: Dec 21st to Dec 31st</p>\n<p>if this is the case is there no way of finding the previous values? or is that what supplemental triain is? is that the data that intersparses with the private testing set?</p>\n<p>Sorry for the confusion.</p>\n<p>Edit: Sorry I just thought of this. Like as an example in that notebook I made public I wanted to use histroical data. Since the training and testing sets overlap I just searched and replaced the row with the corresponding row with the engineered features but it was rejected as a submission. I would image this is due to the fact that on private testing there is no overlap so there is nothing to grab from, meaning would we not have access to the prior 30 minutes of data to add as features right?</p>",
      "rawMarkdown": "Thank you! like I would be able to use the .get_loc() pandas function to get the index and then select the previous 30 minutes of data on the public training data, but do you know if this pipeline would work on the private data?\n\nI can't remember the exact dates but ex.\nTraining Data: Jan 1st to Dec 20th\nPublic Test Data: Dec 1st to Dec 20th\nPrivate Test Data: Dec 21st to Dec 31st\n\nif this is the case is there no way of finding the previous values? or is that what supplemental triain is? is that the data that intersparses with the private testing set?\n\nSorry for the confusion.\n\nEdit: Sorry I just thought of this. Like as an example in that notebook I made public I wanted to use histroical data. Since the training and testing sets overlap I just searched and replaced the row with the corresponding row with the engineered features but it was rejected as a submission. I would image this is due to the fact that on private testing there is no overlap so there is nothing to grab from, meaning would we not have access to the prior 30 minutes of data to add as features right?",
      "votes": null
    },
    {
      "id": "1660665",
      "postDate": "01/22/2022 20:53:35",
      "content": "<p>Indeed, to fill the gap between the training and testing data they added the supplemental_train.csv. For now as they use part of the training data as test, you should initialize your data handler using the closest date from your training data (train.csv + supplemental_train.csv). After the competition starts, you could just initialize it using the last dates from the supplemental_train.csv.</p>",
      "rawMarkdown": "Indeed, to fill the gap between the training and testing data they added the supplemental_train.csv. For now as they use part of the training data as test, you should initialize your data handler using the closest date from your training data (train.csv + supplemental_train.csv). After the competition starts, you could just initialize it using the last dates from the supplemental_train.csv.",
      "votes": null
    },
    {
      "id": "1661979",
      "postDate": "01/23/2022 22:15:14",
      "content": "<p>Gotcha, thank you so much for the explanation, I really appreicate your help!</p>",
      "rawMarkdown": "Gotcha, thank you so much for the explanation, I really appreicate your help!",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 1660643,
      "author_name": "kiernanmcguigan",
      "author_url": "",
      "post_date": "01/22/2022 20:19:40",
      "content": "<p>for reference I dont know how to share notebook, but I made this notebook public. </p>\n<p>Fork of xgboost on all returns </p>\n<p>it demonstrates a successful run with historical data, but failed when running on submission</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 1660648,
      "author_name": "iiyadmea",
      "author_url": "",
      "post_date": "01/22/2022 20:31:22",
      "content": "<p>Hello, for the testing part you only have access to minute by minute data. If you want to make some features using the previous 30 minutes you should build a data handler. You will first have to initialize it using the data from the supplemental_train.csv for instance, then just append the data from the env() function. Hope it will help.</p>",
      "votes": null,
      "replies": [
        {
          "id": 1660659,
          "author_name": "kiernanmcguigan",
          "author_url": "",
          "post_date": "01/22/2022 20:42:38",
          "content": "<p>Thank you! like I would be able to use the .get_loc() pandas function to get the index and then select the previous 30 minutes of data on the public training data, but do you know if this pipeline would work on the private data?</p>\n<p>I can't remember the exact dates but ex.<br>\nTraining Data: Jan 1st to Dec 20th<br>\nPublic Test Data: Dec 1st to Dec 20th<br>\nPrivate Test Data: Dec 21st to Dec 31st</p>\n<p>if this is the case is there no way of finding the previous values? or is that what supplemental triain is? is that the data that intersparses with the private testing set?</p>\n<p>Sorry for the confusion.</p>\n<p>Edit: Sorry I just thought of this. Like as an example in that notebook I made public I wanted to use histroical data. Since the training and testing sets overlap I just searched and replaced the row with the corresponding row with the engineered features but it was rejected as a submission. I would image this is due to the fact that on private testing there is no overlap so there is nothing to grab from, meaning would we not have access to the prior 30 minutes of data to add as features right?</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 1660665,
          "author_name": "iiyadmea",
          "author_url": "",
          "post_date": "01/22/2022 20:53:35",
          "content": "<p>Indeed, to fill the gap between the training and testing data they added the supplemental_train.csv. For now as they use part of the training data as test, you should initialize your data handler using the closest date from your training data (train.csv + supplemental_train.csv). After the competition starts, you could just initialize it using the last dates from the supplemental_train.csv.</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 1661979,
          "author_name": "kiernanmcguigan",
          "author_url": "",
          "post_date": "01/23/2022 22:15:14",
          "content": "<p>Gotcha, thank you so much for the explanation, I really appreicate your help!</p>",
          "votes": null,
          "replies": []
        }
      ]
    }
  ],
  "raw_markdown_by_id": {
    "1660640": "Hi, I'm new to competitions so sorry if this is a dumb question. I see that the dataset used in the private scoring manner is hidden, and the public test data overlaps with the train data.\n\nMy question is can I use a reccurrent approach like an LSTM and have it look at say the previous 30 samples? I know this is possible with the public test data, but is this even acheivable with the private testing data? \n\nAnother example of this is using historical information for feature engineering. I have this notebook which ran successfully on the public test data, but failed on the private test data. Is this becuase the private test data is different and it it not possible to access historical information or prior steps?\n\nThank you, and again sorry if this is an obvious question.",
    "1660643": "for reference I dont know how to share notebook, but I made this notebook public. \n\nFork of xgboost on all returns \n\nit demonstrates a successful run with historical data, but failed when running on submission",
    "1660648": "Hello, for the testing part you only have access to minute by minute data. If you want to make some features using the previous 30 minutes you should build a data handler. You will first have to initialize it using the data from the supplemental_train.csv for instance, then just append the data from the env() function. Hope it will help.",
    "1660659": "Thank you! like I would be able to use the .get_loc() pandas function to get the index and then select the previous 30 minutes of data on the public training data, but do you know if this pipeline would work on the private data?\n\nI can't remember the exact dates but ex.\nTraining Data: Jan 1st to Dec 20th\nPublic Test Data: Dec 1st to Dec 20th\nPrivate Test Data: Dec 21st to Dec 31st\n\nif this is the case is there no way of finding the previous values? or is that what supplemental triain is? is that the data that intersparses with the private testing set?\n\nSorry for the confusion.\n\nEdit: Sorry I just thought of this. Like as an example in that notebook I made public I wanted to use histroical data. Since the training and testing sets overlap I just searched and replaced the row with the corresponding row with the engineered features but it was rejected as a submission. I would image this is due to the fact that on private testing there is no overlap so there is nothing to grab from, meaning would we not have access to the prior 30 minutes of data to add as features right?",
    "1660665": "Indeed, to fill the gap between the training and testing data they added the supplemental_train.csv. For now as they use part of the training data as test, you should initialize your data handler using the closest date from your training data (train.csv + supplemental_train.csv). After the competition starts, you could just initialize it using the last dates from the supplemental_train.csv.",
    "1661979": "Gotcha, thank you so much for the explanation, I really appreicate your help!"
  },
  "source": "meta"
}