Maybe this is a silly question, but when I go through some of the tutorials, most of the data that I see generated is with information that can be extracted from that particular row.
\nThe environment provided by the sponsor only allows you to load in predictions one row at a time. Maybe I'm not fully versed yet with the environment, but is it possible to submit predictions with data that uses windows / ewm's / lags of various types?
\nIt seems like the prediction environment is built against it.
", "messages": [ { "id": "1600062", "postDate": "11/30/2021 04:50:33", "content": "Maybe this is a silly question, but when I go through some of the tutorials, most of the data that I see generated is with information that can be extracted from that particular row.
\nThe environment provided by the sponsor only allows you to load in predictions one row at a time. Maybe I'm not fully versed yet with the environment, but is it possible to submit predictions with data that uses windows / ewm's / lags of various types?
\nIt seems like the prediction environment is built against it.
", "rawMarkdown": "Maybe this is a silly question, but when I go through some of the tutorials, most of the data that I see generated is with information that can be extracted from that particular row. \n\nThe environment provided by the sponsor only allows you to load in predictions one row at a time. Maybe I'm not fully versed yet with the environment, but is it possible to submit predictions with data that uses windows / ewm's / lags of various types? \n\nIt seems like the prediction environment is built against it.", "votes": null }, { "id": "1600299", "postDate": "11/30/2021 09:14:05", "content": "Yes, you can track historical data outside of the environment and use it for computing features. You can even use the data the api provides during inference as new data and update your models (even though you have to calculate the new targets yourself).
", "rawMarkdown": "Yes, you can track historical data outside of the environment and use it for computing features. You can even use the data the api provides during inference as new data and update your models (even though you have to calculate the new targets yourself).", "votes": null }, { "id": "1600595", "postDate": "11/30/2021 14:51:45", "content": "All the API cares about is if you are looking into future timestamps or using internet access , if those two criteria are assured then you are good to go
", "rawMarkdown": "All the API cares about is if you are looking into future timestamps or using internet access , if those two criteria are assured then you are good to go", "votes": null }, { "id": "1603702", "postDate": "12/02/2021 17:20:16", "content": "This is a good question! I have been struggling with this! I have tried this fairly hacky solution, but I don't know if it is working on the secret scoring notebook run.
\nfirst = True\nfor index, (test_df, sample_prediction_df) in enumerate(iter_test):\n\n ## Get the timestamp of the test data\n ts = test_df.timestamp[0]\n\n if first:\n ## Stitch the test data on to the training data up to the test data timestamp\n test = train_df[(train_df.timestamp < ts) & (train_df.timestamp > (ts - 60 * 60 * 24 * 3))].copy()\n test = pandas.concat([test, test_df.dropna()])\n first = False\n else:\n ## Stitch the test data on to the previous test data.\n test = pandas.concat([test, test_df.dropna()])\nPlease let us know if you make any progress with this!
", "rawMarkdown": "This is a good question! I have been struggling with this! I have tried this fairly hacky solution, but I don't know if it is working on the secret scoring notebook run.\n\n```\nfirst = True\nfor index, (test_df, sample_prediction_df) in enumerate(iter_test):\n\n ## Get the timestamp of the test data\n ts = test_df.timestamp[0]\n \n if first:\n ## Stitch the test data on to the training data up to the test data timestamp\n test = train_df[(train_df.timestamp < ts) & (train_df.timestamp > (ts - 60 * 60 * 24 * 3))].copy()\n test = pandas.concat([test, test_df.dropna()])\n first = False\n else:\n ## Stitch the test data on to the previous test data.\n test = pandas.concat([test, test_df.dropna()])\n```\n\nPlease let us know if you make any progress with this!", "votes": null }, { "id": "1604965", "postDate": "12/03/2021 22:17:01", "content": "Is anyone aware of an open notebook that trains on at least one technical indicator (window or no window) and then appends the test row price and recalculates the indicator(s) before predicting? I am trying to avoid reinventing the wheel if someone has already made it publc - if not I am going to take a crack at a public notebook?
", "rawMarkdown": "Is anyone aware of an open notebook that trains on at least one technical indicator (window or no window) and then appends the test row price and recalculates the indicator(s) before predicting? I am trying to avoid reinventing the wheel if someone has already made it publc - if not I am going to take a crack at a public notebook?", "votes": null }, { "id": "1604966", "postDate": "12/03/2021 22:17:39", "content": "Interesting @markajoc
", "rawMarkdown": "Interesting @markajoc", "votes": null }, { "id": "1605157", "postDate": "12/04/2021 02:36:03", "content": "Hello @jonathanbechtel, I was thinking the same, I'm planning to append the data used in the predictions to the training dataset, calculate the target and use some of the information as lag features for the next step in the calculations.
", "rawMarkdown": "Hello @jonathanbechtel, I was thinking the same, I'm planning to append the data used in the predictions to the training dataset, calculate the target and use some of the information as lag features for the next step in the calculations.", "votes": null }, { "id": "1607248", "postDate": "12/05/2021 18:51:41", "content": "Thanks. You need to be careful with the running time here though… I had to strip my feature generation and model design right back to get this to run on the public leaderboard data with the API
", "rawMarkdown": "Thanks. You need to be careful with the running time here though... I had to strip my feature generation and model design right back to get this to run on the public leaderboard data with the API", "votes": null }, { "id": "1611376", "postDate": "12/07/2021 23:11:24", "content": "Using some previous work from the Janestreet competition I built and shared some Online Feature Engineering framework for feature engineering and submission. See the discussion here, or the On-line Feature Engineering Notebook here.
", "rawMarkdown": "Using some previous work from the Janestreet competition I built and shared some Online Feature Engineering framework for feature engineering and submission. See the discussion [here](https://www.kaggle.com/c/g-research-crypto-forecasting/discussion/293994), or the On-line Feature Engineering Notebook [here](https://www.kaggle.com/lucasmorin/on-line-feature-engineering).", "votes": null }, { "id": "1617777", "postDate": "12/14/2021 11:07:26", "content": "Hi @jonathanbechtel, i have shared an implementation of submitting lagged features via api, hope this helps! https://www.kaggle.com/tomforbes/gresearch-submitting-lagged-features-via-api
", "rawMarkdown": "Hi @jonathanbechtel, i have shared an implementation of submitting lagged features via api, hope this helps! [https://www.kaggle.com/tomforbes/gresearch-submitting-lagged-features-via-api](url)", "votes": null } ], "comments": [ { "id": 1600299, "author_name": "christoffer", "author_url": "", "post_date": "11/30/2021 09:14:05", "content": "Yes, you can track historical data outside of the environment and use it for computing features. You can even use the data the api provides during inference as new data and update your models (even though you have to calculate the new targets yourself).
", "votes": null, "replies": [] }, { "id": 1600595, "author_name": "swaralipibose", "author_url": "", "post_date": "11/30/2021 14:51:45", "content": "All the API cares about is if you are looking into future timestamps or using internet access , if those two criteria are assured then you are good to go
", "votes": null, "replies": [] }, { "id": 1603702, "author_name": "markajoc", "author_url": "", "post_date": "12/02/2021 17:20:16", "content": "This is a good question! I have been struggling with this! I have tried this fairly hacky solution, but I don't know if it is working on the secret scoring notebook run.
\nfirst = True\nfor index, (test_df, sample_prediction_df) in enumerate(iter_test):\n\n ## Get the timestamp of the test data\n ts = test_df.timestamp[0]\n\n if first:\n ## Stitch the test data on to the training data up to the test data timestamp\n test = train_df[(train_df.timestamp < ts) & (train_df.timestamp > (ts - 60 * 60 * 24 * 3))].copy()\n test = pandas.concat([test, test_df.dropna()])\n first = False\n else:\n ## Stitch the test data on to the previous test data.\n test = pandas.concat([test, test_df.dropna()])\nPlease let us know if you make any progress with this!
", "votes": null, "replies": [ { "id": 1604966, "author_name": "mlconsult", "author_url": "", "post_date": "12/03/2021 22:17:39", "content": "Interesting @markajoc
", "votes": null, "replies": [] }, { "id": 1607248, "author_name": "markajoc", "author_url": "", "post_date": "12/05/2021 18:51:41", "content": "Thanks. You need to be careful with the running time here though… I had to strip my feature generation and model design right back to get this to run on the public leaderboard data with the API
", "votes": null, "replies": [] } ] }, { "id": 1604965, "author_name": "mlconsult", "author_url": "", "post_date": "12/03/2021 22:17:01", "content": "Is anyone aware of an open notebook that trains on at least one technical indicator (window or no window) and then appends the test row price and recalculates the indicator(s) before predicting? I am trying to avoid reinventing the wheel if someone has already made it publc - if not I am going to take a crack at a public notebook?
", "votes": null, "replies": [] }, { "id": 1605157, "author_name": "cv13j0", "author_url": "", "post_date": "12/04/2021 02:36:03", "content": "Hello @jonathanbechtel, I was thinking the same, I'm planning to append the data used in the predictions to the training dataset, calculate the target and use some of the information as lag features for the next step in the calculations.
", "votes": null, "replies": [] }, { "id": 1611376, "author_name": "lucasmorin", "author_url": "", "post_date": "12/07/2021 23:11:24", "content": "Using some previous work from the Janestreet competition I built and shared some Online Feature Engineering framework for feature engineering and submission. See the discussion here, or the On-line Feature Engineering Notebook here.
", "votes": null, "replies": [] }, { "id": 1617777, "author_name": "tomforbes", "author_url": "", "post_date": "12/14/2021 11:07:26", "content": "Hi @jonathanbechtel, i have shared an implementation of submitting lagged features via api, hope this helps! https://www.kaggle.com/tomforbes/gresearch-submitting-lagged-features-via-api
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "1600062": "Maybe this is a silly question, but when I go through some of the tutorials, most of the data that I see generated is with information that can be extracted from that particular row. \n\nThe environment provided by the sponsor only allows you to load in predictions one row at a time. Maybe I'm not fully versed yet with the environment, but is it possible to submit predictions with data that uses windows / ewm's / lags of various types? \n\nIt seems like the prediction environment is built against it.", "1600299": "Yes, you can track historical data outside of the environment and use it for computing features. You can even use the data the api provides during inference as new data and update your models (even though you have to calculate the new targets yourself).", "1600595": "All the API cares about is if you are looking into future timestamps or using internet access , if those two criteria are assured then you are good to go", "1603702": "This is a good question! I have been struggling with this! I have tried this fairly hacky solution, but I don't know if it is working on the secret scoring notebook run.\n\n```\nfirst = True\nfor index, (test_df, sample_prediction_df) in enumerate(iter_test):\n\n ## Get the timestamp of the test data\n ts = test_df.timestamp[0]\n \n if first:\n ## Stitch the test data on to the training data up to the test data timestamp\n test = train_df[(train_df.timestamp < ts) & (train_df.timestamp > (ts - 60 * 60 * 24 * 3))].copy()\n test = pandas.concat([test, test_df.dropna()])\n first = False\n else:\n ## Stitch the test data on to the previous test data.\n test = pandas.concat([test, test_df.dropna()])\n```\n\nPlease let us know if you make any progress with this!", "1604965": "Is anyone aware of an open notebook that trains on at least one technical indicator (window or no window) and then appends the test row price and recalculates the indicator(s) before predicting? I am trying to avoid reinventing the wheel if someone has already made it publc - if not I am going to take a crack at a public notebook?", "1604966": "Interesting @markajoc", "1605157": "Hello @jonathanbechtel, I was thinking the same, I'm planning to append the data used in the predictions to the training dataset, calculate the target and use some of the information as lag features for the next step in the calculations.", "1607248": "Thanks. You need to be careful with the running time here though... I had to strip my feature generation and model design right back to get this to run on the public leaderboard data with the API", "1611376": "Using some previous work from the Janestreet competition I built and shared some Online Feature Engineering framework for feature engineering and submission. See the discussion [here](https://www.kaggle.com/c/g-research-crypto-forecasting/discussion/293994), or the On-line Feature Engineering Notebook [here](https://www.kaggle.com/lucasmorin/on-line-feature-engineering).", "1617777": "Hi @jonathanbechtel, i have shared an implementation of submitting lagged features via api, hope this helps! [https://www.kaggle.com/tomforbes/gresearch-submitting-lagged-features-via-api](url)" }, "source": "meta" }