Hi All,
\nJust wanted to seek some help from some fellow Kagglers; as this is my first real competition entry, since joining Kaggle for the 30 day intro to ML challange.
\nI've been attempting to submit my code to the LB, after succeding to \"Save & Run All (Commit)\", however I believe the online G-Research API Limitations are causing my code to timeout upon submission.
\nI've read in a couple of other discussions that each iteration is meant to be under 0.25s, however each of my individual feature engineering steps are far above above this value, which is a little frustrating as I believe it helps towards making my predicitions more accurate, so I don't really want to get rid of them completely.
\nStage 1. Feature Engineering steps [row-by-row] ~4s per iteration:
\nQ1. Do you suggest I get rid of these all to meet the time restrictions or is there anway I can keep these and carry them out outside the iteration loop?
\nStage 2. Prediction step [row-by-row] ~19s per iteration:
\nQ2. Is there any way I can maybe speed this up by storing a copy of the model in a file or something else?
\nQ3. Is there a way I could carry this out outside the env iteration loop to avoid triggering the API time limitation?
Copy of my Submission Iteration Loop
\ncrypo_ID_list = list(crypo_id_store_dict.keys())\ntime_delta_list = []\n\nfor i, (df_test, df_pred) in enumerate(iter_test):\n for j , row in df_test.iterrows():\n try:\n start_time = datetime.datetime.now()\n\n if row['Asset_ID'] in crypo_ID_list:\n feature_engineering(row)\n y_pred = model_stacking.predict(row.loc[:, row.columns!='row_id'])\n else: \n y_pred = 0.0\n\n time_delta_list.append(datetime.datetime.now() - start_time)\n except: \n y_pred = 0.0\n traceback.print_exc()\n df_pred.loc[df_pred['row_id'].values == row['row_id'].values, 'Target'] = y_pred\n all_df_test.append(df_test)\n env.predict(df_pred)\nNOTE: Training model offline takes ~16-26mins
\nI'm a little confused on what steps to take as it's reaching the deadline and I haven't even been able to make a single submission due to 'timeout' errors and I'm worried I'll be scored with a fail on this competiton when I've put so many hours in, getting to this point.
\nIf you need me to clarify anything further, in-order to help me, please do let me know.
\nMany Thanks,
\nWilson
Hi All,
\nJust wanted to seek some help from some fellow Kagglers; as this is my first real competition entry, since joining Kaggle for the 30 day intro to ML challange.
\nI've been attempting to submit my code to the LB, after succeding to \"Save & Run All (Commit)\", however I believe the online G-Research API Limitations are causing my code to timeout upon submission.
\nI've read in a couple of other discussions that each iteration is meant to be under 0.25s, however each of my individual feature engineering steps are far above above this value, which is a little frustrating as I believe it helps towards making my predicitions more accurate, so I don't really want to get rid of them completely.
\nStage 1. Feature Engineering steps [row-by-row] ~4s per iteration:
\nQ1. Do you suggest I get rid of these all to meet the time restrictions or is there anway I can keep these and carry them out outside the iteration loop?
\nStage 2. Prediction step [row-by-row] ~19s per iteration:
\nQ2. Is there any way I can maybe speed this up by storing a copy of the model in a file or something else?
\nQ3. Is there a way I could carry this out outside the env iteration loop to avoid triggering the API time limitation?
Copy of my Submission Iteration Loop
\ncrypo_ID_list = list(crypo_id_store_dict.keys())\ntime_delta_list = []\n\nfor i, (df_test, df_pred) in enumerate(iter_test):\n for j , row in df_test.iterrows():\n try:\n start_time = datetime.datetime.now()\n\n if row['Asset_ID'] in crypo_ID_list:\n feature_engineering(row)\n y_pred = model_stacking.predict(row.loc[:, row.columns!='row_id'])\n else: \n y_pred = 0.0\n\n time_delta_list.append(datetime.datetime.now() - start_time)\n except: \n y_pred = 0.0\n traceback.print_exc()\n df_pred.loc[df_pred['row_id'].values == row['row_id'].values, 'Target'] = y_pred\n all_df_test.append(df_test)\n env.predict(df_pred)\nNOTE: Training model offline takes ~16-26mins
\nI'm a little confused on what steps to take as it's reaching the deadline and I haven't even been able to make a single submission due to 'timeout' errors and I'm worried I'll be scored with a fail on this competiton when I've put so many hours in, getting to this point.
\nIf you need me to clarify anything further, in-order to help me, please do let me know.
\nMany Thanks,
\nWilson
Are you sure its timeout? That would mean your notebook takes over 9 hrs. Does your feature engineering take into account that data could be missing for an asset?
", "rawMarkdown": "Are you sure its timeout? That would mean your notebook takes over 9 hrs. Does your feature engineering take into account that data could be missing for an asset?", "votes": null }, { "id": "1661806", "postDate": "01/23/2022 17:55:10", "content": "Hi Chris,
\nThanks for your speedy comment in reply to my issue!
\nWhenever I attempt my submission to the LB the error only ever shows up as 'notebook timeout' with no further information.
\nHowever, with regards to your second question:
\nWouldn't the missing data from the API trigger an exception in my submission iteration loop and mark the y_pred value as 0.0; as can be seen in the code snippet I'd included?
Or would I have to implement further cases to handle the occurrence of missing or infinity values data in my feature engineering function?
19s per iteration ? so more than 1s to predict a given row ? This doesn't seems normal. What framework do you use ? How many model do you have ?
", "rawMarkdown": "19s per iteration ? so more than 1s to predict a given row ? This doesn't seems normal. What framework do you use ? How many model do you have ?", "votes": null }, { "id": "1662013", "postDate": "01/23/2022 23:39:58", "content": "I've checked the timedelta a couple of times and it actually averages out to ~15s
\nSo I'm using the SciKit StackingRegressor; and within this regressor I'm using the x1 SciKit linear model, x1 LGBM model and x1 XGB model.
\nDo you have any suggestions of where I could be going wrong with this, could this maybe be too many models for the notebook to handle at once?
", "rawMarkdown": "I've checked the timedelta a couple of times and it actually averages out to ~15s\n\nSo I'm using the SciKit StackingRegressor; and within this regressor I'm using the x1 SciKit linear model, x1 LGBM model and x1 XGB model.\n\nDo you have any suggestions of where I could be going wrong with this, could this maybe be too many models for the notebook to handle at once?", "votes": null }, { "id": "1662036", "postDate": "01/24/2022 00:56:53", "content": "Is this supposed to change row inplace?
\nfeature_engineering(row)\nbecause then you pass row to your predict function and I'm not sure the features are there.
\nI am sorry, I am really not good enough to be able to give specific and good advice, but I the best advice I can give is this:
\nSimplify your code until you get a valid submission and then build up the complexity. This is the only way to go I think.
\nOne quick thing I saw looking at your code is this: the feature engineering is done on one row at a time. Make sure this matches your feature engineering before training.
\nSorry I can't spend more time debugging, I have so many of my own issues to deal with, but I promise that if you simplify first, and be sure all preprocessing steps before training are possible and the same as during the prediction phase (use the api with 3 timestamps to test this) you will be alright.
", "rawMarkdown": "Is this supposed to change row inplace? \n```\nfeature_engineering(row)\n```\nbecause then you pass row to your predict function and I'm not sure the features are there. \n\nI am sorry, I am really not good enough to be able to give specific and good advice, but I the best advice I can give is this: \n\nSimplify your code until you get a valid submission and then build up the complexity. This is the only way to go I think. \n\nOne quick thing I saw looking at your code is this: the feature engineering is done on one row at a time. Make sure this matches your feature engineering before training. \n\nSorry I can't spend more time debugging, I have so many of my own issues to deal with, but I promise that if you simplify first, and be sure all preprocessing steps before training are possible and the same as during the prediction phase (use the api with 3 timestamps to test this) you will be alright.", "votes": null }, { "id": "1662066", "postDate": "01/24/2022 01:56:12", "content": "Hi Again Chris,
\nThe feature additions and manipulations are carried out inplace to reduce dataframe copying and memory errors, as this was one of the issues I'd ran into previously; but managed to rectify, combined with dataframe memory reductions and garbage collection.
\nI perfectly understand though, no need to be sorry as you've been of tremendous help; all the help you've been able to provide thus far has sparked some ideas for me, which I feel can help towards me making my first valid submission.
\nThanks so much for the assistance and best of luck in the competiton!
", "rawMarkdown": "Hi Again Chris, \n\nThe feature additions and manipulations are carried out inplace to reduce dataframe copying and memory errors, as this was one of the issues I'd ran into previously; but managed to rectify, combined with dataframe memory reductions and garbage collection.\n\nI perfectly understand though, no need to be sorry as you've been of tremendous help; all the help you've been able to provide thus far has sparked some ideas for me, which I feel can help towards me making my first valid submission. \n\nThanks so much for the assistance and best of luck in the competiton!", "votes": null }, { "id": "1662182", "postDate": "01/24/2022 05:19:39", "content": "@wils3456 Great to hear you got some ideas. Keep it up. You will get it. Good luck to you as well.
", "rawMarkdown": "wils3456 Great to hear you got some ideas. Keep it up. You will get it. Good luck to you as well.", "votes": null }, { "id": "1662195", "postDate": "01/24/2022 05:37:43", "content": "I see you are taking the test price and predicting Target. This is incorrect because the Target data should be calculated using 3750 previous price data (~2.5 days). This way we have to predict the future prices of 16 steps (minutes) ahead of the existing test data.
", "rawMarkdown": "I see you are taking the test price and predicting Target. This is incorrect because the Target data should be calculated using 3750 previous price data (~2.5 days). This way we have to predict the future prices of 16 steps (minutes) ahead of the existing test data.", "votes": null }, { "id": "1662200", "postDate": "01/24/2022 05:41:42", "content": "\n\nWouldn't the missing data from the API trigger an exception
\n
What can be missing in the test data? The prices? The inconsistent row_ids? New Asset_ID? Missing Targets or prices in supplemental data?
", "rawMarkdown": "> Wouldn't the missing data from the API trigger an exception\n\nWhat can be missing in the test data? The prices? The inconsistent row_ids? New Asset_ID? Missing Targets or prices in supplemental data?", "votes": null }, { "id": "1662610", "postDate": "01/24/2022 12:24:25", "content": "Hi @markbquant
\nI'm not quite sure actually; at first my only assumption was that an incorrect Asset ID could be passed through to us by the API, but since Chris' comment I've added another case which catches out np.nan & np.inf and sets y_pred = 0.0
Would it be possible for you to elaborate on this a little further please? As I'm fairly new to Data Science and don't quite understand…
\nWhen analysing the data, I saw there was a Target was present; so I dealt with it like I would any other regular Regression problem. However, if there are any further steps I need to carry out to ensure that I'm predicting future values, could you please point me in the right direction. (i.e. a little hint, an article, blog post, a past notebook, etc…)
\nMany Thanks!
", "rawMarkdown": "Would it be possible for you to elaborate on this a little further please? As I'm fairly new to Data Science and don't quite understand...\n\nWhen analysing the data, I saw there was a Target was present; so I dealt with it like I would any other regular Regression problem. However, if there are any further steps I need to carry out to ensure that I'm predicting future values, could you please point me in the right direction. (i.e. a little hint, an article, blog post, a past notebook, etc...)\n\nMany Thanks!", "votes": null }, { "id": "1662745", "postDate": "01/24/2022 14:02:31", "content": "For reference: my offline training is 9-10mins and leaderboard is like 6.5 hours.. So yours at 26mins will defo time out.
", "rawMarkdown": "For reference: my offline training is 9-10mins and leaderboard is like 6.5 hours.. So yours at 26mins will defo time out.", "votes": null }, { "id": "1662988", "postDate": "01/24/2022 17:37:38", "content": "The target is calculated as shown in this function calculate_target from the notebook: https://www.kaggle.com/alexfir/recreating-target
We have to predict 16 future prices or their returns:
\n$$R^a(t) = log (P^a(t+16)\\ /\\ P^a(t+1))$$
\ntargets[asset_names[i]] = price.shift(periods=-16)/price.shift(periods=-1) - 1
and based on them calculate prediction target.
\nIn live there will be 3 month of test data, all without targets. We cannot predict targets 3*30*24*60 intervals ahead.
Hint: target = asset_returns - beta(average_market_returns) = r - b(m)
\nYou could predict target directly and it might work out. Or you could predict r, b and m separately and create the target. (big hint part) b is sometimes zero because there is a nan in the 3750 perioud rolling mean. For more clarification, please find the notebook that says target calculation sota, and figure out exactly how those calculations work.
The big point is, when beta changes, the target changes drastically.
", "rawMarkdown": "Hint: target = asset_returns - beta(average_market_returns) = r - b(m)\nYou could predict target directly and it might work out. Or you could predict r, b and m separately and create the target. (big hint part) b is sometimes zero because there is a nan in the 3750 perioud rolling mean. For more clarification, please find the notebook that says target calculation sota, and figure out exactly how those calculations work. \n\nThe big point is, when beta changes, the target changes drastically.", "votes": null }, { "id": "1663284", "postDate": "01/25/2022 00:16:34", "content": "Hi @chrisrichardmiles / @markbquant
\nThis is quite interesting; I've had a read through the content you've both suggested and can now see why my current approach, of directly predicting the target from the features may not be the best way forward.
\nGiven this new information, I'm currently researching more on making use of rolling point averages programmaticaly, to aid with the manipulation of the training data; prior to training my model. If anyone has any insightful blog posts or notebooks which they feel would help explain this concept with examples, this would be greatly appreciated!
", "rawMarkdown": "Hi @chrisrichardmiles / @markbquant \n\nThis is quite interesting; I've had a read through the content you've both suggested and can now see why my current approach, of directly predicting the target from the features may not be the best way forward. \n\nGiven this new information, I'm currently researching more on making use of rolling point averages programmaticaly, to aid with the manipulation of the training data; prior to training my model. If anyone has any insightful blog posts or notebooks which they feel would help explain this concept with examples, this would be greatly appreciated!", "votes": null }, { "id": "1663293", "postDate": "01/25/2022 00:25:16", "content": "\nThanks soo much for your input, I'm currently working on making certain aspects of my code more efficient through he use of direct numpy manipulations and the removal of unecessary code; and, I've managed to cut down my last offline run to 13mins whilst each row predicition now takes ~9-10secs, which I believe is a huge improvement.
\nHowever, I've still got loads more work to do before it's running at optimal speeds.
\nI have one question, do you feel it's a waste of processing time having the same model in both the first and meta level of the StackingRegressor?
", "rawMarkdown": "julianmukaj,\n\nThanks soo much for your input, I'm currently working on making certain aspects of my code more efficient through he use of direct numpy manipulations and the removal of unecessary code; and, I've managed to cut down my last offline run to 13mins whilst each row predicition now takes ~9-10secs, which I believe is a huge improvement. \n\nHowever, I've still got loads more work to do before it's running at optimal speeds. \n\nI have one question, do you feel it's a waste of processing time having the same model in both the first and meta level of the StackingRegressor?", "votes": null }, { "id": "1663304", "postDate": "01/25/2022 01:00:27", "content": "@wils3456 Sounds like a good idea. I am currently working the basics to make sure I can keep track everything to calculate the betas, returns, and targets correctly.
\n@lucasmorin has some good notebooks and comments about running means.
Are you sure its timeout? That would mean your notebook takes over 9 hrs. Does your feature engineering take into account that data could be missing for an asset?
", "votes": null, "replies": [ { "id": 1661806, "author_name": "wils3456", "author_url": "", "post_date": "01/23/2022 17:55:10", "content": "Hi Chris,
\nThanks for your speedy comment in reply to my issue!
\nWhenever I attempt my submission to the LB the error only ever shows up as 'notebook timeout' with no further information.
\nHowever, with regards to your second question:
\nWouldn't the missing data from the API trigger an exception in my submission iteration loop and mark the y_pred value as 0.0; as can be seen in the code snippet I'd included?
Or would I have to implement further cases to handle the occurrence of missing or infinity values data in my feature engineering function?
Is this supposed to change row inplace?
\nfeature_engineering(row)\nbecause then you pass row to your predict function and I'm not sure the features are there.
\nI am sorry, I am really not good enough to be able to give specific and good advice, but I the best advice I can give is this:
\nSimplify your code until you get a valid submission and then build up the complexity. This is the only way to go I think.
\nOne quick thing I saw looking at your code is this: the feature engineering is done on one row at a time. Make sure this matches your feature engineering before training.
\nSorry I can't spend more time debugging, I have so many of my own issues to deal with, but I promise that if you simplify first, and be sure all preprocessing steps before training are possible and the same as during the prediction phase (use the api with 3 timestamps to test this) you will be alright.
", "votes": null, "replies": [] }, { "id": 1662066, "author_name": "wils3456", "author_url": "", "post_date": "01/24/2022 01:56:12", "content": "Hi Again Chris,
\nThe feature additions and manipulations are carried out inplace to reduce dataframe copying and memory errors, as this was one of the issues I'd ran into previously; but managed to rectify, combined with dataframe memory reductions and garbage collection.
\nI perfectly understand though, no need to be sorry as you've been of tremendous help; all the help you've been able to provide thus far has sparked some ideas for me, which I feel can help towards me making my first valid submission.
\nThanks so much for the assistance and best of luck in the competiton!
", "votes": null, "replies": [] }, { "id": 1662182, "author_name": "chrisrichardmiles", "author_url": "", "post_date": "01/24/2022 05:19:39", "content": "@wils3456 Great to hear you got some ideas. Keep it up. You will get it. Good luck to you as well.
", "votes": null, "replies": [] }, { "id": 1662200, "author_name": "markbquant", "author_url": "", "post_date": "01/24/2022 05:41:42", "content": "\n\nWouldn't the missing data from the API trigger an exception
\n
What can be missing in the test data? The prices? The inconsistent row_ids? New Asset_ID? Missing Targets or prices in supplemental data?
", "votes": null, "replies": [] }, { "id": 1662610, "author_name": "wils3456", "author_url": "", "post_date": "01/24/2022 12:24:25", "content": "Hi @markbquant
\nI'm not quite sure actually; at first my only assumption was that an incorrect Asset ID could be passed through to us by the API, but since Chris' comment I've added another case which catches out np.nan & np.inf and sets y_pred = 0.0
19s per iteration ? so more than 1s to predict a given row ? This doesn't seems normal. What framework do you use ? How many model do you have ?
", "votes": null, "replies": [ { "id": 1662013, "author_name": "wils3456", "author_url": "", "post_date": "01/23/2022 23:39:58", "content": "I've checked the timedelta a couple of times and it actually averages out to ~15s
\nSo I'm using the SciKit StackingRegressor; and within this regressor I'm using the x1 SciKit linear model, x1 LGBM model and x1 XGB model.
\nDo you have any suggestions of where I could be going wrong with this, could this maybe be too many models for the notebook to handle at once?
", "votes": null, "replies": [] } ] }, { "id": 1662195, "author_name": "markbquant", "author_url": "", "post_date": "01/24/2022 05:37:43", "content": "I see you are taking the test price and predicting Target. This is incorrect because the Target data should be calculated using 3750 previous price data (~2.5 days). This way we have to predict the future prices of 16 steps (minutes) ahead of the existing test data.
", "votes": null, "replies": [ { "id": 1662617, "author_name": "wils3456", "author_url": "", "post_date": "01/24/2022 12:32:22", "content": "Would it be possible for you to elaborate on this a little further please? As I'm fairly new to Data Science and don't quite understand…
\nWhen analysing the data, I saw there was a Target was present; so I dealt with it like I would any other regular Regression problem. However, if there are any further steps I need to carry out to ensure that I'm predicting future values, could you please point me in the right direction. (i.e. a little hint, an article, blog post, a past notebook, etc…)
\nMany Thanks!
", "votes": null, "replies": [] }, { "id": 1662988, "author_name": "markbquant", "author_url": "", "post_date": "01/24/2022 17:37:38", "content": "The target is calculated as shown in this function calculate_target from the notebook: https://www.kaggle.com/alexfir/recreating-target
We have to predict 16 future prices or their returns:
\n$$R^a(t) = log (P^a(t+16)\\ /\\ P^a(t+1))$$
\ntargets[asset_names[i]] = price.shift(periods=-16)/price.shift(periods=-1) - 1
and based on them calculate prediction target.
\nIn live there will be 3 month of test data, all without targets. We cannot predict targets 3*30*24*60 intervals ahead.
Hint: target = asset_returns - beta(average_market_returns) = r - b(m)
\nYou could predict target directly and it might work out. Or you could predict r, b and m separately and create the target. (big hint part) b is sometimes zero because there is a nan in the 3750 perioud rolling mean. For more clarification, please find the notebook that says target calculation sota, and figure out exactly how those calculations work.
The big point is, when beta changes, the target changes drastically.
", "votes": null, "replies": [] }, { "id": 1663284, "author_name": "wils3456", "author_url": "", "post_date": "01/25/2022 00:16:34", "content": "Hi @chrisrichardmiles / @markbquant
\nThis is quite interesting; I've had a read through the content you've both suggested and can now see why my current approach, of directly predicting the target from the features may not be the best way forward.
\nGiven this new information, I'm currently researching more on making use of rolling point averages programmaticaly, to aid with the manipulation of the training data; prior to training my model. If anyone has any insightful blog posts or notebooks which they feel would help explain this concept with examples, this would be greatly appreciated!
", "votes": null, "replies": [] }, { "id": 1663304, "author_name": "chrisrichardmiles", "author_url": "", "post_date": "01/25/2022 01:00:27", "content": "@wils3456 Sounds like a good idea. I am currently working the basics to make sure I can keep track everything to calculate the betas, returns, and targets correctly.
\n@lucasmorin has some good notebooks and comments about running means.
For reference: my offline training is 9-10mins and leaderboard is like 6.5 hours.. So yours at 26mins will defo time out.
", "votes": null, "replies": [ { "id": 1663293, "author_name": "wils3456", "author_url": "", "post_date": "01/25/2022 00:25:16", "content": "\nThanks soo much for your input, I'm currently working on making certain aspects of my code more efficient through he use of direct numpy manipulations and the removal of unecessary code; and, I've managed to cut down my last offline run to 13mins whilst each row predicition now takes ~9-10secs, which I believe is a huge improvement.
\nHowever, I've still got loads more work to do before it's running at optimal speeds.
\nI have one question, do you feel it's a waste of processing time having the same model in both the first and meta level of the StackingRegressor?
", "votes": null, "replies": [] } ] } ], "raw_markdown_by_id": { "1661755": "Hi All,\n\nJust wanted to seek some help from some fellow Kagglers; as this is my first real competition entry, since joining Kaggle for the 30 day intro to ML challange. \n\nI've been attempting to submit my code to the LB, after succeding to \"Save & Run All (Commit)\", however I believe the online G-Research API Limitations are causing my code to timeout upon submission.\n\nI've read in a couple of other discussions that each iteration is meant to be under 0.25s, however each of my individual feature engineering steps are far above above this value, which is a little frustrating as I believe it helps towards making my predicitions more accurate, so I don't really want to get rid of them completely.\n\nStage 1. **Feature Engineering steps [row-by-row] ~4s per iteration:**\n- Make epoch human readable [datetime64[ns]]\n- Add four new features\n- Scale the numeric columns [Count to VWAP + new features]\n- OneHot Encode categorical column [Asset ID]\n\nQ1. Do you suggest I get rid of these all to meet the time restrictions or is there anway I can keep these and carry them out outside the iteration loop?\n\nStage 2. **Prediction step [row-by-row] ~19s per iteration:**\n- Stacked Model Prediction\n\nQ2. Is there any way I can maybe speed this up by storing a copy of the model in a file or something else?\nQ3. Is there a way I could carry this out outside the env iteration loop to avoid triggering the API time limitation?\n\n**Copy of my Submission Iteration Loop**\n```\ncrypo_ID_list = list(crypo_id_store_dict.keys())\ntime_delta_list = []\n\nfor i, (df_test, df_pred) in enumerate(iter_test):\n for j , row in df_test.iterrows():\n try:\n start_time = datetime.datetime.now()\n \n if row['Asset_ID'] in crypo_ID_list:\n feature_engineering(row)\n y_pred = model_stacking.predict(row.loc[:, row.columns!='row_id'])\n else: \n y_pred = 0.0\n \n time_delta_list.append(datetime.datetime.now() - start_time)\n except: \n y_pred = 0.0\n traceback.print_exc()\n df_pred.loc[df_pred['row_id'].values == row['row_id'].values, 'Target'] = y_pred\n all_df_test.append(df_test)\n env.predict(df_pred)\n```\n\nNOTE: Training model offline takes ~16-26mins\n\nI'm a little confused on what steps to take as it's reaching the deadline and I haven't even been able to make a single submission due to 'timeout' errors and I'm worried I'll be scored with a fail on this competiton when I've put so many hours in, getting to this point. \n\nIf you need me to clarify anything further, in-order to help me, please do let me know.\n\nMany Thanks,\nWilson", "1661776": "Are you sure its timeout? That would mean your notebook takes over 9 hrs. Does your feature engineering take into account that data could be missing for an asset?", "1661806": "Hi Chris,\n\nThanks for your speedy comment in reply to my issue!\n\nWhenever I attempt my submission to the LB the error only ever shows up as 'notebook timeout' with no further information.\n\nHowever, with regards to your second question:\n\n- Wouldn't the missing data from the API trigger an exception in my submission iteration loop and mark the `y_pred` value as `0.0`; as can be seen in the code snippet I'd included?\n\n- Or would I have to implement further cases to handle the occurrence of missing or infinity values data in my feature engineering function?", "1661962": "19s per iteration ? so more than 1s to predict a given row ? This doesn't seems normal. What framework do you use ? How many model do you have ?", "1662013": "I've checked the timedelta a couple of times and it actually averages out to ~15s\n\nSo I'm using the SciKit StackingRegressor; and within this regressor I'm using the x1 SciKit linear model, x1 LGBM model and x1 XGB model.\n\nDo you have any suggestions of where I could be going wrong with this, could this maybe be too many models for the notebook to handle at once?", "1662036": "Is this supposed to change row inplace? \n```\nfeature_engineering(row)\n```\nbecause then you pass row to your predict function and I'm not sure the features are there. \n\nI am sorry, I am really not good enough to be able to give specific and good advice, but I the best advice I can give is this: \n\nSimplify your code until you get a valid submission and then build up the complexity. This is the only way to go I think. \n\nOne quick thing I saw looking at your code is this: the feature engineering is done on one row at a time. Make sure this matches your feature engineering before training. \n\nSorry I can't spend more time debugging, I have so many of my own issues to deal with, but I promise that if you simplify first, and be sure all preprocessing steps before training are possible and the same as during the prediction phase (use the api with 3 timestamps to test this) you will be alright.", "1662066": "Hi Again Chris, \n\nThe feature additions and manipulations are carried out inplace to reduce dataframe copying and memory errors, as this was one of the issues I'd ran into previously; but managed to rectify, combined with dataframe memory reductions and garbage collection.\n\nI perfectly understand though, no need to be sorry as you've been of tremendous help; all the help you've been able to provide thus far has sparked some ideas for me, which I feel can help towards me making my first valid submission. \n\nThanks so much for the assistance and best of luck in the competiton!", "1662182": "wils3456 Great to hear you got some ideas. Keep it up. You will get it. Good luck to you as well.", "1662195": "I see you are taking the test price and predicting Target. This is incorrect because the Target data should be calculated using 3750 previous price data (~2.5 days). This way we have to predict the future prices of 16 steps (minutes) ahead of the existing test data.", "1662200": "> Wouldn't the missing data from the API trigger an exception\n\nWhat can be missing in the test data? The prices? The inconsistent row_ids? New Asset_ID? Missing Targets or prices in supplemental data?", "1662610": "Hi @markbquant \n\nI'm not quite sure actually; at first my only assumption was that an incorrect Asset ID could be passed through to us by the API, but since Chris' comment I've added another case which catches out `np.nan` & `np.inf` and sets `y_pred = 0.0`", "1662617": "Would it be possible for you to elaborate on this a little further please? As I'm fairly new to Data Science and don't quite understand...\n\nWhen analysing the data, I saw there was a Target was present; so I dealt with it like I would any other regular Regression problem. However, if there are any further steps I need to carry out to ensure that I'm predicting future values, could you please point me in the right direction. (i.e. a little hint, an article, blog post, a past notebook, etc...)\n\nMany Thanks!", "1662745": "For reference: my offline training is 9-10mins and leaderboard is like 6.5 hours.. So yours at 26mins will defo time out.", "1662988": "The target is calculated as shown in this function `calculate_target` from the notebook: https://www.kaggle.com/alexfir/recreating-target\n\nWe have to predict 16 future prices or their returns:\n$$R^a(t) = log (P^a(t+16)\\ /\\ P^a(t+1))$$\n`targets[asset_names[i]] = price.shift(periods=-16)/price.shift(periods=-1) - 1`\n\nand based on them calculate prediction target.\nIn live there will be 3 month of test data, all without targets. We cannot predict targets `3*30*24*60` intervals ahead.", "1663092": "Hint: target = asset_returns - beta(average_market_returns) = r - b(m)\nYou could predict target directly and it might work out. Or you could predict r, b and m separately and create the target. (big hint part) b is sometimes zero because there is a nan in the 3750 perioud rolling mean. For more clarification, please find the notebook that says target calculation sota, and figure out exactly how those calculations work. \n\nThe big point is, when beta changes, the target changes drastically.", "1663284": "Hi @chrisrichardmiles / @markbquant \n\nThis is quite interesting; I've had a read through the content you've both suggested and can now see why my current approach, of directly predicting the target from the features may not be the best way forward. \n\nGiven this new information, I'm currently researching more on making use of rolling point averages programmaticaly, to aid with the manipulation of the training data; prior to training my model. If anyone has any insightful blog posts or notebooks which they feel would help explain this concept with examples, this would be greatly appreciated!", "1663293": "julianmukaj,\n\nThanks soo much for your input, I'm currently working on making certain aspects of my code more efficient through he use of direct numpy manipulations and the removal of unecessary code; and, I've managed to cut down my last offline run to 13mins whilst each row predicition now takes ~9-10secs, which I believe is a huge improvement. \n\nHowever, I've still got loads more work to do before it's running at optimal speeds. \n\nI have one question, do you feel it's a waste of processing time having the same model in both the first and meta level of the StackingRegressor?", "1663304": "wils3456 Sounds like a good idea. I am currently working the basics to make sure I can keep track everything to calculate the betas, returns, and targets correctly. \n@lucasmorin has some good notebooks and comments about running means." }, "source": "meta" }