Sorry to ask this, but recently I found that models that run under 15min seem to timeout during submission after hours.
\nDid I something wrong or are others facing the issue as well?
Thanks a lot,
\nWolfgang
Sorry to ask this, but recently I found that models that run under 15min seem to timeout during submission after hours.
\nDid I something wrong or are others facing the issue as well?
Thanks a lot,
\nWolfgang
Anybody? Now also notebooks that I successfully submitted a week ago are timing out during submission?
", "rawMarkdown": "Anybody? Now also notebooks that I successfully submitted a week ago are timing out during submission?", "votes": null }, { "id": "1618221", "postDate": "12/14/2021 19:20:13", "content": "As someone calculated in one of the discussion threads you need to make sure the timing of a 1 minute prediction (for the 14 assets) is under 0.25s to prevent timing out. This without taking into account the loading and training time if done in same notebook. Suggest you check the timing of your prediction loop for the sample 4 minutes in the LB.
", "rawMarkdown": "As someone calculated in one of the discussion threads you need to make sure the timing of a 1 minute prediction (for the 14 assets) is under 0.25s to prevent timing out. This without taking into account the loading and training time if done in same notebook. Suggest you check the timing of your prediction loop for the sample 4 minutes in the LB.", "votes": null }, { "id": "1618281", "postDate": "12/14/2021 20:41:46", "content": "Ahh ok interesting, thanks a lot for sharing. I was not aware of that.
", "rawMarkdown": "Ahh ok interesting, thanks a lot for sharing. I was not aware of that.", "votes": null }, { "id": "1620430", "postDate": "12/16/2021 18:38:24", "content": "I've personally found it really difficult to infer the sub success from the sample 4 mins, with substantially faster notebooks than previous successes failing. Find it ridiculous that we only get 4 sample minutes in the API when the public LB is a mess anyway
", "rawMarkdown": "I've personally found it really difficult to infer the sub success from the sample 4 mins, with substantially faster notebooks than previous successes failing. Find it ridiculous that we only get 4 sample minutes in the API when the public LB is a mess anyway", "votes": null }, { "id": "1656209", "postDate": "01/19/2022 07:11:35", "content": "My primary issue with the submission timeout, even if I now fully understand the reason behind, is that for me its nearly impossible to achieve the inference times with any kind of LTSM or Keras model. No matter what I try, no matter how simple I design the layers its impossible to be below the timeout threshold.
\nWith XGboost its possible as its model is much faster for the single predictions.
How are your experiences here? Anybody a hint on how to speed up a layered Keras model to be as fast in prediction as necessary?
", "rawMarkdown": "My primary issue with the submission timeout, even if I now fully understand the reason behind, is that for me its nearly impossible to achieve the inference times with any kind of LTSM or Keras model. No matter what I try, no matter how simple I design the layers its impossible to be below the timeout threshold. \nWith XGboost its possible as its model is much faster for the single predictions.\n\nHow are your experiences here? Anybody a hint on how to speed up a layered Keras model to be as fast in prediction as necessary?", "votes": null }, { "id": "1656594", "postDate": "01/19/2022 13:20:13", "content": "Are you using GPU because that would speed it up. Also, if you train a single NN model and use that to predict on the lb data set, that shouldn't cause timeout issues. Just make sure its outside of the lb predicting loop.
", "rawMarkdown": "Are you using GPU because that would speed it up. Also, if you train a single NN model and use that to predict on the lb data set, that shouldn't cause timeout issues. Just make sure its outside of the lb predicting loop.", "votes": null }, { "id": "1656596", "postDate": "01/19/2022 13:20:52", "content": "Because if it is in the loop it runs around 140,000 times.
", "rawMarkdown": "Because if it is in the loop it runs around 140,000 times.", "votes": null }, { "id": "1656765", "postDate": "01/19/2022 16:01:21", "content": "Not sure about that, can you give an example as a code template? I do the training outside and then call the predict method for each of the 14 asset iterations. See below:
\niter_test = env.iter_test() # an iterator which loops over the test set and sample submission
for (test_df, sample_prediction_df) in iter_test:\n start_time = time.time()\n test_df = feature_engineering(test_df)\n sample_prediction_df['Target'] = 0 # make your predictions here\n env.predict(sample_prediction_df) # register your predictions\n print(\"--- %s seconds ---\" % (time.time() - start_time))\nIf feature_engineering takes more than around .25 seconds then you will also timeout. This code is run many times so it has to be quick.
", "rawMarkdown": "If feature_engineering takes more than around .25 seconds then you will also timeout. This code is run many times so it has to be quick.", "votes": null }, { "id": "1656897", "postDate": "01/19/2022 18:46:41", "content": "Yes I am aware of that, thx.
", "rawMarkdown": "Yes I am aware of that, thx.", "votes": null } ], "comments": [ { "id": 1618062, "author_name": "wolfgangb33r", "author_url": "", "post_date": "12/14/2021 15:51:31", "content": "Anybody? Now also notebooks that I successfully submitted a week ago are timing out during submission?
", "votes": null, "replies": [ { "id": 1618221, "author_name": "casras", "author_url": "", "post_date": "12/14/2021 19:20:13", "content": "As someone calculated in one of the discussion threads you need to make sure the timing of a 1 minute prediction (for the 14 assets) is under 0.25s to prevent timing out. This without taking into account the loading and training time if done in same notebook. Suggest you check the timing of your prediction loop for the sample 4 minutes in the LB.
", "votes": null, "replies": [] }, { "id": 1618281, "author_name": "wolfgangb33r", "author_url": "", "post_date": "12/14/2021 20:41:46", "content": "Ahh ok interesting, thanks a lot for sharing. I was not aware of that.
", "votes": null, "replies": [] }, { "id": 1620430, "author_name": "jacobhowardparker", "author_url": "", "post_date": "12/16/2021 18:38:24", "content": "I've personally found it really difficult to infer the sub success from the sample 4 mins, with substantially faster notebooks than previous successes failing. Find it ridiculous that we only get 4 sample minutes in the API when the public LB is a mess anyway
", "votes": null, "replies": [] } ] }, { "id": 1656209, "author_name": "wolfgangb33r", "author_url": "", "post_date": "01/19/2022 07:11:35", "content": "My primary issue with the submission timeout, even if I now fully understand the reason behind, is that for me its nearly impossible to achieve the inference times with any kind of LTSM or Keras model. No matter what I try, no matter how simple I design the layers its impossible to be below the timeout threshold.
\nWith XGboost its possible as its model is much faster for the single predictions.
How are your experiences here? Anybody a hint on how to speed up a layered Keras model to be as fast in prediction as necessary?
", "votes": null, "replies": [ { "id": 1656594, "author_name": "petersk20", "author_url": "", "post_date": "01/19/2022 13:20:13", "content": "Are you using GPU because that would speed it up. Also, if you train a single NN model and use that to predict on the lb data set, that shouldn't cause timeout issues. Just make sure its outside of the lb predicting loop.
", "votes": null, "replies": [] }, { "id": 1656596, "author_name": "petersk20", "author_url": "", "post_date": "01/19/2022 13:20:52", "content": "Because if it is in the loop it runs around 140,000 times.
", "votes": null, "replies": [] }, { "id": 1656765, "author_name": "wolfgangb33r", "author_url": "", "post_date": "01/19/2022 16:01:21", "content": "Not sure about that, can you give an example as a code template? I do the training outside and then call the predict method for each of the 14 asset iterations. See below:
\niter_test = env.iter_test() # an iterator which loops over the test set and sample submission
for (test_df, sample_prediction_df) in iter_test:\n start_time = time.time()\n test_df = feature_engineering(test_df)\n sample_prediction_df['Target'] = 0 # make your predictions here\n env.predict(sample_prediction_df) # register your predictions\n print(\"--- %s seconds ---\" % (time.time() - start_time))\nIf feature_engineering takes more than around .25 seconds then you will also timeout. This code is run many times so it has to be quick.
", "votes": null, "replies": [] }, { "id": 1656897, "author_name": "wolfgangb33r", "author_url": "", "post_date": "01/19/2022 18:46:41", "content": "Yes I am aware of that, thx.
", "votes": null, "replies": [] } ] } ], "raw_markdown_by_id": { "1617561": "Sorry to ask this, but recently I found that models that run under 15min seem to timeout during submission after hours. \nDid I something wrong or are others facing the issue as well?\n\nThanks a lot,\nWolfgang", "1618062": "Anybody? Now also notebooks that I successfully submitted a week ago are timing out during submission?", "1618221": "As someone calculated in one of the discussion threads you need to make sure the timing of a 1 minute prediction (for the 14 assets) is under 0.25s to prevent timing out. This without taking into account the loading and training time if done in same notebook. Suggest you check the timing of your prediction loop for the sample 4 minutes in the LB.", "1618281": "Ahh ok interesting, thanks a lot for sharing. I was not aware of that.", "1620430": "I've personally found it really difficult to infer the sub success from the sample 4 mins, with substantially faster notebooks than previous successes failing. Find it ridiculous that we only get 4 sample minutes in the API when the public LB is a mess anyway", "1656209": "My primary issue with the submission timeout, even if I now fully understand the reason behind, is that for me its nearly impossible to achieve the inference times with any kind of LTSM or Keras model. No matter what I try, no matter how simple I design the layers its impossible to be below the timeout threshold. \nWith XGboost its possible as its model is much faster for the single predictions.\n\nHow are your experiences here? Anybody a hint on how to speed up a layered Keras model to be as fast in prediction as necessary?", "1656594": "Are you using GPU because that would speed it up. Also, if you train a single NN model and use that to predict on the lb data set, that shouldn't cause timeout issues. Just make sure its outside of the lb predicting loop.", "1656596": "Because if it is in the loop it runs around 140,000 times.", "1656765": "Not sure about that, can you give an example as a code template? I do the training outside and then call the predict method for each of the 14 asset iterations. See below:\niter_test = env.iter_test() # an iterator which loops over the test set and sample submission\n\n```\nfor (test_df, sample_prediction_df) in iter_test:\n start_time = time.time()\n test_df = feature_engineering(test_df)\n sample_prediction_df['Target'] = 0 # make your predictions here\n env.predict(sample_prediction_df) # register your predictions\n print(\"--- %s seconds ---\" % (time.time() - start_time))\n```", "1656802": "If feature_engineering takes more than around .25 seconds then you will also timeout. This code is run many times so it has to be quick.", "1656897": "Yes I am aware of that, thx." }, "source": "meta" }