{ "id": 286070, "title": "6-Fold Models Ran for 9 Hours and Received the Timeout Error", "url": "/competitions/g-research-crypto-forecasting/discussion/286070", "author_name": "", "post_date": "2021-11-07T20:19:11.661076100Z", "votes": 21, "comment_count": 18, "views": 0, "content": "

Hi,

\n

For this competition, I created 8 more features during the preprocessing part and built 6-fold xgb models using GPU. However, the whole prediction process is encountering a \"timeout error\". Have you encountered this situation, too? Do you have any suggestions?

\n

Thanks!

", "messages": [ { "id": "1574732", "postDate": "11/07/2021 20:19:11", "content": "

Hi,

\n

For this competition, I created 8 more features during the preprocessing part and built 6-fold xgb models using GPU. However, the whole prediction process is encountering a \"timeout error\". Have you encountered this situation, too? Do you have any suggestions?

\n

Thanks!

", "rawMarkdown": "Hi,\n\nFor this competition, I created 8 more features during the preprocessing part and built 6-fold xgb models using GPU. However, the whole prediction process is encountering a \"timeout error\". Have you encountered this situation, too? Do you have any suggestions?\n\nThanks!", "votes": null }, { "id": "1577349", "postDate": "11/10/2021 02:31:49", "content": "

Endless timeout error. I am nearly giving up.

", "rawMarkdown": "Endless timeout error. I am nearly giving up.", "votes": null }, { "id": "1578470", "postDate": "11/11/2021 04:58:21", "content": "

I just use 2-fold models to make predictions and they fail in the time requirement again.

", "rawMarkdown": "I just use 2-fold models to make predictions and they fail in the time requirement again.", "votes": null }, { "id": "1578486", "postDate": "11/11/2021 05:26:29", "content": "

Most likely, when writing the API, not the most optimal algorithms and data structures were used (pandas instead of numpy … O(n**2) instead O(n) … and so on). I wonder how long does a 'dummy' prediction take? (one number) I haven't had time to try it yet, maybe someone checked it?

", "rawMarkdown": "Most likely, when writing the API, not the most optimal algorithms and data structures were used (pandas instead of numpy ... O(n**2) instead O(n) ... and so on). I wonder how long does a 'dummy' prediction take? (one number) I haven't had time to try it yet, maybe someone checked it?", "votes": null }, { "id": "1578487", "postDate": "11/11/2021 05:28:57", "content": "

a 'dummy' prediction takes about 3 hours to display the score result.

", "rawMarkdown": "a 'dummy' prediction takes about 3 hours to display the score result.", "votes": null }, { "id": "1578495", "postDate": "11/11/2021 05:45:16", "content": "

This should not be) I bet 99.9% that this is a suboptimal code. I suggest the organizers share the \"slow\" pieces of code so that we can optimize them together.

", "rawMarkdown": "This should not be) I bet 99.9% that this is a suboptimal code. I suggest the organizers share the \"slow\" pieces of code so that we can optimize them together.", "votes": null }, { "id": "1578752", "postDate": "11/11/2021 10:12:32", "content": "

I have a same problem. I don't know how to resolve it. 😨

", "rawMarkdown": "I have a same problem. I don't know how to resolve it. 😨", "votes": null }, { "id": "1579337", "postDate": "11/11/2021 18:51:36", "content": "

I ran some additional tests just to confirm there wasn't something wrong with my initial results and I stand by the existing note on the data page:

\n
\n

The API will also consume less than 30 minutes of runtime for loading and serving the data.

\n
\n

@yus002 can you share your dummy notebook with me? I see submissions from you that took much less than 3 hours to complete. If the end results are taking 3 hours to become visible to you then I'll need to look into what's happening outside of the actual notebook rerun and scoring process.

", "rawMarkdown": "I ran some additional tests just to confirm there wasn't something wrong with my initial results and I stand by the existing note on the data page: \n>The API will also consume less than 30 minutes of runtime for loading and serving the data.\n\n@yus002 can you share your dummy notebook with me? I see submissions from you that took much less than 3 hours to complete. If the end results are taking 3 hours to become visible to you then I'll need to look into what's happening outside of the actual notebook rerun and scoring process.", "votes": null }, { "id": "1579354", "postDate": "11/11/2021 19:35:43", "content": "

Hi, yes. The notebook I use is https://www.kaggle.com/yus002/naive-prediction-by-mean-values/notebook?scriptVersionId=78725687. Thank you!!

", "rawMarkdown": "Hi, yes. The notebook I use is https://www.kaggle.com/yus002/naive-prediction-by-mean-values/notebook?scriptVersionId=78725687. Thank you!!", "votes": null }, { "id": "1579355", "postDate": "11/11/2021 19:39:08", "content": "

Hi, I am still testing. This time I reduce the value of hyperparamters that control overfitting. In the end, I got 2-fold models which only hold 1.8 mb. The time I used to ran my submission was 21 seconds, and the submission still got a timeout error.

", "rawMarkdown": "Hi, I am still testing. This time I reduce the value of hyperparamters that control overfitting. In the end, I got 2-fold models which only hold 1.8 mb. The time I used to ran my submission was 21 seconds, and the submission still got a timeout error.", "votes": null }, { "id": "1580433", "postDate": "11/12/2021 18:31:44", "content": "

Would you mind resubmitting that notebook? My own simple test submission just now finished in about eight minutes.

", "rawMarkdown": "Would you mind resubmitting that notebook? My own simple test submission just now finished in about eight minutes.", "votes": null }, { "id": "1580574", "postDate": "11/12/2021 22:49:38", "content": "

Hi, yes. It runs for 6 minutes and finishes with a score. I realize that I actually remember the time cost of dummy approach wrong. 😅

", "rawMarkdown": "Hi, yes. It runs for 6 minutes and finishes with a score. I realize that I actually remember the time cost of dummy approach wrong. 😅", "votes": null }, { "id": "1580577", "postDate": "11/12/2021 22:58:02", "content": "

Phew! Thanks for double checking.

", "rawMarkdown": "Phew! Thanks for double checking.", "votes": null }, { "id": "1617203", "postDate": "12/14/2021 00:08:09", "content": "

Same Problem !! Offline kernel takes about 45min. Online submission timeout! 😭

", "rawMarkdown": "Same Problem !! Offline kernel takes about 45min. Online submission timeout! 😭", "votes": null }, { "id": "1617469", "postDate": "12/14/2021 05:35:09", "content": "

Hi, data preprocessing and model prediction in each loop take time. Make sure you optimize the code efficiency in these two parts.

", "rawMarkdown": "Hi, data preprocessing and model prediction in each loop take time. Make sure you optimize the code efficiency in these two parts.", "votes": null }, { "id": "1661013", "postDate": "01/23/2022 06:48:58", "content": "

Hi, @sohier @yus002 I have a notebook that runs under 40secs in Kaggle notebooks. I have timed my average prediction also to be around 0.07 secs.
\nBut when I submit the notebook times out after 9 hours. Not sure what am I doing wrong here. Any suggestions? :D
\nThe link to notebook: https://www.kaggle.com/ankit001/mlp-prediction?scriptVersionId=85919823

", "rawMarkdown": "Hi, @sohier @yus002 I have a notebook that runs under 40secs in Kaggle notebooks. I have timed my average prediction also to be around 0.07 secs.\nBut when I submit the notebook times out after 9 hours. Not sure what am I doing wrong here. Any suggestions? :D\nThe link to notebook: https://www.kaggle.com/ankit001/mlp-prediction?scriptVersionId=85919823", "votes": null }, { "id": "1661105", "postDate": "01/23/2022 08:19:27", "content": "
time_taken = []\nfor i, (df_test, df_pred) in enumerate(iter_test):\n    for j, row in df_test.iterrows():\n\n        try:\n            #init_time = time.time()\n            model = models[row['Asset_ID']]\n            x_test = get_features(row)\n            #n_steps = x_test.shape[0]\n            #x_test = x_test.to_numpy()\n            #x_test = x_test.reshape((1, n_steps))\n            y_pred = model.predict([x_test])[0]\n\n            df_pred.loc[df_pred['row_id'] == row['row_id'], 'Target'] = y_pred\n            #time_taken.append(time.time() - init_time)\n\n        except:\n            #print(f'{i}-th iteration of the test dataset, {j}-th row - there was the exception, then set Target = 0')\n            df_pred.loc[df_pred['row_id'] == row['row_id'], 'Target'] = 0\n\n#         # Print just one sample row to get a feeling of what it looks like        \n#         if i == 0 and j == 0:\n#             print('Example of the x_test data')\n#             display(x_test)\n\n#     # Display the first prediction dataframe\n#     if i == 0:\n#         print('Example of the prediction for test data')\n#         display(df_pred)\n    #df_pred['Target'] = df_pred['Target'].fillna(0)\n\n    # Send submissions\n    env.predict(df_pred)\n
\n

Hi, the comment out codes should be considered to remove.

", "rawMarkdown": "```\ntime_taken = []\nfor i, (df_test, df_pred) in enumerate(iter_test):\n for j, row in df_test.iterrows():\n \n try:\n #init_time = time.time()\n model = models[row['Asset_ID']]\n x_test = get_features(row)\n #n_steps = x_test.shape[0]\n #x_test = x_test.to_numpy()\n #x_test = x_test.reshape((1, n_steps))\n y_pred = model.predict([x_test])[0]\n\n df_pred.loc[df_pred['row_id'] == row['row_id'], 'Target'] = y_pred\n #time_taken.append(time.time() - init_time)\n \n except:\n #print(f'{i}-th iteration of the test dataset, {j}-th row - there was the exception, then set Target = 0')\n df_pred.loc[df_pred['row_id'] == row['row_id'], 'Target'] = 0\n \n# # Print just one sample row to get a feeling of what it looks like \n# if i == 0 and j == 0:\n# print('Example of the x_test data')\n# display(x_test)\n\n# # Display the first prediction dataframe\n# if i == 0:\n# print('Example of the prediction for test data')\n# display(df_pred)\n #df_pred['Target'] = df_pred['Target'].fillna(0)\n\n # Send submissions\n env.predict(df_pred)\n```\nHi, the comment out codes should be considered to remove.", "votes": null }, { "id": "1669504", "postDate": "01/30/2022 18:37:14", "content": "

Offline kernel takes 45min ?
\nHow ?😳

", "rawMarkdown": "Offline kernel takes 45min ?\nHow ?😳", "votes": null }, { "id": "1669507", "postDate": "01/30/2022 18:38:09", "content": "

Mine takes 35s and it TimeOut i can't understand why ?
\nKnowing that each iteration on test is under 0.14s

", "rawMarkdown": "Mine takes 35s and it TimeOut i can't understand why ?\nKnowing that each iteration on test is under 0.14s", "votes": null } ], "comments": [ { "id": 1577349, "author_name": "yus002", "author_url": "", "post_date": "11/10/2021 02:31:49", "content": "

Endless timeout error. I am nearly giving up.

", "votes": null, "replies": [] }, { "id": 1578470, "author_name": "yus002", "author_url": "", "post_date": "11/11/2021 04:58:21", "content": "

I just use 2-fold models to make predictions and they fail in the time requirement again.

", "votes": null, "replies": [] }, { "id": 1578486, "author_name": "sapr3s", "author_url": "", "post_date": "11/11/2021 05:26:29", "content": "

Most likely, when writing the API, not the most optimal algorithms and data structures were used (pandas instead of numpy … O(n**2) instead O(n) … and so on). I wonder how long does a 'dummy' prediction take? (one number) I haven't had time to try it yet, maybe someone checked it?

", "votes": null, "replies": [ { "id": 1578487, "author_name": "yus002", "author_url": "", "post_date": "11/11/2021 05:28:57", "content": "

a 'dummy' prediction takes about 3 hours to display the score result.

", "votes": null, "replies": [] }, { "id": 1578495, "author_name": "sapr3s", "author_url": "", "post_date": "11/11/2021 05:45:16", "content": "

This should not be) I bet 99.9% that this is a suboptimal code. I suggest the organizers share the \"slow\" pieces of code so that we can optimize them together.

", "votes": null, "replies": [] } ] }, { "id": 1578752, "author_name": "tensorchoko", "author_url": "", "post_date": "11/11/2021 10:12:32", "content": "

I have a same problem. I don't know how to resolve it. 😨

", "votes": null, "replies": [ { "id": 1579355, "author_name": "yus002", "author_url": "", "post_date": "11/11/2021 19:39:08", "content": "

Hi, I am still testing. This time I reduce the value of hyperparamters that control overfitting. In the end, I got 2-fold models which only hold 1.8 mb. The time I used to ran my submission was 21 seconds, and the submission still got a timeout error.

", "votes": null, "replies": [] } ] }, { "id": 1579337, "author_name": "sohier", "author_url": "", "post_date": "11/11/2021 18:51:36", "content": "

I ran some additional tests just to confirm there wasn't something wrong with my initial results and I stand by the existing note on the data page:

\n
\n

The API will also consume less than 30 minutes of runtime for loading and serving the data.

\n
\n

@yus002 can you share your dummy notebook with me? I see submissions from you that took much less than 3 hours to complete. If the end results are taking 3 hours to become visible to you then I'll need to look into what's happening outside of the actual notebook rerun and scoring process.

", "votes": null, "replies": [ { "id": 1579354, "author_name": "yus002", "author_url": "", "post_date": "11/11/2021 19:35:43", "content": "

Hi, yes. The notebook I use is https://www.kaggle.com/yus002/naive-prediction-by-mean-values/notebook?scriptVersionId=78725687. Thank you!!

", "votes": null, "replies": [] }, { "id": 1580433, "author_name": "sohier", "author_url": "", "post_date": "11/12/2021 18:31:44", "content": "

Would you mind resubmitting that notebook? My own simple test submission just now finished in about eight minutes.

", "votes": null, "replies": [] }, { "id": 1580574, "author_name": "yus002", "author_url": "", "post_date": "11/12/2021 22:49:38", "content": "

Hi, yes. It runs for 6 minutes and finishes with a score. I realize that I actually remember the time cost of dummy approach wrong. 😅

", "votes": null, "replies": [] }, { "id": 1580577, "author_name": "sohier", "author_url": "", "post_date": "11/12/2021 22:58:02", "content": "

Phew! Thanks for double checking.

", "votes": null, "replies": [] }, { "id": 1661013, "author_name": "ankit001", "author_url": "", "post_date": "01/23/2022 06:48:58", "content": "

Hi, @sohier @yus002 I have a notebook that runs under 40secs in Kaggle notebooks. I have timed my average prediction also to be around 0.07 secs.
\nBut when I submit the notebook times out after 9 hours. Not sure what am I doing wrong here. Any suggestions? :D
\nThe link to notebook: https://www.kaggle.com/ankit001/mlp-prediction?scriptVersionId=85919823

", "votes": null, "replies": [] }, { "id": 1661105, "author_name": "yus002", "author_url": "", "post_date": "01/23/2022 08:19:27", "content": "
time_taken = []\nfor i, (df_test, df_pred) in enumerate(iter_test):\n    for j, row in df_test.iterrows():\n\n        try:\n            #init_time = time.time()\n            model = models[row['Asset_ID']]\n            x_test = get_features(row)\n            #n_steps = x_test.shape[0]\n            #x_test = x_test.to_numpy()\n            #x_test = x_test.reshape((1, n_steps))\n            y_pred = model.predict([x_test])[0]\n\n            df_pred.loc[df_pred['row_id'] == row['row_id'], 'Target'] = y_pred\n            #time_taken.append(time.time() - init_time)\n\n        except:\n            #print(f'{i}-th iteration of the test dataset, {j}-th row - there was the exception, then set Target = 0')\n            df_pred.loc[df_pred['row_id'] == row['row_id'], 'Target'] = 0\n\n#         # Print just one sample row to get a feeling of what it looks like        \n#         if i == 0 and j == 0:\n#             print('Example of the x_test data')\n#             display(x_test)\n\n#     # Display the first prediction dataframe\n#     if i == 0:\n#         print('Example of the prediction for test data')\n#         display(df_pred)\n    #df_pred['Target'] = df_pred['Target'].fillna(0)\n\n    # Send submissions\n    env.predict(df_pred)\n
\n

Hi, the comment out codes should be considered to remove.

", "votes": null, "replies": [] } ] }, { "id": 1617203, "author_name": "linjincong", "author_url": "", "post_date": "12/14/2021 00:08:09", "content": "

Same Problem !! Offline kernel takes about 45min. Online submission timeout! 😭

", "votes": null, "replies": [ { "id": 1617469, "author_name": "yus002", "author_url": "", "post_date": "12/14/2021 05:35:09", "content": "

Hi, data preprocessing and model prediction in each loop take time. Make sure you optimize the code efficiency in these two parts.

", "votes": null, "replies": [] }, { "id": 1669504, "author_name": "amedprof", "author_url": "", "post_date": "01/30/2022 18:37:14", "content": "

Offline kernel takes 45min ?
\nHow ?😳

", "votes": null, "replies": [] }, { "id": 1669507, "author_name": "amedprof", "author_url": "", "post_date": "01/30/2022 18:38:09", "content": "

Mine takes 35s and it TimeOut i can't understand why ?
\nKnowing that each iteration on test is under 0.14s

", "votes": null, "replies": [] } ] } ], "raw_markdown_by_id": { "1574732": "Hi,\n\nFor this competition, I created 8 more features during the preprocessing part and built 6-fold xgb models using GPU. However, the whole prediction process is encountering a \"timeout error\". Have you encountered this situation, too? Do you have any suggestions?\n\nThanks!", "1577349": "Endless timeout error. I am nearly giving up.", "1578470": "I just use 2-fold models to make predictions and they fail in the time requirement again.", "1578486": "Most likely, when writing the API, not the most optimal algorithms and data structures were used (pandas instead of numpy ... O(n**2) instead O(n) ... and so on). I wonder how long does a 'dummy' prediction take? (one number) I haven't had time to try it yet, maybe someone checked it?", "1578487": "a 'dummy' prediction takes about 3 hours to display the score result.", "1578495": "This should not be) I bet 99.9% that this is a suboptimal code. I suggest the organizers share the \"slow\" pieces of code so that we can optimize them together.", "1578752": "I have a same problem. I don't know how to resolve it. 😨", "1579337": "I ran some additional tests just to confirm there wasn't something wrong with my initial results and I stand by the existing note on the data page: \n>The API will also consume less than 30 minutes of runtime for loading and serving the data.\n\n@yus002 can you share your dummy notebook with me? I see submissions from you that took much less than 3 hours to complete. If the end results are taking 3 hours to become visible to you then I'll need to look into what's happening outside of the actual notebook rerun and scoring process.", "1579354": "Hi, yes. The notebook I use is https://www.kaggle.com/yus002/naive-prediction-by-mean-values/notebook?scriptVersionId=78725687. Thank you!!", "1579355": "Hi, I am still testing. This time I reduce the value of hyperparamters that control overfitting. In the end, I got 2-fold models which only hold 1.8 mb. The time I used to ran my submission was 21 seconds, and the submission still got a timeout error.", "1580433": "Would you mind resubmitting that notebook? My own simple test submission just now finished in about eight minutes.", "1580574": "Hi, yes. It runs for 6 minutes and finishes with a score. I realize that I actually remember the time cost of dummy approach wrong. 😅", "1580577": "Phew! Thanks for double checking.", "1617203": "Same Problem !! Offline kernel takes about 45min. Online submission timeout! 😭", "1617469": "Hi, data preprocessing and model prediction in each loop take time. Make sure you optimize the code efficiency in these two parts.", "1661013": "Hi, @sohier @yus002 I have a notebook that runs under 40secs in Kaggle notebooks. I have timed my average prediction also to be around 0.07 secs.\nBut when I submit the notebook times out after 9 hours. Not sure what am I doing wrong here. Any suggestions? :D\nThe link to notebook: https://www.kaggle.com/ankit001/mlp-prediction?scriptVersionId=85919823", "1661105": "```\ntime_taken = []\nfor i, (df_test, df_pred) in enumerate(iter_test):\n for j, row in df_test.iterrows():\n \n try:\n #init_time = time.time()\n model = models[row['Asset_ID']]\n x_test = get_features(row)\n #n_steps = x_test.shape[0]\n #x_test = x_test.to_numpy()\n #x_test = x_test.reshape((1, n_steps))\n y_pred = model.predict([x_test])[0]\n\n df_pred.loc[df_pred['row_id'] == row['row_id'], 'Target'] = y_pred\n #time_taken.append(time.time() - init_time)\n \n except:\n #print(f'{i}-th iteration of the test dataset, {j}-th row - there was the exception, then set Target = 0')\n df_pred.loc[df_pred['row_id'] == row['row_id'], 'Target'] = 0\n \n# # Print just one sample row to get a feeling of what it looks like \n# if i == 0 and j == 0:\n# print('Example of the x_test data')\n# display(x_test)\n\n# # Display the first prediction dataframe\n# if i == 0:\n# print('Example of the prediction for test data')\n# display(df_pred)\n #df_pred['Target'] = df_pred['Target'].fillna(0)\n\n # Send submissions\n env.predict(df_pred)\n```\nHi, the comment out codes should be considered to remove.", "1669504": "Offline kernel takes 45min ?\nHow ?😳", "1669507": "Mine takes 35s and it TimeOut i can't understand why ?\nKnowing that each iteration on test is under 0.14s" }, "source": "meta" }