My program runs 0.5s for each iteration. I use a groupby() function to apply ml models in predicting different asset's return. Is there any way I can try to improve the speed?
\nUsing joblib seems didn't work for the reason that the dataframe was not too large and it took time to initialize.
My program runs 0.5s for each iteration. I use a groupby() function to apply ml models in predicting different asset's return. Is there any way I can try to improve the speed?
\nUsing joblib seems didn't work for the reason that the dataframe was not too large and it took time to initialize.
If you're using Pandas, try to avoid it. Use numpy arrays instead. They will make use of vectorization, which is way faster.
", "rawMarkdown": "If you're using Pandas, try to avoid it. Use numpy arrays instead. They will make use of vectorization, which is way faster.", "votes": null }, { "id": "1666286", "postDate": "01/27/2022 16:17:56", "content": "agree solved most of my speed issues with Numpy
", "rawMarkdown": "agree solved most of my speed issues with Numpy", "votes": null }, { "id": "1667851", "postDate": "01/29/2022 01:54:51", "content": "Thanks for your advice. I changed my code, and for each iteration, it took 0.20s to finish, but I still got a submission error: Exceeded Allowed Compute after scoring 8 hours.
\nI don't know why. Can you help me? Below is the submission iteration:
iter_test = env.iter_test()\nfor (test_df, sample_prediction_df) in iter_test:\n test_df[\"Target\"] = 0\n test_df.sort_values(by=\"timestamp\", inplace=True)\n res_df = test_df.groupby(\"Asset_ID\").apply(Tree_predicting)\n res_df.reset_index(drop=True, inplace=True)\n sample_prediction_df = pd.merge(sample_prediction_df[[\"row_id\"]], res_df, how=\"left\", on=\"row_id\")\n sample_prediction_df.to_csv(\"submission.csv\", index=False)\n env.predict(sample_prediction_df)
As Antonio said, you are using panda operations \"groupby.apply\" and \"merge\". Try to find a way to avoid using them.
", "rawMarkdown": "As Antonio said, you are using panda operations \"groupby.apply\" and \"merge\". Try to find a way to avoid using them.", "votes": null }, { "id": "1669165", "postDate": "01/30/2022 12:21:46", "content": "Thank you so much, after using numpy and remove the merge inside the iteration, I made it within 0.15s and sucessfully submitted my work. Thanks for your help!
", "rawMarkdown": "Thank you so much, after using numpy and remove the merge inside the iteration, I made it within 0.15s and sucessfully submitted my work. Thanks for your help!", "votes": null } ], "comments": [ { "id": 1665486, "author_name": "antoniointini1980", "author_url": "", "post_date": "01/26/2022 22:09:27", "content": "If you're using Pandas, try to avoid it. Use numpy arrays instead. They will make use of vectorization, which is way faster.
", "votes": null, "replies": [ { "id": 1666286, "author_name": "jonnydossantos", "author_url": "", "post_date": "01/27/2022 16:17:56", "content": "agree solved most of my speed issues with Numpy
", "votes": null, "replies": [] }, { "id": 1667851, "author_name": "zhengyuhao", "author_url": "", "post_date": "01/29/2022 01:54:51", "content": "Thanks for your advice. I changed my code, and for each iteration, it took 0.20s to finish, but I still got a submission error: Exceeded Allowed Compute after scoring 8 hours.
\nI don't know why. Can you help me? Below is the submission iteration:
iter_test = env.iter_test()\nfor (test_df, sample_prediction_df) in iter_test:\n test_df[\"Target\"] = 0\n test_df.sort_values(by=\"timestamp\", inplace=True)\n res_df = test_df.groupby(\"Asset_ID\").apply(Tree_predicting)\n res_df.reset_index(drop=True, inplace=True)\n sample_prediction_df = pd.merge(sample_prediction_df[[\"row_id\"]], res_df, how=\"left\", on=\"row_id\")\n sample_prediction_df.to_csv(\"submission.csv\", index=False)\n env.predict(sample_prediction_df)
As Antonio said, you are using panda operations \"groupby.apply\" and \"merge\". Try to find a way to avoid using them.
", "votes": null, "replies": [] }, { "id": 1669165, "author_name": "zhengyuhao", "author_url": "", "post_date": "01/30/2022 12:21:46", "content": "Thank you so much, after using numpy and remove the merge inside the iteration, I made it within 0.15s and sucessfully submitted my work. Thanks for your help!
", "votes": null, "replies": [] } ] } ], "raw_markdown_by_id": { "1665449": "My program runs 0.5s for each iteration. I use a groupby() function to apply ml models in predicting different asset's return. Is there any way I can try to improve the speed?\nUsing joblib seems didn't work for the reason that the dataframe was not too large and it took time to initialize.", "1665486": "If you're using Pandas, try to avoid it. Use numpy arrays instead. They will make use of vectorization, which is way faster.", "1666286": "agree solved most of my speed issues with Numpy", "1667851": "Thanks for your advice. I changed my code, and for each iteration, it took 0.20s to finish, but I still got a submission error: Exceeded Allowed Compute after scoring 8 hours.\nI don't know why. Can you help me? Below is the submission iteration:\n\n`iter_test = env.iter_test()\nfor (test_df, sample_prediction_df) in iter_test:\n test_df[\"Target\"] = 0\n test_df.sort_values(by=\"timestamp\", inplace=True)\n res_df = test_df.groupby(\"Asset_ID\").apply(Tree_predicting)\n res_df.reset_index(drop=True, inplace=True)\n sample_prediction_df = pd.merge(sample_prediction_df[[\"row_id\"]], res_df, how=\"left\", on=\"row_id\")\n sample_prediction_df.to_csv(\"submission.csv\", index=False)\n env.predict(sample_prediction_df)`", "1668951": "As Antonio said, you are using panda operations \"groupby.apply\" and \"merge\". Try to find a way to avoid using them.", "1669165": "Thank you so much, after using numpy and remove the merge inside the iteration, I made it within 0.15s and sucessfully submitted my work. Thanks for your help!" }, "source": "meta" }