I wanted to use a notebook https://www.kaggle.com/tomforbes/gresearch-submitting-lagged-features-via-api/notebook for the base of my submission, since it prepares the data very well for using lagged features, but I encountered a problem.
\nWhile I can run his notebook without problems, when I add more features it doesn't work anymore.
I first added other data provided by organizers, since my features rely on them
\ndf_feat = df_feat[['timestamp','Asset_ID','Close','Count', 'Open', 'High', 'Low','Volume','Target','train_flg']].copy()
\nThen I created 17 of my own features out of it.
When I run my notebook I get a message: \"Your notebook tried to allocate more memory than is available. It has restarted.\". When submitting for competition: \"Notebook Exceeded Allowed Compute\".
\nI tried using function which I found in other notebooks - reduce_memory_usage(df):
\nBut it did not help, maybe I did something wrong.
\nI tried reducing rolling functions, like I discarded median returns rolling on 1 year of data, but still did not work. Now the furthest shift or rolling I have is 60.
I wonder if you experienced similar issue and if you found a solution to it
", "messages": [ { "id": "1666383", "postDate": "01/27/2022 17:48:12", "content": "I wanted to use a notebook https://www.kaggle.com/tomforbes/gresearch-submitting-lagged-features-via-api/notebook for the base of my submission, since it prepares the data very well for using lagged features, but I encountered a problem.
\nWhile I can run his notebook without problems, when I add more features it doesn't work anymore.
I first added other data provided by organizers, since my features rely on them
\ndf_feat = df_feat[['timestamp','Asset_ID','Close','Count', 'Open', 'High', 'Low','Volume','Target','train_flg']].copy()
\nThen I created 17 of my own features out of it.
When I run my notebook I get a message: \"Your notebook tried to allocate more memory than is available. It has restarted.\". When submitting for competition: \"Notebook Exceeded Allowed Compute\".
\nI tried using function which I found in other notebooks - reduce_memory_usage(df):
\nBut it did not help, maybe I did something wrong.
\nI tried reducing rolling functions, like I discarded median returns rolling on 1 year of data, but still did not work. Now the furthest shift or rolling I have is 60.
I wonder if you experienced similar issue and if you found a solution to it
", "rawMarkdown": "I wanted to use a notebook https://www.kaggle.com/tomforbes/gresearch-submitting-lagged-features-via-api/notebook for the base of my submission, since it prepares the data very well for using lagged features, but I encountered a problem.\nWhile I can run his notebook without problems, when I add more features it doesn't work anymore.\n\nI first added other data provided by organizers, since my features rely on them\ndf_feat = df_feat[['timestamp','Asset_ID','Close','Count', 'Open', 'High', 'Low','Volume','Target','train_flg']].copy()\nThen I created 17 of my own features out of it.\n\nWhen I run my notebook I get a message: \"Your notebook tried to allocate more memory than is available. It has restarted.\". When submitting for competition: \"Notebook Exceeded Allowed Compute\".\n\nI tried using function which I found in other notebooks - reduce_memory_usage(df):\nBut it did not help, maybe I did something wrong.\nI tried reducing rolling functions, like I discarded median returns rolling on 1 year of data, but still did not work. Now the furthest shift or rolling I have is 60.\n\nI wonder if you experienced similar issue and if you found a solution to it", "votes": null }, { "id": "1666803", "postDate": "01/28/2022 04:36:09", "content": "Hi try deleting the old dataframe…. I mean the df_train, after you are done with it… that should do the job
", "rawMarkdown": "Hi try deleting the old dataframe.... I mean the df_train, after you are done with it... that should do the job", "votes": null }, { "id": "1669772", "postDate": "01/30/2022 23:29:52", "content": "I resolved the issues by saving the dataset with .astype(np.float32)
", "rawMarkdown": "I resolved the issues by saving the dataset with .astype(np.float32)", "votes": null } ], "comments": [ { "id": 1666803, "author_name": "pranayteaches", "author_url": "", "post_date": "01/28/2022 04:36:09", "content": "Hi try deleting the old dataframe…. I mean the df_train, after you are done with it… that should do the job
", "votes": null, "replies": [] }, { "id": 1669772, "author_name": "rafalradwanski", "author_url": "", "post_date": "01/30/2022 23:29:52", "content": "I resolved the issues by saving the dataset with .astype(np.float32)
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "1666383": "I wanted to use a notebook https://www.kaggle.com/tomforbes/gresearch-submitting-lagged-features-via-api/notebook for the base of my submission, since it prepares the data very well for using lagged features, but I encountered a problem.\nWhile I can run his notebook without problems, when I add more features it doesn't work anymore.\n\nI first added other data provided by organizers, since my features rely on them\ndf_feat = df_feat[['timestamp','Asset_ID','Close','Count', 'Open', 'High', 'Low','Volume','Target','train_flg']].copy()\nThen I created 17 of my own features out of it.\n\nWhen I run my notebook I get a message: \"Your notebook tried to allocate more memory than is available. It has restarted.\". When submitting for competition: \"Notebook Exceeded Allowed Compute\".\n\nI tried using function which I found in other notebooks - reduce_memory_usage(df):\nBut it did not help, maybe I did something wrong.\nI tried reducing rolling functions, like I discarded median returns rolling on 1 year of data, but still did not work. Now the furthest shift or rolling I have is 60.\n\nI wonder if you experienced similar issue and if you found a solution to it", "1666803": "Hi try deleting the old dataframe.... I mean the df_train, after you are done with it... that should do the job", "1669772": "I resolved the issues by saving the dataset with .astype(np.float32)" }, "source": "meta" }