Due to the length of the code or extra features added during feature engineering , my kernel is restarting again and again(all the 14GB RAM space gets used up).I've seen people have more features than mine and their kernels seems to run just fine.
\n\nAny suggestions to fix this would help!
\n\nThanks
", "messages": [ { "id": 783090, "postDate": "2020-03-23T01:41:12.247Z", "content": "I said to use pyspark because you can store your features in their database instead of locally, you can even run your ml calculations in their cloud system. So I don't understand why you down voted me.
", "rawMarkdown": "I said to use pyspark because you can store your features in their database instead of locally, you can even run your ml calculations in their cloud system. So I don't understand why you down voted me.", "votes": 5 }, { "id": 466314, "postDate": "2019-02-05T05:19:28.843Z", "content": "Due to the length of the code or extra features added during feature engineering , my kernel is restarting again and again(all the 14GB RAM space gets used up).I've seen people have more features than mine and their kernels seems to run just fine.
\n\nAny suggestions to fix this would help!
\n\nThanks
", "rawMarkdown": "Due to the length of the code or extra features added during feature engineering , my kernel is restarting again and again(all the 14GB RAM space gets used up).I've seen people have more features than mine and their kernels seems to run just fine.\n\nAny suggestions to fix this would help!\n\nThanks", "votes": 10 }, { "id": 468661, "postDate": "2019-02-09T12:03:05.533Z", "content": "Hi, if you are using pandas.read_csv function, set the parameters iterator=True and chunksize to a reasonable value to iterate lazily instead of reading the whole file into memory.
\n\nExample:
\n\ntrain = pd.read_csv('train.csv', iterator=True, chunksize=100_000, dtype={'x': np.int16, 'y': np.float64})\nX_train = pd.DataFrame()\ny_train = pd.Series()\n\nfor df in train:\n X_train = X_train.append(your_feature_engineering_function(df))\n y_train = y_train.append(pd.Series(df['y'].values[-1]))\n # this does feature engineering per chunk\nHope this is helpful.
", "rawMarkdown": "Hi, if you are using pandas.read_csv function, set the parameters iterator=True and chunksize to a reasonable value to iterate lazily instead of reading the whole file into memory.\n\nExample:\n\n train = pd.read_csv('train.csv', iterator=True, chunksize=100_000, dtype={'x': np.int16, 'y': np.float64})\n X_train = pd.DataFrame()\n y_train = pd.Series()\n\n for df in train:\n X_train = X_train.append(your_feature_engineering_function(df))\n y_train = y_train.append(pd.Series(df['y'].values[-1]))\n # this does feature engineering per chunk\n\nHope this is helpful.", "votes": 7 }, { "id": 468017, "postDate": "2019-02-08T06:24:30.373Z", "content": "I found this piece of code in one of the kernels (I don't remember, sorry) but it really helps get rid of memory errors by choosing the right data type I think.\nYou can just call this, like new_df= reduce_mem_usage(new_df) etc.
def reduce_mem_usage(df, verbose=True):\n numerics = ['int16', 'int32', 'int64', 'float16', 'float32', 'float64']\n start_mem = df.memory_usage().sum() / 1024**2 \n for col in df.columns:\n col_type = df[col].dtypes\n if col_type in numerics:\n c_min = df[col].min()\n c_max = df[col].max()\n if str(col_type)[:3] == 'int':\n if c_min > np.iinfo(np.int8).min and c_max < np.iinfo(np.int8).max:\n df[col] = df[col].astype(np.int8)\n elif c_min > np.iinfo(np.int16).min and c_max < np.iinfo(np.int16).max:\n df[col] = df[col].astype(np.int16)\n elif c_min > np.iinfo(np.int32).min and c_max < np.iinfo(np.int32).max:\n df[col] = df[col].astype(np.int32)\n elif c_min > np.iinfo(np.int64).min and c_max < np.iinfo(np.int64).max:\n df[col] = df[col].astype(np.int64) \n else:\n if c_min > np.finfo(np.float16).min and c_max < np.finfo(np.float16).max:\n df[col] = df[col].astype(np.float16)\n elif c_min > np.finfo(np.float32).min and c_max < np.finfo(np.float32).max:\n df[col] = df[col].astype(np.float32)\n else:\n df[col] = df[col].astype(np.float64) \n end_mem = df.memory_usage().sum() / 1024**2\n if verbose: print('Mem. usage decreased to {:5.2f} Mb ({:.1f}% reduction)'.format(end_mem, 100 * (start_mem - end_mem) / start_mem))\n return df\nHey, Thanks for your help, I'm already using this function.
", "rawMarkdown": "Hey, Thanks for your help, I'm already using this function." }, { "id": 468119, "postDate": "2019-02-08T10:29:58.233Z", "content": "Do you use gc?
\n\nimport gc
\n\ndel train\ngc.collect()
", "rawMarkdown": "Do you use gc?\n\nimport gc\n\ndel train\ngc.collect()" }, { "id": 468154, "postDate": "2019-02-08T11:35:32.963Z", "content": "Here it is the kernel using that piece of code (at least here I found it): https://www.kaggle.com/mfjwr1/simple-lightgbm-without-blending
", "rawMarkdown": "Here it is the kernel using that piece of code (at least here I found it): https://www.kaggle.com/mfjwr1/simple-lightgbm-without-blending" }, { "id": 468270, "postDate": "2019-02-08T15:50:59.120Z", "content": "yup! I do use gc
", "rawMarkdown": "yup! I do use gc" }, { "id": 843685, "postDate": "2020-05-12T07:48:08.997Z", "content": "hi @pranavmahajan725 & @gpreda, even after using that the problem persists. Even after the garbage collection... Anything else you'd like to add up on that? Maybe calling the function several times in the script??
", "rawMarkdown": "hi @pranavmahajan725 & @gpreda, even after using that the problem persists. Even after the garbage collection... Anything else you'd like to add up on that? Maybe calling the function several times in the script??\n" } ] }, { "id": 468548, "postDate": "2019-02-09T05:57:33.933Z", "content": "Re-installation helped me .\nPlease try if still facing the same issue.
", "rawMarkdown": "Re-installation helped me .\nPlease try if still facing the same issue.", "votes": 1 }, { "id": 467583, "postDate": "2019-02-07T11:28:57.223Z", "content": "@IbtesamAhmed
\n\nnumpy arrays rather than a list everywhere in your code. Hopefully this will help. Happy Kaggling. :)
", "rawMarkdown": "@IbtesamAhmed \n\n 1. If possible use `numpy` arrays rather than a list everywhere in your code. \n 2. Use datable from h2o.ai as dataframe, it has better way of storing larger datasets in memory as compared to pandas. This has been already described here https://www.kaggle.com/c/microsoft-malware-prediction/discussion/75478 FYI you have to connect your kernel to network by default it is not connected, you should be able to install it using pre made wheel see full details here https://www.kaggle.com/tunguz/eda-with-python-datatable\n\nHopefully this will help. Happy Kaggling. :)", "votes": 1, "replies": [ { "id": 467608, "postDate": "2019-02-07T12:16:13.370Z", "content": "This is helpful. Thanks!
", "rawMarkdown": "This is helpful. Thanks!" } ] }, { "id": 467579, "postDate": "2019-02-07T11:09:09.467Z", "content": "use Dask it really helps in dealing with large data
", "rawMarkdown": "use Dask it really helps in dealing with large data", "votes": 1, "replies": [ { "id": 467611, "postDate": "2019-02-07T12:17:40.733Z", "content": "Thanks
", "rawMarkdown": "Thanks" } ] }, { "id": 467073, "postDate": "2019-02-06T12:17:43.780Z", "content": "Hi!\nI had problem with memory too. First I had 4 GB (WIN64) which was enough but very slow. If I used jupiter notebook and Spyder in the same time, the memory error came. After I just use the Spyder IDE the memory error disappeared , but still very slow. After I bought another 4GB RAM and my memory problem solved. \nGeri
", "rawMarkdown": "Hi!\nI had problem with memory too. First I had 4 GB (WIN64) which was enough but very slow. If I used jupiter notebook and Spyder in the same time, the memory error came. After I just use the Spyder IDE the memory error disappeared , but still very slow. After I bought another 4GB RAM and my memory problem solved. \nGeri", "votes": 1, "replies": [ { "id": 467101, "postDate": "2019-02-06T13:09:51.993Z", "content": "Hey,\nI'm using the kaggle kernel environment. I'm not running it locally
", "rawMarkdown": "Hey,\nI'm using the kaggle kernel environment. I'm not running it locally" } ] }, { "id": 466775, "postDate": "2019-02-05T22:56:56.913Z", "content": "I usually split the transaction and new merchant data frame based on train transactions and test transactions. Then, I run the feature engineering on each split.
", "rawMarkdown": "I usually split the transaction and new merchant data frame based on train transactions and test transactions. Then, I run the feature engineering on each split.", "votes": 1 }, { "id": 469187, "postDate": "2019-02-10T17:05:26.393Z", "content": "Use pyspark
", "rawMarkdown": "Use pyspark", "votes": -2 }, { "id": 1385605, "postDate": "2021-07-12T20:37:46.003Z", "content": "Use Modin + Dask, will really going to help with the error.
\nAnd, the best part is, It's very much similar to what we do as -
\nimport pandas as pd
Instead we need to do-
\nimport modin.pandas as pd
I tried almost all the methods mentioned below but its not working .
\n\nIs it specific for this competition only
", "rawMarkdown": "I tried almost all the methods mentioned below but its not working .\n\nIs it specific for this competition only" }, { "id": 466332, "postDate": "2019-02-05T06:24:53.013Z", "content": "If using windows platform, the problem could be solved by virtual memory, but one disadvantage is quite slow speed.
", "rawMarkdown": "If using windows platform, the problem could be solved by virtual memory, but one disadvantage is quite slow speed." }, { "id": 466479, "postDate": "2019-02-05T13:13:33.880Z", "rawMarkdown": "", "isDeleted": true } ], "comments": [ { "id": 783090, "author_name": "Pythonian", "author_url": "", "post_date": "2020-03-23T01:41:12.247000", "content": "I said to use pyspark because you can store your features in their database instead of locally, you can even run your ml calculations in their cloud system. So I don't understand why you down voted me.
", "votes": 5, "replies": [] }, { "id": 468661, "author_name": "Kevin Jilg", "author_url": "", "post_date": "2019-02-09T12:03:05.533000", "content": "Hi, if you are using pandas.read_csv function, set the parameters iterator=True and chunksize to a reasonable value to iterate lazily instead of reading the whole file into memory.
\n\nExample:
\n\ntrain = pd.read_csv('train.csv', iterator=True, chunksize=100_000, dtype={'x': np.int16, 'y': np.float64})\nX_train = pd.DataFrame()\ny_train = pd.Series()\n\nfor df in train:\n X_train = X_train.append(your_feature_engineering_function(df))\n y_train = y_train.append(pd.Series(df['y'].values[-1]))\n # this does feature engineering per chunk\nHope this is helpful.
", "votes": 7, "replies": [] }, { "id": 468017, "author_name": "Pranav Mahajan", "author_url": "", "post_date": "2019-02-08T06:24:30.373000", "content": "I found this piece of code in one of the kernels (I don't remember, sorry) but it really helps get rid of memory errors by choosing the right data type I think.\nYou can just call this, like new_df= reduce_mem_usage(new_df) etc.
def reduce_mem_usage(df, verbose=True):\n numerics = ['int16', 'int32', 'int64', 'float16', 'float32', 'float64']\n start_mem = df.memory_usage().sum() / 1024**2 \n for col in df.columns:\n col_type = df[col].dtypes\n if col_type in numerics:\n c_min = df[col].min()\n c_max = df[col].max()\n if str(col_type)[:3] == 'int':\n if c_min > np.iinfo(np.int8).min and c_max < np.iinfo(np.int8).max:\n df[col] = df[col].astype(np.int8)\n elif c_min > np.iinfo(np.int16).min and c_max < np.iinfo(np.int16).max:\n df[col] = df[col].astype(np.int16)\n elif c_min > np.iinfo(np.int32).min and c_max < np.iinfo(np.int32).max:\n df[col] = df[col].astype(np.int32)\n elif c_min > np.iinfo(np.int64).min and c_max < np.iinfo(np.int64).max:\n df[col] = df[col].astype(np.int64) \n else:\n if c_min > np.finfo(np.float16).min and c_max < np.finfo(np.float16).max:\n df[col] = df[col].astype(np.float16)\n elif c_min > np.finfo(np.float32).min and c_max < np.finfo(np.float32).max:\n df[col] = df[col].astype(np.float32)\n else:\n df[col] = df[col].astype(np.float64) \n end_mem = df.memory_usage().sum() / 1024**2\n if verbose: print('Mem. usage decreased to {:5.2f} Mb ({:.1f}% reduction)'.format(end_mem, 100 * (start_mem - end_mem) / start_mem))\n return df\nHey, Thanks for your help, I'm already using this function.
", "votes": 0, "replies": [] }, { "id": 468119, "author_name": "Kemal Salihçavuşoğlu", "author_url": "", "post_date": "2019-02-08T10:29:58.233000", "content": "Do you use gc?
\n\nimport gc
\n\ndel train\ngc.collect()
", "votes": 0, "replies": [] }, { "id": 468154, "author_name": "Gabriel Preda", "author_url": "", "post_date": "2019-02-08T11:35:32.963000", "content": "Here it is the kernel using that piece of code (at least here I found it): https://www.kaggle.com/mfjwr1/simple-lightgbm-without-blending
", "votes": 0, "replies": [] }, { "id": 468270, "author_name": "Ibtesam Ahmed", "author_url": "", "post_date": "2019-02-08T15:50:59.120000", "content": "yup! I do use gc
", "votes": 0, "replies": [] }, { "id": 843685, "author_name": "Divyansh Agrawal", "author_url": "", "post_date": "2020-05-12T07:48:08.997000", "content": "hi @pranavmahajan725 & @gpreda, even after using that the problem persists. Even after the garbage collection... Anything else you'd like to add up on that? Maybe calling the function several times in the script??
", "votes": 0, "replies": [] } ] }, { "id": 468548, "author_name": "Prachi", "author_url": "", "post_date": "2019-02-09T05:57:33.933000", "content": "Re-installation helped me .\nPlease try if still facing the same issue.
", "votes": 1, "replies": [] }, { "id": 467583, "author_name": "cyberia", "author_url": "", "post_date": "2019-02-07T11:28:57.223000", "content": "@IbtesamAhmed
\n\nnumpy arrays rather than a list everywhere in your code. Hopefully this will help. Happy Kaggling. :)
", "votes": 1, "replies": [ { "id": 467608, "author_name": "Ibtesam Ahmed", "author_url": "", "post_date": "2019-02-07T12:16:13.370000", "content": "This is helpful. Thanks!
", "votes": 0, "replies": [] } ] }, { "id": 467579, "author_name": "ayush", "author_url": "", "post_date": "2019-02-07T11:09:09.467000", "content": "use Dask it really helps in dealing with large data
", "votes": 1, "replies": [ { "id": 467611, "author_name": "Ibtesam Ahmed", "author_url": "", "post_date": "2019-02-07T12:17:40.733000", "content": "Thanks
", "votes": 0, "replies": [] } ] }, { "id": 467073, "author_name": "Gergely Iski", "author_url": "", "post_date": "2019-02-06T12:17:43.780000", "content": "Hi!\nI had problem with memory too. First I had 4 GB (WIN64) which was enough but very slow. If I used jupiter notebook and Spyder in the same time, the memory error came. After I just use the Spyder IDE the memory error disappeared , but still very slow. After I bought another 4GB RAM and my memory problem solved. \nGeri
", "votes": 1, "replies": [ { "id": 467101, "author_name": "Ibtesam Ahmed", "author_url": "", "post_date": "2019-02-06T13:09:51.993000", "content": "Hey,\nI'm using the kaggle kernel environment. I'm not running it locally
", "votes": 0, "replies": [] } ] }, { "id": 466775, "author_name": "Kyle Peters", "author_url": "", "post_date": "2019-02-05T22:56:56.913000", "content": "I usually split the transaction and new merchant data frame based on train transactions and test transactions. Then, I run the feature engineering on each split.
", "votes": 1, "replies": [] }, { "id": 469187, "author_name": "Pythonian", "author_url": "", "post_date": "2019-02-10T17:05:26.393000", "content": "Use pyspark
", "votes": -2, "replies": [] }, { "id": 1385605, "author_name": "Navneet", "author_url": "", "post_date": "2021-07-12T20:37:46.003000", "content": "Use Modin + Dask, will really going to help with the error.
\nAnd, the best part is, It's very much similar to what we do as -
\nimport pandas as pd
Instead we need to do-
\nimport modin.pandas as pd
I tried almost all the methods mentioned below but its not working .
\n\nIs it specific for this competition only
", "votes": 0, "replies": [] }, { "id": 466332, "author_name": "Eric Wu", "author_url": "", "post_date": "2019-02-05T06:24:53.013000", "content": "If using windows platform, the problem could be solved by virtual memory, but one disadvantage is quite slow speed.
", "votes": 0, "replies": [] }, { "id": 466479, "author_name": "", "author_url": "", "post_date": "2019-02-05T13:13:33.880000", "content": "", "votes": 0, "replies": [] } ], "raw_markdown_by_id": { "783090": "I said to use pyspark because you can store your features in their database instead of locally, you can even run your ml calculations in their cloud system. So I don't understand why you down voted me.", "466314": "Due to the length of the code or extra features added during feature engineering , my kernel is restarting again and again(all the 14GB RAM space gets used up).I've seen people have more features than mine and their kernels seems to run just fine.\n\nAny suggestions to fix this would help!\n\nThanks", "468661": "Hi, if you are using pandas.read_csv function, set the parameters iterator=True and chunksize to a reasonable value to iterate lazily instead of reading the whole file into memory.\n\nExample:\n\n train = pd.read_csv('train.csv', iterator=True, chunksize=100_000, dtype={'x': np.int16, 'y': np.float64})\n X_train = pd.DataFrame()\n y_train = pd.Series()\n\n for df in train:\n X_train = X_train.append(your_feature_engineering_function(df))\n y_train = y_train.append(pd.Series(df['y'].values[-1]))\n # this does feature engineering per chunk\n\nHope this is helpful.", "468017": "I found this piece of code in one of the kernels (I don't remember, sorry) but it really helps get rid of memory errors by choosing the right data type I think.\nYou can just call this, like `new_df= reduce_mem_usage(new_df)` etc.\n\n def reduce_mem_usage(df, verbose=True):\n numerics = ['int16', 'int32', 'int64', 'float16', 'float32', 'float64']\n start_mem = df.memory_usage().sum() / 1024**2 \n for col in df.columns:\n col_type = df[col].dtypes\n if col_type in numerics:\n c_min = df[col].min()\n c_max = df[col].max()\n if str(col_type)[:3] == 'int':\n if c_min > np.iinfo(np.int8).min and c_max < np.iinfo(np.int8).max:\n df[col] = df[col].astype(np.int8)\n elif c_min > np.iinfo(np.int16).min and c_max < np.iinfo(np.int16).max:\n df[col] = df[col].astype(np.int16)\n elif c_min > np.iinfo(np.int32).min and c_max < np.iinfo(np.int32).max:\n df[col] = df[col].astype(np.int32)\n elif c_min > np.iinfo(np.int64).min and c_max < np.iinfo(np.int64).max:\n df[col] = df[col].astype(np.int64) \n else:\n if c_min > np.finfo(np.float16).min and c_max < np.finfo(np.float16).max:\n df[col] = df[col].astype(np.float16)\n elif c_min > np.finfo(np.float32).min and c_max < np.finfo(np.float32).max:\n df[col] = df[col].astype(np.float32)\n else:\n df[col] = df[col].astype(np.float64) \n end_mem = df.memory_usage().sum() / 1024**2\n if verbose: print('Mem. usage decreased to {:5.2f} Mb ({:.1f}% reduction)'.format(end_mem, 100 * (start_mem - end_mem) / start_mem))\n return df", "468548": "Re-installation helped me .\nPlease try if still facing the same issue.", "467583": "@IbtesamAhmed \n\n 1. If possible use `numpy` arrays rather than a list everywhere in your code. \n 2. Use datable from h2o.ai as dataframe, it has better way of storing larger datasets in memory as compared to pandas. This has been already described here https://www.kaggle.com/c/microsoft-malware-prediction/discussion/75478 FYI you have to connect your kernel to network by default it is not connected, you should be able to install it using pre made wheel see full details here https://www.kaggle.com/tunguz/eda-with-python-datatable\n\nHopefully this will help. Happy Kaggling. :)", "467579": "use Dask it really helps in dealing with large data", "467073": "Hi!\nI had problem with memory too. First I had 4 GB (WIN64) which was enough but very slow. If I used jupiter notebook and Spyder in the same time, the memory error came. After I just use the Spyder IDE the memory error disappeared , but still very slow. After I bought another 4GB RAM and my memory problem solved. \nGeri", "466775": "I usually split the transaction and new merchant data frame based on train transactions and test transactions. Then, I run the feature engineering on each split.", "469187": "Use pyspark", "1385605": "Use Modin + Dask, will really going to help with the error.\nAnd, the best part is, It's very much similar to what we do as -\nimport pandas as pd\n\nInstead we need to do-\nimport modin.pandas as pd ", "962303": "I tried almost all the methods mentioned below but its not working .\n\nIs it specific for this competition only", "466332": "If using windows platform, the problem could be solved by virtual memory, but one disadvantage is quite slow speed.", "466479": "" } }