{ "id": 112938, "title": "Reducing Memory Size for Great Energy Predictor III", "url": "/competitions/ashrae-energy-prediction/discussion/112938", "author_name": "", "post_date": "2019-10-16T08:18:40.381981200Z", "votes": 6, "comment_count": 2, "views": 0, "content": "

\"\"\n\"\"

\n\n

All codes and details exist in this kernel

\n\n

Reducing Memory Size for Great Energy Predictor III Competition\n*Objective of the Kernel: Save Time & Memory*\nIf you would like to create a kernel for this Competition. This is a good idea to add this kernel as a data set to your own kernel. due to you can save your time and memory.

\n\n

*MEMORY USAGE BEFORE AND AFTER COMPLETION FOR TRAIN:\nMemory usage before running this script : 2.6+ GB\nMemory usage after running this script : 1.1+ GB\nThis is ~ 60 % of the initial size

\n\n

MEMORY USAGE BEFORE AND AFTER COMPLETION FOR TEST:\nMemory usage before running this script : 5.3+ GB\nMemory usage after running this script: 2.3+ GB\nThis is ~ 60 % of the initial size

\n\n

All codes and details exist in this kernel

\n\n

What do we have in input\n\"\"

\n\n

Before Reducing Memory\nWhen I have just read the data set and join them! I saw that the status of my RAM is more than 10GB!\n\"\"

\n\n

Then we shoud just delete some dt!\ndel building, weather_train, weather_test

\n\n

Now let's check the size of our train & test\n\"\"\n\"\"

\n\n

Reducing Memory Size\nIt is necessary that after using this code, carefully check the output results for each column.\n`#Based on this great kernel https://www.kaggle.com/arjanso/reducing-dataframe-memory-size-by-65\ndef reduce_mem_usage(df):\n start_mem_usg = df.memory_usage().sum() / 1024**2 \n print(\"Memory usage of properties dataframe is :\",start_mem_usg,\" MB\")\n NAlist = [] # Keeps track of columns that have missing values filled in. \n for col in df.columns:\n if df[col].dtype != object: # Exclude strings
\n # Print current column type\n print(\"****************\")\n print(\"Column: \",col)\n print(\"dtype before: \",df[col].dtype)
\n # make variables for Int, max and min\n IsInt = False\n mx = df[col].max()\n mn = df[col].min()\n print(\"min for this col: \",mn)\n print(\"max for this col: \",mx)\n # Integer does not support NA, therefore, NA needs to be filled\n if not np.isfinite(df[col]).all(): \n NAlist.append(col)\n df[col].fillna(mn-1,inplace=True)

\n\n
        # test if column can be converted to an integer\n        asint = df[col].fillna(0).astype(np.int64)\n        result = (df[col] - asint)\n        result = result.sum()\n        if result > -0.01 and result < 0.01:\n            IsInt = True            \n        # Make Integer/unsigned Integer datatypes\n        if IsInt:\n            if mn >= 0:\n                if mx < 255:\n                    df[col] = df[col].astype(np.uint8)\n                elif mx < 65535:\n                    df[col] = df[col].astype(np.uint16)\n                elif mx < 4294967295:\n                    df[col] = df[col].astype(np.uint32)\n                else:\n                    df[col] = df[col].astype(np.uint64)\n            else:\n                if mn > np.iinfo(np.int8).min and mx < np.iinfo(np.int8).max:\n                    df[col] = df[col].astype(np.int8)\n                elif mn > np.iinfo(np.int16).min and mx < np.iinfo(np.int16).max:\n                    df[col] = df[col].astype(np.int16)\n                elif mn > np.iinfo(np.int32).min and mx < np.iinfo(np.int32).max:\n                    df[col] = df[col].astype(np.int32)\n                elif mn > np.iinfo(np.int64).min and mx < np.iinfo(np.int64).max:\n                    df[col] = df[col].astype(np.int64)    \n        # Make float datatypes 32 bit\n        else:\n            df[col] = df[col].astype(np.float32)\n\n        # Print new column type\n        print(\"dtype after: \",df[col].dtype)\n        print(\"******************************\")\n# Print final result\nprint(\"___MEMORY USAGE AFTER COMPLETION:___\")\nmem_usg = df.memory_usage().sum() / 1024**2 \nprint(\"Memory usage is: \",mem_usg,\" MB\")\nprint(\"This is \",100*mem_usg/start_mem_usg,\"% of the initial size\")\nreturn df, NAlist`\n
\n\n

Reducing for train data set:\n\"\"\nReducing for test data set:\n\"\"

\n\n

Check again! our RAM. More than 4 GB has got free!\n\"\"

\n\n

\"\"\n\"\"

\n\n

Save your dataframe as pickle file for fast read.\n\"\"

\n\n

Do not Forget to save your work. This way you can access the downcasted df everytime you load it!

\n\n

The idea behind the this is based on the fact that most of the column only takes few values and pandas can limit those values only to the few categorical value to save the memory.

\n\n

Often in real-time, data includes the text columns, which are repetitive. Features like gender, country, and codes are always repetitive. These are the examples for categorical data.

\n\n

Categorical variables can take on only a limited, and usually fixed number of possible values. Besides the fixed length, categorical data might have an order but cannot perform numerical operation. Categorical are a Pandas data type.

\n\n

The categorical data type is useful in the following cases −

\n\n

A string variable consisting of only a few different values. Converting such a string variable to a categorical variable will save some memory.

\n\n

The lexical order of a variable is not the same as the logical order (“one”, “two”, “three”). By converting to a categorical and specifying an order on the categories, sorting and min/max will use the logical order instead of the lexical order.

\n\n

As a signal to other python libraries that this column should be treated as a categorical variable (e.g. to use suitable statistical methods or plot types).\" (https://www.tutorialspoint.com/python_pandas/python_pandas_categorical_data)

\n\n

How about other ways!\nI have used my kernel but there are also other ways such as:

\n\n
    \n
  1. Panda for big data
    2. \n
  2. optimizing-the-size-of-a-pandas-dataframe-for-low-memory-environment
    3. \n
  3. pandas-making-dataframe-smaller-faster
    4. \n
\n\n

Thanks to MJ Bahmani who created this kernel

\n\n

Pickle code refers to this kaggle kernal

\n\n

All codes and details exist in this kernel

\n\n

Good Luck 👍 💪

", "messages": [ { "id": "650255", "postDate": "10/16/2019 08:18:40", "content": "

\"\"\n\"\"

\n\n

All codes and details exist in this kernel

\n\n

Reducing Memory Size for Great Energy Predictor III Competition\n*Objective of the Kernel: Save Time & Memory*\nIf you would like to create a kernel for this Competition. This is a good idea to add this kernel as a data set to your own kernel. due to you can save your time and memory.

\n\n

*MEMORY USAGE BEFORE AND AFTER COMPLETION FOR TRAIN:\nMemory usage before running this script : 2.6+ GB\nMemory usage after running this script : 1.1+ GB\nThis is ~ 60 % of the initial size

\n\n

MEMORY USAGE BEFORE AND AFTER COMPLETION FOR TEST:\nMemory usage before running this script : 5.3+ GB\nMemory usage after running this script: 2.3+ GB\nThis is ~ 60 % of the initial size

\n\n

All codes and details exist in this kernel

\n\n

What do we have in input\n\"\"

\n\n

Before Reducing Memory\nWhen I have just read the data set and join them! I saw that the status of my RAM is more than 10GB!\n\"\"

\n\n

Then we shoud just delete some dt!\ndel building, weather_train, weather_test

\n\n

Now let's check the size of our train & test\n\"\"\n\"\"

\n\n

Reducing Memory Size\nIt is necessary that after using this code, carefully check the output results for each column.\n`#Based on this great kernel https://www.kaggle.com/arjanso/reducing-dataframe-memory-size-by-65\ndef reduce_mem_usage(df):\n start_mem_usg = df.memory_usage().sum() / 1024**2 \n print(\"Memory usage of properties dataframe is :\",start_mem_usg,\" MB\")\n NAlist = [] # Keeps track of columns that have missing values filled in. \n for col in df.columns:\n if df[col].dtype != object: # Exclude strings
\n # Print current column type\n print(\"****************\")\n print(\"Column: \",col)\n print(\"dtype before: \",df[col].dtype)
\n # make variables for Int, max and min\n IsInt = False\n mx = df[col].max()\n mn = df[col].min()\n print(\"min for this col: \",mn)\n print(\"max for this col: \",mx)\n # Integer does not support NA, therefore, NA needs to be filled\n if not np.isfinite(df[col]).all(): \n NAlist.append(col)\n df[col].fillna(mn-1,inplace=True)

\n\n
        # test if column can be converted to an integer\n        asint = df[col].fillna(0).astype(np.int64)\n        result = (df[col] - asint)\n        result = result.sum()\n        if result > -0.01 and result < 0.01:\n            IsInt = True            \n        # Make Integer/unsigned Integer datatypes\n        if IsInt:\n            if mn >= 0:\n                if mx < 255:\n                    df[col] = df[col].astype(np.uint8)\n                elif mx < 65535:\n                    df[col] = df[col].astype(np.uint16)\n                elif mx < 4294967295:\n                    df[col] = df[col].astype(np.uint32)\n                else:\n                    df[col] = df[col].astype(np.uint64)\n            else:\n                if mn > np.iinfo(np.int8).min and mx < np.iinfo(np.int8).max:\n                    df[col] = df[col].astype(np.int8)\n                elif mn > np.iinfo(np.int16).min and mx < np.iinfo(np.int16).max:\n                    df[col] = df[col].astype(np.int16)\n                elif mn > np.iinfo(np.int32).min and mx < np.iinfo(np.int32).max:\n                    df[col] = df[col].astype(np.int32)\n                elif mn > np.iinfo(np.int64).min and mx < np.iinfo(np.int64).max:\n                    df[col] = df[col].astype(np.int64)    \n        # Make float datatypes 32 bit\n        else:\n            df[col] = df[col].astype(np.float32)\n\n        # Print new column type\n        print(\"dtype after: \",df[col].dtype)\n        print(\"******************************\")\n# Print final result\nprint(\"___MEMORY USAGE AFTER COMPLETION:___\")\nmem_usg = df.memory_usage().sum() / 1024**2 \nprint(\"Memory usage is: \",mem_usg,\" MB\")\nprint(\"This is \",100*mem_usg/start_mem_usg,\"% of the initial size\")\nreturn df, NAlist`\n
\n\n

Reducing for train data set:\n\"\"\nReducing for test data set:\n\"\"

\n\n

Check again! our RAM. More than 4 GB has got free!\n\"\"

\n\n

\"\"\n\"\"

\n\n

Save your dataframe as pickle file for fast read.\n\"\"

\n\n

Do not Forget to save your work. This way you can access the downcasted df everytime you load it!

\n\n

The idea behind the this is based on the fact that most of the column only takes few values and pandas can limit those values only to the few categorical value to save the memory.

\n\n

Often in real-time, data includes the text columns, which are repetitive. Features like gender, country, and codes are always repetitive. These are the examples for categorical data.

\n\n

Categorical variables can take on only a limited, and usually fixed number of possible values. Besides the fixed length, categorical data might have an order but cannot perform numerical operation. Categorical are a Pandas data type.

\n\n

The categorical data type is useful in the following cases −

\n\n

A string variable consisting of only a few different values. Converting such a string variable to a categorical variable will save some memory.

\n\n

The lexical order of a variable is not the same as the logical order (“one”, “two”, “three”). By converting to a categorical and specifying an order on the categories, sorting and min/max will use the logical order instead of the lexical order.

\n\n

As a signal to other python libraries that this column should be treated as a categorical variable (e.g. to use suitable statistical methods or plot types).\" (https://www.tutorialspoint.com/python_pandas/python_pandas_categorical_data)

\n\n

How about other ways!\nI have used my kernel but there are also other ways such as:

\n\n
    \n
  1. Panda for big data
    2. \n
  2. optimizing-the-size-of-a-pandas-dataframe-for-low-memory-environment
    3. \n
  3. pandas-making-dataframe-smaller-faster
    4. \n
\n\n

Thanks to MJ Bahmani who created this kernel

\n\n

Pickle code refers to this kaggle kernal

\n\n

All codes and details exist in this kernel

\n\n

Good Luck 👍 💪

", "rawMarkdown": "![](https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F1673888%2Fa8ee8d9f805b78ae521a0fe43d2d0e12%2Freducing1.jpg?generation=1571212073391937&alt=media)\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F1673888%2F9581a97affbfbbfaf7cc8009c7e0b330%2Fmr.jpg?generation=1571247904542379&alt=media)\n\n**All codes and details exist in this [kernel](https://www.kaggle.com/hamditarek/reducing-memory-size-for-great-energy-predictor)**\n\n**Reducing Memory Size for Great Energy Predictor III Competition**\n**Objective of the Kernel: Save Time & Memory**\nIf you would like to create a kernel for this Competition. This is a good idea to add this kernel as a data set to your own kernel. due to you can save your time and memory.\n\n***MEMORY USAGE BEFORE AND AFTER COMPLETION FOR TRAIN:**\nMemory usage before running this script : **2.6+ GB**\nMemory usage after running this script : **1.1+ GB**\nThis is **~ 60 %** of the initial size\n\n**MEMORY USAGE BEFORE AND AFTER COMPLETION FOR TEST:**\nMemory usage before running this script : **5.3+ GB**\nMemory usage after running this script: **2.3+ GB**\nThis is **~ 60 %** of the initial size\n\n**All codes and details exist in this [kernel](https://www.kaggle.com/hamditarek/reducing-memory-size-for-great-energy-predictor)**\n\n**What do we have in input**\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F1673888%2F95085897fc1345f14449d7a92339a040%2Finput.PNG?generation=1571212401294913&alt=media)\n\n**Before Reducing Memory**\nWhen I have just read the data set and join them! I saw that the status of my **RAM** is more than **10GB!**\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F1673888%2F1da86489b84fcc777c6748030cbe0ffb%2Fram1.PNG?generation=1571212477826573&alt=media)\n\n**Then we shoud just delete some dt!**\n`del building, weather_train, weather_test`\n\n**Now let's check the size of our train & test**\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F1673888%2F73d4aced86aac02f0d3799fded0aee3c%2Fsize1.PNG?generation=1571212681353475&alt=media)\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F1673888%2Fb12488dfb0a6163f7fe7716f2284ca2b%2Fsize2.PNG?generation=1571212681009662&alt=media)\n\n**Reducing Memory Size**\nIt is necessary that after using this code, carefully check the output results for each column.\n`#Based on this great kernel https://www.kaggle.com/arjanso/reducing-dataframe-memory-size-by-65\ndef reduce_mem_usage(df):\n start_mem_usg = df.memory_usage().sum() / 1024**2 \n print(\"Memory usage of properties dataframe is :\",start_mem_usg,\" MB\")\n NAlist = [] # Keeps track of columns that have missing values filled in. \n for col in df.columns:\n if df[col].dtype != object: # Exclude strings \n # Print current column type\n print(\"******************************\")\n print(\"Column: \",col)\n print(\"dtype before: \",df[col].dtype) \n # make variables for Int, max and min\n IsInt = False\n mx = df[col].max()\n mn = df[col].min()\n print(\"min for this col: \",mn)\n print(\"max for this col: \",mx)\n # Integer does not support NA, therefore, NA needs to be filled\n if not np.isfinite(df[col]).all(): \n NAlist.append(col)\n df[col].fillna(mn-1,inplace=True) \n \n # test if column can be converted to an integer\n asint = df[col].fillna(0).astype(np.int64)\n result = (df[col] - asint)\n result = result.sum()\n if result > -0.01 and result < 0.01:\n IsInt = True \n # Make Integer/unsigned Integer datatypes\n if IsInt:\n if mn >= 0:\n if mx < 255:\n df[col] = df[col].astype(np.uint8)\n elif mx < 65535:\n df[col] = df[col].astype(np.uint16)\n elif mx < 4294967295:\n df[col] = df[col].astype(np.uint32)\n else:\n df[col] = df[col].astype(np.uint64)\n else:\n if mn > np.iinfo(np.int8).min and mx < np.iinfo(np.int8).max:\n df[col] = df[col].astype(np.int8)\n elif mn > np.iinfo(np.int16).min and mx < np.iinfo(np.int16).max:\n df[col] = df[col].astype(np.int16)\n elif mn > np.iinfo(np.int32).min and mx < np.iinfo(np.int32).max:\n df[col] = df[col].astype(np.int32)\n elif mn > np.iinfo(np.int64).min and mx < np.iinfo(np.int64).max:\n df[col] = df[col].astype(np.int64) \n # Make float datatypes 32 bit\n else:\n df[col] = df[col].astype(np.float32)\n \n # Print new column type\n print(\"dtype after: \",df[col].dtype)\n print(\"******************************\")\n # Print final result\n print(\"___MEMORY USAGE AFTER COMPLETION:___\")\n mem_usg = df.memory_usage().sum() / 1024**2 \n print(\"Memory usage is: \",mem_usg,\" MB\")\n print(\"This is \",100*mem_usg/start_mem_usg,\"% of the initial size\")\n return df, NAlist`\n\n**Reducing for train data set:**\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F1673888%2F1ed61ceabcc82ad4030d93b9e2173186%2Fred1.PNG?generation=1571212907477933&alt=media)\n**Reducing for test data set:**\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F1673888%2F09aaff8bd825daced15dd7080ce1cfa7%2Fred2.PNG?generation=1571212956357397&alt=media)\n\n**Check again! our RAM. More than 4 GB has got free!**\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F1673888%2Fe34d953b44d1f4dd176bc9df9f3d2bc1%2Fram2.PNG?generation=1571213028158670&alt=media)\n\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F1673888%2Fd26993b3eb15a6391728338cbf93a692%2Ftr.PNG?generation=1571213143231326&alt=media)\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F1673888%2F2bcd4bacf247d8bed31cc3af94d2514f%2Ftsr.PNG?generation=1571213144065839&alt=media)\n\n**Save your dataframe as pickle file for fast read.**\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F1673888%2F770e91d57073cc2f86dcac1039fd4ee3%2Fpkl.PNG?generation=1571213279654216&alt=media)\n\n**Do not Forget to save your work. This way you can access the downcasted df everytime you load it!**\n\n**The idea behind the this is based on the fact that most of the column only takes few values and pandas can limit those values only to the few categorical value to save the memory.**\n\n**Often in real-time, data includes the text columns, which are repetitive. Features like gender, country, and codes are always repetitive. These are the examples for categorical data.**\n\n**Categorical variables can take on only a limited, and usually fixed number of possible values. Besides the fixed length, categorical data might have an order but cannot perform numerical operation. Categorical are a Pandas data type.**\n\n**The categorical data type is useful in the following cases −**\n\n**A string variable consisting of only a few different values. Converting such a string variable to a categorical variable will save some memory.**\n\n**The lexical order of a variable is not the same as the logical order (“one”, “two”, “three”). By converting to a categorical and specifying an order on the categories, sorting and min/max will use the logical order instead of the lexical order.**\n\n**As a signal to other python libraries that this column should be treated as a categorical variable (e.g. to use suitable statistical methods or plot types).\" (https://www.tutorialspoint.com/python_pandas/python_pandas_categorical_data)**\n\n**How about other ways!**\nI have used my [kernel](https://www.kaggle.com/hamditarek/reducing-memory-size-for-great-energy-predictor) but there are also other ways such as:\n\n1. [Panda for big data](https://www.dataquest.io/blog/pandas-big-data/)\n2. [optimizing-the-size-of-a-pandas-dataframe-for-low-memory-environment](https://medium.com/@vincentteyssier/optimizing-the-size-of-a-pandas-dataframe-for-low-memory-environment-5f07db3d72e)\n3. [pandas-making-dataframe-smaller-faster](https://www.ritchieng.com/pandas-making-dataframe-smaller-faster/)\n\nThanks to [MJ Bahmani](https://www.kaggle.com/mjbahmani) who created this [kernel](https://www.kaggle.com/mjbahmani/reducing-memory-size-for-ieee)\n\nPickle code refers to this kaggle [kernal](https://www.kaggle.com/ljjblackpig/3-steps-to-reduce-memory-size-for-the-dataset)\n\n**All codes and details exist in this [kernel](https://www.kaggle.com/hamditarek/reducing-memory-size-for-great-energy-predictor)**\n\n**Good Luck** 👍 💪", "votes": null }, { "id": "650713", "postDate": "10/16/2019 16:06:33", "content": "

Helpful Stuff\nThanks @hamditarek

", "rawMarkdown": "Helpful Stuff\nThanks @hamditarek", "votes": null }, { "id": "650716", "postDate": "10/16/2019 16:09:52", "content": "

You are welcome @veeralakrishna, you can check the code in my kernel.

", "rawMarkdown": "You are welcome @veeralakrishna, you can check the code in my kernel.", "votes": null } ], "comments": [ { "id": 650713, "author_name": "veeralakrishna", "author_url": "", "post_date": "10/16/2019 16:06:33", "content": "

Helpful Stuff\nThanks @hamditarek

", "votes": null, "replies": [ { "id": 650716, "author_name": "hamditarek", "author_url": "", "post_date": "10/16/2019 16:09:52", "content": "

You are welcome @veeralakrishna, you can check the code in my kernel.

", "votes": null, "replies": [] } ] } ], "raw_markdown_by_id": { "650255": "![](https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F1673888%2Fa8ee8d9f805b78ae521a0fe43d2d0e12%2Freducing1.jpg?generation=1571212073391937&alt=media)\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F1673888%2F9581a97affbfbbfaf7cc8009c7e0b330%2Fmr.jpg?generation=1571247904542379&alt=media)\n\n**All codes and details exist in this [kernel](https://www.kaggle.com/hamditarek/reducing-memory-size-for-great-energy-predictor)**\n\n**Reducing Memory Size for Great Energy Predictor III Competition**\n**Objective of the Kernel: Save Time & Memory**\nIf you would like to create a kernel for this Competition. This is a good idea to add this kernel as a data set to your own kernel. due to you can save your time and memory.\n\n***MEMORY USAGE BEFORE AND AFTER COMPLETION FOR TRAIN:**\nMemory usage before running this script : **2.6+ GB**\nMemory usage after running this script : **1.1+ GB**\nThis is **~ 60 %** of the initial size\n\n**MEMORY USAGE BEFORE AND AFTER COMPLETION FOR TEST:**\nMemory usage before running this script : **5.3+ GB**\nMemory usage after running this script: **2.3+ GB**\nThis is **~ 60 %** of the initial size\n\n**All codes and details exist in this [kernel](https://www.kaggle.com/hamditarek/reducing-memory-size-for-great-energy-predictor)**\n\n**What do we have in input**\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F1673888%2F95085897fc1345f14449d7a92339a040%2Finput.PNG?generation=1571212401294913&alt=media)\n\n**Before Reducing Memory**\nWhen I have just read the data set and join them! I saw that the status of my **RAM** is more than **10GB!**\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F1673888%2F1da86489b84fcc777c6748030cbe0ffb%2Fram1.PNG?generation=1571212477826573&alt=media)\n\n**Then we shoud just delete some dt!**\n`del building, weather_train, weather_test`\n\n**Now let's check the size of our train & test**\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F1673888%2F73d4aced86aac02f0d3799fded0aee3c%2Fsize1.PNG?generation=1571212681353475&alt=media)\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F1673888%2Fb12488dfb0a6163f7fe7716f2284ca2b%2Fsize2.PNG?generation=1571212681009662&alt=media)\n\n**Reducing Memory Size**\nIt is necessary that after using this code, carefully check the output results for each column.\n`#Based on this great kernel https://www.kaggle.com/arjanso/reducing-dataframe-memory-size-by-65\ndef reduce_mem_usage(df):\n start_mem_usg = df.memory_usage().sum() / 1024**2 \n print(\"Memory usage of properties dataframe is :\",start_mem_usg,\" MB\")\n NAlist = [] # Keeps track of columns that have missing values filled in. \n for col in df.columns:\n if df[col].dtype != object: # Exclude strings \n # Print current column type\n print(\"******************************\")\n print(\"Column: \",col)\n print(\"dtype before: \",df[col].dtype) \n # make variables for Int, max and min\n IsInt = False\n mx = df[col].max()\n mn = df[col].min()\n print(\"min for this col: \",mn)\n print(\"max for this col: \",mx)\n # Integer does not support NA, therefore, NA needs to be filled\n if not np.isfinite(df[col]).all(): \n NAlist.append(col)\n df[col].fillna(mn-1,inplace=True) \n \n # test if column can be converted to an integer\n asint = df[col].fillna(0).astype(np.int64)\n result = (df[col] - asint)\n result = result.sum()\n if result > -0.01 and result < 0.01:\n IsInt = True \n # Make Integer/unsigned Integer datatypes\n if IsInt:\n if mn >= 0:\n if mx < 255:\n df[col] = df[col].astype(np.uint8)\n elif mx < 65535:\n df[col] = df[col].astype(np.uint16)\n elif mx < 4294967295:\n df[col] = df[col].astype(np.uint32)\n else:\n df[col] = df[col].astype(np.uint64)\n else:\n if mn > np.iinfo(np.int8).min and mx < np.iinfo(np.int8).max:\n df[col] = df[col].astype(np.int8)\n elif mn > np.iinfo(np.int16).min and mx < np.iinfo(np.int16).max:\n df[col] = df[col].astype(np.int16)\n elif mn > np.iinfo(np.int32).min and mx < np.iinfo(np.int32).max:\n df[col] = df[col].astype(np.int32)\n elif mn > np.iinfo(np.int64).min and mx < np.iinfo(np.int64).max:\n df[col] = df[col].astype(np.int64) \n # Make float datatypes 32 bit\n else:\n df[col] = df[col].astype(np.float32)\n \n # Print new column type\n print(\"dtype after: \",df[col].dtype)\n print(\"******************************\")\n # Print final result\n print(\"___MEMORY USAGE AFTER COMPLETION:___\")\n mem_usg = df.memory_usage().sum() / 1024**2 \n print(\"Memory usage is: \",mem_usg,\" MB\")\n print(\"This is \",100*mem_usg/start_mem_usg,\"% of the initial size\")\n return df, NAlist`\n\n**Reducing for train data set:**\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F1673888%2F1ed61ceabcc82ad4030d93b9e2173186%2Fred1.PNG?generation=1571212907477933&alt=media)\n**Reducing for test data set:**\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F1673888%2F09aaff8bd825daced15dd7080ce1cfa7%2Fred2.PNG?generation=1571212956357397&alt=media)\n\n**Check again! our RAM. More than 4 GB has got free!**\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F1673888%2Fe34d953b44d1f4dd176bc9df9f3d2bc1%2Fram2.PNG?generation=1571213028158670&alt=media)\n\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F1673888%2Fd26993b3eb15a6391728338cbf93a692%2Ftr.PNG?generation=1571213143231326&alt=media)\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F1673888%2F2bcd4bacf247d8bed31cc3af94d2514f%2Ftsr.PNG?generation=1571213144065839&alt=media)\n\n**Save your dataframe as pickle file for fast read.**\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F1673888%2F770e91d57073cc2f86dcac1039fd4ee3%2Fpkl.PNG?generation=1571213279654216&alt=media)\n\n**Do not Forget to save your work. This way you can access the downcasted df everytime you load it!**\n\n**The idea behind the this is based on the fact that most of the column only takes few values and pandas can limit those values only to the few categorical value to save the memory.**\n\n**Often in real-time, data includes the text columns, which are repetitive. Features like gender, country, and codes are always repetitive. These are the examples for categorical data.**\n\n**Categorical variables can take on only a limited, and usually fixed number of possible values. Besides the fixed length, categorical data might have an order but cannot perform numerical operation. Categorical are a Pandas data type.**\n\n**The categorical data type is useful in the following cases −**\n\n**A string variable consisting of only a few different values. Converting such a string variable to a categorical variable will save some memory.**\n\n**The lexical order of a variable is not the same as the logical order (“one”, “two”, “three”). By converting to a categorical and specifying an order on the categories, sorting and min/max will use the logical order instead of the lexical order.**\n\n**As a signal to other python libraries that this column should be treated as a categorical variable (e.g. to use suitable statistical methods or plot types).\" (https://www.tutorialspoint.com/python_pandas/python_pandas_categorical_data)**\n\n**How about other ways!**\nI have used my [kernel](https://www.kaggle.com/hamditarek/reducing-memory-size-for-great-energy-predictor) but there are also other ways such as:\n\n1. [Panda for big data](https://www.dataquest.io/blog/pandas-big-data/)\n2. [optimizing-the-size-of-a-pandas-dataframe-for-low-memory-environment](https://medium.com/@vincentteyssier/optimizing-the-size-of-a-pandas-dataframe-for-low-memory-environment-5f07db3d72e)\n3. [pandas-making-dataframe-smaller-faster](https://www.ritchieng.com/pandas-making-dataframe-smaller-faster/)\n\nThanks to [MJ Bahmani](https://www.kaggle.com/mjbahmani) who created this [kernel](https://www.kaggle.com/mjbahmani/reducing-memory-size-for-ieee)\n\nPickle code refers to this kaggle [kernal](https://www.kaggle.com/ljjblackpig/3-steps-to-reduce-memory-size-for-the-dataset)\n\n**All codes and details exist in this [kernel](https://www.kaggle.com/hamditarek/reducing-memory-size-for-great-energy-predictor)**\n\n**Good Luck** 👍 💪", "650713": "Helpful Stuff\nThanks @hamditarek", "650716": "You are welcome @veeralakrishna, you can check the code in my kernel." }, "source": "meta" }