Data types determine how much memory is allocated to store each element in your dataset. In our case, original dataset contains 128 million rows. Therefore it's very important to keep track of your memory usage.
\nOne of the simplies ways to lighten dataset is to downsample data types. For example, we can use int8 (stores integers using just 8 bits) and float16 (uses 16 bits to store floating-point numbers) instead of int64 and float64
Now, let's calculate how much memory we can save by downsampling data types:
\nint8 uses 8 bits, which is equivalent to 1 byte.
\nint64 uses 64 bits, which is equivalent to 8 bytes.
\n,, integers x / = ,024,, = MB\n\n\n,, integers x byte/ = ,, = MB\n\n\nMemory savings = MB - MB = MB\nBy switching from int64 to int8 for a column with 128 million integers, you can save approximately 854.49 megabytes of memory and it's only savings from 1 column.
\nP.S. If you have any simple ways on how to reduce the dataset's memory footprint - feel free to write them in the comments.
", "messages": [ { "id": "2432518", "postDate": "09/10/2023 23:23:54", "content": "Data types determine how much memory is allocated to store each element in your dataset. In our case, original dataset contains 128 million rows. Therefore it's very important to keep track of your memory usage.
\nOne of the simplies ways to lighten dataset is to downsample data types. For example, we can use int8 (stores integers using just 8 bits) and float16 (uses 16 bits to store floating-point numbers) instead of int64 and float64
Now, let's calculate how much memory we can save by downsampling data types:
\nint8 uses 8 bits, which is equivalent to 1 byte.
\nint64 uses 64 bits, which is equivalent to 8 bytes.
\n,, integers x / = ,024,, = MB\n\n\n,, integers x byte/ = ,, = MB\n\n\nMemory savings = MB - MB = MB\nBy switching from int64 to int8 for a column with 128 million integers, you can save approximately 854.49 megabytes of memory and it's only savings from 1 column.
\nP.S. If you have any simple ways on how to reduce the dataset's memory footprint - feel free to write them in the comments.
", "rawMarkdown": "Data types determine how much memory is allocated to store each element in your dataset. In our case, original dataset contains 128 million rows. Therefore it's very important to keep track of your memory usage. \n\nOne of the simplies ways to lighten dataset is to downsample data types. For example, we can use `int8` (stores integers using just 8 bits) and `float16` (uses 16 bits to store floating-point numbers) instead of `int64` and `float64`\n\nNow, let's calculate how much memory we can save by downsampling data types:\n\n`int8` uses 8 bits, which is equivalent to 1 byte.\n`int64` uses 64 bits, which is equivalent to 8 bytes.\n\n```python\n# Memory Usage with int64:\n128,000,000 integers x 8 bytes/int = 1,024,000,000 bytes = 976.56 MB\n\n# Memory Usage with int8:\n128,000,000 integers x 1 byte/int = 128,000,000 bytes = 122.07 MB\n\n# Memory Savings:\nMemory savings = 976.56 MB - 122.07 MB = 854.49 MB\n```\n\nBy switching from int64 to int8 for a column with 128 million integers, you can save approximately 854.49 megabytes of memory and it's only savings from 1 column.\n\nP.S. If you have any simple ways on how to reduce the dataset's memory footprint - feel free to write them in the comments.", "votes": null }, { "id": "2432555", "postDate": "09/11/2023 01:18:38", "content": "Will it be a calculation issue if we use 8/16 bit type? Like overflow, underflow, etc. I prefer to use 32 bit type at least.
", "rawMarkdown": "Will it be a calculation issue if we use 8/16 bit type? Like overflow, underflow, etc. I prefer to use 32 bit type at least.", "votes": null }, { "id": "2432590", "postDate": "09/11/2023 02:47:29", "content": "@rimbax Well, I think it all depends on what you are going to do with particular column. Of course 8bits and oftentimes even 16bits datatypes are not suitable for most calculations. However, let's say we have feature called \"hour\" that contains unique values from 1 to 24 and if we know that we are not planning to use it later for any calculations we can easily store it in 8bit type and save some memory.
", "rawMarkdown": "rimbax Well, I think it all depends on what you are going to do with particular column. Of course 8bits and oftentimes even 16bits datatypes are not suitable for most calculations. However, let's say we have feature called \"hour\" that contains unique values from 1 to 24 and if we know that we are not planning to use it later for any calculations we can easily store it in 8bit type and save some memory.", "votes": null }, { "id": "2434601", "postDate": "09/12/2023 11:53:06", "content": "@zhukovoleksiy : thanks for the good suggestion. It is a timely suggestion, given the fact of some of the current discussion sentiments
\nJust for those who like to quickly embrace your suggestion, there are some ready-to-reuse implementations of the RAM reduction that follow your advice, as per the references below
\nI hope it is helpful.
", "rawMarkdown": "zhukovoleksiy : thanks for the good suggestion. It is a timely suggestion, given the fact of some of the current [discussion sentiments](https://www.kaggle.com/competitions/child-mind-institute-detect-sleep-states/discussion/438137)\n\nJust for those who like to quickly embrace your suggestion, there are some ready-to-reuse implementations of the RAM reduction that follow your advice, as per the references below\n\n- https://www.kaggle.com/code/renatoreggiani/reduce-memory-usage-zzzs-cmi/notebook\n- https://www.kaggle.com/code/gvyshnya/to-sleep-or-not-to-sleep-deep-eda-dive\n\nI hope it is helpful.", "votes": null }, { "id": "2435705", "postDate": "09/13/2023 06:17:16", "content": "thanks for sharing. good suggestion.
", "rawMarkdown": "thanks for sharing. good suggestion.", "votes": null }, { "id": "2442124", "postDate": "09/16/2023 18:24:06", "content": "Good advice. Thank you !
", "rawMarkdown": "Good advice. Thank you !", "votes": null } ], "comments": [ { "id": 2432555, "author_name": "rimbax", "author_url": "", "post_date": "09/11/2023 01:18:38", "content": "Will it be a calculation issue if we use 8/16 bit type? Like overflow, underflow, etc. I prefer to use 32 bit type at least.
", "votes": null, "replies": [ { "id": 2432590, "author_name": "zhukovoleksiy", "author_url": "", "post_date": "09/11/2023 02:47:29", "content": "@rimbax Well, I think it all depends on what you are going to do with particular column. Of course 8bits and oftentimes even 16bits datatypes are not suitable for most calculations. However, let's say we have feature called \"hour\" that contains unique values from 1 to 24 and if we know that we are not planning to use it later for any calculations we can easily store it in 8bit type and save some memory.
", "votes": null, "replies": [] } ] }, { "id": 2434601, "author_name": "gvyshnya", "author_url": "", "post_date": "09/12/2023 11:53:06", "content": "@zhukovoleksiy : thanks for the good suggestion. It is a timely suggestion, given the fact of some of the current discussion sentiments
\nJust for those who like to quickly embrace your suggestion, there are some ready-to-reuse implementations of the RAM reduction that follow your advice, as per the references below
\nI hope it is helpful.
", "votes": null, "replies": [] }, { "id": 2435705, "author_name": "mehmetisik", "author_url": "", "post_date": "09/13/2023 06:17:16", "content": "thanks for sharing. good suggestion.
", "votes": null, "replies": [] }, { "id": 2442124, "author_name": "aniketdash7", "author_url": "", "post_date": "09/16/2023 18:24:06", "content": "Good advice. Thank you !
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "2432518": "Data types determine how much memory is allocated to store each element in your dataset. In our case, original dataset contains 128 million rows. Therefore it's very important to keep track of your memory usage. \n\nOne of the simplies ways to lighten dataset is to downsample data types. For example, we can use `int8` (stores integers using just 8 bits) and `float16` (uses 16 bits to store floating-point numbers) instead of `int64` and `float64`\n\nNow, let's calculate how much memory we can save by downsampling data types:\n\n`int8` uses 8 bits, which is equivalent to 1 byte.\n`int64` uses 64 bits, which is equivalent to 8 bytes.\n\n```python\n# Memory Usage with int64:\n128,000,000 integers x 8 bytes/int = 1,024,000,000 bytes = 976.56 MB\n\n# Memory Usage with int8:\n128,000,000 integers x 1 byte/int = 128,000,000 bytes = 122.07 MB\n\n# Memory Savings:\nMemory savings = 976.56 MB - 122.07 MB = 854.49 MB\n```\n\nBy switching from int64 to int8 for a column with 128 million integers, you can save approximately 854.49 megabytes of memory and it's only savings from 1 column.\n\nP.S. If you have any simple ways on how to reduce the dataset's memory footprint - feel free to write them in the comments.", "2432555": "Will it be a calculation issue if we use 8/16 bit type? Like overflow, underflow, etc. I prefer to use 32 bit type at least.", "2432590": "rimbax Well, I think it all depends on what you are going to do with particular column. Of course 8bits and oftentimes even 16bits datatypes are not suitable for most calculations. However, let's say we have feature called \"hour\" that contains unique values from 1 to 24 and if we know that we are not planning to use it later for any calculations we can easily store it in 8bit type and save some memory.", "2434601": "zhukovoleksiy : thanks for the good suggestion. It is a timely suggestion, given the fact of some of the current [discussion sentiments](https://www.kaggle.com/competitions/child-mind-institute-detect-sleep-states/discussion/438137)\n\nJust for those who like to quickly embrace your suggestion, there are some ready-to-reuse implementations of the RAM reduction that follow your advice, as per the references below\n\n- https://www.kaggle.com/code/renatoreggiani/reduce-memory-usage-zzzs-cmi/notebook\n- https://www.kaggle.com/code/gvyshnya/to-sleep-or-not-to-sleep-deep-eda-dive\n\nI hope it is helpful.", "2435705": "thanks for sharing. good suggestion.", "2442124": "Good advice. Thank you !" }, "source": "meta" }