{
  "id": 96655,
  "title": "A pitfall of reduce_mem_usage function",
  "url": "/competitions/champs-scalar-coupling/discussion/96655",
  "author_name": "",
  "post_date": "2019-06-21T18:36:31.578445300Z",
  "votes": 64,
  "comment_count": 14,
  "views": 0,
  "content": "<p>In many competitions we use <code>reduce_mem_usage</code> function to reduce the size of the dataset. Dataframes take less memory and we lose nothing, right? What could go wrong?</p>\n\n<p>Well, @arlanov pointed out one strange thing:</p>\n\n<p><img src=\"https://i.imgur.com/llB4VHZ.png\" alt=\"\"></p>\n\n<p>Turns out that this function doesn't have checks for floats with high precision:</p>\n\n<p><code>\nif c_min &gt; np.finfo(np.float16).min and c_max &lt; np.finfo(np.float16).max:\n    df[col] = df[col].astype(np.float16)\nelif c_min &gt; np.finfo(np.float32).min and c_max &lt; np.finfo(np.float32).max:\n    df[col] = df[col].astype(np.float32)\nelse:\n    df[col] = df[col].astype(np.float64)\n</code></p>\n\n<p>We compare min/max values of columns with min/max values of np float types.</p>\n\n<p>There are some ways to implement necessary checks, I compare precisions:</p>\n\n<p><img src=\"https://i.imgur.com/sYu2KCM.png\" alt=\"\"></p>\n\n<p>The fixed code looks like this:\n<code>\ndef reduce_mem_usage(df, verbose=True):\n    numerics = ['int16', 'int32', 'int64', 'float16', 'float32', 'float64']\n    start_mem = df.memory_usage().sum() / 1024**2\n    for col in df.columns:\n        col_type = df[col].dtypes\n        if col_type in numerics:\n            c_min = df[col].min()\n            c_max = df[col].max()\n            if str(col_type)[:3] == 'int':\n                if c_min &gt; np.iinfo(np.int8).min and c_max &lt; np.iinfo(np.int8).max:\n                    df[col] = df[col].astype(np.int8)\n                elif c_min &gt; np.iinfo(np.int16).min and c_max &lt; np.iinfo(np.int16).max:\n                    df[col] = df[col].astype(np.int16)\n                elif c_min &gt; np.iinfo(np.int32).min and c_max &lt; np.iinfo(np.int32).max:\n                    df[col] = df[col].astype(np.int32)\n                elif c_min &gt; np.iinfo(np.int64).min and c_max &lt; np.iinfo(np.int64).max:\n                    df[col] = df[col].astype(np.int64)\n            else:\n                c_prec = df[col].apply(lambda x: np.finfo(x).precision).max()\n                if c_min &gt; np.finfo(np.float16).min and c_max &lt; np.finfo(np.float16).max and c_prec == np.finfo(np.float16).precision:\n                    df[col] = df[col].astype(np.float16)\n                elif c_min &gt; np.finfo(np.float32).min and c_max &lt; np.finfo(np.float32).max and c_prec == np.finfo(np.float32).precision:\n                    df[col] = df[col].astype(np.float32)\n                else:\n                    df[col] = df[col].astype(np.float64)\n    end_mem = df.memory_usage().sum() / 1024**2\n    if verbose: print('Mem. usage decreased to {:5.2f} Mb ({:.1f}% reduction)'.format(end_mem, 100 * (start_mem - end_mem) / start_mem))\n    return df\n</code></p>\n\n<p>I'll update my kernels with this code and update the dataset.</p>",
  "messages": [
    {
      "id": "558045",
      "postDate": "06/21/2019 18:36:31",
      "content": "<p>In many competitions we use <code>reduce_mem_usage</code> function to reduce the size of the dataset. Dataframes take less memory and we lose nothing, right? What could go wrong?</p>\n\n<p>Well, @arlanov pointed out one strange thing:</p>\n\n<p><img src=\"https://i.imgur.com/llB4VHZ.png\" alt=\"\"></p>\n\n<p>Turns out that this function doesn't have checks for floats with high precision:</p>\n\n<p><code>\nif c_min &gt; np.finfo(np.float16).min and c_max &lt; np.finfo(np.float16).max:\n    df[col] = df[col].astype(np.float16)\nelif c_min &gt; np.finfo(np.float32).min and c_max &lt; np.finfo(np.float32).max:\n    df[col] = df[col].astype(np.float32)\nelse:\n    df[col] = df[col].astype(np.float64)\n</code></p>\n\n<p>We compare min/max values of columns with min/max values of np float types.</p>\n\n<p>There are some ways to implement necessary checks, I compare precisions:</p>\n\n<p><img src=\"https://i.imgur.com/sYu2KCM.png\" alt=\"\"></p>\n\n<p>The fixed code looks like this:\n<code>\ndef reduce_mem_usage(df, verbose=True):\n    numerics = ['int16', 'int32', 'int64', 'float16', 'float32', 'float64']\n    start_mem = df.memory_usage().sum() / 1024**2\n    for col in df.columns:\n        col_type = df[col].dtypes\n        if col_type in numerics:\n            c_min = df[col].min()\n            c_max = df[col].max()\n            if str(col_type)[:3] == 'int':\n                if c_min &gt; np.iinfo(np.int8).min and c_max &lt; np.iinfo(np.int8).max:\n                    df[col] = df[col].astype(np.int8)\n                elif c_min &gt; np.iinfo(np.int16).min and c_max &lt; np.iinfo(np.int16).max:\n                    df[col] = df[col].astype(np.int16)\n                elif c_min &gt; np.iinfo(np.int32).min and c_max &lt; np.iinfo(np.int32).max:\n                    df[col] = df[col].astype(np.int32)\n                elif c_min &gt; np.iinfo(np.int64).min and c_max &lt; np.iinfo(np.int64).max:\n                    df[col] = df[col].astype(np.int64)\n            else:\n                c_prec = df[col].apply(lambda x: np.finfo(x).precision).max()\n                if c_min &gt; np.finfo(np.float16).min and c_max &lt; np.finfo(np.float16).max and c_prec == np.finfo(np.float16).precision:\n                    df[col] = df[col].astype(np.float16)\n                elif c_min &gt; np.finfo(np.float32).min and c_max &lt; np.finfo(np.float32).max and c_prec == np.finfo(np.float32).precision:\n                    df[col] = df[col].astype(np.float32)\n                else:\n                    df[col] = df[col].astype(np.float64)\n    end_mem = df.memory_usage().sum() / 1024**2\n    if verbose: print('Mem. usage decreased to {:5.2f} Mb ({:.1f}% reduction)'.format(end_mem, 100 * (start_mem - end_mem) / start_mem))\n    return df\n</code></p>\n\n<p>I'll update my kernels with this code and update the dataset.</p>",
      "rawMarkdown": "In many competitions we use `reduce_mem_usage` function to reduce the size of the dataset. Dataframes take less memory and we lose nothing, right? What could go wrong?\n\nWell, @arlanov pointed out one strange thing:\n\n![](https://i.imgur.com/llB4VHZ.png)\n\nTurns out that this function doesn't have checks for floats with high precision:\n\n```\nif c_min &gt; np.finfo(np.float16).min and c_max &lt; np.finfo(np.float16).max:\n    df[col] = df[col].astype(np.float16)\nelif c_min &gt; np.finfo(np.float32).min and c_max &lt; np.finfo(np.float32).max:\n    df[col] = df[col].astype(np.float32)\nelse:\n    df[col] = df[col].astype(np.float64)\n```\n\nWe compare min/max values of columns with min/max values of np float types.\n\nThere are some ways to implement necessary checks, I compare precisions:\n\n![](https://i.imgur.com/sYu2KCM.png)\n\nThe fixed code looks like this:\n```\ndef reduce_mem_usage(df, verbose=True):\n    numerics = ['int16', 'int32', 'int64', 'float16', 'float32', 'float64']\n    start_mem = df.memory_usage().sum() / 1024**2\n    for col in df.columns:\n        col_type = df[col].dtypes\n        if col_type in numerics:\n            c_min = df[col].min()\n            c_max = df[col].max()\n            if str(col_type)[:3] == 'int':\n                if c_min &gt; np.iinfo(np.int8).min and c_max &lt; np.iinfo(np.int8).max:\n                    df[col] = df[col].astype(np.int8)\n                elif c_min &gt; np.iinfo(np.int16).min and c_max &lt; np.iinfo(np.int16).max:\n                    df[col] = df[col].astype(np.int16)\n                elif c_min &gt; np.iinfo(np.int32).min and c_max &lt; np.iinfo(np.int32).max:\n                    df[col] = df[col].astype(np.int32)\n                elif c_min &gt; np.iinfo(np.int64).min and c_max &lt; np.iinfo(np.int64).max:\n                    df[col] = df[col].astype(np.int64)\n            else:\n                c_prec = df[col].apply(lambda x: np.finfo(x).precision).max()\n                if c_min &gt; np.finfo(np.float16).min and c_max &lt; np.finfo(np.float16).max and c_prec == np.finfo(np.float16).precision:\n                    df[col] = df[col].astype(np.float16)\n                elif c_min &gt; np.finfo(np.float32).min and c_max &lt; np.finfo(np.float32).max and c_prec == np.finfo(np.float32).precision:\n                    df[col] = df[col].astype(np.float32)\n                else:\n                    df[col] = df[col].astype(np.float64)\n    end_mem = df.memory_usage().sum() / 1024**2\n    if verbose: print('Mem. usage decreased to {:5.2f} Mb ({:.1f}% reduction)'.format(end_mem, 100 * (start_mem - end_mem) / start_mem))\n    return df\n```\n\nI'll update my kernels with this code and update the dataset.",
      "votes": null
    },
    {
      "id": "558076",
      "postDate": "06/21/2019 20:08:56",
      "content": "<p>All hail our savior Andrew, and I will report back see if this increases the CV of my best performed LGB model.</p>\n\n<p>EDIT: using fixed seed, 3 of my best LGB models improves CV only by 0.02, 0.02, and 0.03, which is not surprising for tree-based regressors as the feature itself is not used in estimating the targets.</p>",
      "rawMarkdown": "All hail our savior Andrew, and I will report back see if this increases the CV of my best performed LGB model.\n\nEDIT: using fixed seed, 3 of my best LGB models improves CV only by 0.02, 0.02, and 0.03, which is not surprising for tree-based regressors as the feature itself is not used in estimating the targets.",
      "votes": null
    },
    {
      "id": "558122",
      "postDate": "06/21/2019 21:39:37",
      "content": "<p>I tested the function and found that <code>c_prec = df[col].apply(lambda x: np.finfo(x).precision).max()</code> takes too long time to apply <code>np.finfo</code> on over nearly 7 million rows of train and test combined. Since pandas uses a unique <code>dtype</code> for each column, I think it suffices to just check 1 number <code>c_prec =  np.finfo(df[col]).precision</code>.</p>\n\n<p>EDIT: okay, this still does not work, since every numerical columns' <code>np.finfo(df[col]).precision</code> will be 15 which is <code>np.float64</code>'s precision, and the memory usage is not really reduced. The \"strange thing\" <a href=\"/arlanov\">@arlanov</a> encountered is quite normal. Since the range of the <code>scalar_coupling_constant</code> is well within [-65535, 65536] range, Andrew's original memory reducer will treat it as <code>np.float16</code> and convert it to half precision. Try\n<code>\ntrain['scalar_coupling_constant'].astype(np.float32)\ntrain['scalar_coupling_constant'].astype(np.float16)\n</code>\nThe second one is exactly 84.8125 for the entry 0. </p>",
      "rawMarkdown": "I tested the function and found that `c_prec = df[col].apply(lambda x: np.finfo(x).precision).max()` takes too long time to apply `np.finfo` on over nearly 7 million rows of train and test combined. Since pandas uses a unique `dtype` for each column, I think it suffices to just check 1 number `c_prec =  np.finfo(df[col]).precision`.\n\nEDIT: okay, this still does not work, since every numerical columns' `np.finfo(df[col]).precision` will be 15 which is `np.float64`'s precision, and the memory usage is not really reduced. The \"strange thing\" @arlanov encountered is quite normal. Since the range of the `scalar_coupling_constant` is well within [-65535, 65536] range, Andrew's original memory reducer will treat it as `np.float16` and convert it to half precision. Try\n```\ntrain['scalar_coupling_constant'].astype(np.float32)\ntrain['scalar_coupling_constant'].astype(np.float16)\n```\nThe second one is exactly 84.8125 for the entry 0.",
      "votes": null
    },
    {
      "id": "558239",
      "postDate": "06/22/2019 03:22:53",
      "content": "<p>Hm, I see. My first priority was not losing information.\nThe main question is: do we need this precision? If I run the same code with <code>reduced</code> dtypes and float64, I see a slight improvement in metrics for all 8 types. So maybe increased memory usage is worth it.</p>",
      "rawMarkdown": "Hm, I see. My first priority was not losing information.\nThe main question is: do we need this precision? If I run the same code with `reduced` dtypes and float64, I see a slight improvement in metrics for all 8 types. So maybe increased memory usage is worth it.",
      "votes": null
    },
    {
      "id": "558560",
      "postDate": "06/22/2019 15:06:14",
      "content": "<p>I also found that with this code, I was getting overflow warnings when running LightGBM (actually from numpy sum) since numbers were overflowing the <code>float16</code> limit. I removed the <code>float16</code> lines so that the smallest data type is now <code>float32</code></p>",
      "rawMarkdown": "I also found that with this code, I was getting overflow warnings when running LightGBM (actually from numpy sum) since numbers were overflowing the `float16` limit. I removed the `float16` lines so that the smallest data type is now `float32`",
      "votes": null
    },
    {
      "id": "558737",
      "postDate": "06/22/2019 20:40:38",
      "content": "<p>np.float16 is a very dangerous type. e.g. when you use mean over float16, what obviously happens in the background is the sum. if sum exceeds float16 precision, you get NA. it is better to avoid float16 overall</p>",
      "rawMarkdown": "np.float16 is a very dangerous type. e.g. when you use mean over float16, what obviously happens in the background is the sum. if sum exceeds float16 precision, you get NA. it is better to avoid float16 overall",
      "votes": null
    },
    {
      "id": "558745",
      "postDate": "06/22/2019 21:09:29",
      "content": "<p>There is another unrelated bug in  <code>reduce_mem_usage()</code> when <code>df</code> has duplicate column names. This arises because <code>df[col].dtypes</code> returns a series and then <code>col_type in numerics</code> throws an error. The simple solution, which is recommended in any case, is to not have duplicate column names in <code>df</code>.</p>",
      "rawMarkdown": "There is another unrelated bug in  `reduce_mem_usage()` when `df` has duplicate column names. This arises because `df[col].dtypes` returns a series and then `col_type in numerics` throws an error. The simple solution, which is recommended in any case, is to not have duplicate column names in `df`.",
      "votes": null
    },
    {
      "id": "558801",
      "postDate": "06/23/2019 00:05:12",
      "content": "<p>I totally agree with the other guys that say not to use float16. Even if you can represent that number as a float16 type you will get a big error after doing some operations with those numbers.</p>\n\n<p>As <a href=\"/scaomath\">@scaomath</a> said, the <code>np.finfo(df[col]).precision</code> returns always 15 for all the <code>float64</code> because it works with <code>dtype</code>, not with an specific number. The documentations says \"kind of float\" and not \"float\" or \"specific float\":</p>\n\n<blockquote>\n  <p>The approximate number of decimal digits to which this kind of float is precise</p>\n</blockquote>\n\n<p>I'm using this version of the <code>reduce_mem_usage</code>:</p>\n\n<p>```python\ndef reduce_memory_usage(df, deep=True, verbose=True, categories=True):\n    # All types that we want to change for \"lighter\" ones.\n    # int8 and float16 are not include because we cannot reduce\n    # those data types.\n    # float32 is not include because float16 has too low precision.\n    numeric2reduce = [\"int16\", \"int32\", \"int64\", \"float64\"]\n    start_mem = 0\n    if verbose:\n        start_mem = memory_usage_mb(df, deep=deep)</p>\n\n<pre><code>for col, col_type in df.dtypes.iteritems():\n    best_type = None\n    if col_type == \"object\":\n        df[col] = df[col].astype(\"category\")\n        best_type = \"category\"\n    elif col_type in numeric2reduce:\n        downcast = \"integer\" if \"int\" in str(col_type) else \"float\"\n        df[col] = pd.to_numeric(df[col], downcast=downcast)\n        best_type = df[col].dtype.name\n    # Log the conversion performed.\n    if verbose and best_type is not None and best_type != str(col_type):\n        print(f\"Column '{col}' converted from {col_type} to {best_type}\")\n\nif verbose:\n    end_mem = memory_usage_mb(df, deep=deep)\n    diff_mem = start_mem - end_mem\n    percent_mem = 100 * diff_mem / start_mem\n    print(f\"Memory usage decreased from\"\n          f\" {start_mem:.2f}MB to {end_mem:.2f}MB\"\n          f\" ({diff_mem:.2f}MB, {percent_mem:.2f}% reduction)\")\n</code></pre>\n\n<p><code>``\nI'm delegating the datatype conversion to</code>pandas.to_numeric<code>that doesn't perform downcasting to</code>float16` (<a href=\"https://stackoverflow.com/questions/56640472/downcast-to-float16-in-pandas-to-numeric\">https://stackoverflow.com/questions/56640472/downcast-to-float16-in-pandas-to-numeric</a>). It also simplifies the code and makes it more readable :)</p>\n\n<p>NOTE: I'm also converting columns of type <code>object</code> to Pandas categories.</p>",
      "rawMarkdown": "I totally agree with the other guys that say not to use float16. Even if you can represent that number as a float16 type you will get a big error after doing some operations with those numbers.\n\nAs @scaomath said, the `np.finfo(df[col]).precision` returns always 15 for all the `float64` because it works with `dtype`, not with an specific number. The documentations says \"kind of float\" and not \"float\" or \"specific float\":\n&gt; The approximate number of decimal digits to which this kind of float is precise\n\nI'm using this version of the `reduce_mem_usage`:\n\n```python\ndef reduce_memory_usage(df, deep=True, verbose=True, categories=True):\n    # All types that we want to change for \"lighter\" ones.\n    # int8 and float16 are not include because we cannot reduce\n    # those data types.\n    # float32 is not include because float16 has too low precision.\n    numeric2reduce = [\"int16\", \"int32\", \"int64\", \"float64\"]\n    start_mem = 0\n    if verbose:\n        start_mem = memory_usage_mb(df, deep=deep)\n\n    for col, col_type in df.dtypes.iteritems():\n        best_type = None\n        if col_type == \"object\":\n            df[col] = df[col].astype(\"category\")\n            best_type = \"category\"\n        elif col_type in numeric2reduce:\n            downcast = \"integer\" if \"int\" in str(col_type) else \"float\"\n            df[col] = pd.to_numeric(df[col], downcast=downcast)\n            best_type = df[col].dtype.name\n        # Log the conversion performed.\n        if verbose and best_type is not None and best_type != str(col_type):\n            print(f\"Column '{col}' converted from {col_type} to {best_type}\")\n\n    if verbose:\n        end_mem = memory_usage_mb(df, deep=deep)\n        diff_mem = start_mem - end_mem\n        percent_mem = 100 * diff_mem / start_mem\n        print(f\"Memory usage decreased from\"\n              f\" {start_mem:.2f}MB to {end_mem:.2f}MB\"\n              f\" ({diff_mem:.2f}MB, {percent_mem:.2f}% reduction)\")\n```\nI'm delegating the datatype conversion to `pandas.to_numeric` that doesn't perform downcasting to `float16` (https://stackoverflow.com/questions/56640472/downcast-to-float16-in-pandas-to-numeric). It also simplifies the code and makes it more readable :)\n\nNOTE: I'm also converting columns of type `object` to Pandas categories.",
      "votes": null
    },
    {
      "id": "559214",
      "postDate": "06/23/2019 16:46:01",
      "content": "<p>I removed float16 from the original version and had no problems - so I agree with the posters below who commented on float16.  I will look at the new version, but I still think I will ditch float16.  Float16 is too scary...</p>",
      "rawMarkdown": "I removed float16 from the original version and had no problems - so I agree with the posters below who commented on float16.  I will look at the new version, but I still think I will ditch float16.  Float16 is too scary...",
      "votes": null
    },
    {
      "id": "559720",
      "postDate": "06/24/2019 13:52:28",
      "content": "<p>In Santander competition this improvement could grant you jump 1-2k positions up the LB and silver medal :)</p>",
      "rawMarkdown": "In Santander competition this improvement could grant you jump 1-2k positions up the LB and silver medal :)",
      "votes": null
    },
    {
      "id": "561806",
      "postDate": "06/26/2019 18:49:50",
      "content": "<p>Correct me if I'm wrong but doesn't NumPy convert  <code>float16</code> to <code>float32</code>by default anyway. Wouldn't it be more efficient, as well as safer, to just use <code>float32</code>?</p>",
      "rawMarkdown": "Correct me if I'm wrong but doesn't NumPy convert  `float16` to `float32 `by default anyway. Wouldn't it be more efficient, as well as safer, to just use `float32`?",
      "votes": null
    },
    {
      "id": "565815",
      "postDate": "07/01/2019 12:09:27",
      "content": "<p>Thanks for sharing. How have you defined memory_usage_mb?</p>",
      "rawMarkdown": "Thanks for sharing. How have you defined memory\\_usage\\_mb?",
      "votes": null
    },
    {
      "id": "566225",
      "postDate": "07/01/2019 23:22:56",
      "content": "<p>It's just a call to the Pandas <code>memory_usage</code> and a conversion to MB :)</p>\n\n<p><code>\ndef memory_usage_mb(df, *args, **kwargs):\n    \"\"\"Dataframe memory usage in MB. \"\"\"\n    return df.memory_usage(*args, **kwargs).sum() / 1024**2\n</code></p>",
      "rawMarkdown": "It's just a call to the Pandas `memory_usage` and a conversion to MB :)\n\n```\ndef memory_usage_mb(df, *args, **kwargs):\n    \"\"\"Dataframe memory usage in MB. \"\"\"\n    return df.memory_usage(*args, **kwargs).sum() / 1024**2\n```",
      "votes": null
    },
    {
      "id": "619322",
      "postDate": "09/06/2019 04:37:31",
      "content": "<p>Thanks for sharing! I want to try it on my kernel~</p>",
      "rawMarkdown": "Thanks for sharing! I want to try it on my kernel~",
      "votes": null
    },
    {
      "id": "620729",
      "postDate": "09/07/2019 23:01:42",
      "content": "<p>Thanks for sharing, I'll try it in a new competition !</p>",
      "rawMarkdown": "Thanks for sharing, I'll try it in a new competition !",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 558076,
      "author_name": "scaomath",
      "author_url": "",
      "post_date": "06/21/2019 20:08:56",
      "content": "<p>All hail our savior Andrew, and I will report back see if this increases the CV of my best performed LGB model.</p>\n\n<p>EDIT: using fixed seed, 3 of my best LGB models improves CV only by 0.02, 0.02, and 0.03, which is not surprising for tree-based regressors as the feature itself is not used in estimating the targets.</p>",
      "votes": null,
      "replies": [
        {
          "id": 559720,
          "author_name": "stanislavblinov",
          "author_url": "",
          "post_date": "06/24/2019 13:52:28",
          "content": "<p>In Santander competition this improvement could grant you jump 1-2k positions up the LB and silver medal :)</p>",
          "votes": null,
          "replies": []
        }
      ]
    },
    {
      "id": 558122,
      "author_name": "scaomath",
      "author_url": "",
      "post_date": "06/21/2019 21:39:37",
      "content": "<p>I tested the function and found that <code>c_prec = df[col].apply(lambda x: np.finfo(x).precision).max()</code> takes too long time to apply <code>np.finfo</code> on over nearly 7 million rows of train and test combined. Since pandas uses a unique <code>dtype</code> for each column, I think it suffices to just check 1 number <code>c_prec =  np.finfo(df[col]).precision</code>.</p>\n\n<p>EDIT: okay, this still does not work, since every numerical columns' <code>np.finfo(df[col]).precision</code> will be 15 which is <code>np.float64</code>'s precision, and the memory usage is not really reduced. The \"strange thing\" <a href=\"/arlanov\">@arlanov</a> encountered is quite normal. Since the range of the <code>scalar_coupling_constant</code> is well within [-65535, 65536] range, Andrew's original memory reducer will treat it as <code>np.float16</code> and convert it to half precision. Try\n<code>\ntrain['scalar_coupling_constant'].astype(np.float32)\ntrain['scalar_coupling_constant'].astype(np.float16)\n</code>\nThe second one is exactly 84.8125 for the entry 0. </p>",
      "votes": null,
      "replies": [
        {
          "id": 558239,
          "author_name": "artgor",
          "author_url": "",
          "post_date": "06/22/2019 03:22:53",
          "content": "<p>Hm, I see. My first priority was not losing information.\nThe main question is: do we need this precision? If I run the same code with <code>reduced</code> dtypes and float64, I see a slight improvement in metrics for all 8 types. So maybe increased memory usage is worth it.</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 561806,
          "author_name": "jaredsavage",
          "author_url": "",
          "post_date": "06/26/2019 18:49:50",
          "content": "<p>Correct me if I'm wrong but doesn't NumPy convert  <code>float16</code> to <code>float32</code>by default anyway. Wouldn't it be more efficient, as well as safer, to just use <code>float32</code>?</p>",
          "votes": null,
          "replies": []
        }
      ]
    },
    {
      "id": 558560,
      "author_name": "anjum48",
      "author_url": "",
      "post_date": "06/22/2019 15:06:14",
      "content": "<p>I also found that with this code, I was getting overflow warnings when running LightGBM (actually from numpy sum) since numbers were overflowing the <code>float16</code> limit. I removed the <code>float16</code> lines so that the smallest data type is now <code>float32</code></p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 558737,
      "author_name": "snovik1975",
      "author_url": "",
      "post_date": "06/22/2019 20:40:38",
      "content": "<p>np.float16 is a very dangerous type. e.g. when you use mean over float16, what obviously happens in the background is the sum. if sum exceeds float16 precision, you get NA. it is better to avoid float16 overall</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 558745,
      "author_name": "robertburbidge",
      "author_url": "",
      "post_date": "06/22/2019 21:09:29",
      "content": "<p>There is another unrelated bug in  <code>reduce_mem_usage()</code> when <code>df</code> has duplicate column names. This arises because <code>df[col].dtypes</code> returns a series and then <code>col_type in numerics</code> throws an error. The simple solution, which is recommended in any case, is to not have duplicate column names in <code>df</code>.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 558801,
      "author_name": "guiferviz",
      "author_url": "",
      "post_date": "06/23/2019 00:05:12",
      "content": "<p>I totally agree with the other guys that say not to use float16. Even if you can represent that number as a float16 type you will get a big error after doing some operations with those numbers.</p>\n\n<p>As <a href=\"/scaomath\">@scaomath</a> said, the <code>np.finfo(df[col]).precision</code> returns always 15 for all the <code>float64</code> because it works with <code>dtype</code>, not with an specific number. The documentations says \"kind of float\" and not \"float\" or \"specific float\":</p>\n\n<blockquote>\n  <p>The approximate number of decimal digits to which this kind of float is precise</p>\n</blockquote>\n\n<p>I'm using this version of the <code>reduce_mem_usage</code>:</p>\n\n<p>```python\ndef reduce_memory_usage(df, deep=True, verbose=True, categories=True):\n    # All types that we want to change for \"lighter\" ones.\n    # int8 and float16 are not include because we cannot reduce\n    # those data types.\n    # float32 is not include because float16 has too low precision.\n    numeric2reduce = [\"int16\", \"int32\", \"int64\", \"float64\"]\n    start_mem = 0\n    if verbose:\n        start_mem = memory_usage_mb(df, deep=deep)</p>\n\n<pre><code>for col, col_type in df.dtypes.iteritems():\n    best_type = None\n    if col_type == \"object\":\n        df[col] = df[col].astype(\"category\")\n        best_type = \"category\"\n    elif col_type in numeric2reduce:\n        downcast = \"integer\" if \"int\" in str(col_type) else \"float\"\n        df[col] = pd.to_numeric(df[col], downcast=downcast)\n        best_type = df[col].dtype.name\n    # Log the conversion performed.\n    if verbose and best_type is not None and best_type != str(col_type):\n        print(f\"Column '{col}' converted from {col_type} to {best_type}\")\n\nif verbose:\n    end_mem = memory_usage_mb(df, deep=deep)\n    diff_mem = start_mem - end_mem\n    percent_mem = 100 * diff_mem / start_mem\n    print(f\"Memory usage decreased from\"\n          f\" {start_mem:.2f}MB to {end_mem:.2f}MB\"\n          f\" ({diff_mem:.2f}MB, {percent_mem:.2f}% reduction)\")\n</code></pre>\n\n<p><code>``\nI'm delegating the datatype conversion to</code>pandas.to_numeric<code>that doesn't perform downcasting to</code>float16` (<a href=\"https://stackoverflow.com/questions/56640472/downcast-to-float16-in-pandas-to-numeric\">https://stackoverflow.com/questions/56640472/downcast-to-float16-in-pandas-to-numeric</a>). It also simplifies the code and makes it more readable :)</p>\n\n<p>NOTE: I'm also converting columns of type <code>object</code> to Pandas categories.</p>",
      "votes": null,
      "replies": [
        {
          "id": 565815,
          "author_name": "pietermostert",
          "author_url": "",
          "post_date": "07/01/2019 12:09:27",
          "content": "<p>Thanks for sharing. How have you defined memory_usage_mb?</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 566225,
          "author_name": "guiferviz",
          "author_url": "",
          "post_date": "07/01/2019 23:22:56",
          "content": "<p>It's just a call to the Pandas <code>memory_usage</code> and a conversion to MB :)</p>\n\n<p><code>\ndef memory_usage_mb(df, *args, **kwargs):\n    \"\"\"Dataframe memory usage in MB. \"\"\"\n    return df.memory_usage(*args, **kwargs).sum() / 1024**2\n</code></p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 619322,
          "author_name": "zning1994",
          "author_url": "",
          "post_date": "09/06/2019 04:37:31",
          "content": "<p>Thanks for sharing! I want to try it on my kernel~</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 620729,
          "author_name": "chenxinye",
          "author_url": "",
          "post_date": "09/07/2019 23:01:42",
          "content": "<p>Thanks for sharing, I'll try it in a new competition !</p>",
          "votes": null,
          "replies": []
        }
      ]
    },
    {
      "id": 559214,
      "author_name": "vettejeep",
      "author_url": "",
      "post_date": "06/23/2019 16:46:01",
      "content": "<p>I removed float16 from the original version and had no problems - so I agree with the posters below who commented on float16.  I will look at the new version, but I still think I will ditch float16.  Float16 is too scary...</p>",
      "votes": null,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "558045": "In many competitions we use `reduce_mem_usage` function to reduce the size of the dataset. Dataframes take less memory and we lose nothing, right? What could go wrong?\n\nWell, @arlanov pointed out one strange thing:\n\n![](https://i.imgur.com/llB4VHZ.png)\n\nTurns out that this function doesn't have checks for floats with high precision:\n\n```\nif c_min &gt; np.finfo(np.float16).min and c_max &lt; np.finfo(np.float16).max:\n    df[col] = df[col].astype(np.float16)\nelif c_min &gt; np.finfo(np.float32).min and c_max &lt; np.finfo(np.float32).max:\n    df[col] = df[col].astype(np.float32)\nelse:\n    df[col] = df[col].astype(np.float64)\n```\n\nWe compare min/max values of columns with min/max values of np float types.\n\nThere are some ways to implement necessary checks, I compare precisions:\n\n![](https://i.imgur.com/sYu2KCM.png)\n\nThe fixed code looks like this:\n```\ndef reduce_mem_usage(df, verbose=True):\n    numerics = ['int16', 'int32', 'int64', 'float16', 'float32', 'float64']\n    start_mem = df.memory_usage().sum() / 1024**2\n    for col in df.columns:\n        col_type = df[col].dtypes\n        if col_type in numerics:\n            c_min = df[col].min()\n            c_max = df[col].max()\n            if str(col_type)[:3] == 'int':\n                if c_min &gt; np.iinfo(np.int8).min and c_max &lt; np.iinfo(np.int8).max:\n                    df[col] = df[col].astype(np.int8)\n                elif c_min &gt; np.iinfo(np.int16).min and c_max &lt; np.iinfo(np.int16).max:\n                    df[col] = df[col].astype(np.int16)\n                elif c_min &gt; np.iinfo(np.int32).min and c_max &lt; np.iinfo(np.int32).max:\n                    df[col] = df[col].astype(np.int32)\n                elif c_min &gt; np.iinfo(np.int64).min and c_max &lt; np.iinfo(np.int64).max:\n                    df[col] = df[col].astype(np.int64)\n            else:\n                c_prec = df[col].apply(lambda x: np.finfo(x).precision).max()\n                if c_min &gt; np.finfo(np.float16).min and c_max &lt; np.finfo(np.float16).max and c_prec == np.finfo(np.float16).precision:\n                    df[col] = df[col].astype(np.float16)\n                elif c_min &gt; np.finfo(np.float32).min and c_max &lt; np.finfo(np.float32).max and c_prec == np.finfo(np.float32).precision:\n                    df[col] = df[col].astype(np.float32)\n                else:\n                    df[col] = df[col].astype(np.float64)\n    end_mem = df.memory_usage().sum() / 1024**2\n    if verbose: print('Mem. usage decreased to {:5.2f} Mb ({:.1f}% reduction)'.format(end_mem, 100 * (start_mem - end_mem) / start_mem))\n    return df\n```\n\nI'll update my kernels with this code and update the dataset.",
    "558076": "All hail our savior Andrew, and I will report back see if this increases the CV of my best performed LGB model.\n\nEDIT: using fixed seed, 3 of my best LGB models improves CV only by 0.02, 0.02, and 0.03, which is not surprising for tree-based regressors as the feature itself is not used in estimating the targets.",
    "558122": "I tested the function and found that `c_prec = df[col].apply(lambda x: np.finfo(x).precision).max()` takes too long time to apply `np.finfo` on over nearly 7 million rows of train and test combined. Since pandas uses a unique `dtype` for each column, I think it suffices to just check 1 number `c_prec =  np.finfo(df[col]).precision`.\n\nEDIT: okay, this still does not work, since every numerical columns' `np.finfo(df[col]).precision` will be 15 which is `np.float64`'s precision, and the memory usage is not really reduced. The \"strange thing\" @arlanov encountered is quite normal. Since the range of the `scalar_coupling_constant` is well within [-65535, 65536] range, Andrew's original memory reducer will treat it as `np.float16` and convert it to half precision. Try\n```\ntrain['scalar_coupling_constant'].astype(np.float32)\ntrain['scalar_coupling_constant'].astype(np.float16)\n```\nThe second one is exactly 84.8125 for the entry 0.",
    "558239": "Hm, I see. My first priority was not losing information.\nThe main question is: do we need this precision? If I run the same code with `reduced` dtypes and float64, I see a slight improvement in metrics for all 8 types. So maybe increased memory usage is worth it.",
    "558560": "I also found that with this code, I was getting overflow warnings when running LightGBM (actually from numpy sum) since numbers were overflowing the `float16` limit. I removed the `float16` lines so that the smallest data type is now `float32`",
    "558737": "np.float16 is a very dangerous type. e.g. when you use mean over float16, what obviously happens in the background is the sum. if sum exceeds float16 precision, you get NA. it is better to avoid float16 overall",
    "558745": "There is another unrelated bug in  `reduce_mem_usage()` when `df` has duplicate column names. This arises because `df[col].dtypes` returns a series and then `col_type in numerics` throws an error. The simple solution, which is recommended in any case, is to not have duplicate column names in `df`.",
    "558801": "I totally agree with the other guys that say not to use float16. Even if you can represent that number as a float16 type you will get a big error after doing some operations with those numbers.\n\nAs @scaomath said, the `np.finfo(df[col]).precision` returns always 15 for all the `float64` because it works with `dtype`, not with an specific number. The documentations says \"kind of float\" and not \"float\" or \"specific float\":\n&gt; The approximate number of decimal digits to which this kind of float is precise\n\nI'm using this version of the `reduce_mem_usage`:\n\n```python\ndef reduce_memory_usage(df, deep=True, verbose=True, categories=True):\n    # All types that we want to change for \"lighter\" ones.\n    # int8 and float16 are not include because we cannot reduce\n    # those data types.\n    # float32 is not include because float16 has too low precision.\n    numeric2reduce = [\"int16\", \"int32\", \"int64\", \"float64\"]\n    start_mem = 0\n    if verbose:\n        start_mem = memory_usage_mb(df, deep=deep)\n\n    for col, col_type in df.dtypes.iteritems():\n        best_type = None\n        if col_type == \"object\":\n            df[col] = df[col].astype(\"category\")\n            best_type = \"category\"\n        elif col_type in numeric2reduce:\n            downcast = \"integer\" if \"int\" in str(col_type) else \"float\"\n            df[col] = pd.to_numeric(df[col], downcast=downcast)\n            best_type = df[col].dtype.name\n        # Log the conversion performed.\n        if verbose and best_type is not None and best_type != str(col_type):\n            print(f\"Column '{col}' converted from {col_type} to {best_type}\")\n\n    if verbose:\n        end_mem = memory_usage_mb(df, deep=deep)\n        diff_mem = start_mem - end_mem\n        percent_mem = 100 * diff_mem / start_mem\n        print(f\"Memory usage decreased from\"\n              f\" {start_mem:.2f}MB to {end_mem:.2f}MB\"\n              f\" ({diff_mem:.2f}MB, {percent_mem:.2f}% reduction)\")\n```\nI'm delegating the datatype conversion to `pandas.to_numeric` that doesn't perform downcasting to `float16` (https://stackoverflow.com/questions/56640472/downcast-to-float16-in-pandas-to-numeric). It also simplifies the code and makes it more readable :)\n\nNOTE: I'm also converting columns of type `object` to Pandas categories.",
    "559214": "I removed float16 from the original version and had no problems - so I agree with the posters below who commented on float16.  I will look at the new version, but I still think I will ditch float16.  Float16 is too scary...",
    "559720": "In Santander competition this improvement could grant you jump 1-2k positions up the LB and silver medal :)",
    "561806": "Correct me if I'm wrong but doesn't NumPy convert  `float16` to `float32 `by default anyway. Wouldn't it be more efficient, as well as safer, to just use `float32`?",
    "565815": "Thanks for sharing. How have you defined memory\\_usage\\_mb?",
    "566225": "It's just a call to the Pandas `memory_usage` and a conversion to MB :)\n\n```\ndef memory_usage_mb(df, *args, **kwargs):\n    \"\"\"Dataframe memory usage in MB. \"\"\"\n    return df.memory_usage(*args, **kwargs).sum() / 1024**2\n```",
    "619322": "Thanks for sharing! I want to try it on my kernel~",
    "620729": "Thanks for sharing, I'll try it in a new competition !"
  },
  "source": "meta"
}