{
  "id": 289138,
  "title": "Using reduce_mem_usage Significantly Reduces Score for Linear Regression",
  "url": "/competitions/g-research-crypto-forecasting/discussion/289138",
  "author_name": "",
  "post_date": "2021-11-19T10:38:20.961131800Z",
  "votes": 5,
  "comment_count": 4,
  "views": 0,
  "content": "<p>Hi, I've observed that using reduce_mem_usage reduces my score from 0.021 to around 0.013. I'm not sure why this happens. Does anyone have any idea?</p>\n<p>The code for reduce_mem_usage is as follows:</p>\n<pre><code># Memory saving function credit to https://www.kaggle.com/gemartin/load-data-reduce-memory-usage\ndef reduce_mem_usage(df):\n    \"\"\" iterate through all the columns of a dataframe and modify the data type\n        to reduce memory usage.\n    \"\"\"\n    start_mem = df.memory_usage().sum() / 1024**2\n    print('Memory usage of dataframe is {:.2f} MB'.format(start_mem))\n\n    for col in df.columns:\n        col_type = df[col].dtype.name\n\n        if col_type not in ['object', 'category', 'datetime64[ns, UTC]']:\n            c_min = df[col].min()\n            c_max = df[col].max()\n            if str(col_type)[:3] == 'int':\n                if c_min &gt; np.iinfo(np.int8).min and c_max &lt; np.iinfo(np.int8).max:\n                    df[col] = df[col].astype(np.int16)\n                elif c_min &gt; np.iinfo(np.int16).min and c_max &lt; np.iinfo(np.int16).max:\n                    df[col] = df[col].astype(np.int16)\n                elif c_min &gt; np.iinfo(np.int32).min and c_max &lt; np.iinfo(np.int32).max:\n                    df[col] = df[col].astype(np.int32)\n                elif c_min &gt; np.iinfo(np.int64).min and c_max &lt; np.iinfo(np.int64).max:\n                    df[col] = df[col].astype(np.int64)  \n            else:\n                if c_min &gt; np.finfo(np.float32).min and c_max &lt; np.finfo(np.float32).max:\n                    df[col] = df[col].astype(np.float32)\n                else:\n                    df[col] = df[col].astype(np.float64)\n\n    end_mem = df.memory_usage().sum() / 1024**2\n    print('Memory usage after optimization is: {:.2f} MB'.format(end_mem))\n    print('Decreased by {:.1f}%'.format(100 * (start_mem - end_mem) / start_mem))\n\n    return df\n</code></pre>",
  "messages": [
    {
      "id": "1588305",
      "postDate": "11/19/2021 10:38:20",
      "content": "<p>Hi, I've observed that using reduce_mem_usage reduces my score from 0.021 to around 0.013. I'm not sure why this happens. Does anyone have any idea?</p>\n<p>The code for reduce_mem_usage is as follows:</p>\n<pre><code># Memory saving function credit to https://www.kaggle.com/gemartin/load-data-reduce-memory-usage\ndef reduce_mem_usage(df):\n    \"\"\" iterate through all the columns of a dataframe and modify the data type\n        to reduce memory usage.\n    \"\"\"\n    start_mem = df.memory_usage().sum() / 1024**2\n    print('Memory usage of dataframe is {:.2f} MB'.format(start_mem))\n\n    for col in df.columns:\n        col_type = df[col].dtype.name\n\n        if col_type not in ['object', 'category', 'datetime64[ns, UTC]']:\n            c_min = df[col].min()\n            c_max = df[col].max()\n            if str(col_type)[:3] == 'int':\n                if c_min &gt; np.iinfo(np.int8).min and c_max &lt; np.iinfo(np.int8).max:\n                    df[col] = df[col].astype(np.int16)\n                elif c_min &gt; np.iinfo(np.int16).min and c_max &lt; np.iinfo(np.int16).max:\n                    df[col] = df[col].astype(np.int16)\n                elif c_min &gt; np.iinfo(np.int32).min and c_max &lt; np.iinfo(np.int32).max:\n                    df[col] = df[col].astype(np.int32)\n                elif c_min &gt; np.iinfo(np.int64).min and c_max &lt; np.iinfo(np.int64).max:\n                    df[col] = df[col].astype(np.int64)  \n            else:\n                if c_min &gt; np.finfo(np.float32).min and c_max &lt; np.finfo(np.float32).max:\n                    df[col] = df[col].astype(np.float32)\n                else:\n                    df[col] = df[col].astype(np.float64)\n\n    end_mem = df.memory_usage().sum() / 1024**2\n    print('Memory usage after optimization is: {:.2f} MB'.format(end_mem))\n    print('Decreased by {:.1f}%'.format(100 * (start_mem - end_mem) / start_mem))\n\n    return df\n</code></pre>",
      "rawMarkdown": "Hi, I've observed that using reduce_mem_usage reduces my score from 0.021 to around 0.013. I'm not sure why this happens. Does anyone have any idea?\n\nThe code for reduce_mem_usage is as follows:\n\n```\n# Memory saving function credit to https://www.kaggle.com/gemartin/load-data-reduce-memory-usage\ndef reduce_mem_usage(df):\n    \"\"\" iterate through all the columns of a dataframe and modify the data type\n        to reduce memory usage.\n    \"\"\"\n    start_mem = df.memory_usage().sum() / 1024**2\n    print('Memory usage of dataframe is {:.2f} MB'.format(start_mem))\n\n    for col in df.columns:\n        col_type = df[col].dtype.name\n\n        if col_type not in ['object', 'category', 'datetime64[ns, UTC]']:\n            c_min = df[col].min()\n            c_max = df[col].max()\n            if str(col_type)[:3] == 'int':\n                if c_min > np.iinfo(np.int8).min and c_max < np.iinfo(np.int8).max:\n                    df[col] = df[col].astype(np.int16)\n                elif c_min > np.iinfo(np.int16).min and c_max < np.iinfo(np.int16).max:\n                    df[col] = df[col].astype(np.int16)\n                elif c_min > np.iinfo(np.int32).min and c_max < np.iinfo(np.int32).max:\n                    df[col] = df[col].astype(np.int32)\n                elif c_min > np.iinfo(np.int64).min and c_max < np.iinfo(np.int64).max:\n                    df[col] = df[col].astype(np.int64)  \n            else:\n                if c_min > np.finfo(np.float32).min and c_max < np.finfo(np.float32).max:\n                    df[col] = df[col].astype(np.float32)\n                else:\n                    df[col] = df[col].astype(np.float64)\n\n    end_mem = df.memory_usage().sum() / 1024**2\n    print('Memory usage after optimization is: {:.2f} MB'.format(end_mem))\n    print('Decreased by {:.1f}%'.format(100 * (start_mem - end_mem) / start_mem))\n\n    return df\n```",
      "votes": null
    },
    {
      "id": "1588692",
      "postDate": "11/19/2021 15:02:46",
      "content": "<p>Hello, <a href=\"https://www.kaggle.com/mingyangyi\" target=\"_blank\">@mingyangyi</a> I saw this happen to me when I lost precision on some of the variables (my own reduce memory implementation), for example, a float 64 to float 32 this shouldn't be the case because the function should be able to reduce variables in order to not lose precision but maybe there is something there to look for, <br>\nI recommend completing detailed stats before and after the conversion to see what you lost.</p>",
      "rawMarkdown": "Hello, @mingyangyi I saw this happen to me when I lost precision on some of the variables (my own reduce memory implementation), for example, a float 64 to float 32 this shouldn't be the case because the function should be able to reduce variables in order to not lose precision but maybe there is something there to look for, \nI recommend completing detailed stats before and after the conversion to see what you lost.",
      "votes": null
    },
    {
      "id": "1591408",
      "postDate": "11/22/2021 09:56:36",
      "content": "<p>Hello, probably because your data have high precision. You must apply these memory reduction only on columns where the precision is not really important. Maybe the range of your data is tiny, on very high or very low values ? </p>",
      "rawMarkdown": "Hello, probably because your data have high precision. You must apply these memory reduction only on columns where the precision is not really important. Maybe the range of your data is tiny, on very high or very low values ?",
      "votes": null
    },
    {
      "id": "1662264",
      "postDate": "01/24/2022 06:38:15",
      "content": "<p>Hi, this function would change the data content .</p>",
      "rawMarkdown": "Hi, this function would change the data content .",
      "votes": null
    },
    {
      "id": "1664110",
      "postDate": "01/25/2022 17:48:06",
      "content": "<p>There are limits to the size of the number with the use of this routine, I always ensure that all pre processing is done before minimizing the data size.</p>",
      "rawMarkdown": "There are limits to the size of the number with the use of this routine, I always ensure that all pre processing is done before minimizing the data size.",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 1588692,
      "author_name": "cv13j0",
      "author_url": "",
      "post_date": "11/19/2021 15:02:46",
      "content": "<p>Hello, <a href=\"https://www.kaggle.com/mingyangyi\" target=\"_blank\">@mingyangyi</a> I saw this happen to me when I lost precision on some of the variables (my own reduce memory implementation), for example, a float 64 to float 32 this shouldn't be the case because the function should be able to reduce variables in order to not lose precision but maybe there is something there to look for, <br>\nI recommend completing detailed stats before and after the conversion to see what you lost.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 1591408,
      "author_name": "dsemmau",
      "author_url": "",
      "post_date": "11/22/2021 09:56:36",
      "content": "<p>Hello, probably because your data have high precision. You must apply these memory reduction only on columns where the precision is not really important. Maybe the range of your data is tiny, on very high or very low values ? </p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 1662264,
      "author_name": "yus002",
      "author_url": "",
      "post_date": "01/24/2022 06:38:15",
      "content": "<p>Hi, this function would change the data content .</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 1664110,
      "author_name": "jonnydossantos",
      "author_url": "",
      "post_date": "01/25/2022 17:48:06",
      "content": "<p>There are limits to the size of the number with the use of this routine, I always ensure that all pre processing is done before minimizing the data size.</p>",
      "votes": null,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "1588305": "Hi, I've observed that using reduce_mem_usage reduces my score from 0.021 to around 0.013. I'm not sure why this happens. Does anyone have any idea?\n\nThe code for reduce_mem_usage is as follows:\n\n```\n# Memory saving function credit to https://www.kaggle.com/gemartin/load-data-reduce-memory-usage\ndef reduce_mem_usage(df):\n    \"\"\" iterate through all the columns of a dataframe and modify the data type\n        to reduce memory usage.\n    \"\"\"\n    start_mem = df.memory_usage().sum() / 1024**2\n    print('Memory usage of dataframe is {:.2f} MB'.format(start_mem))\n\n    for col in df.columns:\n        col_type = df[col].dtype.name\n\n        if col_type not in ['object', 'category', 'datetime64[ns, UTC]']:\n            c_min = df[col].min()\n            c_max = df[col].max()\n            if str(col_type)[:3] == 'int':\n                if c_min > np.iinfo(np.int8).min and c_max < np.iinfo(np.int8).max:\n                    df[col] = df[col].astype(np.int16)\n                elif c_min > np.iinfo(np.int16).min and c_max < np.iinfo(np.int16).max:\n                    df[col] = df[col].astype(np.int16)\n                elif c_min > np.iinfo(np.int32).min and c_max < np.iinfo(np.int32).max:\n                    df[col] = df[col].astype(np.int32)\n                elif c_min > np.iinfo(np.int64).min and c_max < np.iinfo(np.int64).max:\n                    df[col] = df[col].astype(np.int64)  \n            else:\n                if c_min > np.finfo(np.float32).min and c_max < np.finfo(np.float32).max:\n                    df[col] = df[col].astype(np.float32)\n                else:\n                    df[col] = df[col].astype(np.float64)\n\n    end_mem = df.memory_usage().sum() / 1024**2\n    print('Memory usage after optimization is: {:.2f} MB'.format(end_mem))\n    print('Decreased by {:.1f}%'.format(100 * (start_mem - end_mem) / start_mem))\n\n    return df\n```",
    "1588692": "Hello, @mingyangyi I saw this happen to me when I lost precision on some of the variables (my own reduce memory implementation), for example, a float 64 to float 32 this shouldn't be the case because the function should be able to reduce variables in order to not lose precision but maybe there is something there to look for, \nI recommend completing detailed stats before and after the conversion to see what you lost.",
    "1591408": "Hello, probably because your data have high precision. You must apply these memory reduction only on columns where the precision is not really important. Maybe the range of your data is tiny, on very high or very low values ?",
    "1662264": "Hi, this function would change the data content .",
    "1664110": "There are limits to the size of the number with the use of this routine, I always ensure that all pre processing is done before minimizing the data size."
  },
  "source": "meta"
}