{
  "id": 285721,
  "title": "Memory Reduction",
  "url": "/competitions/g-research-crypto-forecasting/discussion/285721",
  "author_name": "",
  "post_date": "2021-11-05T23:15:01.870199300Z",
  "votes": 34,
  "comment_count": 3,
  "views": 0,
  "content": "<p>You can use <a href=\"https://www.kaggle.com/jpmiller/skmem\" target=\"_blank\">skmem</a> as a utility script if memory is a concern. I wrote it a while back because the scripts floating around Kaggle had strange effects on floats and didn't have the flexibility I wanted. You can drop the memory footprint of train.csv by half, which equates to ~1GB, if you're OK with losing float precision at around the 5th decimal place.</p>\n<pre><code>import datatable as dt\nfrom skmem import MemReducer\n\ndef drop_memory(path):\n    df = dt.fread(path).to_pandas()\n    mr = MemReducer()\n    floats = df.select_dtypes('float').columns.tolist()\n    df = mr.fit_transform(df, float_cols=floats)\n    return df\n\ndf_reduced = drop_memory('../input/g-research-crypto-forecasting/train.csv')\n</code></pre>",
  "messages": [
    {
      "id": "1572759",
      "postDate": "11/05/2021 23:15:01",
      "content": "<p>You can use <a href=\"https://www.kaggle.com/jpmiller/skmem\" target=\"_blank\">skmem</a> as a utility script if memory is a concern. I wrote it a while back because the scripts floating around Kaggle had strange effects on floats and didn't have the flexibility I wanted. You can drop the memory footprint of train.csv by half, which equates to ~1GB, if you're OK with losing float precision at around the 5th decimal place.</p>\n<pre><code>import datatable as dt\nfrom skmem import MemReducer\n\ndef drop_memory(path):\n    df = dt.fread(path).to_pandas()\n    mr = MemReducer()\n    floats = df.select_dtypes('float').columns.tolist()\n    df = mr.fit_transform(df, float_cols=floats)\n    return df\n\ndf_reduced = drop_memory('../input/g-research-crypto-forecasting/train.csv')\n</code></pre>",
      "rawMarkdown": "You can use [skmem](https://www.kaggle.com/jpmiller/skmem) as a utility script if memory is a concern. I wrote it a while back because the scripts floating around Kaggle had strange effects on floats and didn't have the flexibility I wanted. You can drop the memory footprint of train.csv by half, which equates to ~1GB, if you're OK with losing float precision at around the 5th decimal place.\n\n```\nimport datatable as dt\nfrom skmem import MemReducer\n\ndef drop_memory(path):\n    df = dt.fread(path).to_pandas()\n    mr = MemReducer()\n    floats = df.select_dtypes('float').columns.tolist()\n    df = mr.fit_transform(df, float_cols=floats)\n    return df\n\ndf_reduced = drop_memory('../input/g-research-crypto-forecasting/train.csv')\n```",
      "votes": null
    },
    {
      "id": "1653188",
      "postDate": "01/17/2022 10:15:29",
      "content": "<p>Nice share, I have been personally using this <a href=\"https://github.com/ianozsvald/dtype_diet\" target=\"_blank\">https://github.com/ianozsvald/dtype_diet</a></p>",
      "rawMarkdown": "Nice share, I have been personally using this https://github.com/ianozsvald/dtype_diet",
      "votes": null
    },
    {
      "id": "2488034",
      "postDate": "10/19/2023 01:08:59",
      "content": "<p>Great stuff. Thanks a lot for sharing. </p>",
      "rawMarkdown": "Great stuff. Thanks a lot for sharing.",
      "votes": null
    },
    {
      "id": "2926881",
      "postDate": "07/18/2024 05:12:07",
      "content": "<p>very nice trick</p>",
      "rawMarkdown": "very nice trick",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 1653188,
      "author_name": "neomatrix369",
      "author_url": "",
      "post_date": "01/17/2022 10:15:29",
      "content": "<p>Nice share, I have been personally using this <a href=\"https://github.com/ianozsvald/dtype_diet\" target=\"_blank\">https://github.com/ianozsvald/dtype_diet</a></p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 2488034,
      "author_name": "adegladius",
      "author_url": "",
      "post_date": "10/19/2023 01:08:59",
      "content": "<p>Great stuff. Thanks a lot for sharing. </p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 2926881,
      "author_name": "drlukasz",
      "author_url": "",
      "post_date": "07/18/2024 05:12:07",
      "content": "<p>very nice trick</p>",
      "votes": null,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "1572759": "You can use [skmem](https://www.kaggle.com/jpmiller/skmem) as a utility script if memory is a concern. I wrote it a while back because the scripts floating around Kaggle had strange effects on floats and didn't have the flexibility I wanted. You can drop the memory footprint of train.csv by half, which equates to ~1GB, if you're OK with losing float precision at around the 5th decimal place.\n\n```\nimport datatable as dt\nfrom skmem import MemReducer\n\ndef drop_memory(path):\n    df = dt.fread(path).to_pandas()\n    mr = MemReducer()\n    floats = df.select_dtypes('float').columns.tolist()\n    df = mr.fit_transform(df, float_cols=floats)\n    return df\n\ndf_reduced = drop_memory('../input/g-research-crypto-forecasting/train.csv')\n```",
    "1653188": "Nice share, I have been personally using this https://github.com/ianozsvald/dtype_diet",
    "2488034": "Great stuff. Thanks a lot for sharing.",
    "2926881": "very nice trick"
  },
  "source": "meta"
}