{
  "id": 114029,
  "title": "Bear with me! with 64 Go in RAM,  why i am having a memory error with this dataset?",
  "url": "/competitions/ashrae-energy-prediction/discussion/114029",
  "author_name": "",
  "post_date": "2019-10-23T17:26:04.305264300Z",
  "votes": null,
  "comment_count": 3,
  "views": 0,
  "content": "<p>when I check memory usage I get : 1.7%  </p>\n\n<p>But when I try to merge <code>train.csv</code>  with<code>weathe_train.csv</code> I get memory error ! I have 64 Gb of RAM memory and 1T in disk ? </p>\n\n<p>I tried in different machine but I get the same problem. </p>\n\n<p>Is there anything wrong about my merge ? </p>\n\n<p>here is a code snippets for your references: \n```</p>\n\n<h1>check memory usage</h1>\n\n<p>def check_memory_usage(): \n    import os\n    import psutil\n    process = psutil.Process(os.getpid())\n    print('memory usage is: ',round(process.memory_percent(), 3),'%')</p>\n\n<h1>load data</h1>\n\n<p>train = reduce_mem_usage(pd.read_csv('train.csv'))\nweather_train = reduce_mem_usage(pd.read_csv('weather_train.csv'))\nbuilding_metadata = reduce_mem_usage(pd.read_csv('building_metadata.csv'))\nsample_submission= reduce_mem_usage(pd.read_csv('sample_submission.csv'))</p>\n\n<p>df = train.merge(building_metadata, on= ['building_id' ])</p>\n\n<p>check_memory_usage()</p>\n\n<h1>Output : 1.753 %</h1>\n\n<h1>until here everything is fine !</h1>\n\n<h1>take a chunk of df</h1>\n\n<p>train_df = df.sample(10**6, random_state=0)\ndel df</p>\n\n<h1>here is the problem</h1>\n\n<p>train_df.merge(weather_train, on= ['site_id'])</p>\n\n<p>```</p>\n\n<p>Error message : \n <img src=\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F3386911%2F41ffac4b18209944667b1212010e5761%2Ferror.png?generation=1571851925691945&amp;alt=media\" alt=\"\"></p>",
  "messages": [
    {
      "id": "655930",
      "postDate": "10/23/2019 17:26:04",
      "content": "<p>when I check memory usage I get : 1.7%  </p>\n\n<p>But when I try to merge <code>train.csv</code>  with<code>weathe_train.csv</code> I get memory error ! I have 64 Gb of RAM memory and 1T in disk ? </p>\n\n<p>I tried in different machine but I get the same problem. </p>\n\n<p>Is there anything wrong about my merge ? </p>\n\n<p>here is a code snippets for your references: \n```</p>\n\n<h1>check memory usage</h1>\n\n<p>def check_memory_usage(): \n    import os\n    import psutil\n    process = psutil.Process(os.getpid())\n    print('memory usage is: ',round(process.memory_percent(), 3),'%')</p>\n\n<h1>load data</h1>\n\n<p>train = reduce_mem_usage(pd.read_csv('train.csv'))\nweather_train = reduce_mem_usage(pd.read_csv('weather_train.csv'))\nbuilding_metadata = reduce_mem_usage(pd.read_csv('building_metadata.csv'))\nsample_submission= reduce_mem_usage(pd.read_csv('sample_submission.csv'))</p>\n\n<p>df = train.merge(building_metadata, on= ['building_id' ])</p>\n\n<p>check_memory_usage()</p>\n\n<h1>Output : 1.753 %</h1>\n\n<h1>until here everything is fine !</h1>\n\n<h1>take a chunk of df</h1>\n\n<p>train_df = df.sample(10**6, random_state=0)\ndel df</p>\n\n<h1>here is the problem</h1>\n\n<p>train_df.merge(weather_train, on= ['site_id'])</p>\n\n<p>```</p>\n\n<p>Error message : \n <img src=\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F3386911%2F41ffac4b18209944667b1212010e5761%2Ferror.png?generation=1571851925691945&amp;alt=media\" alt=\"\"></p>",
      "rawMarkdown": "when I check memory usage I get : 1.7%  \n\nBut when I try to merge ` train.csv`  with` weathe_train.csv` I get memory error ! I have 64 Gb of RAM memory and 1T in disk ? \n\nI tried in different machine but I get the same problem. \n\n\n Is there anything wrong about my merge ? \n\n\nhere is a code snippets for your references: \n```\n# check memory usage\ndef check_memory_usage(): \n    import os\n    import psutil\n    process = psutil.Process(os.getpid())\n    print('memory usage is: ',round(process.memory_percent(), 3),'%')\n\n# load data \ntrain = reduce_mem_usage(pd.read_csv('train.csv'))\nweather_train = reduce_mem_usage(pd.read_csv('weather_train.csv'))\nbuilding_metadata = reduce_mem_usage(pd.read_csv('building_metadata.csv'))\nsample_submission= reduce_mem_usage(pd.read_csv('sample_submission.csv'))\n\ndf = train.merge(building_metadata, on= ['building_id' ])\n \ncheck_memory_usage()\n# Output : 1.753 %\n# until here everything is fine !\n\n# take a chunk of df \ntrain_df = df.sample(10**6, random_state=0)\ndel df\n\n# here is the problem \ntrain_df.merge(weather_train, on= ['site_id'])\n \n```\n\nError message : \n ![](https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F3386911%2F41ffac4b18209944667b1212010e5761%2Ferror.png?generation=1571851925691945&amp;alt=media)",
      "votes": null
    },
    {
      "id": "655939",
      "postDate": "10/23/2019 17:37:12",
      "content": "<p>You need 256 GB as we have in our job 👀  Also 64 cores working 100% 😣 </p>\n\n<p>Merge with weather using site_id and timestamp.</p>",
      "rawMarkdown": "You need 256 GB as we have in our job 👀  Also 64 cores working 100% 😣 \n\n\nMerge with weather using site_id and timestamp.",
      "votes": null
    },
    {
      "id": "655976",
      "postDate": "10/23/2019 18:33:55",
      "content": "<p>You should change <code>train_df.merge(weather_train, on= ['site_id'])</code> to <code>train_df.merge(weather_train, on= ['site_id','timestamp'])</code> like Ivan mentioned. As it stands, your merge is creating a row for every match in <code>site_id</code> between train and the weather data, which means that each row of train will generate a row in the merge for every single weather timestamp associated with that <code>site_id</code>. That ends up massively reduplicating all of the different timestamps, which will cause issues on even very large RAM machines and isn't what you want. The fix lets you keep unique rows per <code>building_id</code>, <code>site_id</code>, and <code>timestamp</code>. </p>",
      "rawMarkdown": "You should change `train_df.merge(weather_train, on= ['site_id'])` to `train_df.merge(weather_train, on= ['site_id','timestamp'])` like Ivan mentioned. As it stands, your merge is creating a row for every match in `site_id` between train and the weather data, which means that each row of train will generate a row in the merge for every single weather timestamp associated with that `site_id`. That ends up massively reduplicating all of the different timestamps, which will cause issues on even very large RAM machines and isn't what you want. The fix lets you keep unique rows per `building_id`, `site_id`, and `timestamp`.",
      "votes": null
    },
    {
      "id": "656892",
      "postDate": "10/24/2019 18:16:36",
      "content": "<p>Thank you <a href=\"/drexpz\">@drexpz</a> and <a href=\"/aquatic\">@aquatic</a> 👍 </p>",
      "rawMarkdown": "Thank you @drexpz and @aquatic 👍",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 655939,
      "author_name": "drexpz",
      "author_url": "",
      "post_date": "10/23/2019 17:37:12",
      "content": "<p>You need 256 GB as we have in our job 👀  Also 64 cores working 100% 😣 </p>\n\n<p>Merge with weather using site_id and timestamp.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 655976,
      "author_name": "aquatic",
      "author_url": "",
      "post_date": "10/23/2019 18:33:55",
      "content": "<p>You should change <code>train_df.merge(weather_train, on= ['site_id'])</code> to <code>train_df.merge(weather_train, on= ['site_id','timestamp'])</code> like Ivan mentioned. As it stands, your merge is creating a row for every match in <code>site_id</code> between train and the weather data, which means that each row of train will generate a row in the merge for every single weather timestamp associated with that <code>site_id</code>. That ends up massively reduplicating all of the different timestamps, which will cause issues on even very large RAM machines and isn't what you want. The fix lets you keep unique rows per <code>building_id</code>, <code>site_id</code>, and <code>timestamp</code>. </p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 656892,
      "author_name": "smerllo",
      "author_url": "",
      "post_date": "10/24/2019 18:16:36",
      "content": "<p>Thank you <a href=\"/drexpz\">@drexpz</a> and <a href=\"/aquatic\">@aquatic</a> 👍 </p>",
      "votes": null,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "655930": "when I check memory usage I get : 1.7%  \n\nBut when I try to merge ` train.csv`  with` weathe_train.csv` I get memory error ! I have 64 Gb of RAM memory and 1T in disk ? \n\nI tried in different machine but I get the same problem. \n\n\n Is there anything wrong about my merge ? \n\n\nhere is a code snippets for your references: \n```\n# check memory usage\ndef check_memory_usage(): \n    import os\n    import psutil\n    process = psutil.Process(os.getpid())\n    print('memory usage is: ',round(process.memory_percent(), 3),'%')\n\n# load data \ntrain = reduce_mem_usage(pd.read_csv('train.csv'))\nweather_train = reduce_mem_usage(pd.read_csv('weather_train.csv'))\nbuilding_metadata = reduce_mem_usage(pd.read_csv('building_metadata.csv'))\nsample_submission= reduce_mem_usage(pd.read_csv('sample_submission.csv'))\n\ndf = train.merge(building_metadata, on= ['building_id' ])\n \ncheck_memory_usage()\n# Output : 1.753 %\n# until here everything is fine !\n\n# take a chunk of df \ntrain_df = df.sample(10**6, random_state=0)\ndel df\n\n# here is the problem \ntrain_df.merge(weather_train, on= ['site_id'])\n \n```\n\nError message : \n ![](https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F3386911%2F41ffac4b18209944667b1212010e5761%2Ferror.png?generation=1571851925691945&amp;alt=media)",
    "655939": "You need 256 GB as we have in our job 👀  Also 64 cores working 100% 😣 \n\n\nMerge with weather using site_id and timestamp.",
    "655976": "You should change `train_df.merge(weather_train, on= ['site_id'])` to `train_df.merge(weather_train, on= ['site_id','timestamp'])` like Ivan mentioned. As it stands, your merge is creating a row for every match in `site_id` between train and the weather data, which means that each row of train will generate a row in the merge for every single weather timestamp associated with that `site_id`. That ends up massively reduplicating all of the different timestamps, which will cause issues on even very large RAM machines and isn't what you want. The fix lets you keep unique rows per `building_id`, `site_id`, and `timestamp`.",
    "656892": "Thank you @drexpz and @aquatic 👍"
  },
  "source": "meta"
}