{
  "id": 112917,
  "title": "Safe memory optimization in constant time",
  "url": "/competitions/ashrae-energy-prediction/discussion/112917",
  "author_name": "",
  "post_date": "2019-10-16T06:25:07.734490900Z",
  "votes": 12,
  "comment_count": 5,
  "views": 0,
  "content": "<p>```\n%%time</p>\n\n<p>df_train_dtypes = {'building_id': np.uint16, 'meter': np.uint8, 'meter_reading': np.float32}\ndf_test_dtypes = {'building_id': np.uint16, 'meter': np.uint8}\ndf_building_metadata_dtypes = {'site_id': np.uint8, 'building_id': np.uint16, 'square_feet': np.int32, 'year_built': np.float32, 'floor_count': np.float32}\ndf_weather_dtypes = {'site_id': np.uint8, 'air_temperature': np.float32, 'cloud_coverage': np.float32, 'dew_temperature': np.float32,\n                     'precip_depth_1_hr': np.float32, 'sea_level_pressure': np.float32, 'wind_direction': np.float32, 'wind_speed': np.float32}</p>\n\n<p>df_train = pd.read_csv('../input/ashrae-energy-prediction/train.csv', dtype=df_train_dtypes)\ndf_test = pd.read_csv('../input/ashrae-energy-prediction/test.csv', dtype=df_train_dtypes)\ndf_building_metadata = pd.read_csv('../input/ashrae-energy-prediction/building_metadata.csv', dtype=df_building_metadata_dtypes)\ndf_weather_train = pd.read_csv('../input/ashrae-energy-prediction/weather_train.csv', dtype=df_weather_dtypes)\ndf_weather_test = pd.read_csv('../input/ashrae-energy-prediction/weather_test.csv', dtype=df_weather_dtypes)</p>\n\n<p>df_test.drop(columns=['row_id'], inplace=True)</p>\n\n<p>df_train = df_train.merge(df_building_metadata, on='building_id', how='left')\ndf_train = df_train.merge(df_weather_train, on=['site_id', 'timestamp'], how='left')\ndf_test = df_test.merge(df_building_metadata, on='building_id', how='left')\ndf_test = df_test.merge(df_weather_test, on=['site_id', 'timestamp'], how='left')</p>\n\n<p>del df_building_metadata, df_weather_train, df_weather_test\ngc.collect()</p>\n\n<p>print('Training Set Shape = {}'.format(df_train.shape))\nprint('Test Set Shape = {}'.format(df_test.shape))\nprint('Training Set Memory Usage = {:.2f} MB'.format(df_train.memory_usage().sum() / 1024**2))\nprint('Test Set Memory Usage = {:.2f} MB'.format(df_test.memory_usage().sum() / 1024**2))\n```</p>\n\n<p>This is a thing I used in IEEE a lot. If you already know what data types you are going to use, it is better to assign them while reading from csv file. You don't need to iterate over columns and downcast them one by one. This is the loading time and memory consumption of the data.</p>\n\n<p><code>\nTraining Set Shape = (20216100, 16)\nTest Set Shape = (41697600, 15)\nTraining Set Memory Usage = 1388.13 MB\nTest Set Memory Usage = 2704.08 MB\nCPU times: user 1min, sys: 20.8 s, total: 1min 21s\nWall time: 1min 20s\n</code></p>",
  "messages": [
    {
      "id": "650137",
      "postDate": "10/16/2019 06:25:07",
      "content": "<p>```\n%%time</p>\n\n<p>df_train_dtypes = {'building_id': np.uint16, 'meter': np.uint8, 'meter_reading': np.float32}\ndf_test_dtypes = {'building_id': np.uint16, 'meter': np.uint8}\ndf_building_metadata_dtypes = {'site_id': np.uint8, 'building_id': np.uint16, 'square_feet': np.int32, 'year_built': np.float32, 'floor_count': np.float32}\ndf_weather_dtypes = {'site_id': np.uint8, 'air_temperature': np.float32, 'cloud_coverage': np.float32, 'dew_temperature': np.float32,\n                     'precip_depth_1_hr': np.float32, 'sea_level_pressure': np.float32, 'wind_direction': np.float32, 'wind_speed': np.float32}</p>\n\n<p>df_train = pd.read_csv('../input/ashrae-energy-prediction/train.csv', dtype=df_train_dtypes)\ndf_test = pd.read_csv('../input/ashrae-energy-prediction/test.csv', dtype=df_train_dtypes)\ndf_building_metadata = pd.read_csv('../input/ashrae-energy-prediction/building_metadata.csv', dtype=df_building_metadata_dtypes)\ndf_weather_train = pd.read_csv('../input/ashrae-energy-prediction/weather_train.csv', dtype=df_weather_dtypes)\ndf_weather_test = pd.read_csv('../input/ashrae-energy-prediction/weather_test.csv', dtype=df_weather_dtypes)</p>\n\n<p>df_test.drop(columns=['row_id'], inplace=True)</p>\n\n<p>df_train = df_train.merge(df_building_metadata, on='building_id', how='left')\ndf_train = df_train.merge(df_weather_train, on=['site_id', 'timestamp'], how='left')\ndf_test = df_test.merge(df_building_metadata, on='building_id', how='left')\ndf_test = df_test.merge(df_weather_test, on=['site_id', 'timestamp'], how='left')</p>\n\n<p>del df_building_metadata, df_weather_train, df_weather_test\ngc.collect()</p>\n\n<p>print('Training Set Shape = {}'.format(df_train.shape))\nprint('Test Set Shape = {}'.format(df_test.shape))\nprint('Training Set Memory Usage = {:.2f} MB'.format(df_train.memory_usage().sum() / 1024**2))\nprint('Test Set Memory Usage = {:.2f} MB'.format(df_test.memory_usage().sum() / 1024**2))\n```</p>\n\n<p>This is a thing I used in IEEE a lot. If you already know what data types you are going to use, it is better to assign them while reading from csv file. You don't need to iterate over columns and downcast them one by one. This is the loading time and memory consumption of the data.</p>\n\n<p><code>\nTraining Set Shape = (20216100, 16)\nTest Set Shape = (41697600, 15)\nTraining Set Memory Usage = 1388.13 MB\nTest Set Memory Usage = 2704.08 MB\nCPU times: user 1min, sys: 20.8 s, total: 1min 21s\nWall time: 1min 20s\n</code></p>",
      "rawMarkdown": "```\n%%time\n\ndf_train_dtypes = {'building_id': np.uint16, 'meter': np.uint8, 'meter_reading': np.float32}\ndf_test_dtypes = {'building_id': np.uint16, 'meter': np.uint8}\ndf_building_metadata_dtypes = {'site_id': np.uint8, 'building_id': np.uint16, 'square_feet': np.int32, 'year_built': np.float32, 'floor_count': np.float32}\ndf_weather_dtypes = {'site_id': np.uint8, 'air_temperature': np.float32, 'cloud_coverage': np.float32, 'dew_temperature': np.float32,\n                     'precip_depth_1_hr': np.float32, 'sea_level_pressure': np.float32, 'wind_direction': np.float32, 'wind_speed': np.float32}\n\ndf_train = pd.read_csv('../input/ashrae-energy-prediction/train.csv', dtype=df_train_dtypes)\ndf_test = pd.read_csv('../input/ashrae-energy-prediction/test.csv', dtype=df_train_dtypes)\ndf_building_metadata = pd.read_csv('../input/ashrae-energy-prediction/building_metadata.csv', dtype=df_building_metadata_dtypes)\ndf_weather_train = pd.read_csv('../input/ashrae-energy-prediction/weather_train.csv', dtype=df_weather_dtypes)\ndf_weather_test = pd.read_csv('../input/ashrae-energy-prediction/weather_test.csv', dtype=df_weather_dtypes)\n\ndf_test.drop(columns=['row_id'], inplace=True)\n\ndf_train = df_train.merge(df_building_metadata, on='building_id', how='left')\ndf_train = df_train.merge(df_weather_train, on=['site_id', 'timestamp'], how='left')\ndf_test = df_test.merge(df_building_metadata, on='building_id', how='left')\ndf_test = df_test.merge(df_weather_test, on=['site_id', 'timestamp'], how='left')\n\ndel df_building_metadata, df_weather_train, df_weather_test\ngc.collect()\n\nprint('Training Set Shape = {}'.format(df_train.shape))\nprint('Test Set Shape = {}'.format(df_test.shape))\nprint('Training Set Memory Usage = {:.2f} MB'.format(df_train.memory_usage().sum() / 1024**2))\nprint('Test Set Memory Usage = {:.2f} MB'.format(df_test.memory_usage().sum() / 1024**2))\n```\n\nThis is a thing I used in IEEE a lot. If you already know what data types you are going to use, it is better to assign them while reading from csv file. You don't need to iterate over columns and downcast them one by one. This is the loading time and memory consumption of the data.\n\n```\nTraining Set Shape = (20216100, 16)\nTest Set Shape = (41697600, 15)\nTraining Set Memory Usage = 1388.13 MB\nTest Set Memory Usage = 2704.08 MB\nCPU times: user 1min, sys: 20.8 s, total: 1min 21s\nWall time: 1min 20s\n```",
      "votes": null
    },
    {
      "id": "650352",
      "postDate": "10/16/2019 09:44:44",
      "content": "<p>You can check my kernel <a href=\"https://www.kaggle.com/hamditarek/reducing-memory-size-for-great-energy-predictor\">Reducing Memory Size for Great Energy Predictor</a>\n<img src=\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F1673888%2Fed5250d250d8a9fd7358ef8ada73f8b9%2Fmr.jpg?generation=1571248028549446&amp;alt=media\" alt=\"\"></p>",
      "rawMarkdown": "You can check my kernel [Reducing Memory Size for Great Energy Predictor](https://www.kaggle.com/hamditarek/reducing-memory-size-for-great-energy-predictor)\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F1673888%2Fed5250d250d8a9fd7358ef8ada73f8b9%2Fmr.jpg?generation=1571248028549446&amp;alt=media)",
      "votes": null
    },
    {
      "id": "650814",
      "postDate": "10/16/2019 17:53:49",
      "content": "<p>Upvoted your kernel</p>",
      "rawMarkdown": "Upvoted your kernel",
      "votes": null
    },
    {
      "id": "651092",
      "postDate": "10/17/2019 03:24:23",
      "content": "<p>Thanks <a href=\"/gunesevitan\">@gunesevitan</a> , I hope it's useful</p>",
      "rawMarkdown": "Thanks @gunesevitan , I hope it's useful",
      "votes": null
    },
    {
      "id": "651436",
      "postDate": "10/17/2019 13:37:03",
      "content": "<p>Thank you :) </p>",
      "rawMarkdown": "Thank you :)",
      "votes": null
    },
    {
      "id": "663094",
      "postDate": "11/01/2019 12:50:58",
      "content": "<p>Thanks for this!</p>",
      "rawMarkdown": "Thanks for this!",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 650352,
      "author_name": "hamditarek",
      "author_url": "",
      "post_date": "10/16/2019 09:44:44",
      "content": "<p>You can check my kernel <a href=\"https://www.kaggle.com/hamditarek/reducing-memory-size-for-great-energy-predictor\">Reducing Memory Size for Great Energy Predictor</a>\n<img src=\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F1673888%2Fed5250d250d8a9fd7358ef8ada73f8b9%2Fmr.jpg?generation=1571248028549446&amp;alt=media\" alt=\"\"></p>",
      "votes": null,
      "replies": [
        {
          "id": 650814,
          "author_name": "gunesevitan",
          "author_url": "",
          "post_date": "10/16/2019 17:53:49",
          "content": "<p>Upvoted your kernel</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 651092,
          "author_name": "hamditarek",
          "author_url": "",
          "post_date": "10/17/2019 03:24:23",
          "content": "<p>Thanks <a href=\"/gunesevitan\">@gunesevitan</a> , I hope it's useful</p>",
          "votes": null,
          "replies": []
        }
      ]
    },
    {
      "id": 651436,
      "author_name": "xtelima",
      "author_url": "",
      "post_date": "10/17/2019 13:37:03",
      "content": "<p>Thank you :) </p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 663094,
      "author_name": "balzy88",
      "author_url": "",
      "post_date": "11/01/2019 12:50:58",
      "content": "<p>Thanks for this!</p>",
      "votes": null,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "650137": "```\n%%time\n\ndf_train_dtypes = {'building_id': np.uint16, 'meter': np.uint8, 'meter_reading': np.float32}\ndf_test_dtypes = {'building_id': np.uint16, 'meter': np.uint8}\ndf_building_metadata_dtypes = {'site_id': np.uint8, 'building_id': np.uint16, 'square_feet': np.int32, 'year_built': np.float32, 'floor_count': np.float32}\ndf_weather_dtypes = {'site_id': np.uint8, 'air_temperature': np.float32, 'cloud_coverage': np.float32, 'dew_temperature': np.float32,\n                     'precip_depth_1_hr': np.float32, 'sea_level_pressure': np.float32, 'wind_direction': np.float32, 'wind_speed': np.float32}\n\ndf_train = pd.read_csv('../input/ashrae-energy-prediction/train.csv', dtype=df_train_dtypes)\ndf_test = pd.read_csv('../input/ashrae-energy-prediction/test.csv', dtype=df_train_dtypes)\ndf_building_metadata = pd.read_csv('../input/ashrae-energy-prediction/building_metadata.csv', dtype=df_building_metadata_dtypes)\ndf_weather_train = pd.read_csv('../input/ashrae-energy-prediction/weather_train.csv', dtype=df_weather_dtypes)\ndf_weather_test = pd.read_csv('../input/ashrae-energy-prediction/weather_test.csv', dtype=df_weather_dtypes)\n\ndf_test.drop(columns=['row_id'], inplace=True)\n\ndf_train = df_train.merge(df_building_metadata, on='building_id', how='left')\ndf_train = df_train.merge(df_weather_train, on=['site_id', 'timestamp'], how='left')\ndf_test = df_test.merge(df_building_metadata, on='building_id', how='left')\ndf_test = df_test.merge(df_weather_test, on=['site_id', 'timestamp'], how='left')\n\ndel df_building_metadata, df_weather_train, df_weather_test\ngc.collect()\n\nprint('Training Set Shape = {}'.format(df_train.shape))\nprint('Test Set Shape = {}'.format(df_test.shape))\nprint('Training Set Memory Usage = {:.2f} MB'.format(df_train.memory_usage().sum() / 1024**2))\nprint('Test Set Memory Usage = {:.2f} MB'.format(df_test.memory_usage().sum() / 1024**2))\n```\n\nThis is a thing I used in IEEE a lot. If you already know what data types you are going to use, it is better to assign them while reading from csv file. You don't need to iterate over columns and downcast them one by one. This is the loading time and memory consumption of the data.\n\n```\nTraining Set Shape = (20216100, 16)\nTest Set Shape = (41697600, 15)\nTraining Set Memory Usage = 1388.13 MB\nTest Set Memory Usage = 2704.08 MB\nCPU times: user 1min, sys: 20.8 s, total: 1min 21s\nWall time: 1min 20s\n```",
    "650352": "You can check my kernel [Reducing Memory Size for Great Energy Predictor](https://www.kaggle.com/hamditarek/reducing-memory-size-for-great-energy-predictor)\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F1673888%2Fed5250d250d8a9fd7358ef8ada73f8b9%2Fmr.jpg?generation=1571248028549446&amp;alt=media)",
    "650814": "Upvoted your kernel",
    "651092": "Thanks @gunesevitan , I hope it's useful",
    "651436": "Thank you :)",
    "663094": "Thanks for this!"
  },
  "source": "meta"
}