{
  "id": 114754,
  "title": "Very significant memory storage",
  "url": "/competitions/ashrae-energy-prediction/discussion/114754",
  "author_name": "Vitalii Mokin",
  "post_date": "2019-10-29T02:43:18.857000",
  "votes": 2,
  "comment_count": 0,
  "views": 0,
  "content": "<p>In addition to the already known and discussed simplifications and rounds, I propose to round up everything you can, including all  meteorological parameters, and squeeze in 256 integers (int8/uint8), which minimizes the size of the dataset while reducing the risk of overfitting (I got LB = 1.19 with <a href=\"https://www.kaggle.com/vbmokin/very-significant-\">https://www.kaggle.com/vbmokin/very-significant-</a> safe-memory-lightgbm) through the following transformations:</p>\n\n<h1>store values with precision 0.5</h1>\n\n<p><code>train['air_temperature'] = np.int8(round(2*train['air_temperature'],0))</code></p>\n\n<h1>store values with precision 0.1</h1>\n\n<p><code>train['cloud_coverage'] = np.uint8(round(10*train['cloud_coverage'],0))</code></p>\n\n<h1>store values with precision 0.2</h1>\n\n<p><code>train['dew_temperature'] = np.int8(round(5*train['dew_temperature'],0))</code></p>\n\n<h1>transform [-1,343] to [0,255]</h1>\n\n<p><code>train['precip_depth_1_hr'] = np.uint8(np.clip(round(train['precip_depth_1_hr'],0),0,255))</code></p>\n\n<h1>and others</h1>\n\n<p><code>train['square_feet'] = (10*np.log(train['square_feet'])).astype(np.uint8)\ntrain[\"building_id\"] = train['building_id'].astype(np.uint16)</code></p>\n\n<h1>after that</h1>\n\n<p><code>train = train.fillna(0)</code>\n<code>train['year_built'] = train['year_built'].astype(np.uint8)</code>\n<code>train['floor_count'] = train['floor_count'].astype(np.uint8)</code></p>\n\n<p>The result of this:\n<img src=\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F1798416%2F3fc9914721116fab5aa999bb588e2841%2FRezult_of_squeeze.png?generation=1572316994973355&amp;alt=media\" alt=\"\"></p>",
  "messages": [
    {
      "id": 660345,
      "postDate": "2019-10-29T02:43:18.857Z",
      "content": "<p>In addition to the already known and discussed simplifications and rounds, I propose to round up everything you can, including all  meteorological parameters, and squeeze in 256 integers (int8/uint8), which minimizes the size of the dataset while reducing the risk of overfitting (I got LB = 1.19 with <a href=\"https://www.kaggle.com/vbmokin/very-significant-\">https://www.kaggle.com/vbmokin/very-significant-</a> safe-memory-lightgbm) through the following transformations:</p>\n\n<h1>store values with precision 0.5</h1>\n\n<p><code>train['air_temperature'] = np.int8(round(2*train['air_temperature'],0))</code></p>\n\n<h1>store values with precision 0.1</h1>\n\n<p><code>train['cloud_coverage'] = np.uint8(round(10*train['cloud_coverage'],0))</code></p>\n\n<h1>store values with precision 0.2</h1>\n\n<p><code>train['dew_temperature'] = np.int8(round(5*train['dew_temperature'],0))</code></p>\n\n<h1>transform [-1,343] to [0,255]</h1>\n\n<p><code>train['precip_depth_1_hr'] = np.uint8(np.clip(round(train['precip_depth_1_hr'],0),0,255))</code></p>\n\n<h1>and others</h1>\n\n<p><code>train['square_feet'] = (10*np.log(train['square_feet'])).astype(np.uint8)\ntrain[\"building_id\"] = train['building_id'].astype(np.uint16)</code></p>\n\n<h1>after that</h1>\n\n<p><code>train = train.fillna(0)</code>\n<code>train['year_built'] = train['year_built'].astype(np.uint8)</code>\n<code>train['floor_count'] = train['floor_count'].astype(np.uint8)</code></p>\n\n<p>The result of this:\n<img src=\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F1798416%2F3fc9914721116fab5aa999bb588e2841%2FRezult_of_squeeze.png?generation=1572316994973355&amp;alt=media\" alt=\"\"></p>",
      "rawMarkdown": "In addition to the already known and discussed simplifications and rounds, I propose to round up everything you can, including all  meteorological parameters, and squeeze in 256 integers (int8/uint8), which minimizes the size of the dataset while reducing the risk of overfitting (I got LB = 1.19 with https://www.kaggle.com/vbmokin/very-significant- safe-memory-lightgbm) through the following transformations:\n\n# store values with precision 0.5\n`train['air_temperature'] = np.int8(round(2*train['air_temperature'],0)) `\n\n# store values with precision 0.1\n`train['cloud_coverage'] = np.uint8(round(10*train['cloud_coverage'],0)) `\n\n# store values with precision 0.2\n`train['dew_temperature'] = np.int8(round(5*train['dew_temperature'],0)) `\n\n# transform [-1,343] to [0,255]\n`train['precip_depth_1_hr'] = np.uint8(np.clip(round(train['precip_depth_1_hr'],0),0,255)) `\n\n# and others\n`train['square_feet'] = (10*np.log(train['square_feet'])).astype(np.uint8)\ntrain[\"building_id\"] = train['building_id'].astype(np.uint16)`\n\n# after that\n`train = train.fillna(0)`\n`train['year_built'] = train['year_built'].astype(np.uint8)`\n`train['floor_count'] = train['floor_count'].astype(np.uint8)`\n\nThe result of this:\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F1798416%2F3fc9914721116fab5aa999bb588e2841%2FRezult_of_squeeze.png?generation=1572316994973355&amp;alt=media)\n",
      "votes": 2
    }
  ],
  "comments": [],
  "raw_markdown_by_id": {
    "660345": "In addition to the already known and discussed simplifications and rounds, I propose to round up everything you can, including all  meteorological parameters, and squeeze in 256 integers (int8/uint8), which minimizes the size of the dataset while reducing the risk of overfitting (I got LB = 1.19 with https://www.kaggle.com/vbmokin/very-significant- safe-memory-lightgbm) through the following transformations:\n\n# store values with precision 0.5\n`train['air_temperature'] = np.int8(round(2*train['air_temperature'],0)) `\n\n# store values with precision 0.1\n`train['cloud_coverage'] = np.uint8(round(10*train['cloud_coverage'],0)) `\n\n# store values with precision 0.2\n`train['dew_temperature'] = np.int8(round(5*train['dew_temperature'],0)) `\n\n# transform [-1,343] to [0,255]\n`train['precip_depth_1_hr'] = np.uint8(np.clip(round(train['precip_depth_1_hr'],0),0,255)) `\n\n# and others\n`train['square_feet'] = (10*np.log(train['square_feet'])).astype(np.uint8)\ntrain[\"building_id\"] = train['building_id'].astype(np.uint16)`\n\n# after that\n`train = train.fillna(0)`\n`train['year_built'] = train['year_built'].astype(np.uint8)`\n`train['floor_count'] = train['floor_count'].astype(np.uint8)`\n\nThe result of this:\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F1798416%2F3fc9914721116fab5aa999bb588e2841%2FRezult_of_squeeze.png?generation=1572316994973355&amp;alt=media)\n"
  }
}