{"cells":[{"metadata":{},"cell_type":"markdown","source":"## Introduction\n\nReducing the memory usage by `reduce_mem_usage` appears a lot in kernels. However, as [alexeykupershtokh](https://www.kaggle.com/alexeykupershtokh) pointed out at the previous IEEE competition, it has some drawbacks such as the transformed values doesn't exactly match with the original values. This competition's dataset is not exceptional. If you check [this kernel](https://www.kaggle.com/kyakovlev/ashrae-data-minification) by Konstantin Yakovlev, values of some columns( `air_temperature`, `dew_temperature`, `sea_level_pressure`, `wind_speed`) in weather dataset changes after implementing the function.\n\nIn this notebook, I'm going to compare the two data minification methods between `reduce_mem_usage` and `reduce_mem_usage_sd`.\n\n---\n- The original alexeykupershtokh's kernel : https://www.kaggle.com/alexeykupershtokh/safe-memory-reduction\n- Konstantin Yakovlev's kernel : https://www.kaggle.com/kyakovlev/ashrae-data-minification\n\nI was motivated by above two kernels, please upvote those if this kernel is beneficial for you."},{"metadata":{},"cell_type":"markdown","source":"## Import library"},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"code","source":"import pandas as pd\nimport numpy as np\nimport os","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Functions for reducing memory"},{"metadata":{"trusted":true},"cell_type":"code","source":"# safe downcast\ndef sd(col, max_loss_limit=0.001, avg_loss_limit=0.001, na_loss_limit=0, n_uniq_loss_limit=0, fillna=0):\n    \"\"\"\n    max_loss_limit - don't allow any float to lose precision more than this value. Any values are ok for GBT algorithms as long as you don't unique values.\n                     See https://en.wikipedia.org/wiki/Half-precision_floating-point_format#Precision_limitations_on_decimal_values_in_[0,_1]\n    avg_loss_limit - same but calculates avg throughout the series.\n    na_loss_limit - not really useful.\n    n_uniq_loss_limit - very important parameter. If you have a float field with very high cardinality you can set this value to something like n_records * 0.01 in order to allow some field relaxing.\n    \"\"\"\n    is_float = str(col.dtypes)[:5] == 'float'\n    na_count = col.isna().sum()\n    n_uniq = col.nunique(dropna=False)\n    try_types = ['float16', 'float32']\n\n    if na_count <= na_loss_limit:\n        try_types = ['int8', 'int16', 'float16', 'int32', 'float32']\n\n    for type in try_types:\n        col_tmp = col\n\n        # float to int conversion => try to round to minimize casting error\n        if is_float and (str(type)[:3] == 'int'):\n            col_tmp = col_tmp.copy().fillna(fillna).round()\n\n        col_tmp = col_tmp.astype(type)\n        max_loss = (col_tmp - col).abs().max()\n        avg_loss = (col_tmp - col).abs().mean()\n        na_loss = np.abs(na_count - col_tmp.isna().sum())\n        n_uniq_loss = np.abs(n_uniq - col_tmp.nunique(dropna=False))\n\n        if max_loss <= max_loss_limit and avg_loss <= avg_loss_limit and na_loss <= na_loss_limit and n_uniq_loss <= n_uniq_loss_limit:\n            return col_tmp\n\n    # field can't be converted\n    return col\n\n\ndef reduce_mem_usage_sd(df, deep=True, verbose=False, obj_to_cat=False):\n    numerics = ['int16', 'uint16', 'int32', 'uint32', 'int64', 'uint64', 'float16', 'float32', 'float64']\n    start_mem = df.memory_usage(deep=deep).sum() / 1024 ** 2\n    for col in df.columns:\n        col_type = df[col].dtypes\n\n        # collect stats\n        na_count = df[col].isna().sum()\n        n_uniq = df[col].nunique(dropna=False)\n        \n        # numerics\n        if col_type in numerics:\n            df[col] = sd(df[col])\n\n        # strings\n        if (col_type == 'object') and obj_to_cat:\n            df[col] = df[col].astype('category')\n        \n        if verbose:\n            print(f'Column {col}: {col_type} -> {df[col].dtypes}, na_count={na_count}, n_uniq={n_uniq}')\n        new_na_count = df[col].isna().sum()\n        if (na_count != new_na_count):\n            print(f'Warning: column {col}, {col_type} -> {df[col].dtypes} lost na values. Before: {na_count}, after: {new_na_count}')\n        new_n_uniq = df[col].nunique(dropna=False)\n        if (n_uniq != new_n_uniq):\n            print(f'Warning: column {col}, {col_type} -> {df[col].dtypes} lost unique values. Before: {n_uniq}, after: {new_n_uniq}')\n\n    end_mem = df.memory_usage(deep=deep).sum() / 1024 ** 2\n    percent = 100 * (start_mem - end_mem) / start_mem\n    if verbose:\n        print('Mem. usage decreased from {:5.2f} Mb to {:5.2f} Mb ({:.1f}% reduction)'.format(start_mem, end_mem, percent))\n    return df\n\ndef reduce_mem_usage(df, verbose=True):\n    numerics = ['int16', 'int32', 'int64', 'float16', 'float32', 'float64']\n    start_mem = df.memory_usage(deep=True).sum() / 1024 ** 2 # just added \n    for col in df.columns:\n        col_type = df[col].dtypes\n        if col_type in numerics:\n            c_min = df[col].min()\n            c_max = df[col].max()\n            if str(col_type)[:3] == 'int':\n                if c_min > np.iinfo(np.int8).min and c_max < np.iinfo(np.int8).max:\n                    df[col] = df[col].astype(np.int8)\n                elif c_min > np.iinfo(np.int16).min and c_max < np.iinfo(np.int16).max:\n                    df[col] = df[col].astype(np.int16)\n                elif c_min > np.iinfo(np.int32).min and c_max < np.iinfo(np.int32).max:\n                    df[col] = df[col].astype(np.int32)\n                elif c_min > np.iinfo(np.int64).min and c_max < np.iinfo(np.int64).max:\n                    df[col] = df[col].astype(np.int64)  \n            else:\n                if c_min > np.finfo(np.float16).min and c_max < np.finfo(np.float16).max:\n                    df[col] = df[col].astype(np.float16)\n                elif c_min > np.finfo(np.float32).min and c_max < np.finfo(np.float32).max:\n                    df[col] = df[col].astype(np.float32)\n                else:\n                    df[col] = df[col].astype(np.float64)    \n    end_mem = df.memory_usage(deep=True).sum() / 1024 ** 2\n    percent = 100 * (start_mem - end_mem) / start_mem\n    print('Mem. usage decreased from {:5.2f} Mb to {:5.2f} Mb ({:.1f}% reduction)'.format(start_mem, end_mem, percent))\n    return df","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Load Dataset"},{"metadata":{"trusted":true},"cell_type":"code","source":"# Load Dataset\ndef load_dataset():\n    path = '../input/ashrae-energy-prediction/'\n    train = pd.read_csv(path+'train.csv')\n    test  = pd.read_csv(path+'test.csv')\n    weather_train = pd.read_csv(path+'weather_train.csv')\n    weather_test  = pd.read_csv(path+'weather_test.csv')\n    building_metadata = pd.read_csv(path+'building_metadata.csv')\n    return train, test, weather_train, weather_test, building_metadata\n\n%time raw_train, raw_test, raw_weather_train, raw_weather_test, raw_building_metadata = load_dataset()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Copy each table, to compare whether the transformed values macth with the original ones.\ntrain = raw_train.copy()\ntest  = raw_test.copy()\nweather_train = raw_weather_train.copy()\nweather_test  = raw_weather_test.copy()\nbuilding_metadata = raw_building_metadata.copy()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":" ## Compare `reduce_mem_usage` vs `reduce_mem_usage_sd`\n"},{"metadata":{},"cell_type":"markdown","source":"### reduce_mem_usage\n\nFirst of all, let's check the `reduce_mem_usage`. As you can see, there're some bad-transformations."},{"metadata":{"trusted":true},"cell_type":"code","source":"# Implement `reduce_mem_usage`\ntrain = reduce_mem_usage(train)\ntest = reduce_mem_usage(test)\nweather_train = reduce_mem_usage(weather_train)\nweather_test = reduce_mem_usage(weather_test)\nbuilding_metadata = reduce_mem_usage(building_metadata)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Check whether the transformed values exactly match with the original ones.\ntable_names = ['train','test','weather_train','weather_test', 'building_metadata']\nraw_tables = [ raw_train, raw_test, raw_weather_train, raw_weather_test, raw_building_metadata ]\ntransformed_tables = [ train, test, weather_train , weather_test, building_metadata]\n\n\nfor table_name, raw_table, transformed_table in zip( table_names, raw_tables, transformed_tables  ):\n    print('<' + table_name.upper() +'>')\n    for column in list(raw_table):\n        print(f\"Columns Name : {column}\")\n        if np.mean( raw_table[column] == transformed_table[column] ) == 1:\n            print(\"\\tperfectly matches with the originals\")\n        else:\n            print(\"\\tnot perfectly matches with the originals\")\n            print(\"\\tBad Transformation : {:.2f}%\".format(100 * (1 -np.mean( raw_table[column] == transformed_table[column] ) )))\n    print('='*100)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### reduce_mem_usage_sd\nNow it's time for `reduce_mem_usage_sd`. The result is not that different with the above one. But the target variable was badly transformed by `reduce_mem_usage`, but it's well transformed by this function."},{"metadata":{"trusted":true},"cell_type":"code","source":"# Copy each table, to compare whether the transformed values macth with the original ones.\ntrain = raw_train.copy()\ntest  = raw_test.copy()\nweather_train = raw_weather_train.copy()\nweather_test  = raw_weather_test.copy()\nbuilding_metadata = raw_building_metadata.copy()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Implement `reduce_mem_usage_sd`\ntrain = reduce_mem_usage_sd(train, verbose=True)\nprint('='*80)\ntest = reduce_mem_usage_sd(test, verbose=True)\nprint('='*80)\nweather_train = reduce_mem_usage_sd(weather_train, verbose=True)\nprint('='*80)\nweather_test = reduce_mem_usage_sd(weather_test, verbose=True)\nprint('='*80)\nraw_building_metadata = reduce_mem_usage_sd(building_metadata, verbose=True)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Check whether the transformed values exactly match with the original ones.\ntable_names = ['train','test','weather_train','weather_test', 'building_metadata']\nraw_tables = [ raw_train, raw_test, raw_weather_train, raw_weather_test, raw_building_metadata ]\ntransformed_tables = [ train, test, weather_train , weather_test, building_metadata]\n\n\nfor table_name, raw_table, transformed_table in zip( table_names, raw_tables, transformed_tables  ):\n    print('<' + table_name.upper() +'>')\n    for column in list(raw_table):\n        print(f\"Columns Name : {column}\")\n        if np.mean( raw_table[column] == transformed_table[column] ) == 1:\n            print(\"\\tperfectly matches with the originals\")\n        else:\n            print(\"\\tnot perfectly matches with the originals\")\n            print(\"\\tBad Transformation : {:.2f}%\".format(100 * (1 -np.mean( raw_table[column] == transformed_table[column] ) )))\n    print('='*100)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Conclusion"},{"metadata":{},"cell_type":"markdown","source":"The result shows us there's not huge difference between two methods. But the target variable is only intact with the `reduce_mem_usage_sd`.  And by lowering the memory size, we need to make it sure that some of the values could be badly transformed."}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":1}