{"cells":[{"metadata":{},"cell_type":"markdown","source":"Merge operation would generate very large dataframes and need huge memory. [LiteMORT](https://github.com/closest-git/LiteMORT)  support implicit merging operation. That is,only send the merge information to LiteMORT. Then LiteMORT will generate every merged feature and its histogram  automatically. Modern GBDT only needs these histograms. So saved huge memory, while the accuracy remains the same. \n\nIn this case(to merge 25 features), implicit merging only need about 4G memory,  which is much less than 12G memory needed by standard merge operation.  It's easy to implicit merge hundreds of features if you want.\n\nThe following code showes the merge information to deal with weather and building dataframe. \n\n```python\nmerge_infos = [  \n\t{'on': ['site_id', 'timestamp'], 'dataset': weather_df, \"desc\": \"weather\"},  \n\t{'on': ['building_id'], 'dataset': building_merge_, \"desc\": \"building\", \"feat_info\": \t\t\tfeat_infos},  \n]\n```\n"},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"code","source":"from sklearn.preprocessing import LabelEncoder\nfrom sklearn.model_selection import StratifiedKFold, GroupKFold,KFold\nfrom sklearn.metrics import log_loss, mean_squared_error\nimport datetime\nimport time\nimport seaborn as sns; sns.set(style=\"ticks\", color_codes=True)\nimport matplotlib.pyplot as plt\nimport random\nimport gc\nimport pandas as pd\nimport numpy as np\nimport lightgbm as lgb\n# import shap as shap\nimport os\nimport sys\nimport pickle\nfrom tqdm import tqdm\nfrom pandas.api.types import is_datetime64_any_dtype as is_datetime\nfrom pandas.api.types import is_categorical_dtype\n%load_ext wurlitzer\n\n!ls '../input/ashrae-energy-prediction/'\n!ls '.'","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"LiteMORT is a new open source gradient boosting lib( https://github.com/closest-git/LiteMORT).  Install and import it.\n"},{"metadata":{"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","trusted":true},"cell_type":"code","source":"!pip install -i https://test.pypi.org/simple/  litemort==0.1.18\nimport litemort\nfrom litemort import * \nprint(litemort.__version__)\n#profile = LiteMORT_profile()\n#profile.Snapshot(\":\");          profile.Stat(\":\",\"::\")","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Some control flags. Switch these flags, you can compare the memory with and without implicit merging.  \nYou can also compare the performance between LiteMORT and lightGBM."},{"metadata":{"trusted":true},"cell_type":"code","source":"#isMORT = len(sys.argv)>1 and sys.argv[1] == \"mort\"\nisMORT = True    #Switch this flag to compare the performance between LiteMORT and lightGBM\nisImplicitMerge = True   #Switch to compare the memory usage \ngbm='MORT' if isMORT else 'LGB'\nuse_ucf=True\nnTargetMeter=4\ndata_root = '../input/ashrae-energy-prediction/'\nprint(f\"====== ImplicitMerge={isImplicitMerge} gbm={gbm} ======\\n\\n\")","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"UCF functions,  from https://www.kaggle.com/yamsam/new-ucf-starter-kernel"},{"metadata":{"trusted":true},"cell_type":"code","source":"def LoadUCF(data_root):\n    ucf_root = '../input/ashrae-ucf-spider-and-eda-full-test-labels'\n    ucf_leak_df = pd.read_pickle(f'{ucf_root}/site0.pkl')\n    ucf_leak_df['meter_reading'] = ucf_leak_df.meter_reading_scraped\n    ucf_leak_df.drop(['meter_reading_original', 'meter_reading_scraped'], axis=1, inplace=True)\n    ucf_leak_df.fillna(0, inplace=True)\n    ucf_leak_df.loc[ucf_leak_df.meter_reading < 0, 'meter_reading'] = 0\n    ucf_leak_df = ucf_leak_df[ucf_leak_df.timestamp.dt.year > 2016]\n    print(len(ucf_leak_df))\n    return ucf_leak_df\nLoadUCF(\"\")    #some testing code\n    \ndef ReplaceUCF():\n    print('ReplaceUCF......')  \n    leak_score = 0\n    leak_df = LoadUCF(data_root)\n    sample_submission.loc[sample_submission.meter_reading < 0, 'meter_reading'] = 0\n    for bid in leak_df.building_id.unique():\n        temp_df = leak_df[(leak_df.building_id == bid)]\n        for m in temp_df.meter.unique():\n            v0 = sample_submission.loc[(test_df.building_id == bid) & (test_df.meter == m), 'meter_reading'].values\n            v1 = temp_df[temp_df.meter == m].meter_reading.values\n            leak_score += mean_squared_error(np.log1p(v0), np.log1p(v1)) * len(v0)\n            sample_submission.loc[(test_df.building_id == bid) & (test_df.meter == m), 'meter_reading'] = temp_df[\n                temp_df.meter == m].meter_reading.values\n    print('UCF score = ', np.sqrt(leak_score / len(leak_df)))\n    sample_submission.to_csv('submission.csv', index=False, float_format='%.4f')\n    print(sample_submission.head(100),sample_submission.tail(100))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"This class for whether data and some lag features.\nJust modify lag_day to add more features.   \nFor example self.lag_day==[3,72] or self.lag_day==[3,24,72]"},{"metadata":{"trusted":true},"cell_type":"code","source":"class Whether(object):\n    def __init__(self, source, data_root,params=None):\n        self.source = source\n        self.data_root = data_root\n        self.lag_day=[3,72]     #,[]72\n        #self.pkl_path = f'{pickle_root}/Whether_{source}_[{self.lag_day}]_.pickle'\n        self.lag_feat_list=[]\n\n    def TimeAlignment(self,weather_df):   #https://www.kaggle.com/nz0722/aligned-timestamp-lgbm-by-meter-type\n        print(f\"TimeAlignment@{self.source}\\tdf{weather_df.shape}......\")\n        weather_key = ['site_id', 'timestamp']\n        temp_skeleton = weather_df[weather_key + ['air_temperature']].drop_duplicates(subset=weather_key).\\\n            sort_values(by=weather_key).copy()\n        temp_skeleton['temp_rank'] = temp_skeleton.groupby(['site_id', temp_skeleton.timestamp.dt.date])[\n            'air_temperature'].rank('average')\n        # create a dataframe of site_ids (0-16) x mean hour rank of temperature within day (0-23)\n        df_2d = temp_skeleton.groupby(['site_id', temp_skeleton.timestamp.dt.hour])['temp_rank'].mean().unstack(level=1)\n        # Subtract the columnID of temperature peak by 14, getting the timestamp alignment gap.\n        site_ids_offsets = pd.Series(df_2d.values.argmax(axis=1) - 14)\n        site_ids_offsets.index.name = 'site_id'\n\n        weather_df['offset'] = weather_df.site_id.map(site_ids_offsets)\n        weather_df['timestamp_aligned'] = (weather_df.timestamp - pd.to_timedelta(weather_df.offset, unit='H'))\n        weather_df['timestamp'] = weather_df['timestamp_aligned']\n        del weather_df['timestamp_aligned']        \n        return weather_df\n\n    def df(self):        \n        weather_df = pd.read_csv(f'{self.data_root}/weather_{self.source}.csv', parse_dates=['timestamp'],\n                    dtype={'site_id': np.uint8, 'air_temperature': np.float16,\n                           'cloud_coverage': np.float16, 'dew_temperature': np.float16,'precip_depth_1_hr': np.float16})\n        print(f\"{weather_df.shape}\\n{weather_df.isna().sum()}\")\n        weather_df = self.TimeAlignment(weather_df)\n        #w_sum = weather_df.groupby('site_id').apply(lambda group: group.isna().sum())\n        weather_df = weather_df.groupby('site_id').apply(lambda group: group.interpolate(limit_direction='both'))\n        w_sum = weather_df.groupby('site_id').apply(lambda group: group.isna().sum())\n        for days in self.lag_day:\n            f_list = self.add_lag_feature(weather_df, window=days)\n            self.lag_feat_list.extend(f_list)\n        print(f\"weather_{self.source}={weather_df.shape} features={weather_df.columns}\")  #weather_df.head()\n        return weather_df\n\n    def add_lag_feature(self,weather_df, window=3):\n        group_df = weather_df.groupby('site_id')\n        feat_list=[]\n        cols = ['air_temperature', 'cloud_coverage', 'dew_temperature', 'precip_depth_1_hr', 'sea_level_pressure',\n                'wind_direction', 'wind_speed']\n        rolled = group_df[cols].rolling(window=window, min_periods=0)\n        lag_mean = rolled.mean().reset_index().astype(np.float16)\n        lag_max = rolled.max().reset_index().astype(np.float16)\n        lag_min = rolled.min().reset_index().astype(np.float16)\n        lag_std = rolled.std().reset_index().astype(np.float16)\n        for col in cols:\n            feat_list.append(f'{col}_mean_lag{window}')\n            weather_df[f'{col}_mean_lag{window}'] = lag_mean[col]\n            if col=='air_temperature':\n                feat_list.append(f'{col}_max_lag{window}')\n                weather_df[f'{col}_max_lag{window}'] = lag_max[col]\n                feat_list.append(f'{col}_min_lag{window}')\n                weather_df[f'{col}_min_lag{window}'] = lag_min[col]\n                feat_list.append(f'{col}_std_lag{window}')\n                weather_df[f'{col}_std_lag{window}'] = lag_std[col]\n        return feat_list","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"If isImplicitMerge is False, the ASHRAE_data class would generate dataframe with all-features. The following is the standard merge code:\n\n\n```python\ntarget_train_df = target_train_df.merge(self.building_meta_df, on='building_id', how='left')\n                target_train_df = target_train_df.merge(self.weather_df, on=['site_id', 'timestamp'], how='left')\n```\n\n\n\n\nIf isImplicitMerge is True, no merging operation. Only record the merge infomation like this  \n\n\n```python\nself.merge_infos = [\n                    {'on': ['site_id', 'timestamp'], 'dataset': self.weather_df, \"desc\": \"weather\"},\n                    {'on': ['building_id'], 'dataset': self.building_merge_, \"desc\": \"building\",\n                     \"feat_info\": feat_infos},\n                ]\n```\n\n"},{"metadata":{"trusted":true},"cell_type":"code","source":"class ASHRAE_data(object):\n    def __init__(self, source,data_root,building_meta_df,weather_df):\n        self.category_cols = ['building_id', 'site_id', 'primary_use']  # , 'meter'\n\n\n        self.source = source\n        self.data_root = data_root\n        self.building_meta_df = building_meta_df\n        self.weather_df = weather_df\n        feats_whether =[e for e in list(self.weather_df.columns) if e not in ('site_id','offset', 'timestamp')]\n        self.feature_cols = ['square_feet', 'year_built'] + [\n            'hour', 'weekend',  # 'month' , 'dayofweek'\n            'building_median']+feats_whether\n        print(f\"ASHRAE_data_{self.source}......category={self.category_cols}\\nfeature_cols={self.feature_cols}......\")\n        #self.some_rows = 15000\n        self.some_rows = None\n        self.df_base = self.Load_Processing()\n        self.df_base_shape = self.df_base.shape\n        print(f\"ASHRAE_data_ df_base={self.df_base_shape}\")\n\n    def fit(cls, df):\n        pass\n\n    def data_X_y(self,target_meter):\n        feat_v0 = self.feature_cols + self.category_cols\n        feat_infos = {\"categorical\": self.category_cols}\n        train_df = self.df_base\n        print(f\"{self.source}_X_y@{target_meter} df_base={train_df.shape}......\")\n        pkl_path = f'./_ashrae_{self.source}_T{target_meter}_{self.some_rows}_{\"Mg\"if isImplicitMerge else \"\"}_.pickle'\n        self.merge_infos = []\n\n        target_train_df = train_df[train_df['meter'] == target_meter]\n        print(f\"target@{target_meter}={target_train_df.shape}\")\n        if isImplicitMerge:\n            building_site = self.building_meta_df[['building_id', 'site_id']]\n            target_train_df = target_train_df.merge(building_site, on='building_id', how='left')  # add 'site_id'\n            self.building_merge_ = self.building_meta_df[['building_id', 'primary_use', 'square_feet', 'year_built']]  # only need 3 col for merge\n            feat_v0 = feat_v0 + ['timestamp']\n            # self.weather_df = self.weather_df[:1100]\n            feat_v1 = list(set(feat_v0).intersection(set(list(self.weather_df.columns))))\n            # feat_v1 = ['site_id','timestamp','precip_depth_1_hr']       #测试需要\n            self.weather_df = self.weather_df[feat_v1]\n            self.merge_infos = [\n                {'on': ['site_id', 'timestamp'], 'dataset': self.weather_df, \"desc\": \"weather\"},\n                {'on': ['building_id'], 'dataset': self.building_merge_, \"desc\": \"building\",\n                 \"feat_info\": feat_infos},\n            ]\n        else:\n            target_train_df = target_train_df.merge(self.building_meta_df, on='building_id', how='left')\n            target_train_df = target_train_df.merge(self.weather_df, on=['site_id', 'timestamp'], how='left')\n        feat_v1 = list(set(feat_v0).intersection(set(list(target_train_df.columns))))\n        X_train = target_train_df[feat_v1]\n        print(f\"data_X__@{target_meter}={X_train.shape}\\toriginal={target_train_df.shape}\\tmerge={isImplicitMerge}\")\n        if (self.source == \"train\"):\n            y_train = target_train_df['meter_reading_log1p'].values\n        else:\n            y_train=None\n        del target_train_df\n        gc.collect()\n\n        return X_train, y_train\n\n    def Load_Processing(self):\n        df = pd.read_csv(f'{self.data_root}/{self.source}.csv', dtype={'building_id': np.uint16, 'meter': np.uint8},\n                         parse_dates=['timestamp'])\n        ucf_leak_df = LoadUCF(data_root)\n        #df = pd.concat([df, ucf_leak_df])\n        if self.source==\"train\":    #All electricity meter is 0 until May 20 for site_id == 0\n            df = df.query('not (building_id <= 104 & meter == 0 & timestamp <= \"2016-05-20\")')\n            if use_ucf:\n                ucf_year = [2017, 2018]  # ucf data year used in train\n                if True:  # del_2016:\n                    print('delete all buildings site0 in 2016')\n                    bids = ucf_leak_df.building_id.unique()\n                    df = df[df.building_id.isin(bids) == False]\n                ucf_leak_df = ucf_leak_df[ucf_leak_df.timestamp.dt.year.isin(ucf_year)]\n                df = pd.concat([df, ucf_leak_df])\n                df.reset_index(inplace=True)\n            if self.some_rows is not None:\n                df, _ = Mort_PickSamples(self.some_rows, df, None)\n                print(f'====== Some Samples@{self.source} ... data={df.shape}')\n        #df['date'] = df['timestamp'].dt.date\n        df[\"hour\"] = np.uint8(df[\"timestamp\"].dt.hour)\n        df[\"weekend\"] = np.uint8(df[\"timestamp\"].dt.weekday)    #cys\n        df[\"month\"] = np.uint8(df[\"timestamp\"].dt.month)\n        df[\"dayofweek\"] = np.uint8(df[\"timestamp\"].dt.dayofweek)\n        if self.source == \"train\":\n            df['meter_reading_log1p'] = np.log1p(df['meter_reading'])\n            df_group = df.groupby('building_id')['meter_reading_log1p']\n            self.building_mean = df_group.mean().astype(np.float16)\n            self.building_median = df_group.median().astype(np.float16)\n            self.building_min = df_group.min().astype(np.float16)\n            self.building_max = df_group.max().astype(np.float16)\n            self.building_std = df_group.std().astype(np.float16)\n            print(f\"building_mean={self.building_mean.head()}\")\n        else:        #for the testing dataframe,just use group infomation from training dataframe\n            self.building_mean=train_datas.building_mean\n            print(f\"test_datas.building_mean={self.building_mean.shape}\")\n            self.building_median=train_datas.building_median\n            self.building_min=train_datas.building_min\n            self.building_max=train_datas.building_max\n            self.building_std=train_datas.building_std\n\n        df['building_mean'] = df['building_id'].map(self.building_mean)\n        df['building_median'] = df['building_id'].map(self.building_median)\n        df['building_min'] = df['building_id'].map(self.building_min)\n        df['building_max'] = df['building_id'].map(self.building_max)\n        df['building_std'] = df['building_id'].map(self.building_std)\n        return df","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Load building_meta and whether datasets. Then use Whether class to generate many lag features."},{"metadata":{"trusted":true},"cell_type":"code","source":"def LoadBuilding(data_root):\n    building_meta_df = pd.read_csv(f'{data_root}/building_metadata.csv')\n    primary_use_list = building_meta_df['primary_use'].unique()\n    primary_use_dict = {key: value for value, key in enumerate(primary_use_list)}\n    print('primary_use_dict: ', primary_use_dict)\n    building_meta_df['primary_use'] = building_meta_df['primary_use'].map(primary_use_dict)\n    print(f\"{building_meta_df.shape}\\n{building_meta_df.head()}\")\n    return building_meta_df\n\nbuilding_meta_df=LoadBuilding(data_root)\nweather_test_df = Whether('test', data_root).df()\nprint(f\"weather_test_df={weather_test_df.shape} \")    #weather_test_df.head()\nweather_train_df = Whether('train', data_root).df()\nprint(f\"weather_train_df={weather_train_df.shape}\")   #weather_train_df.head()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"The core GBDT functons(LiteMORT/lightGBM) and its parameters"},{"metadata":{"trusted":true},"cell_type":"code","source":"#!pip install wurlitzer\nearly_stop = 20\nverbose_eval = 5\nmetric = 'l2'\n#num_rounds=1000, lr=0.05, bf=0.3\nnum_rounds = 1000;      lr = 0.05;          bf = 0.3\nparams = {'num_leaves': 31, 'n_estimators': num_rounds,\n              'objective': 'regression',\n              'max_bin': 256,\n              #               'max_depth': -1,\n              'learning_rate': lr,\n              \"boosting\": \"gbdt\",\n              \"bagging_freq\": 5,\n              \"bagging_fraction\": bf,\n              \"feature_fraction\": 0.9,  # STRANGE GBDT  why(\"bagging_freq\": 5 \"feature_fraction\": 0.9)!!!\n              \"metric\": metric, \"verbose_eval\": verbose_eval, 'n_jobs': 8, \"elitism\": 0,\"debug\":'1',\n              \"early_stopping_rounds\": early_stop, \"adaptive\": 'weight1', 'verbose': 0, 'min_data_in_leaf': 200,\n              #               \"verbosity\": -1,\n              #               'reg_alpha': 0.1,\n              #               'reg_lambda': 0.3\n              }\n#model = LiteMORT(params)\n\ndef fit_regressor(train, val,target_meter,fold, some_params, devices=(-1,), merge_info=None, cat_features=None):\n    t0=time.time()\n    X_train, y_train = train\n    X_valid, y_valid = val\n\n    device = devices[0]\n    if device == -1:        # use cpu\n        pass\n    else:        # use gpu\n        print(f'using gpu device_id {device}...')\n        params.update({'device': 'gpu', 'gpu_device_id': device})\n\n\n    if False:\n        col_y = pd.DataFrame(y_train)\n        col_X = X_train.reset_index(drop=True)\n        d_train = pd.concat([col_y, col_X], ignore_index=True, axis=1)\n        np.savetxt(\"E:/2/LightGBM-master/examples/regression/case_cys_.csv\", d_train, delimiter='\\t')\n        print(\"X_train={}, y_train={} d_train={}\".format(col_X.shape, col_y.shape, d_train.shape))\n\n    if isMORT:        \n        some_params['verbose']=666 if fold==0 else 0\n        merge_datas=[]\n        model = LiteMORT(some_params,merge_infos=merge_info)   # all train,eval,predict would use same merge infomation\n        model.fit(X_train, y_train, eval_set=[(X_valid, y_valid)], categorical_feature=cat_features)\n        fold_importance = None\n        log = \"\"\n    else:\n        some_params['verbose'] = 0\n        d_train = lgb.Dataset(X_train, label=y_train, categorical_feature=cat_features)\n        d_valid = lgb.Dataset(X_valid, label=y_valid, categorical_feature=cat_features)\n        watchlist = [d_train, d_valid]\n        print('training LGB: parmas=',params)\n        model = lgb.train(some_params,\n                          train_set=d_train,\n                          num_boost_round=num_rounds,\n                          valid_sets=watchlist,\n                          verbose_eval=verbose_eval,\n                          early_stopping_rounds=early_stop)\n        print('best_score', model.best_score)\n        log = {'train/mae': model.best_score['training'][metric], 'valid/mae': model.best_score['valid_1'][metric]}\n    # predictions\n    y_pred_valid = model.predict(X_valid, num_iteration=model.best_iteration)\n    oof_loss = mean_squared_error(y_valid, y_pred_valid)  # target is already in log scale\n    print(f'METER:{target_meter} Fold:{fold} MSE: {oof_loss:.4f} time={time.time() - t0:.5g}', flush=True)\n    #input(\"......\");   os._exit(-200)      #\n    return model, y_pred_valid, log","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Generate train dataframe"},{"metadata":{"trusted":true},"cell_type":"code","source":"train_datas = ASHRAE_data(\"train\",data_root,building_meta_df,weather_train_df)\nprint(train_datas.building_mean.shape)\nprint(train_datas.building_mean.head(5))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"5-folds training on 4 target meters"},{"metadata":{"trusted":true},"cell_type":"code","source":"folds = 8\nseed = 666\nshuffle = False\nkf = KFold(n_splits=folds, shuffle=shuffle, random_state=seed)\n#kf = StratifiedKFold(n_splits=folds, shuffle=True, random_state=seed)\ncat_features=None\nmeter_models=[]\n\nlosses=[]\n\nfor target_meter in range(nTargetMeter):\n    X_train, y_train = train_datas.data_X_y(target_meter)\n    #X_train = X_train[feat_fix]\n    y_valid_pred_total = np.zeros(X_train.shape[0])\n    gc.collect()\n    print(f'target_meter={target_meter} X_train={X_train.shape}\\nfeatures={X_train.columns}')\n    cat_features = train_datas.category_cols\n    # cat_features = ['building_id']\n    # [X_train.columns.get_loc(cat_col) for cat_col in train_datas.category_cols]\n    print('cat_features', cat_features)\n    if False :\n        feat_select = X_train.columns\n        feat_select = list(set(feat_select) - set(feat_fix))\n        params['early_stopping_rounds'] = 50        #不宜太大，掉到坑里\n        params['category_features'] = cat_features\n        MORT_feat_select_(X_train, y_train, feat_fix, feat_select,params,nMostSelect=(int)(len(feat_select)/2))\n        input(\"......MORT_feat_search......\")\n        sys.exit(-100)\n\n    t0=time.time()\n    fold = 0\n    models_ = []\n    for train_idx, valid_idx in kf.split(X_train, y_train):\n    #for (train_idx, valid_idx) in kf.split(X_train, X_train['building_id']):\n        train_data = X_train.iloc[train_idx, :], y_train[train_idx]\n        valid_data = X_train.iloc[valid_idx, :], y_train[valid_idx]\n        params['seed'] = seed\n        print(f'fold={fold} train={train_data[0].shape},valid={valid_data[0].shape}')\n        #     model, y_pred_valid, log = fit_cb(train_data, valid_data, cat_features=cat_features, devices=[0,])\n        model, y_pred_valid, log = fit_regressor(train_data, valid_data,target_meter,fold,some_params=params,merge_info=train_datas.merge_infos, cat_features=cat_features)\n        y_valid_pred_total[valid_idx] = y_pred_valid\n        models_.append(model)\n        del train_data,valid_data\n        gc.collect()\n        fold=fold+1\n        #break\n    meter_loss = mean_squared_error(y_train, y_valid_pred_total)\n    print(f'======METER:{target_meter} MSE: {meter_loss:.4f} time={time.time() - t0:.5g}\\n')\n    losses.append(meter_loss)\n    meter_models.append(models_)\n    #sns.distplot(y_train)\n    del X_train, y_train\n    gc.collect()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Load sample_submission and reduce its memomy usage."},{"metadata":{"trusted":true},"cell_type":"code","source":"def reduce_mem_usage(df, use_float16=False):\n    \"\"\" iterate through all the columns of a dataframe and modify the data type\n        to reduce memory usage.\n    \"\"\"\n    start_mem = df.memory_usage().sum() / 1024 ** 2\n    print('Memory usage of dataframe is {:.2f} MB'.format(start_mem))\n\n    for col in df.columns:\n        if is_datetime(df[col]) or is_categorical_dtype(df[col]):\n            # skip datetime type or categorical type\n            continue\n        col_type = df[col].dtype\n\n        if col_type != object:\n            c_min = df[col].min()\n            c_max = df[col].max()\n            if str(col_type)[:3] == 'int':\n                if c_min > np.iinfo(np.int8).min and c_max < np.iinfo(np.int8).max:\n                    df[col] = df[col].astype(np.int8)\n                elif c_min > np.iinfo(np.int16).min and c_max < np.iinfo(np.int16).max:\n                    df[col] = df[col].astype(np.int16)\n                elif c_min > np.iinfo(np.int32).min and c_max < np.iinfo(np.int32).max:\n                    df[col] = df[col].astype(np.int32)\n                elif c_min > np.iinfo(np.int64).min and c_max < np.iinfo(np.int64).max:\n                    df[col] = df[col].astype(np.int64)\n            else:\n                if use_float16 and c_min > np.finfo(np.float16).min and c_max < np.finfo(np.float16).max:\n                    df[col] = df[col].astype(np.float16)\n                elif c_min > np.finfo(np.float32).min and c_max < np.finfo(np.float32).max:\n                    df[col] = df[col].astype(np.float32)\n                else:\n                    df[col] = df[col].astype(np.float64)\n        else:\n            df[col] = df[col].astype('category')\n\n    end_mem = df.memory_usage().sum() / 1024 ** 2\n    print('Memory usage after optimization is: {:.2f} MB'.format(end_mem))\n    print('Decreased by {:.1f}%'.format(100 * (start_mem - end_mem) / start_mem))\n    return df\n\nsample_submission = pd.read_csv(os.path.join(data_root, 'sample_submission.csv'))\nreduce_mem_usage(sample_submission)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Generate testing dataframe.\nIf isImplicitMerge is False, the test_datas has all-features and is very large. If isImplicitMerge is True, no merging operation and its size is small. So we can use much larger batch size(*10)."},{"metadata":{"trusted":true},"cell_type":"code","source":"test_datas = ASHRAE_data(\"test\",data_root,building_meta_df,weather_test_df)\ndel train_datas\ngc.collect()\ntest_df = test_datas.df_base\n\ndef pred(X_test, models, batch_size=1000000):\n    if isMORT and isImplicitMerge:\n        batch_size=batch_size*10\n    iterations = (X_test.shape[0] + batch_size -1) // batch_size\n    nSamp = X_test.shape[0]\n    print(f'iterations={iterations}\\tnSamp={nSamp}\\tbatch_size={batch_size}' )\n\n    y_test_pred_total = np.zeros(X_test.shape[0])\n    for i, model in enumerate(models):\n        print(f'predicting {i}-th model')\n        for k in tqdm(range(iterations)):\n            n_1=min(nSamp,(k+1)*batch_size)\n            y_pred_test = model.predict(X_test[k*batch_size:n_1], num_iteration=model.best_iteration)\n            y_test_pred_total[k*batch_size:n_1] += y_pred_test\n\n    y_test_pred_total /= len(models)\n    return y_test_pred_total","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"For each target_meter, call each models to get predictions.  \nSince the testing dataset merged with different whether data. **Don't forget to call  model.MergeDataSets(test_datas.merge_infos)**\n\n\n\n\n\n```python\nif isMORT and isImplicitMerge:\n        for i, model in enumerate(meter_models[target_meter]):\n            model.MergeDataSets(test_datas.merge_infos,comment=\"_predict\")\n```\n\n"},{"metadata":{"trusted":true},"cell_type":"code","source":"for target_meter in range(nTargetMeter):\n    print(f'\\t target_meter={target_meter}......')\n    X_test,_ = test_datas.data_X_y(target_meter)\n    print(f'\\t target_meter={target_meter} X_test={X_test.shape}\\nfeatures={X_test.columns}')\n    gc.collect()\n    if isMORT and isImplicitMerge:\n        for i, model in enumerate(meter_models[target_meter]):\n            model.MergeDataSets(test_datas.merge_infos,comment=\"_predict\")\n    y_test0 = pred(X_test, meter_models[target_meter])\n    #sns.distplot(y_test0); plt.show()\n    sample_submission.loc[test_df['meter'] == target_meter, 'meter_reading'] = np.expm1(y_test0)\n    del X_test\n    gc.collect()\nif use_ucf:\n    pass\nelse:\n    submit_path = f'submission_{gbm}_.csv.gz'#f'./[{gbm}]_[{losses}].csv.gz'\n    sample_submission.to_csv(submit_path, index=False, float_format='%.4f',compression='gzip')\n    print(sample_submission.head(10),sample_submission.tail(10))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Try the data leak from ucf to get more."},{"metadata":{"trusted":true},"cell_type":"code","source":"if use_ucf:\n    ReplaceUCF()","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":1}