{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# 新洋教育Kaggle零基础教学计划 - 数据挖掘项目\n# 预测建筑物的能源消耗\n\nASHRAE（American Society of Heating, Refrigerating and Air-Conditioning Engineers），中文名称“美国采暖、制冷与空调工程师学会”，于1894年在美国纽约成立，是由暖通空调（HVAC）工程师所组成的学会，全球拥有超过54,000名成员。协会及其成员专注于建筑系统、能源效率、室内空气质量、制冷和行业内的可持续性。通过调研、标准编写、出版和继续教育，ASHRAE发展至现在的规模。\n\n![image](https://www.shell.com/energy-and-innovation/the-energy-future/shell-energy-transition-report/_jcr_content/par/pageHeader/image.img.960.jpeg/1523515186785/cityscape-river-sunshine-hong-kong-china.jpeg?imformat=chrome&imwidth=1280)\n\n问：夏天给大楼降温需要多少钱？\n\n答：非常多！政府正在进行投资，以降低能源成本，减少排放。但是问题是，这些改进是否真的有效？\n\n在这次竞赛中，我们通过**预测冷水表、电表、热水表和蒸汽表的读数**来对这些节能投资进行更好的估计。数据来自近三年来1000栋建筑中的各表读数。大型投资者和金融机构将更倾向于在这一领域投资，以提高建筑能源使用效率。\n\n>**提示：**Code 和 Markdown 区域可通过 **Shift + Enter** 快捷键运行。此外，Markdown可以通过双击进入编辑模式。\n\n我们将这个notebook分为不同的步骤，你可以使用下面的链接来浏览此notebook。\n\n* [Step 1](#step1): 导入数据\n* [Step 2](#step2): 探索性数据分析\n* [Step 3](#step3): 数据预处理\n* [Step 4](#step4): LightGBM\n* [Step 5](#step5): 结果预测\n\n在该项目中包含了如下的问题：\n\n* [问题 1](#question1): 回顾课上内容并查阅资料，归纳总结数据预处理需要的步骤。\n* [问题 2](#question2): 思考此处为何要进行对数转换。\n* [问题 3](#question3): 查阅资料，总结LightGBM与CatBoost的差异。","metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true}},{"cell_type":"code","source":"# This Python 3 environment comes with many helpful analytics libraries installed\n# It is defined by the kaggle/python Docker image: https://github.com/kaggle/docker-python\n# For example, here's several helpful packages to load\n\nimport numpy as np # linear algebra\nimport pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\n\n# Input data files are available in the read-only \"../input/\" directory\n# For example, running this (by clicking run or pressing Shift+Enter) will list all files under the input directory\n\nimport os\nfor dirname, _, filenames in os.walk('/kaggle/input'):\n    for filename in filenames:\n        print(os.path.join(dirname, filename))\n\n# You can write up to 20GB to the current directory (/kaggle/working/) that gets preserved as output when you create a version using \"Save & Run All\" \n# You can also write temporary files to /kaggle/temp/, but they won't be saved outside of the current session","metadata":{"execution":{"iopub.status.busy":"2022-03-06T14:00:34.090141Z","iopub.execute_input":"2022-03-06T14:00:34.090747Z","iopub.status.idle":"2022-03-06T14:00:34.104636Z","shell.execute_reply.started":"2022-03-06T14:00:34.090668Z","shell.execute_reply":"2022-03-06T14:00:34.103515Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#导入必要的库\nimport pandas as pd\nimport numpy as np\nimport os\nimport gc\nimport copy\nimport warnings\n\nimport lightgbm as lgb\nfrom lightgbm import LGBMRegressor\nfrom sklearn.metrics import mean_squared_log_error\nfrom sklearn.model_selection import StratifiedKFold, KFold\nfrom tqdm.notebook import tqdm\nimport matplotlib.pyplot as plt\nimport seaborn as sns\n\nwarnings.filterwarnings('ignore')\npd.set_option(\"max_columns\", 500)\n%matplotlib inline","metadata":{"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","execution":{"iopub.status.busy":"2022-03-06T14:00:34.10661Z","iopub.execute_input":"2022-03-06T14:00:34.107163Z","iopub.status.idle":"2022-03-06T14:00:34.118746Z","shell.execute_reply.started":"2022-03-06T14:00:34.10711Z","shell.execute_reply":"2022-03-06T14:00:34.117495Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"<a id='step1'></a>\n# 1. 导入数据\n\nASHRAE给出的数据包含大量的特征数据，包括仪表读数，天气和建筑的数据等等。该问题为典型的监督学习问题。比赛举办方提供了6个csv文件，包括5个数据集与1个提交样本。其中数据集的字段含义为：\n\n`[train/test].csv`\n- building_id：建筑原数据的外键\n- meter : 仪表的id码, {0: 电表 , 1: 冷水表, 2: 蒸汽表, 3: 热水表}，不是每栋建筑都有全部类型的仪表\n- timestamp：读表的时间\n- meter_reading：目标变量, 用千瓦时（或等效值）表示的能耗。这是带有测量误差的真实数据，其中site0的电表读数出现问题，单位是千英热\n\n`building_meta.csv`\n- site_id: 天气文件的外键\n- building_id: training.csv对应的外键\n- primary_use: 基于EnergyStar property type definitions的建筑物活动的主要类别的指标（education, office…)\n- square_feet: 建筑物的总建筑面积\n- year_built: 建筑完成的年份\n- floor_count: 建筑物层数\n\n`weather_[train/test].csv`：气象站提供的气象数据,尽可能接近现场。\n- site_id: 天气文件的外键\n- air_temperature: 气温，单位为摄氏度\n- cloud_coverage: 天空中被云层覆盖的部分，单位为oktas\n- dew_temperature: 露点温度，单位为摄氏度\n- precip_depth_1_hr: 降水深度，单位为毫米\n- sea_level_pressure: 海平面压力，单位为毫巴/公顷\n- wind_direction: 风向，使用的是指南针方向（0-360）\n- wind_speed: 风速，单位为米每秒","metadata":{}},{"cell_type":"code","source":"train = pd.read_csv(\"../input/ashrae-energy-prediction/train.csv\", parse_dates=[\"timestamp\"])\ntest = pd.read_csv(\"../input/ashrae-energy-prediction/test.csv\", parse_dates=[\"timestamp\"])\nbuilding = pd.read_csv('../input/ashrae-energy-prediction/building_metadata.csv')\nweather_train = pd.read_csv('../input/ashrae-energy-prediction/weather_train.csv', parse_dates=[\"timestamp\"])\nweather_test = pd.read_csv(\"../input/ashrae-energy-prediction/weather_test.csv\", parse_dates=[\"timestamp\"])","metadata":{"execution":{"iopub.status.busy":"2022-03-06T14:00:34.120841Z","iopub.execute_input":"2022-03-06T14:00:34.12172Z","iopub.status.idle":"2022-03-06T14:01:11.45305Z","shell.execute_reply.started":"2022-03-06T14:00:34.121645Z","shell.execute_reply":"2022-03-06T14:01:11.452008Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"<a id='step2'></a>\n# 2. 探索性数据分析\n\n- 根据主办方提供的数据，\"meter\"代表仪表的类型，对应关系为 {0: 电表 , 1: 冷水表, 2: 蒸汽表, 3: 热水表}\n- 观察各个建筑的仪表读数，可以发现，其中一些建筑的读数在某些区间出现了持续为0的异常情况,也有读数异常的高的值\n- 可以通过修改`site`,`meter_type`,`primary_use`三个参数选择想要绘制的数据\n\n关联train表格和building表格用于作图","metadata":{}},{"cell_type":"markdown","source":"# 根据三种限制条件 地区，建筑用途，那种仪表来绘制符合条件的某些建筑的时间序列变化","metadata":{}},{"cell_type":"code","source":"train_plot = train.merge(building, on='building_id', how='left')\n\nsite = 0 #建筑物的地点\nmeter_type = 1 #仪表的类型\nprimary_use = 'Education' #建筑物的用途\n\nr = int(np.ceil(len(train_plot[(train_plot['site_id'] == site) & (train_plot['primary_use'] == primary_use) & (train_plot['meter'] == meter_type)]['building_id'].value_counts(dropna=False).index.to_list())/2))\nfig, axes = plt.subplots(r,2,figsize=(14, 36), dpi=100)\nfor i, building_id in enumerate(train_plot[(train_plot['site_id'] == site) & (train_plot['primary_use'] == primary_use) & (train_plot['meter'] == meter_type)]['building_id'].value_counts(dropna=False).index.to_list()):\n    train_plot[(train_plot['site_id'] == site) & (train_plot['primary_use'] == primary_use) & (train_plot['meter'] == meter_type) & \n               (train_plot['building_id'] == building_id )][['timestamp', 'meter_reading']].set_index('timestamp').resample('H').mean()['meter_reading'].plot(ax=axes[i%r][i//r], \n               alpha=0.8, label='By hour', color='tab:blue').set_ylabel('Mean meter reading', fontsize=13);\n    train_plot[(train_plot['site_id'] == site) & (train_plot['primary_use'] == primary_use) & (train_plot['meter'] == meter_type) & \n               (train_plot['building_id'] == building_id )][['timestamp', 'meter_reading']].set_index('timestamp').resample('D').mean()['meter_reading'].plot(ax=axes[i%r][i//r], \n               alpha=1, label='By day', color='tab:orange').set_xlabel('');\n    axes[i%r][i//r].legend();\n    axes[i%r][i//r].set_title('building_id: ' + str(building_id ), fontsize=13);\n    plt.subplots_adjust(hspace=0.45)\n    \ndel train_plot,fig,axes,r\ngc.collect();","metadata":{"execution":{"iopub.status.busy":"2022-03-06T14:01:11.456472Z","iopub.execute_input":"2022-03-06T14:01:11.457246Z","iopub.status.idle":"2022-03-06T14:01:54.610828Z","shell.execute_reply.started":"2022-03-06T14:01:11.45718Z","shell.execute_reply":"2022-03-06T14:01:54.609551Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"<a id='step3'></a>\n# 3. 数据预处理\n\n<a id='question1'></a>\n### __问题 1:__\n\n回顾课上内容并查阅资料，归纳总结数据预处理需要的步骤。\n\n__回答:__ \n- 缺失值处理（一般用众数、平均数、中位数，或者一些插值的方法处理，决策树类模型不用）\n- 数值转换（标准化，归一化，对于偏度大于0.75的数值特征/预测值进行log（x+1）处理，np.log1p())\n- 独热编码（处理类别特征，决策树类模型不用）\n- 时间戳处理（把时间戳分割开来）\n- bool改成0/1，\n","metadata":{}},{"cell_type":"markdown","source":"## 3.1 数据类型转换","metadata":{}},{"cell_type":"code","source":"def compress_dataframe(df):\n    '''将所有数据的类型都转换为数值型'''\n    result = df.copy()\n    for col in result.columns:\n        col_data = result[col]\n        dn = col_data.dtype.name\n        if dn == \"object\":\n            result[col] = pd.to_numeric(col_data.astype(\"category\").cat.codes, downcast=\"integer\")\n        elif dn == \"bool\":\n            result[col] = col_data.astype(\"int8\")\n        elif dn.startswith(\"int\") or (col_data.round() == col_data).all():\n            result[col] = pd.to_numeric(col_data, downcast=\"integer\")\n        else:\n            result[col] = pd.to_numeric(col_data, downcast='float')\n    return result","metadata":{"execution":{"iopub.status.busy":"2022-03-06T14:01:54.613558Z","iopub.execute_input":"2022-03-06T14:01:54.613941Z","iopub.status.idle":"2022-03-06T14:01:54.622272Z","shell.execute_reply.started":"2022-03-06T14:01:54.613879Z","shell.execute_reply":"2022-03-06T14:01:54.621017Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## 3.2 缺失值填充与特征扩展","metadata":{}},{"cell_type":"markdown","source":"1. 处理时间特征\n2. 转换时间序列\n3. 根据时区修正时间","metadata":{}},{"cell_type":"code","source":"def set_time(df):\n    df.timestamp = (df.timestamp - pd.to_datetime(\"2016-01-01\")).dt.total_seconds() // 3600\n    #这里将timestamp转换成了16年1月1日0点开始计算的小时数‘//’代表除法运算后取整\n    return df\n\n# 根据分析得出各个site来自哪个时区，来修正时间\n# https://www.kaggle.com/patrick0302/locate-cities-according-weather-temperature\nsite_GMT_offsets = [-5, 0, -7, -5, -8, 0, -5, -5, -5, -6, -7, -5, 0, -6, -5, -5]\n\n#转换天气数据表格中的时间,并填充缺失值\ndef weather_set_time(df,time_zone):\n    df.timestamp = (df.timestamp - pd.to_datetime(\"2016-01-01\")).dt.total_seconds() // 3600\n    \n    GMT_offset_map = {site: offset for site, offset in enumerate(site_GMT_offsets)}\n    df.timestamp = df.timestamp + df.site_id.map(GMT_offset_map)\n    #根据时区的不同，统一时间\n    site_dfs = []\n    for site_id in df.site_id.unique():\n        # 确保包括所有可能的小时数\n        site_df = df[df.site_id == site_id].set_index(\"timestamp\").reindex(time_zone)\n        site_df.site_id = site_id\n        for col in [c for c in site_df.columns if c != \"site_id\"]:\n            site_df[f\"had_{col}\"] = ~site_df[col].isna()\n            site_df[col] = site_df[col].interpolate(limit_direction='both', method='linear')\n            # 这里使用中位数来填充缺失值\n            site_df[col] = site_df[col].fillna(df[col].median())\n        site_dfs.append(site_df)\n    df = pd.concat(site_dfs).reset_index()  # make timestamp back into a regular column\n    for col in df.columns:\n        if df[col].isna().any(): df[f\"had_{col}\"] = ~df[col].isna()\n    #如果某列其中有缺失值，就增加一列新的特征：had_xxx 表示这一行在xxx这一列是否有记录\n    return df\n    \n#增加星期，月份，时间的特征\ndef _add_time_features(X):\n    return X.assign(tm_day_of_week=((X.timestamp // 24) % 7), tm_hour_of_day=(X.timestamp % 24))\n\nbuilding = compress_dataframe(building.fillna(-1)).set_index(\"building_id\")\n\ntrain = compress_dataframe(set_time(train)) # 处理标签数据\ntest = compress_dataframe(set_time(test)).set_index(\"row_id\") # 处理测试集标签数据\nweather_train = compress_dataframe(weather_set_time(weather_train,range(8784))).set_index([\"site_id\", \"timestamp\"])\nweather_test = compress_dataframe(weather_set_time(weather_test,range(8784,26304))).set_index([\"site_id\", \"timestamp\"])","metadata":{"execution":{"iopub.status.busy":"2022-03-06T14:01:54.624006Z","iopub.execute_input":"2022-03-06T14:01:54.624436Z","iopub.status.idle":"2022-03-06T14:02:08.200052Z","shell.execute_reply.started":"2022-03-06T14:01:54.62437Z","shell.execute_reply":"2022-03-06T14:02:08.198684Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## 3.3 关联数据","metadata":{}},{"cell_type":"code","source":"def combined_data(df,weather):\n    df = compress_dataframe(df.join(building, on=\"building_id\").join(weather,\n        on=[\"site_id\", \"timestamp\"]).fillna(-1))\n    return df.drop(columns=[\"meter_reading\"]),df.meter_reading","metadata":{"execution":{"iopub.status.busy":"2022-03-06T14:02:08.201703Z","iopub.execute_input":"2022-03-06T14:02:08.202113Z","iopub.status.idle":"2022-03-06T14:02:08.208371Z","shell.execute_reply.started":"2022-03-06T14:02:08.202055Z","shell.execute_reply":"2022-03-06T14:02:08.206932Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## 3.4 异常值处理 ","metadata":{}},{"cell_type":"code","source":"def make_is_bad_zero(Xy_subset, min_interval=48, summer_start=3000, summer_end=7500):\n    #夏天，3000/24=125，7500/24=312.5,第125天到第312.5天为夏天。\n\n    meter = Xy_subset.meter_id.iloc[0]\n    is_zero = Xy_subset.meter_reading == 0 #返回读数为0的电表的indices\n    if meter == 0:\n        #电表的度数不应该为0，所以电表（meter为0）读数为0的行从training dataframe中drop掉\n        return is_zero\n\n    transitions = (is_zero != is_zero.shift(1))#出现0和非0变化的位置\n    all_sequence_ids = transitions.cumsum()#到各位置出现的变化的和，是一个pd.Seires\n    ids = all_sequence_ids[is_zero].rename(\"ids\")#将其中读数为0的提取出来\n    if meter in [2, 3]:\n        # 蒸汽和热水有可能在夏天被关闭\n        keep = set(ids[(Xy_subset.timestamp < summer_start) |\n                       (Xy_subset.timestamp > summer_end)].unique())#不在夏天的indices\n        is_bad = ids.isin(keep) & (ids.map(ids.value_counts()) >= min_interval) \n        #将不在夏天却被关闭的蒸汽和热水表提取出来，至少被关闭了48小时以上的\n    elif meter == 1:\n        time_ids = ids.to_frame().join(Xy_subset.timestamp).set_index(\"timestamp\").ids#将ids和timestamp对应起来\n        is_bad = ids.map(ids.value_counts()) >= min_interval#关闭时间大于48小时的\n\n        # 冷水在冬天可能被关闭\n        jan_id = time_ids.get(0, False)#一月份的开始的id\n        dec_id = time_ids.get(8283, False)#十二月份开始的id\n        if (jan_id and dec_id and jan_id == time_ids.get(500, False) and\n                dec_id == time_ids.get(8783, False)):\n        #如果一月500小时和十二月500小时的读表都为0的话\n            is_bad = is_bad & (~(ids.isin(set([jan_id, dec_id]))))\n            #将这一部分的的行从is_bad中删除\n    else:\n        raise Exception(f\"Unexpected meter type: {meter}\")\n\n    result = is_zero.copy()\n    result.update(is_bad)\n    return result\n\ndef find_bad_zeros(X, y):\n    \"\"\"返回仅包含应该删除的行的Index\"\"\"\n    Xy = X.assign(meter_reading=y, meter_id=X.meter)\n    is_bad_zero = Xy.groupby([\"building_id\", \"meter\"]).apply(make_is_bad_zero)\n    return is_bad_zero[is_bad_zero].index.droplevel([0, 1])\n\ndef find_bad_sitezero(X):\n    \"\"\"返回Site 0 读数异常的行的index.\"\"\"\n    return X[(X.timestamp < 3378) & (X.site_id == 0) & (X.meter == 0)].index\n\ndef find_bad_building1099(X, y):\n    \"\"\"返回建筑1099的读数异常高的行的index .\"\"\"\n    return X[(X.building_id == 1099) & (X.meter == 2) & (y > 3e4)].index\n\ndef find_bad_rows(X, y):\n    return find_bad_zeros(X, y).union(find_bad_sitezero(X)).union(find_bad_building1099(X, y))","metadata":{"execution":{"iopub.status.busy":"2022-03-06T14:02:08.2103Z","iopub.execute_input":"2022-03-06T14:02:08.210852Z","iopub.status.idle":"2022-03-06T14:02:08.227618Z","shell.execute_reply.started":"2022-03-06T14:02:08.210771Z","shell.execute_reply":"2022-03-06T14:02:08.226881Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"X, y = combined_data(train,weather_train)\n\nbad_rows = find_bad_rows(X, y)\n#输出异常值的index\n# pd.Series(bad_rows.sort_values()).to_csv(\"rows_to_drop.csv\", header=False, index=False)\n\nX = X.drop(index=bad_rows)\ny = y.reindex_like(X)\n\nX = _add_time_features(X)\nX = compress_dataframe(X)\nX = X.drop(columns=\"timestamp\")  # drop掉原本的timestamp\n\ndel bad_rows,train,weather_train\ngc.collect();","metadata":{"execution":{"iopub.status.busy":"2022-03-06T14:02:08.228782Z","iopub.execute_input":"2022-03-06T14:02:08.229227Z","iopub.status.idle":"2022-03-06T14:02:56.486754Z","shell.execute_reply.started":"2022-03-06T14:02:08.229174Z","shell.execute_reply":"2022-03-06T14:02:56.485656Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## 3.5 评价函数\n\n由于需要预测连续值，因此需要采用回归模型。由于该项目是Kaggle赛题，测试集是使用均方根对数误差 RMSLE（Root Mean Squared Logarithmic Error, RMSLE)评测的，因此这里只能使用RMSLE。RMSLE的计算公式为：\n\n$${\\rm RMSLE} = \\sqrt{\\frac{1}{n} \\sum_{i=1}^n (\\log(p_i + 1) - \\log(a_i+1))^2 }$$\n\n其中\n- $n$（public/private）数据集中的样本总数,\n- $p_i$ 是目标的预测值\n- $a_i$ 第i个目标的真实值.\n- $\\log(x)$ 是自然对数\n\n我们只需要对目标值进行$y = \\log(y+1)$的变换，就可以使用常见的RMSE作为评价函数，我们使用numpy中的log1p就可以实现。\n\n注意：进行预测的时候需要使用$y = e^y-1$将目标值转换回去，可以使用 y = np.exp1m(y)。","metadata":{}},{"cell_type":"code","source":"#对目标值进行变换\ny = np.log1p(y)","metadata":{"execution":{"iopub.status.busy":"2022-03-06T14:02:56.488487Z","iopub.execute_input":"2022-03-06T14:02:56.489119Z","iopub.status.idle":"2022-03-06T14:02:56.836024Z","shell.execute_reply.started":"2022-03-06T14:02:56.4889Z","shell.execute_reply":"2022-03-06T14:02:56.83478Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"<a id='question2'></a>\n### __问题 2:__\n\n思考此处为何要进行对数转换。\n\n__回答:__ \n- 对数变换(log transformation)是特殊的一种数据变换方式，它可以将一类我们理论上未解决的模型问题转化为已经解决的问题,对于随着自变量的增加，因变量的方差也增大的模型，我们可以通过对数变换让方差恒定，即让波动相对稳定，误差服从独立同分布的正态分布，时间序列要求平稳","metadata":{}},{"cell_type":"markdown","source":"<a id='step4'></a>\n# 4. LightGBM\n## 4.1 模型参数\nLightGBM 主要调节的参数包括：\n- `learning_rate`：迭代步长,学习率；\n- `num_leaves`：LightGBM使用leaf-wise的算法，在调节树的复杂度时，使用num_leaves，较小导致欠拟合，较大导致过拟合；\n- `subsample`：0-1之间，控制每棵树随机采样的比例，减小这个参数的值，算法会更加保守，避免过拟合。但如果这个值设置得过小，可能会导致欠拟合；\n- `lambda_l2`：L2正则化系数，用来控制过拟合；\n- `num_trees`：迭代步数。","metadata":{}},{"cell_type":"code","source":"# params = {\n#     'task': 'train',\n#     'boosting_type': 'gbdt',  \n#     'objective': 'regression',  \n#     'metric': 'rmse',  \n#     'num_leaves': 40,  \n#     'subsample':0.8,\n#     'learning_rate': 0.03,  \n#     'verbose': 1,\n#     'lambda_l2':3\n# }\n\n# num_trees = 1000\n\n# #设置分类变量\n# categorical_features=['building_id', 'site_id', 'primary_use', 'had_air_temperature', 'had_cloud_coverage', \n#                       'had_dew_temperature', 'had_precip_depth_1_hr','had_sea_level_pressure', 'had_wind_direction',\n#                       'had_wind_speed', 'tm_day_of_week', 'tm_hour_of_day']\n","metadata":{"execution":{"iopub.status.busy":"2022-03-06T14:02:56.837567Z","iopub.execute_input":"2022-03-06T14:02:56.838031Z","iopub.status.idle":"2022-03-06T14:02:56.843015Z","shell.execute_reply.started":"2022-03-06T14:02:56.837958Z","shell.execute_reply":"2022-03-06T14:02:56.84152Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## 4.2 模型训练","metadata":{}},{"cell_type":"code","source":"# n_splits = 3\n\n# for val in X['meter'].unique():\n#     X1 = X[X['meter'] == val].drop(columns=['meter'])\n#     kf = StratifiedKFold(n_splits=n_splits,random_state=42)\n#     #使用StratifiedKFold，让指定列在每一个fold中的分布相同，这里设置分为3个fold\n#     t = 0\n#     for train_index, test_index in kf.split(X1, X1['tm_hour_of_day']):\n#         #让每个fold中['tm_hour_of_day']的分布相同\n#         train_features = X1.iloc[train_index]\n#         train_target = y[X1.iloc[train_index].index]\n        \n#         test_features = X1.iloc[test_index]\n#         test_target = y[X1.iloc[test_index].index]\n        \n#         d_train = lgb.Dataset(train_features, train_target, categorical_feature=categorical_features)\n#         d_eval = lgb.Dataset(test_features,test_target, categorical_feature=categorical_features)\n#         print(\"Building model meter :\",val,'fold:',t)        \n        \n#         md = lgb.train(params, d_train, num_boost_round=num_trees, valid_sets=(d_train, d_eval), \n#                        early_stopping_rounds=200,verbose_eval=20)\n#         md.save_model('lgb_val{}_fold{}.bin'.format(val,t))\n#         t += 1\n#     del X1  \n        \n# del d_train, d_eval, train_features, test_features, md\n# gc.collect();","metadata":{"execution":{"iopub.status.busy":"2022-03-06T14:02:56.844576Z","iopub.execute_input":"2022-03-06T14:02:56.845172Z","iopub.status.idle":"2022-03-06T14:02:56.857029Z","shell.execute_reply.started":"2022-03-06T14:02:56.845117Z","shell.execute_reply":"2022-03-06T14:02:56.856066Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"<a id='question3'></a>\n### __问题 3:__\n\n查阅资料，总结LightGBM与CatBoost的差异。\n\n__回答:__ \n1. CatBoost能够基于GPU实现快速学习\n2. CatBoost第一阶段采用梯度步长的无偏估计，第二阶段使用传统的GBDT方案执行。\n\n","metadata":{}},{"cell_type":"markdown","source":"<a id='step5'></a>\n# 5. 结果预测","metadata":{}},{"cell_type":"code","source":"# X = compress_dataframe(test.join(building, on=\"building_id\").join(weather_test, on=[\"site_id\", \"timestamp\"]).fillna(-1))\n# X = compress_dataframe(_add_time_features(X))\n# X = X.drop(columns=\"timestamp\")  # drop掉原本的timestamp\n\n# del test, weather_test\n# gc.collect();","metadata":{"execution":{"iopub.status.busy":"2022-03-06T14:02:56.858738Z","iopub.execute_input":"2022-03-06T14:02:56.859486Z","iopub.status.idle":"2022-03-06T14:02:56.877801Z","shell.execute_reply.started":"2022-03-06T14:02:56.85942Z","shell.execute_reply":"2022-03-06T14:02:56.876077Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# #输出预测结果\n# result = np.zeros(len(X))\n# for val in X['meter'].unique():\n#     ix = np.nonzero((X['meter'] == val).to_numpy())\n#     for i in tqdm(range(n_splits)):\n#         #加载刚才保存的模型\n#         model = lgb.Booster(model_file='lgb_val{}_fold{}.bin'.format(val, i))\n#         result[ix] += model.predict(X.iloc[ix].drop(columns=['meter']), num_iteration=model.best_iteration)/n_splits\n#         del model\n#         gc.collect();\n    \n# predictions = pd.DataFrame({\n#     \"row_id\": X.index,\n#     \"meter_reading\": np.clip(np.expm1(result), 0, None)\n# })\n\n# # float_format设置保留四位小数，减少文件大小，为文件上传节省时间\n# predictions.to_csv(\"./submission.csv\", index=False, float_format=\"%.4f\")","metadata":{"execution":{"iopub.status.busy":"2022-03-06T14:02:56.87994Z","iopub.execute_input":"2022-03-06T14:02:56.8806Z","iopub.status.idle":"2022-03-06T14:02:56.89267Z","shell.execute_reply.started":"2022-03-06T14:02:56.880276Z","shell.execute_reply":"2022-03-06T14:02:56.891215Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# 改写的作业","metadata":{}},{"cell_type":"code","source":"import lightgbm as lgb\n\nfrom sklearn.externals import joblib","metadata":{"execution":{"iopub.status.busy":"2022-03-06T14:02:56.894053Z","iopub.execute_input":"2022-03-06T14:02:56.894576Z","iopub.status.idle":"2022-03-06T14:02:56.911096Z","shell.execute_reply.started":"2022-03-06T14:02:56.894511Z","shell.execute_reply":"2022-03-06T14:02:56.91021Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"params = {\n    'task': 'train',\n    'boosting_type': 'gbdt',  \n    'objective': 'regression',  \n    'metric': 'rmse',  \n    'num_leaves': 40,  \n    'subsample':0.8,\n    'learning_rate': 0.03,  \n    'verbose': 1,\n    'lambda_l2':3\n}\n\nnum_trees = 1000\n\n#设置分类变量\ncategorical_features=['building_id', 'site_id', 'primary_use', 'had_air_temperature', 'had_cloud_coverage', \n                      'had_dew_temperature', 'had_precip_depth_1_hr','had_sea_level_pressure', 'had_wind_direction',\n                      'had_wind_speed', 'tm_day_of_week', 'tm_hour_of_day']","metadata":{"execution":{"iopub.status.busy":"2022-03-06T14:02:56.914024Z","iopub.execute_input":"2022-03-06T14:02:56.914583Z","iopub.status.idle":"2022-03-06T14:02:56.922857Z","shell.execute_reply.started":"2022-03-06T14:02:56.914514Z","shell.execute_reply":"2022-03-06T14:02:56.92191Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"n_splits = 3\nmodel = lgb.LGBMRegressor(**params)\nfor val in X['meter'].unique():\n    X1 = X[X['meter'] == val].drop(columns=['meter'])\n    kf = StratifiedKFold(n_splits=n_splits,random_state=42)\n    #使用StratifiedKFold，让指定列在每一个fold中的分布相同，这里设置分为3个fold\n    t = 0\n    for train_index, test_index in kf.split(X1, X1['tm_hour_of_day']):\n        #让每个fold中['tm_hour_of_day']的分布相同\n        train_features = X1.iloc[train_index]\n        train_target = y[X1.iloc[train_index].index]\n        \n        test_features = X1.iloc[test_index]\n        test_target = y[X1.iloc[test_index].index]\n        \n        model.fit(train_features,  train_target, \n            eval_set=[(test_features, test_target)],  \n            early_stopping_rounds=30,verbose=20)\n        joblib.dump(model, 'lgb_val{}_fold{}.pkl'.format(val,t))\n        t += 1\n\n        \n    print(lgb.plot_importance(model))\n    \n    \n    del X1  ","metadata":{"execution":{"iopub.status.busy":"2022-03-06T14:02:56.924527Z","iopub.execute_input":"2022-03-06T14:02:56.925051Z","iopub.status.idle":"2022-03-06T14:10:41.598067Z","shell.execute_reply.started":"2022-03-06T14:02:56.925001Z","shell.execute_reply":"2022-03-06T14:10:41.596575Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## 结果预测","metadata":{}},{"cell_type":"code","source":"X = compress_dataframe(test.join(building, on=\"building_id\").join(weather_test, on=[\"site_id\", \"timestamp\"]).fillna(-1))\nX = compress_dataframe(_add_time_features(X))\nX = X.drop(columns=\"timestamp\")  # drop掉原本的timestamp\n\n\nresult = np.zeros(len(X))\nfor val in X['meter'].unique():\n    ix = np.nonzero((X['meter'] == val).to_numpy())\n    for i in tqdm(range(n_splits)):\n        model = joblib.load('lgb_val{}_fold{}.pkl'.format(val, i))\n        result[ix] += model.predict(X.iloc[ix].drop(columns=['meter']))/n_splits\n        gc.collect();\n    \npredictions = pd.DataFrame({\n    \"row_id\": X.index,\n    \"meter_reading\": np.clip(np.expm1(result), 0, None)\n})\n\n# float_format设置保留四位小数，减少文件大小，为文件上传节省时间\npredictions.to_csv(\"./submission.csv\", index=False, float_format=\"%.4f\")","metadata":{"execution":{"iopub.status.busy":"2022-03-06T14:10:41.600444Z","iopub.execute_input":"2022-03-06T14:10:41.601448Z","iopub.status.idle":"2022-03-06T14:18:58.327373Z","shell.execute_reply.started":"2022-03-06T14:10:41.601365Z","shell.execute_reply":"2022-03-06T14:18:58.326156Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# 画特征图","metadata":{}},{"cell_type":"code","source":"print(lgb.plot_importance(model))","metadata":{"execution":{"iopub.status.busy":"2022-03-06T14:21:35.629619Z","iopub.execute_input":"2022-03-06T14:21:35.630406Z","iopub.status.idle":"2022-03-06T14:21:35.96432Z","shell.execute_reply.started":"2022-03-06T14:21:35.63032Z","shell.execute_reply":"2022-03-06T14:21:35.962925Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"","metadata":{},"execution_count":null,"outputs":[]}]}