{"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"name":"python","version":"3.10.12","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"},"kaggle":{"accelerator":"none","dataSources":[{"sourceId":10445,"databundleVersionId":200747,"sourceType":"competition"}],"dockerImageVersionId":30558,"isInternetEnabled":true,"language":"python","sourceType":"notebook","isGpuEnabled":false}},"nbformat_minor":5,"nbformat":4,"cells":[{"cell_type":"markdown","source":"#
《Kaggle Top 1%方案精讲与实践》","metadata":{}},{"cell_type":"markdown","source":"# 课程说明：","metadata":{}},{"cell_type":"markdown","source":"  小伙伴好呀\\~欢迎来到《2021机器学习实战训练营》试学体验课！我是课程主讲老师，九天。 \n  本次体验课为期三天（12月8-10号），期间每晚8点在我的B站直播间公开直播，直播间地址:https://live.bilibili.com/22678166 \n  本期公开课的内容是在上一轮Kaggle竞赛公开课（《Elo Merchant Category Recommendation》）的基础上，进一步探讨如何进一步将排名提升至1%。在接下来的三天内容中，我们将进一步尝试多模型建模与优化、模型融合方法、特征优化与Trick优化等方法，从而在此前的结果基础上大幅提高模型预测准确率。 \n  当然，没有参与上一轮公开课的小伙伴也不用担心，本轮公开课将在开始时我们将快速回顾上一轮公开课内容，并将提供上一轮公开课最终完成的数据处理结果与相关代码，帮助大家无门槛进入本轮课程内容的学习中。当然，如果时间允许，也希望大家能够通过观看上一轮公开课的直播录屏，以巩固相关知识。上一轮公开课直播录屏地址：https://www.bilibili.com/video/BV1QU4y1u7Ph \n  课程资料/赛题数据/课程代码/付费课程信息，扫码添加客服“小可爱”回复【kaggle】即可领取哦~","metadata":{}},{"cell_type":"markdown","source":"
\"二维码\"","metadata":{}},{"cell_type":"markdown","source":"  另外，双十二年终大促持续进行中，十八周80+课时体系大课限时七折，扫码咨询小可爱回复“优惠”，还可领取额外折上折优惠，课程主页：https://appze9inzwc2314.pc.xiaoe-tech.com","metadata":{}},{"cell_type":"markdown","source":"---","metadata":{}},{"cell_type":"markdown","source":"#
【Kaggle】Elo Merchant Category Recommendation \n#
竞赛案例解析公开课 Part II","metadata":{}},{"cell_type":"markdown","source":"#
Day 5.集成学习与模型融合","metadata":{}},{"cell_type":"code","source":"cd /kaggle/input/elo-merchant-category-recommendation","metadata":{"execution":{"iopub.status.busy":"2023-10-21T13:31:01.813457Z","iopub.execute_input":"2023-10-21T13:31:01.814264Z","iopub.status.idle":"2023-10-21T13:31:01.844826Z","shell.execute_reply.started":"2023-10-21T13:31:01.814221Z","shell.execute_reply":"2023-10-21T13:31:01.843655Z"},"trusted":true},"execution_count":1,"outputs":[{"name":"stdout","text":"/kaggle/input/elo-merchant-category-recommendation\n","output_type":"stream"}]},{"cell_type":"code","source":"import numpy as np\nimport pandas as pd\nimport lightgbm as lgb\nfrom sklearn.model_selection import KFold\nfrom hyperopt import hp, fmin, tpe\nfrom numpy.random import RandomState\nfrom sklearn.metrics import mean_squared_error","metadata":{"execution":{"iopub.status.busy":"2023-10-21T13:31:06.360588Z","iopub.execute_input":"2023-10-21T13:31:06.360924Z","iopub.status.idle":"2023-10-21T13:31:09.718253Z","shell.execute_reply.started":"2023-10-21T13:31:06.360897Z","shell.execute_reply":"2023-10-21T13:31:09.717263Z"},"trusted":true},"execution_count":2,"outputs":[]},{"cell_type":"code","source":"train = pd.read_csv('train.csv')\ntest = pd.read_csv('test.csv')","metadata":{"execution":{"iopub.status.busy":"2023-10-21T13:31:23.439211Z","iopub.execute_input":"2023-10-21T13:31:23.439647Z","iopub.status.idle":"2023-10-21T13:31:23.933205Z","shell.execute_reply.started":"2023-10-21T13:31:23.439604Z","shell.execute_reply":"2023-10-21T13:31:23.932275Z"},"trusted":true},"execution_count":3,"outputs":[]},{"cell_type":"markdown","source":"---","metadata":{}},{"cell_type":"markdown","source":"##
一、单模训练策略（二）\n##
**Wrapper特征筛选+LightGBM建模+TPE调优**","metadata":{}},{"cell_type":"markdown","source":"  接下来我们进一步尝试Wrapper特征筛选+LightGBM建模+TPE调优的建模策略。相比随机森林这种老牌集成模型，LightGBM则算是Boosting家族中的新秀，LightGBM源自微软旗下的一个项目（Distributed Machine Learning Toolkit （DMKT）），同样是GBDT的一种实现方式，LightGBM支持高效率的并行训练，并且具有更快的训练速度、更低的内存消耗、更好的准确率、支持分布式可以快速处理海量数据等优点。","metadata":{}},{"cell_type":"markdown","source":"  首次使用LightGBM时需要对其单独进行安装，可以直接使用pip工具按照如下指令完成安装：","metadata":{}},{"cell_type":"markdown","source":"
pip install LightGBM","metadata":{}},{"cell_type":"markdown","source":"  安装完即可按照如下方式进行导入：","metadata":{}},{"cell_type":"code","source":"import lightgbm as lgb","metadata":{"execution":{"iopub.status.busy":"2023-10-21T13:31:28.850446Z","iopub.execute_input":"2023-10-21T13:31:28.850843Z","iopub.status.idle":"2023-10-21T13:31:28.855385Z","shell.execute_reply.started":"2023-10-21T13:31:28.850806Z","shell.execute_reply":"2023-10-21T13:31:28.85434Z"},"trusted":true},"execution_count":4,"outputs":[]},{"cell_type":"markdown","source":"  此外，本次建模过程中将使用hyperopt优化器进行超参数搜索，hyperopt优化器也是贝叶斯优化器的一种，可以进行连续变量和离散变量的搜索，目前支持的搜索算法包括随机搜索（random search）、模拟退火（simulated annealing）和TPE（Tree of Parzen Estimator）算法，相比网格搜索，hyperopt效率更快、精度更高。首次使用hyperopt库可使用pip进行安装：","metadata":{}},{"cell_type":"markdown","source":"
pip install hyperopt","metadata":{}},{"cell_type":"markdown","source":"安装完成后按照如下方式进行导入：","metadata":{}},{"cell_type":"code","source":"from hyperopt import hp, fmin, tpe","metadata":{"execution":{"iopub.status.busy":"2023-10-21T13:32:16.118126Z","iopub.execute_input":"2023-10-21T13:32:16.118505Z","iopub.status.idle":"2023-10-21T13:32:16.123377Z","shell.execute_reply.started":"2023-10-21T13:32:16.118476Z","shell.execute_reply":"2023-10-21T13:32:16.122353Z"},"trusted":true},"execution_count":8,"outputs":[]},{"cell_type":"markdown","source":"其中hp是参数空间创建函数，fmin是参数搜索函数，tpe则是一种基于贝叶斯过程的搜索策略。","metadata":{}},{"cell_type":"markdown","source":"  同时，在本次建模中，我们也将采用wrapper方法进行特征筛选，即根据模型输出结果来进行特征筛选，由于很多时候相关系数并不能很好的衡量特征实际对于标签的重要性，因此wrapper筛选的特征往往更加有效。当然，如果希望我们特征筛选结果更加具有可信度，则可以配合交叉验证过程对其进行筛选。","metadata":{}},{"cell_type":"markdown","source":"### 1.Wrapper特征筛选","metadata":{}},{"cell_type":"markdown","source":"  接下来是特征筛选过程，此处先择使用Wrapper方法进行特征筛选，通过带入全部数据训练一个LightGBM模型，然后通过观察特征重要性，选取最重要的300个特征。当然，为了进一步确保挑选过程的有效性，此处我们考虑使用交叉验证的方法来进行多轮验证。实际多轮验证特征重要性的过程也较为清晰，我们只需要记录每一轮特征重要性，并在最后进行简单汇总即可。我们可以通过定义如下函数完成该过程：","metadata":{}},{"cell_type":"code","source":"def feature_select_wrapper(train, test):\n \"\"\"\n lgm特征重要性筛选函数\n :param train:训练数据集\n :param test:测试数据集\n :return:特征筛选后的训练集和测试集\n \"\"\"\n \n # Part 1.划分特征名称，删除ID列和标签列\n print('feature_select_wrapper...')\n label = 'target'\n features = train.columns.tolist()\n features.remove('card_id')\n features.remove('target')\n\n # Step 2.配置lgb参数\n # 模型参数\n params_initial = {\n 'num_leaves': 31,\n 'learning_rate': 0.1,\n 'boosting': 'gbdt',\n 'min_child_samples': 20,\n 'bagging_seed': 2020,\n 'bagging_fraction': 0.7,\n 'bagging_freq': 1,\n 'feature_fraction': 0.7,\n 'max_depth': -1,\n 'metric': 'rmse',\n 'reg_alpha': 0,\n 'reg_lambda': 1,\n 'objective': 'regression'\n }\n # 控制参数\n # 提前验证迭代效果或停止\n ESR = 30\n # 迭代次数\n NBR = 10000\n # 打印间隔\n VBE = 50\n \n # Part 3.交叉验证过程\n # 实例化评估器\n kf = KFold(n_splits=5, random_state=2020, shuffle=True)\n # 创建空容器\n fse = pd.Series(0, index=features)\n \n for train_part_index, eval_index in kf.split(train[features], train[label]):\n # 封装训练数据集\n train_part = lgb.Dataset(train[features].loc[train_part_index],\n train[label].loc[train_part_index])\n # 封装验证数据集\n eval = lgb.Dataset(train[features].loc[eval_index],\n train[label].loc[eval_index])\n # 在训练集上进行训练，并同时进行验证\n bst = lgb.train(params_initial, train_part, num_boost_round=NBR,\n valid_sets=[train_part, eval],\n valid_names=['train', 'valid'],\n early_stopping_rounds=ESR, verbose_eval=VBE)\n # 输出特征重要性计算结果，并进行累加\n fse += pd.Series(bst.feature_importance(), features)\n \n # Part 4.选择最重要的300个特征\n feature_select = ['card_id'] + fse.sort_values(ascending=False).index.tolist()[:300]\n print('done')\n return train[feature_select + ['target']], test[feature_select]","metadata":{},"execution_count":84,"outputs":[]},{"cell_type":"code","source":"train_LGBM, test_LGBM = feature_select_wrapper(train, test)","metadata":{"tags":[],"collapsed":true,"jupyter":{"outputs_hidden":true}},"execution_count":8,"outputs":[{"name":"stdout","output_type":"stream","text":"feature_select_wrapper...\n"},{"name":"stderr","output_type":"stream","text":"C:\\Users\\vdmion\\anaconda3\\lib\\site-packages\\lightgbm\\engine.py:181: UserWarning: 'early_stopping_rounds' argument is deprecated and will be removed in a future release of LightGBM. Pass 'early_stopping()' callback via 'callbacks' argument instead.\n\n _log_warning(\"'early_stopping_rounds' argument is deprecated and will be removed in a future release of LightGBM. \"\n\nC:\\Users\\vdmion\\anaconda3\\lib\\site-packages\\lightgbm\\engine.py:239: UserWarning: 'verbose_eval' argument is deprecated and will be removed in a future release of LightGBM. Pass 'log_evaluation()' callback via 'callbacks' argument instead.\n\n _log_warning(\"'verbose_eval' argument is deprecated and will be removed in a future release of LightGBM. \"\n"},{"name":"stdout","output_type":"stream","text":"[LightGBM] [Warning] Auto-choosing col-wise multi-threading, the overhead of testing was 0.676599 seconds.\n\nYou can set `force_col_wise=true` to remove the overhead.\n\n[LightGBM] [Info] Total Bins 227016\n\n[LightGBM] [Info] Number of data points in the train set: 161533, number of used features: 1626\n\n[LightGBM] [Info] Start training from score -0.390986\n\nTraining until validation scores don't improve for 30 rounds\n\n[50]\ttrain's rmse: 3.43695\tvalid's rmse: 3.70629\n\nEarly stopping, best iteration is:\n\n[66]\ttrain's rmse: 3.39251\tvalid's rmse: 3.70281\n\n[LightGBM] [Warning] Auto-choosing col-wise multi-threading, the overhead of testing was 0.583384 seconds.\n\nYou can set `force_col_wise=true` to remove the overhead.\n\n[LightGBM] [Info] Total Bins 227122\n\n[LightGBM] [Info] Number of data points in the train set: 161533, number of used features: 1629\n\n[LightGBM] [Info] Start training from score -0.396781\n\nTraining until validation scores don't improve for 30 rounds\n\n[50]\ttrain's rmse: 3.45546\tvalid's rmse: 3.67176\n\n[100]\ttrain's rmse: 3.33017\tvalid's rmse: 3.67221\n\nEarly stopping, best iteration is:\n\n[82]\ttrain's rmse: 3.37387\tvalid's rmse: 3.66794\n\n[LightGBM] [Warning] Auto-choosing col-wise multi-threading, the overhead of testing was 0.677969 seconds.\n\nYou can set `force_col_wise=true` to remove the overhead.\n\n[LightGBM] [Info] Total Bins 227089\n\n[LightGBM] [Info] Number of data points in the train set: 161534, number of used features: 1631\n\n[LightGBM] [Info] Start training from score -0.390348\n\nTraining until validation scores don't improve for 30 rounds\n\n[50]\ttrain's rmse: 3.43208\tvalid's rmse: 3.72216\n\nEarly stopping, best iteration is:\n\n[68]\ttrain's rmse: 3.38524\tvalid's rmse: 3.71798\n\n[LightGBM] [Warning] Auto-choosing col-wise multi-threading, the overhead of testing was 0.687552 seconds.\n\nYou can set `force_col_wise=true` to remove the overhead.\n\n[LightGBM] [Info] Total Bins 227023\n\n[LightGBM] [Info] Number of data points in the train set: 161534, number of used features: 1624\n\n[LightGBM] [Info] Start training from score -0.391392\n\nTraining until validation scores don't improve for 30 rounds\n\n[50]\ttrain's rmse: 3.42022\tvalid's rmse: 3.7901\n\nEarly stopping, best iteration is:\n\n[59]\ttrain's rmse: 3.39133\tvalid's rmse: 3.78884\n\n[LightGBM] [Warning] Auto-choosing col-wise multi-threading, the overhead of testing was 0.689366 seconds.\n\nYou can set `force_col_wise=true` to remove the overhead.\n\n[LightGBM] [Info] Total Bins 227023\n\n[LightGBM] [Info] Number of data points in the train set: 161534, number of used features: 1625\n\n[LightGBM] [Info] Start training from score -0.398675\n\nTraining until validation scores don't improve for 30 rounds\n\n[50]\ttrain's rmse: 3.46321\tvalid's rmse: 3.60039\n\nEarly stopping, best iteration is:\n\n[61]\ttrain's rmse: 3.43002\tvalid's rmse: 3.59887\n\ndone\n"}]},{"cell_type":"markdown","source":"查看最终输出结果：","metadata":{}},{"cell_type":"code","source":"train_LGBM.shape","metadata":{},"execution_count":9,"outputs":[{"execution_count":9,"output_type":"execute_result","data":{"text/plain":["(201917, 302)"]},"metadata":{}}]},{"cell_type":"markdown","source":"  接下来，我们即可带入经过筛选的特征进行建模。","metadata":{}},{"cell_type":"markdown","source":"### 2.LightGBM模型训练与TPE参数优化","metadata":{}},{"cell_type":"markdown","source":"  接下来，我们进行LightGBM的模型训练过程，和此前的随机森林建模过程类似，我们需要在训练模型的过程同时进行超参数的搜索调优。为了能够更好的借助hyperopt进行超参数搜索，此处我们考虑使用LightGBM的原生算法库进行建模，并将整个算法建模流程封装在若干个函数 内执行。","metadata":{}},{"cell_type":"markdown","source":"- 参数回调函数","metadata":{}},{"cell_type":"markdown","source":"  首先对于lgb模型来说，并不是所有的超参数都需要进行搜索，为了防止多次实例化模型过程中部分超参数被设置成默认参数，此处我们首先需要创建一个参数回调函数，用于在后续多次实例化模型过程中反复申明这部分参数的固定取值：","metadata":{}},{"cell_type":"code","source":"def params_append(params):\n \"\"\"\n 动态回调参数函数，params视作字典\n :param params:lgb参数字典\n :return params:修正后的lgb参数字典\n \"\"\"\n params['feature_pre_filter'] = False\n params['objective'] = 'regression'\n params['metric'] = 'rmse'\n params['bagging_seed'] = 2020\n return params","metadata":{"execution":{"iopub.status.busy":"2023-10-21T13:33:46.049244Z","iopub.execute_input":"2023-10-21T13:33:46.049631Z","iopub.status.idle":"2023-10-21T13:33:46.05628Z","shell.execute_reply.started":"2023-10-21T13:33:46.049598Z","shell.execute_reply":"2023-10-21T13:33:46.055117Z"},"trusted":true},"execution_count":9,"outputs":[]},{"cell_type":"markdown","source":"- 模型训练与参数优化函数","metadata":{}},{"cell_type":"markdown","source":"  接下来就是更加复杂的模型训练与超参数调优的的过程。不同于sklearn内部的调参过程，此处由于涉及多个不同的库相互协同，外加本身lgb模型参数就较为复杂，因此整体模型训练与优化过程较为复杂，我们可以通过下述函数来执行该过程：","metadata":{}},{"cell_type":"code","source":"def param_hyperopt(train):\n \"\"\"\n 模型参数搜索与优化函数\n :param train:训练数据集\n :return params_best:lgb最优参数\n \"\"\"\n # Part 1.划分特征名称，删除ID列和标签列\n label = 'target'\n features = train.columns.tolist()\n features.remove('card_id')\n features.remove('target')\n \n # Part 2.封装训练数据\n train_data = lgb.Dataset(train[features], train[label])\n \n # Part 3.内部函数，输入模型超参数损失值输出函数\n def hyperopt_objective(params):\n \"\"\"\n 输入超参数，输出对应损失值\n :param params:\n :return:最小rmse\n \"\"\"\n # 创建参数集\n params = params_append(params)\n print(params)\n \n # 借助lgb的cv过程，输出某一组超参数下损失值的最小值\n res = lgb.cv(params, train_data, 1000,\n nfold=2,\n stratified=False,\n shuffle=True,\n metrics='rmse',\n early_stopping_rounds=20,\n verbose_eval=False,\n show_stdv=False,\n seed=2020)\n return min(res['rmse-mean']) # res是个字典\n\n # Part 4.lgb超参数空间\n params_space = {\n 'learning_rate': hp.uniform('learning_rate', 1e-2, 5e-1),\n 'bagging_fraction': hp.uniform('bagging_fraction', 0.5, 1),\n 'feature_fraction': hp.uniform('feature_fraction', 0.5, 1),\n 'num_leaves': hp.choice('num_leaves', list(range(10, 300, 10))),\n 'reg_alpha': hp.randint('reg_alpha', 0, 10),\n 'reg_lambda': hp.uniform('reg_lambda', 0, 10),\n 'bagging_freq': hp.randint('bagging_freq', 1, 10),\n 'min_child_samples': hp.choice('min_child_samples', list(range(1, 30, 5)))\n }\n \n # Part 5.TPE超参数搜索\n params_best = fmin(\n hyperopt_objective,\n space=params_space,\n algo=tpe.suggest,\n max_evals=30,\n rstate=RandomState(2020))\n \n # 返回最佳参数\n return params_best","metadata":{"execution":{"iopub.status.busy":"2023-10-21T13:35:40.838912Z","iopub.execute_input":"2023-10-21T13:35:40.839318Z","iopub.status.idle":"2023-10-21T13:35:40.849703Z","shell.execute_reply.started":"2023-10-21T13:35:40.839287Z","shell.execute_reply":"2023-10-21T13:35:40.848484Z"},"trusted":true},"execution_count":10,"outputs":[]},{"cell_type":"markdown","source":"接下来我们带入训练数据，测试函数性能：","metadata":{}},{"cell_type":"code","source":"cd /kaggle/input/elo-merchant-category-recommendation","metadata":{"execution":{"iopub.status.busy":"2023-10-21T13:45:14.199205Z","iopub.execute_input":"2023-10-21T13:45:14.199623Z","iopub.status.idle":"2023-10-21T13:45:14.207487Z","shell.execute_reply.started":"2023-10-21T13:45:14.19959Z","shell.execute_reply":"2023-10-21T13:45:14.206168Z"},"trusted":true},"execution_count":26,"outputs":[{"name":"stdout","text":"/kaggle/input/elo-merchant-category-recommendation\n","output_type":"stream"}]},{"cell_type":"code","source":"!numpy.--version__\n# !pip show hyperopt\n# !pip install --force-reinstall --no-deps numpy==1.25.2\n# !pip install --upgrade pip\n# !pip install hyperopt==0.2.5","metadata":{"execution":{"iopub.status.busy":"2023-10-21T13:49:28.914295Z","iopub.execute_input":"2023-10-21T13:49:28.914684Z","iopub.status.idle":"2023-10-21T13:49:29.920499Z","shell.execute_reply.started":"2023-10-21T13:49:28.914652Z","shell.execute_reply":"2023-10-21T13:49:29.919181Z"},"trusted":true},"execution_count":31,"outputs":[{"name":"stdout","text":"/bin/bash: numpy.--version__: command not found\n","output_type":"stream"}]},{"cell_type":"code","source":"best_clf = param_hyperopt(train)","metadata":{"tags":[],"execution":{"iopub.status.busy":"2023-10-21T13:40:30.757324Z","iopub.execute_input":"2023-10-21T13:40:30.75798Z","iopub.status.idle":"2023-10-21T13:40:30.877254Z","shell.execute_reply.started":"2023-10-21T13:40:30.757929Z","shell.execute_reply":"2023-10-21T13:40:30.875699Z"},"trusted":true},"execution_count":18,"outputs":[{"name":"stdout","text":" 0%| | 0/30 [00:00 1\u001b[0m best_clf \u001b[38;5;241m=\u001b[39m \u001b[43mparam_hyperopt\u001b[49m\u001b[43m(\u001b[49m\u001b[43mtrain\u001b[49m\u001b[43m)\u001b[49m\n","Cell \u001b[0;32mIn[10], line 52\u001b[0m, in \u001b[0;36mparam_hyperopt\u001b[0;34m(train)\u001b[0m\n\u001b[1;32m 40\u001b[0m params_space \u001b[38;5;241m=\u001b[39m {\n\u001b[1;32m 41\u001b[0m \u001b[38;5;124m'\u001b[39m\u001b[38;5;124mlearning_rate\u001b[39m\u001b[38;5;124m'\u001b[39m: hp\u001b[38;5;241m.\u001b[39muniform(\u001b[38;5;124m'\u001b[39m\u001b[38;5;124mlearning_rate\u001b[39m\u001b[38;5;124m'\u001b[39m, \u001b[38;5;241m1e-2\u001b[39m, \u001b[38;5;241m5e-1\u001b[39m),\n\u001b[1;32m 42\u001b[0m \u001b[38;5;124m'\u001b[39m\u001b[38;5;124mbagging_fraction\u001b[39m\u001b[38;5;124m'\u001b[39m: hp\u001b[38;5;241m.\u001b[39muniform(\u001b[38;5;124m'\u001b[39m\u001b[38;5;124mbagging_fraction\u001b[39m\u001b[38;5;124m'\u001b[39m, \u001b[38;5;241m0.5\u001b[39m, \u001b[38;5;241m1\u001b[39m),\n\u001b[0;32m (...)\u001b[0m\n\u001b[1;32m 48\u001b[0m \u001b[38;5;124m'\u001b[39m\u001b[38;5;124mmin_child_samples\u001b[39m\u001b[38;5;124m'\u001b[39m: hp\u001b[38;5;241m.\u001b[39mchoice(\u001b[38;5;124m'\u001b[39m\u001b[38;5;124mmin_child_samples\u001b[39m\u001b[38;5;124m'\u001b[39m, \u001b[38;5;28mlist\u001b[39m(\u001b[38;5;28mrange\u001b[39m(\u001b[38;5;241m1\u001b[39m, \u001b[38;5;241m30\u001b[39m, \u001b[38;5;241m5\u001b[39m)))\n\u001b[1;32m 49\u001b[0m }\n\u001b[1;32m 51\u001b[0m \u001b[38;5;66;03m# Part 5.TPE超参数搜索\u001b[39;00m\n\u001b[0;32m---> 52\u001b[0m params_best \u001b[38;5;241m=\u001b[39m \u001b[43mfmin\u001b[49m\u001b[43m(\u001b[49m\n\u001b[1;32m 53\u001b[0m \u001b[43m \u001b[49m\u001b[43mhyperopt_objective\u001b[49m\u001b[43m,\u001b[49m\n\u001b[1;32m 54\u001b[0m \u001b[43m \u001b[49m\u001b[43mspace\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43mparams_space\u001b[49m\u001b[43m,\u001b[49m\n\u001b[1;32m 55\u001b[0m \u001b[43m \u001b[49m\u001b[43malgo\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43mtpe\u001b[49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43msuggest\u001b[49m\u001b[43m,\u001b[49m\n\u001b[1;32m 56\u001b[0m \u001b[43m \u001b[49m\u001b[43mmax_evals\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[38;5;241;43m30\u001b[39;49m\u001b[43m,\u001b[49m\n\u001b[1;32m 57\u001b[0m \u001b[43m \u001b[49m\u001b[43mrstate\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43mRandomState\u001b[49m\u001b[43m(\u001b[49m\u001b[38;5;241;43m2020\u001b[39;49m\u001b[43m)\u001b[49m\u001b[43m)\u001b[49m\n\u001b[1;32m 59\u001b[0m \u001b[38;5;66;03m# 返回最佳参数\u001b[39;00m\n\u001b[1;32m 60\u001b[0m \u001b[38;5;28;01mreturn\u001b[39;00m params_best\n","File \u001b[0;32m/opt/conda/lib/python3.10/site-packages/hyperopt/fmin.py:586\u001b[0m, in \u001b[0;36mfmin\u001b[0;34m(fn, space, algo, max_evals, timeout, loss_threshold, trials, rstate, allow_trials_fmin, pass_expr_memo_ctrl, catch_eval_exceptions, verbose, return_argmin, points_to_evaluate, max_queue_len, show_progressbar, early_stop_fn, trials_save_file)\u001b[0m\n\u001b[1;32m 583\u001b[0m rval\u001b[38;5;241m.\u001b[39mcatch_eval_exceptions \u001b[38;5;241m=\u001b[39m catch_eval_exceptions\n\u001b[1;32m 585\u001b[0m \u001b[38;5;66;03m# next line is where the fmin is actually executed\u001b[39;00m\n\u001b[0;32m--> 586\u001b[0m \u001b[43mrval\u001b[49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mexhaust\u001b[49m\u001b[43m(\u001b[49m\u001b[43m)\u001b[49m\n\u001b[1;32m 588\u001b[0m \u001b[38;5;28;01mif\u001b[39;00m return_argmin:\n\u001b[1;32m 589\u001b[0m \u001b[38;5;28;01mif\u001b[39;00m \u001b[38;5;28mlen\u001b[39m(trials\u001b[38;5;241m.\u001b[39mtrials) \u001b[38;5;241m==\u001b[39m \u001b[38;5;241m0\u001b[39m:\n","File \u001b[0;32m/opt/conda/lib/python3.10/site-packages/hyperopt/fmin.py:364\u001b[0m, in \u001b[0;36mFMinIter.exhaust\u001b[0;34m(self)\u001b[0m\n\u001b[1;32m 362\u001b[0m \u001b[38;5;28;01mdef\u001b[39;00m \u001b[38;5;21mexhaust\u001b[39m(\u001b[38;5;28mself\u001b[39m):\n\u001b[1;32m 363\u001b[0m n_done \u001b[38;5;241m=\u001b[39m \u001b[38;5;28mlen\u001b[39m(\u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39mtrials)\n\u001b[0;32m--> 364\u001b[0m \u001b[38;5;28;43mself\u001b[39;49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mrun\u001b[49m\u001b[43m(\u001b[49m\u001b[38;5;28;43mself\u001b[39;49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mmax_evals\u001b[49m\u001b[43m \u001b[49m\u001b[38;5;241;43m-\u001b[39;49m\u001b[43m \u001b[49m\u001b[43mn_done\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[43mblock_until_done\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[38;5;28;43mself\u001b[39;49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43masynchronous\u001b[49m\u001b[43m)\u001b[49m\n\u001b[1;32m 365\u001b[0m \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39mtrials\u001b[38;5;241m.\u001b[39mrefresh()\n\u001b[1;32m 366\u001b[0m \u001b[38;5;28;01mreturn\u001b[39;00m \u001b[38;5;28mself\u001b[39m\n","File \u001b[0;32m/opt/conda/lib/python3.10/site-packages/hyperopt/fmin.py:279\u001b[0m, in \u001b[0;36mFMinIter.run\u001b[0;34m(self, N, block_until_done)\u001b[0m\n\u001b[1;32m 273\u001b[0m \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39mtrials\u001b[38;5;241m.\u001b[39mrefresh()\n\u001b[1;32m 274\u001b[0m \u001b[38;5;66;03m# Based on existing trials and the domain, use `algo` to probe in\u001b[39;00m\n\u001b[1;32m 275\u001b[0m \u001b[38;5;66;03m# new hp points. Save the results of those inspections into\u001b[39;00m\n\u001b[1;32m 276\u001b[0m \u001b[38;5;66;03m# `new_trials`. This is the core of `run`, all the rest is just\u001b[39;00m\n\u001b[1;32m 277\u001b[0m \u001b[38;5;66;03m# processes orchestration\u001b[39;00m\n\u001b[1;32m 278\u001b[0m new_trials \u001b[38;5;241m=\u001b[39m algo(\n\u001b[0;32m--> 279\u001b[0m new_ids, \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39mdomain, trials, \u001b[38;5;28;43mself\u001b[39;49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mrstate\u001b[49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mintegers\u001b[49m(\u001b[38;5;241m2\u001b[39m \u001b[38;5;241m*\u001b[39m\u001b[38;5;241m*\u001b[39m \u001b[38;5;241m31\u001b[39m \u001b[38;5;241m-\u001b[39m \u001b[38;5;241m1\u001b[39m)\n\u001b[1;32m 280\u001b[0m )\n\u001b[1;32m 281\u001b[0m \u001b[38;5;28;01massert\u001b[39;00m \u001b[38;5;28mlen\u001b[39m(new_ids) \u001b[38;5;241m>\u001b[39m\u001b[38;5;241m=\u001b[39m \u001b[38;5;28mlen\u001b[39m(new_trials)\n\u001b[1;32m 283\u001b[0m \u001b[38;5;28;01mif\u001b[39;00m \u001b[38;5;28mlen\u001b[39m(new_trials):\n","\u001b[0;31mAttributeError\u001b[0m: 'numpy.random.mtrand.RandomState' object has no attribute 'integers'"],"ename":"AttributeError","evalue":"'numpy.random.mtrand.RandomState' object has no attribute 'integers'","output_type":"error"}]},{"cell_type":"markdown","source":"此时best_clf即为lgb模型的最优参数组。","metadata":{}},{"cell_type":"code","source":"best_clf","metadata":{},"execution_count":15,"outputs":[{"execution_count":15,"output_type":"execute_result","data":{"text/plain":["{'bagging_fraction': 0.9022336069269954,\n"," 'bagging_freq': 2,\n"," 'feature_fraction': 0.9373662317255621,\n"," 'learning_rate': 0.014947332175194025,\n"," 'min_child_samples': 5,\n"," 'num_leaves': 7,\n"," 'reg_alpha': 2,\n"," 'reg_lambda': 3.5907566887206896}"]},"metadata":{}}]},{"cell_type":"markdown","source":"### 3.LightGBM模型预测与结果排名","metadata":{}},{"cell_type":"markdown","source":"  在搜索出最优参数后，接下来即可进行模型预测了。和此前一样，在实际执行预测时有两种思路，其一是单模型预测，即直接针对测试集进行预测并提交结果，其二则是通过交叉验证提交平均得分，并且在此过程中能同时保留下后续用于stacking集成时所需要用到的数据。","metadata":{}},{"cell_type":"markdown","source":"- 单模型预测","metadata":{}},{"cell_type":"markdown","source":"  首先测试单独模型在测试集上的预测效果：","metadata":{}},{"cell_type":"code","source":"# 再次申明固定参数\nbest_clf = params_append(best_clf)\n\n# 数据准备过程\nlabel = 'target'\nfeatures = train_LGBM.columns.tolist()\nfeatures.remove('card_id')\nfeatures.remove('target')\n\n# 数据封装\nlgb_train = lgb.Dataset(train_LGBM[features], train_LGBM[label])","metadata":{},"execution_count":16,"outputs":[]},{"cell_type":"code","source":"# 在全部数据集上训练模型\nbst = lgb.train(best_clf, lgb_train)","metadata":{},"execution_count":18,"outputs":[{"name":"stdout","output_type":"stream","text":"[LightGBM] [Warning] Auto-choosing col-wise multi-threading, the overhead of testing was 0.426494 seconds.\n\nYou can set `force_col_wise=true` to remove the overhead.\n\n[LightGBM] [Info] Total Bins 65975\n\n[LightGBM] [Info] Number of data points in the train set: 201917, number of used features: 300\n\n[LightGBM] [Info] Start training from score -0.393636\n"}]},{"cell_type":"code","source":"# 在测试集上完成预测\nbst.predict(train_LGBM[features])","metadata":{},"execution_count":19,"outputs":[{"execution_count":19,"output_type":"execute_result","data":{"text/plain":["array([-0.24511938, -2.01595283, 0.1053809 , ..., -0.18190679,\n"," -1.11870804, -0.24511938])"]},"metadata":{}}]},{"cell_type":"code","source":"# 简单查看训练集RMSE\nnp.sqrt(mean_squared_error(train_LGBM[label], bst.predict(train_LGBM[features])))","metadata":{},"execution_count":20,"outputs":[{"execution_count":20,"output_type":"execute_result","data":{"text/plain":["3.7213768397255365"]},"metadata":{}}]},{"cell_type":"markdown","source":"接下来，对测试集进行预测，并将结果写入本地文件","metadata":{}},{"cell_type":"code","source":"test_LGBM['target'] = bst.predict(test_LGBM[features])\ntest_LGBM[['card_id', 'target']].to_csv(\"result/submission_LGBM.csv\", index=False)","metadata":{"tags":[],"collapsed":true,"jupyter":{"outputs_hidden":true}},"execution_count":21,"outputs":[{"name":"stderr","output_type":"stream","text":":1: SettingWithCopyWarning: \n\nA value is trying to be set on a copy of a slice from a DataFrame.\n\nTry using .loc[row_indexer,col_indexer] = value instead\n\n\n\nSee the caveats in the documentation: https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy\n\n test_LGBM['target'] = bst.predict(test_LGBM[features])\n"}]},{"cell_type":"code","source":"test_LGBM[['card_id', 'target']].head(5)","metadata":{"tags":[]},"execution_count":23,"outputs":[{"execution_count":23,"output_type":"execute_result","data":{"text/html":["
\n","\n","\n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n","
card_idtarget
0C_ID_0ab67a22ab-2.418856
1C_ID_130fd0cbdd-0.752626
2C_ID_b709037bc5-0.030933
3C_ID_d27d835a9f-0.245119
4C_ID_2b5e3df5c2-0.366990
\n","
"],"text/plain":[" card_id target\n","0 C_ID_0ab67a22ab -2.418856\n","1 C_ID_130fd0cbdd -0.752626\n","2 C_ID_b709037bc5 -0.030933\n","3 C_ID_d27d835a9f -0.245119\n","4 C_ID_2b5e3df5c2 -0.366990"]},"metadata":{}}]},{"cell_type":"markdown","source":"提交该结果，得到公榜、私榜结果如下：","metadata":{}},{"cell_type":"markdown","source":"
\"image-20211209172112868\"","metadata":{}},{"cell_type":"markdown","source":"对比此前的随机森林提交的两组结果，汇总情况如下：","metadata":{}},{"cell_type":"markdown","source":"| 模型 | Private Score | Public Score |\n| ------ | ------ | ------ |\n| randomforest | 3.65455 | 3.74969 |\n| randomforest+validation | 3.65173 | 3.74954 |\n| LightGBM | 3.69723 | 3.80436 |","metadata":{}},{"cell_type":"markdown","source":"能够发现，在单模型预测情况下，lgb要略弱于rf，接下来考虑进行交叉验证，以提高lgb模型预测效果。","metadata":{}},{"cell_type":"markdown","source":"- 结合交叉验证进行模型预测","metadata":{}},{"cell_type":"markdown","source":"  和随机森林借助交叉验证进行模型预测的过程类似，lgb也需要遵照如下流程进行训练和预测，并同时创建后续集成所需数据集以及预测结果的平均值（作为最终预测结果）","metadata":{}},{"cell_type":"markdown","source":"
\"image-20211208192640281\"","metadata":{}},{"cell_type":"markdown","source":"执行过程如下：","metadata":{}},{"cell_type":"code","source":"def train_predict(train, test, params):\n \"\"\"\n\n :param train:\n :param test:\n :param params:\n :return:\n \"\"\"\n # Part 1.选择特征\n label = 'target'\n features = train.columns.tolist()\n features.remove('card_id')\n features.remove('target')\n \n # Part 2.再次申明固定参数与控制迭代参数\n params = params_append(params)\n ESR = 30\n NBR = 10000\n VBE = 50\n \n # Part 3.创建结果存储容器\n # 测试集预测结果存储器，后保存至本地文件\n prediction_test = 0\n # 验证集的模型表现，作为展示用\n cv_score = []\n # 验证集的预测结果存储器，后保存至本地文件\n prediction_train = pd.Series()\n \n # Part 3.交叉验证\n kf = KFold(n_splits=5, random_state=2020, shuffle=True)\n for train_part_index, eval_index in kf.split(train[features], train[label]):\n # 训练数据封装\n train_part = lgb.Dataset(train[features].loc[train_part_index],\n train[label].loc[train_part_index])\n # 测试数据封装\n eval = lgb.Dataset(train[features].loc[eval_index],\n train[label].loc[eval_index])\n # 依据验证集训练模型\n bst = lgb.train(params, train_part, num_boost_round=NBR,\n valid_sets=[train_part, eval],\n valid_names=['train', 'valid'],\n early_stopping_rounds=ESR, verbose_eval=VBE)\n # 测试集预测结果并纳入prediction_test容器\n prediction_test += bst.predict(test[features])\n # 验证集预测结果并纳入prediction_train容器\n prediction_train = prediction_train.append(pd.Series(bst.predict(train[features].loc[eval_index]),\n index=eval_index))\n # 验证集预测结果\n eval_pre = bst.predict(train[features].loc[eval_index])\n # 计算验证集上得分\n score = np.sqrt(mean_squared_error(train[label].loc[eval_index].values, eval_pre))\n # 纳入cv_score容器\n cv_score.append(score)\n \n # Part 4.打印/输出结果\n # 打印验证集得分与平均得分\n print(cv_score, sum(cv_score) / 5)\n # 将验证集上预测结果写入本地文件\n pd.Series(prediction_train.sort_index().values).to_csv(\"preprocess/train_lightgbm.csv\", index=False)\n # 将测试集上预测结果写入本地文件\n pd.Series(prediction_test / 5).to_csv(\"preprocess/test_lightgbm.csv\", index=False)\n # 测试集平均得分作为模型最终预测结果\n test['target'] = prediction_test / 5\n # 将测试集预测结果写成竞赛要求格式并保存至本地\n test[['card_id', 'target']].to_csv(\"result/submission_lightgbm.csv\", index=False)\n return","metadata":{},"execution_count":24,"outputs":[]},{"cell_type":"code","source":"train_LGBM, test_LGBM = feature_select_wrapper(train, test)\nbest_clf = param_hyperopt(train_LGBM)\ntrain_predict(train_LGBM, test_LGBM, best_clf)","metadata":{"tags":[],"collapsed":true,"jupyter":{"outputs_hidden":true}},"execution_count":25,"outputs":[{"name":"stdout","output_type":"stream","text":"feature_select_wrapper...\n"},{"name":"stderr","output_type":"stream","text":"C:\\Users\\vdmion\\anaconda3\\lib\\site-packages\\lightgbm\\engine.py:181: UserWarning: 'early_stopping_rounds' argument is deprecated and will be removed in a future release of LightGBM. Pass 'early_stopping()' callback via 'callbacks' argument instead.\n\n _log_warning(\"'early_stopping_rounds' argument is deprecated and will be removed in a future release of LightGBM. \"\n\nC:\\Users\\vdmion\\anaconda3\\lib\\site-packages\\lightgbm\\engine.py:239: UserWarning: 'verbose_eval' argument is deprecated and will be removed in a future release of LightGBM. Pass 'log_evaluation()' callback via 'callbacks' argument instead.\n\n _log_warning(\"'verbose_eval' argument is deprecated and will be removed in a future release of LightGBM. \"\n"},{"name":"stdout","output_type":"stream","text":"[LightGBM] [Warning] Auto-choosing col-wise multi-threading, the overhead of testing was 0.676517 seconds.\n\nYou can set `force_col_wise=true` to remove the overhead.\n\n[LightGBM] [Info] Total Bins 227016\n\n[LightGBM] [Info] Number of data points in the train set: 161533, number of used features: 1626\n\n[LightGBM] [Info] Start training from score -0.390986\n\nTraining until validation scores don't improve for 30 rounds\n\n[50]\ttrain's rmse: 3.43695\tvalid's rmse: 3.70629\n\nEarly stopping, best iteration is:\n\n[66]\ttrain's rmse: 3.39251\tvalid's rmse: 3.70281\n\n[LightGBM] [Warning] Auto-choosing col-wise multi-threading, the overhead of testing was 0.595659 seconds.\n\nYou can set `force_col_wise=true` to remove the overhead.\n\n[LightGBM] [Info] Total Bins 227122\n\n[LightGBM] [Info] Number of data points in the train set: 161533, number of used features: 1629\n\n[LightGBM] [Info] Start training from score -0.396781\n\nTraining until validation scores don't improve for 30 rounds\n\n[50]\ttrain's rmse: 3.45546\tvalid's rmse: 3.67176\n\n[100]\ttrain's rmse: 3.33017\tvalid's rmse: 3.67221\n\nEarly stopping, best iteration is:\n\n[82]\ttrain's rmse: 3.37387\tvalid's rmse: 3.66794\n\n[LightGBM] [Warning] Auto-choosing col-wise multi-threading, the overhead of testing was 0.572918 seconds.\n\nYou can set `force_col_wise=true` to remove the overhead.\n\n[LightGBM] [Info] Total Bins 227089\n\n[LightGBM] [Info] Number of data points in the train set: 161534, number of used features: 1631\n\n[LightGBM] [Info] Start training from score -0.390348\n\nTraining until validation scores don't improve for 30 rounds\n\n[50]\ttrain's rmse: 3.43208\tvalid's rmse: 3.72216\n\nEarly stopping, best iteration is:\n\n[68]\ttrain's rmse: 3.38524\tvalid's rmse: 3.71798\n\n[LightGBM] [Warning] Auto-choosing col-wise multi-threading, the overhead of testing was 0.694756 seconds.\n\nYou can set `force_col_wise=true` to remove the overhead.\n\n[LightGBM] [Info] Total Bins 227023\n\n[LightGBM] [Info] Number of data points in the train set: 161534, number of used features: 1624\n\n[LightGBM] [Info] Start training from score -0.391392\n\nTraining until validation scores don't improve for 30 rounds\n\n[50]\ttrain's rmse: 3.42022\tvalid's rmse: 3.7901\n\nEarly stopping, best iteration is:\n\n[59]\ttrain's rmse: 3.39133\tvalid's rmse: 3.78884\n\n[LightGBM] [Warning] Auto-choosing col-wise multi-threading, the overhead of testing was 0.567025 seconds.\n\nYou can set `force_col_wise=true` to remove the overhead.\n\n[LightGBM] [Info] Total Bins 227023\n\n[LightGBM] [Info] Number of data points in the train set: 161534, number of used features: 1625\n\n[LightGBM] [Info] Start training from score -0.398675\n\nTraining until validation scores don't improve for 30 rounds\n\n[50]\ttrain's rmse: 3.46321\tvalid's rmse: 3.60039\n\nEarly stopping, best iteration is:\n\n[61]\ttrain's rmse: 3.43002\tvalid's rmse: 3.59887\n\ndone\n\n{'bagging_fraction': 0.7253952770621912, 'bagging_freq': 5, 'feature_fraction': 0.6972128940985931, 'learning_rate': 0.43437628238508774, 'min_child_samples': 6, 'num_leaves': 60, 'reg_alpha': 0, 'reg_lambda': 1.6139256132729207, 'feature_pre_filter': False, 'objective': 'regression', 'metric': 'rmse', 'bagging_seed': 2020}\n\n 0%| | 0/30 [00:00:27: DeprecationWarning: The default dtype for empty Series will be 'object' instead of 'float64' in a future version. Specify a dtype explicitly to silence this warning.\n\n prediction_train = pd.Series()\n"},{"name":"stdout","output_type":"stream","text":"[LightGBM] [Warning] Auto-choosing col-wise multi-threading, the overhead of testing was 0.340990 seconds.\n\nYou can set `force_col_wise=true` to remove the overhead.\n\n[LightGBM] [Info] Total Bins 65666\n\n[LightGBM] [Info] Number of data points in the train set: 161533, number of used features: 300\n\n[LightGBM] [Info] Start training from score -0.390986\n\nTraining until validation scores don't improve for 30 rounds\n\n[50]\ttrain's rmse: 3.75892\tvalid's rmse: 3.77103\n\n[100]\ttrain's rmse: 3.71868\tvalid's rmse: 3.73851\n\n[150]\ttrain's rmse: 3.69462\tvalid's rmse: 3.721\n\n[200]\ttrain's rmse: 3.67853\tvalid's rmse: 3.71087\n\n[250]\ttrain's rmse: 3.66669\tvalid's rmse: 3.70432\n\n[300]\ttrain's rmse: 3.65751\tvalid's rmse: 3.69995\n\n[350]\ttrain's rmse: 3.65049\tvalid's rmse: 3.69762\n\n[400]\ttrain's rmse: 3.64309\tvalid's rmse: 3.69551\n\n[450]\ttrain's rmse: 3.63597\tvalid's rmse: 3.69377\n\n[500]\ttrain's rmse: 3.63008\tvalid's rmse: 3.6925\n\n[550]\ttrain's rmse: 3.62462\tvalid's rmse: 3.69105\n\n[600]\ttrain's rmse: 3.6198\tvalid's rmse: 3.69013\n\n[650]\ttrain's rmse: 3.61443\tvalid's rmse: 3.68931\n\n[700]\ttrain's rmse: 3.61005\tvalid's rmse: 3.68884\n\n[750]\ttrain's rmse: 3.60491\tvalid's rmse: 3.68812\n\n[800]\ttrain's rmse: 3.59974\tvalid's rmse: 3.68759\n\n[850]\ttrain's rmse: 3.59474\tvalid's rmse: 3.68714\n\nEarly stopping, best iteration is:\n\n[831]\ttrain's rmse: 3.5965\tvalid's rmse: 3.68697\n"},{"name":"stderr","output_type":"stream","text":"C:\\Users\\vdmion\\anaconda3\\lib\\site-packages\\lightgbm\\engine.py:181: UserWarning: 'early_stopping_rounds' argument is deprecated and will be removed in a future release of LightGBM. Pass 'early_stopping()' callback via 'callbacks' argument instead.\n\n _log_warning(\"'early_stopping_rounds' argument is deprecated and will be removed in a future release of LightGBM. \"\n\nC:\\Users\\vdmion\\anaconda3\\lib\\site-packages\\lightgbm\\engine.py:239: UserWarning: 'verbose_eval' argument is deprecated and will be removed in a future release of LightGBM. Pass 'log_evaluation()' callback via 'callbacks' argument instead.\n\n _log_warning(\"'verbose_eval' argument is deprecated and will be removed in a future release of LightGBM. \"\n"},{"name":"stdout","output_type":"stream","text":"[LightGBM] [Warning] Auto-choosing col-wise multi-threading, the overhead of testing was 0.349296 seconds.\n\nYou can set `force_col_wise=true` to remove the overhead.\n\n[LightGBM] [Info] Total Bins 65677\n\n[LightGBM] [Info] Number of data points in the train set: 161533, number of used features: 300\n\n[LightGBM] [Info] Start training from score -0.396781\n\nTraining until validation scores don't improve for 30 rounds\n\n[50]\ttrain's rmse: 3.76719\tvalid's rmse: 3.73854\n\n[100]\ttrain's rmse: 3.72832\tvalid's rmse: 3.70102\n\n[150]\ttrain's rmse: 3.70615\tvalid's rmse: 3.68157\n\n[200]\ttrain's rmse: 3.68981\tvalid's rmse: 3.6711\n\n[250]\ttrain's rmse: 3.67731\tvalid's rmse: 3.6642\n\n[300]\ttrain's rmse: 3.6682\tvalid's rmse: 3.66021\n\n[350]\ttrain's rmse: 3.66021\tvalid's rmse: 3.65732\n\n[400]\ttrain's rmse: 3.653\tvalid's rmse: 3.65522\n\n[450]\ttrain's rmse: 3.6464\tvalid's rmse: 3.65337\n\n[500]\ttrain's rmse: 3.63997\tvalid's rmse: 3.65184\n\n[550]\ttrain's rmse: 3.63353\tvalid's rmse: 3.6508\n\n[600]\ttrain's rmse: 3.62768\tvalid's rmse: 3.64985\n\n[650]\ttrain's rmse: 3.62235\tvalid's rmse: 3.64887\n\n[700]\ttrain's rmse: 3.61719\tvalid's rmse: 3.64818\n\n[750]\ttrain's rmse: 3.61244\tvalid's rmse: 3.64785\n\n[800]\ttrain's rmse: 3.60741\tvalid's rmse: 3.6471\n\n[850]\ttrain's rmse: 3.60239\tvalid's rmse: 3.64632\n\n[900]\ttrain's rmse: 3.59807\tvalid's rmse: 3.64592\n\n[950]\ttrain's rmse: 3.59319\tvalid's rmse: 3.64498\n\n[1000]\ttrain's rmse: 3.58882\tvalid's rmse: 3.64452\n\nEarly stopping, best iteration is:\n\n[998]\ttrain's rmse: 3.58903\tvalid's rmse: 3.64451\n"},{"name":"stderr","output_type":"stream","text":"C:\\Users\\vdmion\\anaconda3\\lib\\site-packages\\lightgbm\\engine.py:181: UserWarning: 'early_stopping_rounds' argument is deprecated and will be removed in a future release of LightGBM. Pass 'early_stopping()' callback via 'callbacks' argument instead.\n\n _log_warning(\"'early_stopping_rounds' argument is deprecated and will be removed in a future release of LightGBM. \"\n\nC:\\Users\\vdmion\\anaconda3\\lib\\site-packages\\lightgbm\\engine.py:239: UserWarning: 'verbose_eval' argument is deprecated and will be removed in a future release of LightGBM. Pass 'log_evaluation()' callback via 'callbacks' argument instead.\n\n _log_warning(\"'verbose_eval' argument is deprecated and will be removed in a future release of LightGBM. \"\n"},{"name":"stdout","output_type":"stream","text":"[LightGBM] [Warning] Auto-choosing col-wise multi-threading, the overhead of testing was 0.353090 seconds.\n\nYou can set `force_col_wise=true` to remove the overhead.\n\n[LightGBM] [Info] Total Bins 65684\n\n[LightGBM] [Info] Number of data points in the train set: 161534, number of used features: 300\n\n[LightGBM] [Info] Start training from score -0.390348\n\nTraining until validation scores don't improve for 30 rounds\n\n[50]\ttrain's rmse: 3.75231\tvalid's rmse: 3.78794\n\n[100]\ttrain's rmse: 3.71093\tvalid's rmse: 3.75345\n\n[150]\ttrain's rmse: 3.68701\tvalid's rmse: 3.73658\n\n[200]\ttrain's rmse: 3.67191\tvalid's rmse: 3.72698\n\n[250]\ttrain's rmse: 3.66002\tvalid's rmse: 3.72123\n\n[300]\ttrain's rmse: 3.65114\tvalid's rmse: 3.71716\n\n[350]\ttrain's rmse: 3.64347\tvalid's rmse: 3.71463\n\n[400]\ttrain's rmse: 3.63661\tvalid's rmse: 3.71266\n\n[450]\ttrain's rmse: 3.63053\tvalid's rmse: 3.71135\n\n[500]\ttrain's rmse: 3.62438\tvalid's rmse: 3.71013\n\n[550]\ttrain's rmse: 3.61892\tvalid's rmse: 3.70927\n\n[600]\ttrain's rmse: 3.61344\tvalid's rmse: 3.70822\n\n[650]\ttrain's rmse: 3.60779\tvalid's rmse: 3.70713\n\n[700]\ttrain's rmse: 3.60292\tvalid's rmse: 3.70661\n\n[750]\ttrain's rmse: 3.59785\tvalid's rmse: 3.70588\n\n[800]\ttrain's rmse: 3.59275\tvalid's rmse: 3.70488\n\n[850]\ttrain's rmse: 3.588\tvalid's rmse: 3.70413\n\n[900]\ttrain's rmse: 3.58365\tvalid's rmse: 3.70404\n\nEarly stopping, best iteration is:\n\n[870]\ttrain's rmse: 3.58625\tvalid's rmse: 3.70395\n"},{"name":"stderr","output_type":"stream","text":"C:\\Users\\vdmion\\anaconda3\\lib\\site-packages\\lightgbm\\engine.py:181: UserWarning: 'early_stopping_rounds' argument is deprecated and will be removed in a future release of LightGBM. Pass 'early_stopping()' callback via 'callbacks' argument instead.\n\n _log_warning(\"'early_stopping_rounds' argument is deprecated and will be removed in a future release of LightGBM. \"\n\nC:\\Users\\vdmion\\anaconda3\\lib\\site-packages\\lightgbm\\engine.py:239: UserWarning: 'verbose_eval' argument is deprecated and will be removed in a future release of LightGBM. Pass 'log_evaluation()' callback via 'callbacks' argument instead.\n\n _log_warning(\"'verbose_eval' argument is deprecated and will be removed in a future release of LightGBM. \"\n"},{"name":"stdout","output_type":"stream","text":"[LightGBM] [Warning] Auto-choosing col-wise multi-threading, the overhead of testing was 0.305438 seconds.\n\nYou can set `force_col_wise=true` to remove the overhead.\n\n[LightGBM] [Info] Total Bins 65642\n\n[LightGBM] [Info] Number of data points in the train set: 161534, number of used features: 300\n\n[LightGBM] [Info] Start training from score -0.391392\n\nTraining until validation scores don't improve for 30 rounds\n\n[50]\ttrain's rmse: 3.73315\tvalid's rmse: 3.86928\n\n[100]\ttrain's rmse: 3.69309\tvalid's rmse: 3.83508\n\n[150]\ttrain's rmse: 3.66977\tvalid's rmse: 3.81704\n\n[200]\ttrain's rmse: 3.65402\tvalid's rmse: 3.80625\n\n[250]\ttrain's rmse: 3.64247\tvalid's rmse: 3.79953\n\n[300]\ttrain's rmse: 3.63288\tvalid's rmse: 3.79472\n\n[350]\ttrain's rmse: 3.62507\tvalid's rmse: 3.79137\n\n[400]\ttrain's rmse: 3.61782\tvalid's rmse: 3.78885\n\n[450]\ttrain's rmse: 3.61148\tvalid's rmse: 3.78654\n\n[500]\ttrain's rmse: 3.60518\tvalid's rmse: 3.78439\n\n[550]\ttrain's rmse: 3.59867\tvalid's rmse: 3.78269\n\n[600]\ttrain's rmse: 3.59283\tvalid's rmse: 3.78127\n\n[650]\ttrain's rmse: 3.58721\tvalid's rmse: 3.77968\n\n[700]\ttrain's rmse: 3.58204\tvalid's rmse: 3.7786\n\n[750]\ttrain's rmse: 3.57702\tvalid's rmse: 3.77795\n\n[800]\ttrain's rmse: 3.57227\tvalid's rmse: 3.7771\n\n[850]\ttrain's rmse: 3.56756\tvalid's rmse: 3.77617\n\n[900]\ttrain's rmse: 3.56254\tvalid's rmse: 3.77507\n\n[950]\ttrain's rmse: 3.55836\tvalid's rmse: 3.77424\n\n[1000]\ttrain's rmse: 3.55367\tvalid's rmse: 3.7737\n\n[1050]\ttrain's rmse: 3.54943\tvalid's rmse: 3.77328\n\n[1100]\ttrain's rmse: 3.54517\tvalid's rmse: 3.77277\n\n[1150]\ttrain's rmse: 3.54089\tvalid's rmse: 3.77242\n\nEarly stopping, best iteration is:\n\n[1144]\ttrain's rmse: 3.54163\tvalid's rmse: 3.77227\n"},{"name":"stderr","output_type":"stream","text":"C:\\Users\\vdmion\\anaconda3\\lib\\site-packages\\lightgbm\\engine.py:181: UserWarning: 'early_stopping_rounds' argument is deprecated and will be removed in a future release of LightGBM. Pass 'early_stopping()' callback via 'callbacks' argument instead.\n\n _log_warning(\"'early_stopping_rounds' argument is deprecated and will be removed in a future release of LightGBM. \"\n\nC:\\Users\\vdmion\\anaconda3\\lib\\site-packages\\lightgbm\\engine.py:239: UserWarning: 'verbose_eval' argument is deprecated and will be removed in a future release of LightGBM. Pass 'log_evaluation()' callback via 'callbacks' argument instead.\n\n _log_warning(\"'verbose_eval' argument is deprecated and will be removed in a future release of LightGBM. \"\n"},{"name":"stdout","output_type":"stream","text":"[LightGBM] [Warning] Auto-choosing col-wise multi-threading, the overhead of testing was 0.350972 seconds.\n\nYou can set `force_col_wise=true` to remove the overhead.\n\n[LightGBM] [Info] Total Bins 65695\n\n[LightGBM] [Info] Number of data points in the train set: 161534, number of used features: 300\n\n[LightGBM] [Info] Start training from score -0.398675\n\nTraining until validation scores don't improve for 30 rounds\n\n[50]\ttrain's rmse: 3.78477\tvalid's rmse: 3.65732\n\n[100]\ttrain's rmse: 3.74396\tvalid's rmse: 3.62525\n\n[150]\ttrain's rmse: 3.72006\tvalid's rmse: 3.6094\n\n[200]\ttrain's rmse: 3.7039\tvalid's rmse: 3.59961\n\n[250]\ttrain's rmse: 3.69286\tvalid's rmse: 3.59368\n\n[300]\ttrain's rmse: 3.68357\tvalid's rmse: 3.59002\n\n[350]\ttrain's rmse: 3.67565\tvalid's rmse: 3.58705\n\n[400]\ttrain's rmse: 3.66878\tvalid's rmse: 3.58453\n\n[450]\ttrain's rmse: 3.66276\tvalid's rmse: 3.5828\n\n[500]\ttrain's rmse: 3.65653\tvalid's rmse: 3.58089\n\n[550]\ttrain's rmse: 3.65107\tvalid's rmse: 3.57947\n\n[600]\ttrain's rmse: 3.64527\tvalid's rmse: 3.57869\n\n[650]\ttrain's rmse: 3.64029\tvalid's rmse: 3.57741\n\n[700]\ttrain's rmse: 3.63434\tvalid's rmse: 3.57659\n\n[750]\ttrain's rmse: 3.62887\tvalid's rmse: 3.57541\n\n[800]\ttrain's rmse: 3.62374\tvalid's rmse: 3.57487\n\n[850]\ttrain's rmse: 3.61877\tvalid's rmse: 3.57409\n\n[900]\ttrain's rmse: 3.61421\tvalid's rmse: 3.57325\n\n[950]\ttrain's rmse: 3.60936\tvalid's rmse: 3.57294\n\n[1000]\ttrain's rmse: 3.60497\tvalid's rmse: 3.57253\n\nEarly stopping, best iteration is:\n\n[1000]\ttrain's rmse: 3.60497\tvalid's rmse: 3.57253\n\n[3.6869706135632367, 3.6445135024361908, 3.7039487803864932, 3.772271999086466, 3.5725331932165627] 3.6760476177377894\n"},{"name":"stderr","output_type":"stream","text":":63: SettingWithCopyWarning: \n\nA value is trying to be set on a copy of a slice from a DataFrame.\n\nTry using .loc[row_indexer,col_indexer] = value instead\n\n\n\nSee the caveats in the documentation: https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy\n\n test['target'] = prediction_test / 5\n"}]},{"cell_type":"markdown","source":"接下来即可在竞赛主页提交预测结果。最终公榜私榜评分如下：","metadata":{}},{"cell_type":"markdown","source":"
\"image-20211209173249232\"","metadata":{}},{"cell_type":"markdown","source":"对比此前结果：","metadata":{}},{"cell_type":"markdown","source":"| 模型 | Private Score | Public Score |\n| ------ | ------ | ------ |\n| randomforest | 3.65455 | 3.74969 |\n| randomforest+validation | 3.65173 | 3.74954 |\n| LightGBM | 3.69723 | 3.80436 |\n| LightGBM+validation | 3.64403 | 3.73875 |","metadata":{}},{"cell_type":"markdown","source":"能够看出，经过交叉验证后输出的平均值结果，较此前的预测评分，有较大提升，这也是目前我们跑出的最好成绩。同时，交叉验证的作用已得到充分征明，后续在进行其他模型训练时仅考虑模型+交叉验证的输出结果，不再进行单模型结果输出。","metadata":{}},{"cell_type":"markdown","source":"---","metadata":{}},{"cell_type":"markdown","source":"##
二、单模训练策略（三）\n##
**NLP特征优化+XGBoost建模+贝叶斯优化器**","metadata":{}},{"cell_type":"markdown","source":"  在执行完随机森林与LightGBM后，我们已经对不同集成算法的竞赛建模流程有了一定的了解，大家可以照此思路继续尝试其他集成模型。当然，如果想进一步优化提升模型效果，我们可以考虑围绕数据集中的部分ID字段进行NLP特征优化。因此，接下来，我们考虑采用CountVectorizer, TfidfVectorizer两种方法对数据集中部分特征进行NLP特征衍生，并且采用XGBoost模型进行预测，同时考虑进一步使用另一种贝叶斯优化器（bayes_opt）来进行模型参数调优。","metadata":{}},{"cell_type":"markdown","source":"> 当然，此处训练的三个模型分别采用了不同的优化器、甚至是采用了不同的特征衍生的方法，也是为了令这三个模型尽可能的存在一定的差异性，从而为后续的模型融合的融合效果做铺垫。","metadata":{}},{"cell_type":"markdown","source":"### 1.NLP特征优化","metadata":{}},{"cell_type":"markdown","source":"  首先我们注意到，在数据集中存在大量的ID相关的列（除了card_id外），包括'merchant_id'、'merchant_category_id'、'state_id'、'subsector_id'、'city_id'等，考虑到这些ID在出现频率方面都和用户实际的交易行为息息相关，例如对于单独用户A来说，在其交易记录中频繁出现某商户id（假设为B），则说明该用户A对商户B情有独钟，而如果在不同的用户交易数据中，都频繁的出现了商户B，则说明这家商户受到广泛欢迎，而进一步的说明A的喜好可能和大多数用户一致，而反之则说明A用户的喜好较为独特。为了能够挖掘出类似信息，我们可以考虑采用NLP中CountVector和TF-IDF两种方法来进行进一步特征衍生，其中CountVector可以挖掘类似某用户钟爱某商铺的信息，而TF-IDF则可进一步挖掘出类似某用户的喜好是否普遍或一致等信息。","metadata":{}},{"cell_type":"markdown","source":"  此外，若要借助NLP方法进行进一步特征衍生，则需要考虑到新创建的特征数量过大所导致的问题，因此我们建议在使用上述方法的同时，考虑借助scipy中的稀疏矩阵相关方法，来进行新特征的存储与读取。即采用CSR格式来创建稀疏矩阵，用npz格式来进行本地数据文件保存。","metadata":{}},{"cell_type":"markdown","source":"需要导入的包如下：","metadata":{}},{"cell_type":"code","source":"from sklearn.feature_extraction.text import CountVectorizer, TfidfVectorizer\nfrom sklearn.preprocessing import OneHotEncoder, LabelEncoder\nfrom scipy import sparse","metadata":{},"execution_count":85,"outputs":[]},{"cell_type":"markdown","source":"  接下来进行NLP特征创建，首先读取原始数据：","metadata":{}},{"cell_type":"code","source":"# 注意，该数据集是最初始的数据集\ntrain = pd.read_csv('data/train.csv')\ntest = pd.read_csv('data/test.csv')\nmerchant = pd.read_csv('data/merchants.csv')\nnew_transaction = pd.read_csv('data/new_merchant_transactions.csv')\nhistory_transaction = pd.read_csv('data/historical_transactions.csv')\ntransaction = pd.concat([new_transaction, history_transaction], axis=0, ignore_index=True)\ndel new_transaction\ndel history_transaction\ngc.collect()","metadata":{},"execution_count":2,"outputs":[{"execution_count":2,"output_type":"execute_result","data":{"text/plain":["0"]},"metadata":{}}]},{"cell_type":"markdown","source":"然后进行数据预处理及增加NLP特征标识：","metadata":{}},{"cell_type":"code","source":"nlp_features = ['merchant_id', 'merchant_category_id', 'state_id', 'subsector_id', 'city_id']\n\nfor co in nlp_features:\n print(co)\n transaction[co] = transaction[co].astype(str)\n temp = transaction[transaction['month_lag']>=0].groupby(\"card_id\")[co].apply(list).apply(lambda x:' '.join(x)).reset_index()\n temp.columns = ['card_id', co+'_new']\n train = pd.merge(train, temp, how='left', on='card_id')\n test = pd.merge(test, temp, how='left', on='card_id')\n\n temp = transaction[transaction['month_lag']<0].groupby(\"card_id\")[co].apply(list).apply(lambda x:' '.join(x)).reset_index()\n temp.columns = ['card_id', co+'_hist']\n train = pd.merge(train, temp, how='left', on='card_id')\n test = pd.merge(test, temp, how='left', on='card_id')\n\n temp = transaction.groupby(\"card_id\")[co].apply(list).apply(lambda x:' '.join(x)).reset_index()\n temp.columns = ['card_id', co+'_all']\n train = pd.merge(train, temp, how='left', on='card_id').fillna(\"-1\")\n test = pd.merge(test, temp, how='left', on='card_id').fillna(\"-1\")","metadata":{"scrolled":true},"execution_count":3,"outputs":[{"name":"stdout","output_type":"stream","text":"merchant_id\n\nmerchant_category_id\n\nstate_id\n\nsubsector_id\n\ncity_id\n"}]},{"cell_type":"code","source":"# 创建空DataFrame用于保存NLP特征\ntrain_x = pd.DataFrame()\ntest_x = pd.DataFrame()\n\n# 实例化CountVectorizer评估器与TfidfVectorizer评估器\ncntv = CountVectorizer()\ntfv = TfidfVectorizer(ngram_range=(1, 2), min_df=3, max_df=0.9, use_idf=1, smooth_idf=1, sublinear_tf=1)\n\n# 创建空列表用户保存修正后的列名称\nvector_feature =[]\nfor co in ['merchant_id', 'merchant_category_id', 'state_id', 'subsector_id', 'city_id']:\n vector_feature.extend([co+'_new', co+'_hist', co+'_all'])\n \n# 提取每一列进行新特征衍生\nfor feature in vector_feature:\n print(feature)\n cntv.fit([feature].append(test[feature]))\n train_x = sparse.hstack((train_x, cntv.transform(train[feature]))).tocsr()\n test_x = sparse.hstack((test_x, cntv.transform(test[feature]))).tocsr()\n \n tfv.fit(train[feature].append(test[feature]))\n train_x = sparse.hstack((train_x, tfv.transform(train[feature]))).tocsr()\n test_x = sparse.hstack((test_x, tfv.transform(test[feature]))).tocsr()\n \n# 保存NLP特征衍生结果\nsparse.save_npz(\"preprocess/train_nlp.npz\", train_x)\nsparse.save_npz(\"preprocess/test_nlp.npz\", test_x)","metadata":{},"execution_count":4,"outputs":[{"name":"stdout","output_type":"stream","text":"merchant_id_new\n\nmerchant_id_hist\n\nmerchant_id_all\n\nmerchant_category_id_new\n\nmerchant_category_id_hist\n\nmerchant_category_id_all\n\nstate_id_new\n\nstate_id_hist\n\nstate_id_all\n\nsubsector_id_new\n\nsubsector_id_hist\n\nsubsector_id_all\n\ncity_id_new\n\ncity_id_hist\n\ncity_id_all\n"}]},{"cell_type":"code","source":"train_x.shape","metadata":{},"execution_count":5,"outputs":[{"execution_count":5,"output_type":"execute_result","data":{"text/plain":["(201917, 1846286)"]},"metadata":{}}]},{"cell_type":"markdown","source":"### 2.XGBoost模型训练与优化","metadata":{}},{"cell_type":"markdown","source":"- 数据读取","metadata":{}},{"cell_type":"markdown","source":"  接下来，来进行XGBoost的模型训练与优化，首先需要导入必要的包以及进行数据读取：","metadata":{}},{"cell_type":"code","source":"import xgboost as xgb\nfrom sklearn.feature_selection import f_regression\nfrom numpy.random import RandomState\nfrom bayes_opt import BayesianOptimization","metadata":{},"execution_count":11,"outputs":[]},{"cell_type":"code","source":"train = pd.read_csv('preprocess/train.csv')\ntest = pd.read_csv('preprocess/test.csv')","metadata":{},"execution_count":13,"outputs":[]},{"cell_type":"code","source":"features = train.columns.tolist()\nfeatures.remove('card_id')\nfeatures.remove('target')\n\ntrain_x = sparse.load_npz(\"preprocess/train_nlp.npz\")\ntest_x = sparse.load_npz(\"preprocess/test_nlp.npz\")\n\ntrain_x = sparse.hstack((train_x, train[features])).tocsr()\ntest_x = sparse.hstack((test_x, test[features])).tocsr()","metadata":{},"execution_count":14,"outputs":[]},{"cell_type":"markdown","source":"- 模型训练与优化","metadata":{}},{"cell_type":"markdown","source":"  接下来进行模型训练，本轮训练的流程和lgb模型流程类似，首先需要创建用于重复申明固定参数的函数，然后定义搜索和优化函数，并在优化函数内部调用参数回调函数，最后定义模型预测函数，该函数将借助交叉验证过程来进行测试集预测，并同步创建验证集预测结果与每个模型对测试集的预测结果。","metadata":{}},{"cell_type":"code","source":"# 参数回调函数\ndef params_append(params):\n \"\"\"\n\n :param params:\n :return:\n \"\"\"\n params['objective'] = 'reg:squarederror'\n params['eval_metric'] = 'rmse'\n params[\"min_child_weight\"] = int(params[\"min_child_weight\"])\n params['max_depth'] = int(params['max_depth'])\n return params\n\n# 模型优化函数\ndef param_beyesian(train):\n \"\"\"\n\n :param train:\n :return:\n \"\"\"\n # Part 1.数据准备\n train_y = pd.read_csv(\"data/train.csv\")['target']\n # 数据封装\n sample_index = train_y.sample(frac=0.1, random_state=2020).index.tolist()\n train_data = xgb.DMatrix(train.tocsr()[sample_index, :\n ], train_y.loc[sample_index].values, silent=True)\n \n # 借助cv过程构建目标函数\n def xgb_cv(colsample_bytree, subsample, min_child_weight, max_depth,\n reg_alpha, eta,\n reg_lambda):\n \"\"\"\n\n :param colsample_bytree:\n :param subsample:\n :param min_child_weight:\n :param max_depth:\n :param reg_alpha:\n :param eta:\n :param reg_lambda:\n :return:\n \"\"\"\n params = {'objective': 'reg:squarederror',\n 'early_stopping_round': 50,\n 'eval_metric': 'rmse'}\n params['colsample_bytree'] = max(min(colsample_bytree, 1), 0)\n params['subsample'] = max(min(subsample, 1), 0)\n params[\"min_child_weight\"] = int(min_child_weight)\n params['max_depth'] = int(max_depth)\n params['eta'] = float(eta)\n params['reg_alpha'] = max(reg_alpha, 0)\n params['reg_lambda'] = max(reg_lambda, 0)\n print(params)\n cv_result = xgb.cv(params, train_data,\n num_boost_round=1000,\n nfold=2, seed=2,\n stratified=False,\n shuffle=True,\n early_stopping_rounds=30,\n verbose_eval=False)\n return -min(cv_result['test-rmse-mean'])\n \n # 调用贝叶斯优化器进行模型优化\n xgb_bo = BayesianOptimization(\n xgb_cv,\n {'colsample_bytree': (0.5, 1),\n 'subsample': (0.5, 1),\n 'min_child_weight': (1, 30),\n 'max_depth': (5, 12),\n 'reg_alpha': (0, 5),\n 'eta':(0.02, 0.2),\n 'reg_lambda': (0, 5)}\n )\n xgb_bo.maximize(init_points=21, n_iter=5) # init_points表示初始点，n_iter代表迭代次数（即采样数）\n print(xgb_bo.max['target'], xgb_bo.max['params'])\n return xgb_bo.max['params']\n\n# 交叉验证预测函数\ndef train_predict(train, test, params):\n \"\"\"\n\n :param train:\n :param test:\n :param params:\n :return:\n \"\"\"\n train_y = pd.read_csv(\"data/train.csv\")['target']\n test_data = xgb.DMatrix(test)\n\n params = params_append(params)\n kf = KFold(n_splits=5, random_state=2020, shuffle=True)\n prediction_test = 0\n cv_score = []\n prediction_train = pd.Series()\n ESR = 30\n NBR = 10000\n VBE = 50\n for train_part_index, eval_index in kf.split(train, train_y):\n # 模型训练\n train_part = xgb.DMatrix(train.tocsr()[train_part_index, :],\n train_y.loc[train_part_index])\n eval = xgb.DMatrix(train.tocsr()[eval_index, :],\n train_y.loc[eval_index])\n bst = xgb.train(params, train_part, NBR, [(train_part, 'train'),\n (eval, 'eval')], verbose_eval=VBE,\n maximize=False, early_stopping_rounds=ESR, )\n prediction_test += bst.predict(test_data)\n eval_pre = bst.predict(eval)\n prediction_train = prediction_train.append(pd.Series(eval_pre, index=eval_index))\n score = np.sqrt(mean_squared_error(train_y.loc[eval_index].values, eval_pre))\n cv_score.append(score)\n print(cv_score, sum(cv_score) / 5)\n pd.Series(prediction_train.sort_index().values).to_csv(\"preprocess/train_xgboost.csv\", index=False)\n pd.Series(prediction_test / 5).to_csv(\"preprocess/test_xgboost.csv\", index=False)\n test = pd.read_csv('data/test.csv')\n test['target'] = prediction_test / 5\n test[['card_id', 'target']].to_csv(\"result/submission_xgboost.csv\", index=False)\n return","metadata":{},"execution_count":15,"outputs":[]},{"cell_type":"code","source":"best_clf = param_beyesian(train_x)","metadata":{"tags":[],"collapsed":true,"jupyter":{"outputs_hidden":true}},"execution_count":16,"outputs":[{"name":"stdout","output_type":"stream","text":"| iter | target | colsam... | eta | max_depth | min_ch... | reg_alpha | reg_la... | subsample |\n\n-------------------------------------------------------------------------------------------------------------\n\n{'objective': 'reg:squarederror', 'early_stopping_round': 50, 'eval_metric': 'rmse', 'colsample_bytree': 0.9008280232486963, 'subsample': 0.8904106813206965, 'min_child_weight': 29, 'max_depth': 9, 'eta': 0.15949688000040066, 'reg_alpha': 1.8850960576404892, 'reg_lambda': 4.436160789463865}\n\n[12:52:57] WARNING: C:/Users/Administrator/workspace/xgboost-win64_release_1.5.0/src/learner.cc:576: \n\nParameters: { \"early_stopping_round\" } might not be used.\n\n\n\n This could be a false alarm, with some parameters getting used by language bindings but\n\n then being mistakenly passed down to XGBoost core, or some parameter actually being used\n\n but getting flagged wrongly here. Please open an issue if you find any such cases.\n\n\n\n\n\n[12:52:59] WARNING: C:/Users/Administrator/workspace/xgboost-win64_release_1.5.0/src/learner.cc:576: \n\nParameters: { \"early_stopping_round\" } might not be used.\n\n\n\n This could be a false alarm, with some parameters getting used by language bindings but\n\n then being mistakenly passed down to XGBoost core, or some parameter actually being used\n\n but getting flagged wrongly here. Please open an issue if you find any such cases.\n\n\n\n\n\n| \u001b[0m 1 \u001b[0m | \u001b[0m-3.828 \u001b[0m | \u001b[0m 0.9008 \u001b[0m | \u001b[0m 0.1595 \u001b[0m | \u001b[0m 9.636 \u001b[0m | \u001b[0m 29.93 \u001b[0m | \u001b[0m 1.885 \u001b[0m | \u001b[0m 4.436 \u001b[0m | \u001b[0m 0.8904 \u001b[0m |\n\n{'objective': 'reg:squarederror', 'early_stopping_round': 50, 'eval_metric': 'rmse', 'colsample_bytree': 0.8733613720817149, 'subsample': 0.6142495949337223, 'min_child_weight': 29, 'max_depth': 8, 'eta': 0.1802456543940385, 'reg_alpha': 0.8072782146576124, 'reg_lambda': 2.284382168716956}\n\n[12:54:07] WARNING: C:/Users/Administrator/workspace/xgboost-win64_release_1.5.0/src/learner.cc:576: \n\nParameters: { \"early_stopping_round\" } might not be used.\n\n\n\n This could be a false alarm, with some parameters getting used by language bindings but\n\n then being mistakenly passed down to XGBoost core, or some parameter actually being used\n\n but getting flagged wrongly here. Please open an issue if you find any such cases.\n\n\n\n\n\n[12:54:08] WARNING: C:/Users/Administrator/workspace/xgboost-win64_release_1.5.0/src/learner.cc:576: \n\nParameters: { \"early_stopping_round\" } might not be used.\n\n\n\n This could be a false alarm, with some parameters getting used by language bindings but\n\n then being mistakenly passed down to XGBoost core, or some parameter actually being used\n\n but getting flagged wrongly here. Please open an issue if you find any such cases.\n\n\n\n\n\n| \u001b[0m 2 \u001b[0m | \u001b[0m-3.861 \u001b[0m | \u001b[0m 0.8734 \u001b[0m | \u001b[0m 0.1802 \u001b[0m | \u001b[0m 8.376 \u001b[0m | \u001b[0m 29.33 \u001b[0m | \u001b[0m 0.8073 \u001b[0m | \u001b[0m 2.284 \u001b[0m | \u001b[0m 0.6142 \u001b[0m |\n\n{'objective': 'reg:squarederror', 'early_stopping_round': 50, 'eval_metric': 'rmse', 'colsample_bytree': 0.9268730478860339, 'subsample': 0.9187182128076231, 'min_child_weight': 18, 'max_depth': 10, 'eta': 0.10485164882690341, 'reg_alpha': 1.4052740152170013, 'reg_lambda': 4.420615175732838}\n\n[12:55:01] WARNING: C:/Users/Administrator/workspace/xgboost-win64_release_1.5.0/src/learner.cc:576: \n\nParameters: { \"early_stopping_round\" } might not be used.\n\n\n\n This could be a false alarm, with some parameters getting used by language bindings but\n\n then being mistakenly passed down to XGBoost core, or some parameter actually being used\n\n but getting flagged wrongly here. Please open an issue if you find any such cases.\n\n\n\n\n\n[12:55:02] WARNING: C:/Users/Administrator/workspace/xgboost-win64_release_1.5.0/src/learner.cc:576: \n\nParameters: { \"early_stopping_round\" } might not be used.\n\n\n\n This could be a false alarm, with some parameters getting used by language bindings but\n\n then being mistakenly passed down to XGBoost core, or some parameter actually being used\n\n but getting flagged wrongly here. Please open an issue if you find any such cases.\n\n\n\n\n\n| \u001b[0m 3 \u001b[0m | \u001b[0m-3.848 \u001b[0m | \u001b[0m 0.9269 \u001b[0m | \u001b[0m 0.1049 \u001b[0m | \u001b[0m 10.6 \u001b[0m | \u001b[0m 18.53 \u001b[0m | \u001b[0m 1.405 \u001b[0m | \u001b[0m 4.421 \u001b[0m | \u001b[0m 0.9187 \u001b[0m |\n\n{'objective': 'reg:squarederror', 'early_stopping_round': 50, 'eval_metric': 'rmse', 'colsample_bytree': 0.625422443194849, 'subsample': 0.6710318901133161, 'min_child_weight': 27, 'max_depth': 8, 'eta': 0.05185743596319062, 'reg_alpha': 1.1690459082772615, 'reg_lambda': 3.562136294750095}\n\n[12:56:38] WARNING: C:/Users/Administrator/workspace/xgboost-win64_release_1.5.0/src/learner.cc:576: \n\nParameters: { \"early_stopping_round\" } might not be used.\n\n\n\n This could be a false alarm, with some parameters getting used by language bindings but\n\n then being mistakenly passed down to XGBoost core, or some parameter actually being used\n\n but getting flagged wrongly here. Please open an issue if you find any such cases.\n\n\n\n\n\n[12:56:39] WARNING: C:/Users/Administrator/workspace/xgboost-win64_release_1.5.0/src/learner.cc:576: \n\nParameters: { \"early_stopping_round\" } might not be used.\n\n\n\n This could be a false alarm, with some parameters getting used by language bindings but\n\n then being mistakenly passed down to XGBoost core, or some parameter actually being used\n\n but getting flagged wrongly here. Please open an issue if you find any such cases.\n\n\n\n\n\n| \u001b[0m 4 \u001b[0m | \u001b[0m-3.829 \u001b[0m | \u001b[0m 0.6254 \u001b[0m | \u001b[0m 0.05186 \u001b[0m | \u001b[0m 8.038 \u001b[0m | \u001b[0m 27.47 \u001b[0m | \u001b[0m 1.169 \u001b[0m | \u001b[0m 3.562 \u001b[0m | \u001b[0m 0.671 \u001b[0m |\n\n{'objective': 'reg:squarederror', 'early_stopping_round': 50, 'eval_metric': 'rmse', 'colsample_bytree': 0.902497211065604, 'subsample': 0.8413464797085647, 'min_child_weight': 27, 'max_depth': 8, 'eta': 0.09748251874712145, 'reg_alpha': 2.9238818257239756, 'reg_lambda': 0.6901256237971481}\n\n[12:58:06] WARNING: C:/Users/Administrator/workspace/xgboost-win64_release_1.5.0/src/learner.cc:576: \n\nParameters: { \"early_stopping_round\" } might not be used.\n\n\n\n This could be a false alarm, with some parameters getting used by language bindings but\n\n then being mistakenly passed down to XGBoost core, or some parameter actually being used\n\n but getting flagged wrongly here. Please open an issue if you find any such cases.\n\n\n\n\n\n[12:58:07] WARNING: C:/Users/Administrator/workspace/xgboost-win64_release_1.5.0/src/learner.cc:576: \n\nParameters: { \"early_stopping_round\" } might not be used.\n\n\n\n This could be a false alarm, with some parameters getting used by language bindings but\n\n then being mistakenly passed down to XGBoost core, or some parameter actually being used\n\n but getting flagged wrongly here. Please open an issue if you find any such cases.\n\n\n\n\n\n| \u001b[95m 5 \u001b[0m | \u001b[95m-3.824 \u001b[0m | \u001b[95m 0.9025 \u001b[0m | \u001b[95m 0.09748 \u001b[0m | \u001b[95m 8.835 \u001b[0m | \u001b[95m 27.77 \u001b[0m | \u001b[95m 2.924 \u001b[0m | \u001b[95m 0.6901 \u001b[0m | \u001b[95m 0.8413 \u001b[0m |\n\n{'objective': 'reg:squarederror', 'early_stopping_round': 50, 'eval_metric': 'rmse', 'colsample_bytree': 0.9977024155365132, 'subsample': 0.8115473489069764, 'min_child_weight': 29, 'max_depth': 8, 'eta': 0.08573985701358497, 'reg_alpha': 4.39911724928385, 'reg_lambda': 0.98249230645374}\n\n[12:59:25] WARNING: C:/Users/Administrator/workspace/xgboost-win64_release_1.5.0/src/learner.cc:576: \n\nParameters: { \"early_stopping_round\" } might not be used.\n\n\n\n This could be a false alarm, with some parameters getting used by language bindings but\n\n then being mistakenly passed down to XGBoost core, or some parameter actually being used\n\n but getting flagged wrongly here. Please open an issue if you find any such cases.\n\n\n\n\n\n[12:59:26] WARNING: C:/Users/Administrator/workspace/xgboost-win64_release_1.5.0/src/learner.cc:576: \n\nParameters: { \"early_stopping_round\" } might not be used.\n\n\n\n This could be a false alarm, with some parameters getting used by language bindings but\n\n then being mistakenly passed down to XGBoost core, or some parameter actually being used\n\n but getting flagged wrongly here. Please open an issue if you find any such cases.\n\n\n\n\n\n| \u001b[0m 6 \u001b[0m | \u001b[0m-3.827 \u001b[0m | \u001b[0m 0.9977 \u001b[0m | \u001b[0m 0.08574 \u001b[0m | \u001b[0m 8.21 \u001b[0m | \u001b[0m 29.41 \u001b[0m | \u001b[0m 4.399 \u001b[0m | \u001b[0m 0.9825 \u001b[0m | \u001b[0m 0.8115 \u001b[0m |\n\n{'objective': 'reg:squarederror', 'early_stopping_round': 50, 'eval_metric': 'rmse', 'colsample_bytree': 0.9407066205905263, 'subsample': 0.827880546540135, 'min_child_weight': 6, 'max_depth': 9, 'eta': 0.1061485706403698, 'reg_alpha': 2.139211485717212, 'reg_lambda': 3.2008526648164737}\n\n[13:00:53] WARNING: C:/Users/Administrator/workspace/xgboost-win64_release_1.5.0/src/learner.cc:576: \n\nParameters: { \"early_stopping_round\" } might not be used.\n\n\n\n This could be a false alarm, with some parameters getting used by language bindings but\n\n then being mistakenly passed down to XGBoost core, or some parameter actually being used\n\n but getting flagged wrongly here. Please open an issue if you find any such cases.\n\n\n\n\n\n[13:00:55] WARNING: C:/Users/Administrator/workspace/xgboost-win64_release_1.5.0/src/learner.cc:576: \n\nParameters: { \"early_stopping_round\" } might not be used.\n\n\n\n This could be a false alarm, with some parameters getting used by language bindings but\n\n then being mistakenly passed down to XGBoost core, or some parameter actually being used\n\n but getting flagged wrongly here. Please open an issue if you find any such cases.\n\n\n\n\n\n| \u001b[0m 7 \u001b[0m | \u001b[0m-3.851 \u001b[0m | \u001b[0m 0.9407 \u001b[0m | \u001b[0m 0.1061 \u001b[0m | \u001b[0m 9.67 \u001b[0m | \u001b[0m 6.107 \u001b[0m | \u001b[0m 2.139 \u001b[0m | \u001b[0m 3.201 \u001b[0m | \u001b[0m 0.8279 \u001b[0m |\n\n{'objective': 'reg:squarederror', 'early_stopping_round': 50, 'eval_metric': 'rmse', 'colsample_bytree': 0.6566786860590916, 'subsample': 0.6452363126120251, 'min_child_weight': 18, 'max_depth': 8, 'eta': 0.14417434852557892, 'reg_alpha': 4.3160287883092066, 'reg_lambda': 3.5328339088472847}\n\n[13:02:17] WARNING: C:/Users/Administrator/workspace/xgboost-win64_release_1.5.0/src/learner.cc:576: \n\nParameters: { \"early_stopping_round\" } might not be used.\n\n\n\n This could be a false alarm, with some parameters getting used by language bindings but\n\n then being mistakenly passed down to XGBoost core, or some parameter actually being used\n\n but getting flagged wrongly here. Please open an issue if you find any such cases.\n\n\n\n\n\n[13:02:18] WARNING: C:/Users/Administrator/workspace/xgboost-win64_release_1.5.0/src/learner.cc:576: \n\nParameters: { \"early_stopping_round\" } might not be used.\n\n\n\n This could be a false alarm, with some parameters getting used by language bindings but\n\n then being mistakenly passed down to XGBoost core, or some parameter actually being used\n\n but getting flagged wrongly here. Please open an issue if you find any such cases.\n\n\n\n\n\n| \u001b[0m 8 \u001b[0m | \u001b[0m-3.848 \u001b[0m | \u001b[0m 0.6567 \u001b[0m | \u001b[0m 0.1442 \u001b[0m | \u001b[0m 8.496 \u001b[0m | \u001b[0m 18.79 \u001b[0m | \u001b[0m 4.316 \u001b[0m | \u001b[0m 3.533 \u001b[0m | \u001b[0m 0.6452 \u001b[0m |\n\n{'objective': 'reg:squarederror', 'early_stopping_round': 50, 'eval_metric': 'rmse', 'colsample_bytree': 0.611191859916479, 'subsample': 0.9607934404730234, 'min_child_weight': 10, 'max_depth': 5, 'eta': 0.16887507984941832, 'reg_alpha': 1.8949997250691952, 'reg_lambda': 1.9700264363894808}\n\n[13:03:08] WARNING: C:/Users/Administrator/workspace/xgboost-win64_release_1.5.0/src/learner.cc:576: \n\nParameters: { \"early_stopping_round\" } might not be used.\n\n\n\n This could be a false alarm, with some parameters getting used by language bindings but\n\n then being mistakenly passed down to XGBoost core, or some parameter actually being used\n\n but getting flagged wrongly here. Please open an issue if you find any such cases.\n\n\n\n\n\n[13:03:09] WARNING: C:/Users/Administrator/workspace/xgboost-win64_release_1.5.0/src/learner.cc:576: \n\nParameters: { \"early_stopping_round\" } might not be used.\n\n\n\n This could be a false alarm, with some parameters getting used by language bindings but\n\n then being mistakenly passed down to XGBoost core, or some parameter actually being used\n\n but getting flagged wrongly here. Please open an issue if you find any such cases.\n\n\n\n\n\n| \u001b[0m 9 \u001b[0m | \u001b[0m-3.84 \u001b[0m | \u001b[0m 0.6112 \u001b[0m | \u001b[0m 0.1689 \u001b[0m | \u001b[0m 5.955 \u001b[0m | \u001b[0m 10.8 \u001b[0m | \u001b[0m 1.895 \u001b[0m | \u001b[0m 1.97 \u001b[0m | \u001b[0m 0.9608 \u001b[0m |\n\n{'objective': 'reg:squarederror', 'early_stopping_round': 50, 'eval_metric': 'rmse', 'colsample_bytree': 0.5688557898775366, 'subsample': 0.792738671048387, 'min_child_weight': 8, 'max_depth': 8, 'eta': 0.029707284678996262, 'reg_alpha': 2.13929302944516, 'reg_lambda': 1.716127401031825}\n\n[13:03:51] WARNING: C:/Users/Administrator/workspace/xgboost-win64_release_1.5.0/src/learner.cc:576: \n\nParameters: { \"early_stopping_round\" } might not be used.\n\n\n\n This could be a false alarm, with some parameters getting used by language bindings but\n\n then being mistakenly passed down to XGBoost core, or some parameter actually being used\n\n but getting flagged wrongly here. Please open an issue if you find any such cases.\n\n\n\n\n\n[13:03:51] WARNING: C:/Users/Administrator/workspace/xgboost-win64_release_1.5.0/src/learner.cc:576: \n\nParameters: { \"early_stopping_round\" } might not be used.\n\n\n\n This could be a false alarm, with some parameters getting used by language bindings but\n\n then being mistakenly passed down to XGBoost core, or some parameter actually being used\n\n but getting flagged wrongly here. Please open an issue if you find any such cases.\n\n\n\n\n\n| \u001b[0m 10 \u001b[0m | \u001b[0m-3.833 \u001b[0m | \u001b[0m 0.5689 \u001b[0m | \u001b[0m 0.02971 \u001b[0m | \u001b[0m 8.436 \u001b[0m | \u001b[0m 8.559 \u001b[0m | \u001b[0m 2.139 \u001b[0m | \u001b[0m 1.716 \u001b[0m | \u001b[0m 0.7927 \u001b[0m |\n\n{'objective': 'reg:squarederror', 'early_stopping_round': 50, 'eval_metric': 'rmse', 'colsample_bytree': 0.8909115661247835, 'subsample': 0.838354468762074, 'min_child_weight': 24, 'max_depth': 6, 'eta': 0.11054484780152028, 'reg_alpha': 4.1364437329554455, 'reg_lambda': 3.1715376094875305}\n\n[13:05:35] WARNING: C:/Users/Administrator/workspace/xgboost-win64_release_1.5.0/src/learner.cc:576: \n\nParameters: { \"early_stopping_round\" } might not be used.\n\n\n\n This could be a false alarm, with some parameters getting used by language bindings but\n\n then being mistakenly passed down to XGBoost core, or some parameter actually being used\n\n but getting flagged wrongly here. Please open an issue if you find any such cases.\n\n\n\n\n\n[13:05:36] WARNING: C:/Users/Administrator/workspace/xgboost-win64_release_1.5.0/src/learner.cc:576: \n\nParameters: { \"early_stopping_round\" } might not be used.\n\n\n\n This could be a false alarm, with some parameters getting used by language bindings but\n\n then being mistakenly passed down to XGBoost core, or some parameter actually being used\n\n but getting flagged wrongly here. Please open an issue if you find any such cases.\n\n\n\n\n\n| \u001b[95m 11 \u001b[0m | \u001b[95m-3.821 \u001b[0m | \u001b[95m 0.8909 \u001b[0m | \u001b[95m 0.1105 \u001b[0m | \u001b[95m 6.204 \u001b[0m | \u001b[95m 24.25 \u001b[0m | \u001b[95m 4.136 \u001b[0m | \u001b[95m 3.172 \u001b[0m | \u001b[95m 0.8384 \u001b[0m |\n\n{'objective': 'reg:squarederror', 'early_stopping_round': 50, 'eval_metric': 'rmse', 'colsample_bytree': 0.9465569893212517, 'subsample': 0.5866450532823893, 'min_child_weight': 13, 'max_depth': 7, 'eta': 0.1596273004501417, 'reg_alpha': 2.364218860201553, 'reg_lambda': 1.7722559715723234}\n\n[13:06:37] WARNING: C:/Users/Administrator/workspace/xgboost-win64_release_1.5.0/src/learner.cc:576: \n\nParameters: { \"early_stopping_round\" } might not be used.\n\n\n\n This could be a false alarm, with some parameters getting used by language bindings but\n\n then being mistakenly passed down to XGBoost core, or some parameter actually being used\n\n but getting flagged wrongly here. Please open an issue if you find any such cases.\n\n\n\n\n\n[13:06:38] WARNING: C:/Users/Administrator/workspace/xgboost-win64_release_1.5.0/src/learner.cc:576: \n\nParameters: { \"early_stopping_round\" } might not be used.\n\n\n\n This could be a false alarm, with some parameters getting used by language bindings but\n\n then being mistakenly passed down to XGBoost core, or some parameter actually being used\n\n but getting flagged wrongly here. Please open an issue if you find any such cases.\n\n\n\n\n\n| \u001b[0m 12 \u001b[0m | \u001b[0m-3.881 \u001b[0m | \u001b[0m 0.9466 \u001b[0m | \u001b[0m 0.1596 \u001b[0m | \u001b[0m 7.621 \u001b[0m | \u001b[0m 13.79 \u001b[0m | \u001b[0m 2.364 \u001b[0m | \u001b[0m 1.772 \u001b[0m | \u001b[0m 0.5866 \u001b[0m |\n\n{'objective': 'reg:squarederror', 'early_stopping_round': 50, 'eval_metric': 'rmse', 'colsample_bytree': 0.9627435399195992, 'subsample': 0.9808950735528328, 'min_child_weight': 27, 'max_depth': 6, 'eta': 0.05741987032403584, 'reg_alpha': 3.9968620363457683, 'reg_lambda': 0.4830516515499872}\n\n[13:07:28] WARNING: C:/Users/Administrator/workspace/xgboost-win64_release_1.5.0/src/learner.cc:576: \n\nParameters: { \"early_stopping_round\" } might not be used.\n\n\n\n This could be a false alarm, with some parameters getting used by language bindings but\n\n then being mistakenly passed down to XGBoost core, or some parameter actually being used\n\n but getting flagged wrongly here. Please open an issue if you find any such cases.\n\n\n\n\n\n[13:07:29] WARNING: C:/Users/Administrator/workspace/xgboost-win64_release_1.5.0/src/learner.cc:576: \n\nParameters: { \"early_stopping_round\" } might not be used.\n\n\n\n This could be a false alarm, with some parameters getting used by language bindings but\n\n then being mistakenly passed down to XGBoost core, or some parameter actually being used\n\n but getting flagged wrongly here. Please open an issue if you find any such cases.\n\n\n\n\n\n| \u001b[95m 13 \u001b[0m | \u001b[95m-3.812 \u001b[0m | \u001b[95m 0.9627 \u001b[0m | \u001b[95m 0.05742 \u001b[0m | \u001b[95m 6.407 \u001b[0m | \u001b[95m 27.35 \u001b[0m | \u001b[95m 3.997 \u001b[0m | \u001b[95m 0.4831 \u001b[0m | \u001b[95m 0.9809 \u001b[0m |\n\n{'objective': 'reg:squarederror', 'early_stopping_round': 50, 'eval_metric': 'rmse', 'colsample_bytree': 0.9044844729820345, 'subsample': 0.5762009686960115, 'min_child_weight': 24, 'max_depth': 5, 'eta': 0.16975603999498512, 'reg_alpha': 3.085476522155029, 'reg_lambda': 3.5171910395636363}\n\n[13:09:11] WARNING: C:/Users/Administrator/workspace/xgboost-win64_release_1.5.0/src/learner.cc:576: \n\nParameters: { \"early_stopping_round\" } might not be used.\n\n\n\n This could be a false alarm, with some parameters getting used by language bindings but\n\n then being mistakenly passed down to XGBoost core, or some parameter actually being used\n\n but getting flagged wrongly here. Please open an issue if you find any such cases.\n\n\n\n\n\n[13:09:12] WARNING: C:/Users/Administrator/workspace/xgboost-win64_release_1.5.0/src/learner.cc:576: \n\nParameters: { \"early_stopping_round\" } might not be used.\n\n\n\n This could be a false alarm, with some parameters getting used by language bindings but\n\n then being mistakenly passed down to XGBoost core, or some parameter actually being used\n\n but getting flagged wrongly here. Please open an issue if you find any such cases.\n\n\n\n\n\n| \u001b[0m 14 \u001b[0m | \u001b[0m-3.838 \u001b[0m | \u001b[0m 0.9045 \u001b[0m | \u001b[0m 0.1698 \u001b[0m | \u001b[0m 5.856 \u001b[0m | \u001b[0m 24.39 \u001b[0m | \u001b[0m 3.085 \u001b[0m | \u001b[0m 3.517 \u001b[0m | \u001b[0m 0.5762 \u001b[0m |\n\n{'objective': 'reg:squarederror', 'early_stopping_round': 50, 'eval_metric': 'rmse', 'colsample_bytree': 0.7325643867708108, 'subsample': 0.5099111689834741, 'min_child_weight': 2, 'max_depth': 6, 'eta': 0.1764589326251524, 'reg_alpha': 1.802265951008657, 'reg_lambda': 3.2543009145085158}\n\n[13:09:55] WARNING: C:/Users/Administrator/workspace/xgboost-win64_release_1.5.0/src/learner.cc:576: \n\nParameters: { \"early_stopping_round\" } might not be used.\n\n\n\n This could be a false alarm, with some parameters getting used by language bindings but\n\n then being mistakenly passed down to XGBoost core, or some parameter actually being used\n\n but getting flagged wrongly here. Please open an issue if you find any such cases.\n\n\n\n\n\n[13:09:56] WARNING: C:/Users/Administrator/workspace/xgboost-win64_release_1.5.0/src/learner.cc:576: \n\nParameters: { \"early_stopping_round\" } might not be used.\n\n\n\n This could be a false alarm, with some parameters getting used by language bindings but\n\n then being mistakenly passed down to XGBoost core, or some parameter actually being used\n\n but getting flagged wrongly here. Please open an issue if you find any such cases.\n\n\n\n\n\n| \u001b[0m 15 \u001b[0m | \u001b[0m-3.897 \u001b[0m | \u001b[0m 0.7326 \u001b[0m | \u001b[0m 0.1765 \u001b[0m | \u001b[0m 6.418 \u001b[0m | \u001b[0m 2.489 \u001b[0m | \u001b[0m 1.802 \u001b[0m | \u001b[0m 3.254 \u001b[0m | \u001b[0m 0.5099 \u001b[0m |\n\n{'objective': 'reg:squarederror', 'early_stopping_round': 50, 'eval_metric': 'rmse', 'colsample_bytree': 0.6131120218876751, 'subsample': 0.5253206916632518, 'min_child_weight': 11, 'max_depth': 6, 'eta': 0.1719844380323167, 'reg_alpha': 2.357407785582822, 'reg_lambda': 3.375957679817562}\n\n[13:10:38] WARNING: C:/Users/Administrator/workspace/xgboost-win64_release_1.5.0/src/learner.cc:576: \n\nParameters: { \"early_stopping_round\" } might not be used.\n\n\n\n This could be a false alarm, with some parameters getting used by language bindings but\n\n then being mistakenly passed down to XGBoost core, or some parameter actually being used\n\n but getting flagged wrongly here. Please open an issue if you find any such cases.\n\n\n\n\n\n[13:10:39] WARNING: C:/Users/Administrator/workspace/xgboost-win64_release_1.5.0/src/learner.cc:576: \n\nParameters: { \"early_stopping_round\" } might not be used.\n\n\n\n This could be a false alarm, with some parameters getting used by language bindings but\n\n then being mistakenly passed down to XGBoost core, or some parameter actually being used\n\n but getting flagged wrongly here. Please open an issue if you find any such cases.\n\n\n\n\n\n| \u001b[0m 16 \u001b[0m | \u001b[0m-3.865 \u001b[0m | \u001b[0m 0.6131 \u001b[0m | \u001b[0m 0.172 \u001b[0m | \u001b[0m 6.231 \u001b[0m | \u001b[0m 11.49 \u001b[0m | \u001b[0m 2.357 \u001b[0m | \u001b[0m 3.376 \u001b[0m | \u001b[0m 0.5253 \u001b[0m |\n\n{'objective': 'reg:squarederror', 'early_stopping_round': 50, 'eval_metric': 'rmse', 'colsample_bytree': 0.6831871509139502, 'subsample': 0.9297109754890269, 'min_child_weight': 2, 'max_depth': 10, 'eta': 0.04473277839394052, 'reg_alpha': 1.3430060896665403, 'reg_lambda': 3.918534812739465}\n\n[13:11:15] WARNING: C:/Users/Administrator/workspace/xgboost-win64_release_1.5.0/src/learner.cc:576: \n\nParameters: { \"early_stopping_round\" } might not be used.\n\n\n\n This could be a false alarm, with some parameters getting used by language bindings but\n\n then being mistakenly passed down to XGBoost core, or some parameter actually being used\n\n but getting flagged wrongly here. Please open an issue if you find any such cases.\n\n\n\n\n\n[13:11:16] WARNING: C:/Users/Administrator/workspace/xgboost-win64_release_1.5.0/src/learner.cc:576: \n\nParameters: { \"early_stopping_round\" } might not be used.\n\n\n\n This could be a false alarm, with some parameters getting used by language bindings but\n\n then being mistakenly passed down to XGBoost core, or some parameter actually being used\n\n but getting flagged wrongly here. Please open an issue if you find any such cases.\n\n\n\n\n\n| \u001b[0m 17 \u001b[0m | \u001b[0m-3.822 \u001b[0m | \u001b[0m 0.6832 \u001b[0m | \u001b[0m 0.04473 \u001b[0m | \u001b[0m 10.27 \u001b[0m | \u001b[0m 2.572 \u001b[0m | \u001b[0m 1.343 \u001b[0m | \u001b[0m 3.919 \u001b[0m | \u001b[0m 0.9297 \u001b[0m |\n\n{'objective': 'reg:squarederror', 'early_stopping_round': 50, 'eval_metric': 'rmse', 'colsample_bytree': 0.523125333707535, 'subsample': 0.7392177712247874, 'min_child_weight': 8, 'max_depth': 5, 'eta': 0.139490251639574, 'reg_alpha': 0.14523540355518005, 'reg_lambda': 0.5513235601440791}\n\n[13:13:51] WARNING: C:/Users/Administrator/workspace/xgboost-win64_release_1.5.0/src/learner.cc:576: \n\nParameters: { \"early_stopping_round\" } might not be used.\n\n\n\n This could be a false alarm, with some parameters getting used by language bindings but\n\n then being mistakenly passed down to XGBoost core, or some parameter actually being used\n\n but getting flagged wrongly here. Please open an issue if you find any such cases.\n\n\n\n\n\n[13:13:51] WARNING: C:/Users/Administrator/workspace/xgboost-win64_release_1.5.0/src/learner.cc:576: \n\nParameters: { \"early_stopping_round\" } might not be used.\n\n\n\n This could be a false alarm, with some parameters getting used by language bindings but\n\n then being mistakenly passed down to XGBoost core, or some parameter actually being used\n\n but getting flagged wrongly here. Please open an issue if you find any such cases.\n\n\n\n\n\n| \u001b[0m 18 \u001b[0m | \u001b[0m-3.856 \u001b[0m | \u001b[0m 0.5231 \u001b[0m | \u001b[0m 0.1395 \u001b[0m | \u001b[0m 5.779 \u001b[0m | \u001b[0m 8.058 \u001b[0m | \u001b[0m 0.1452 \u001b[0m | \u001b[0m 0.5513 \u001b[0m | \u001b[0m 0.7392 \u001b[0m |\n\n{'objective': 'reg:squarederror', 'early_stopping_round': 50, 'eval_metric': 'rmse', 'colsample_bytree': 0.7647893523603579, 'subsample': 0.9464736694936582, 'min_child_weight': 23, 'max_depth': 5, 'eta': 0.11531533678838223, 'reg_alpha': 4.517440315400995, 'reg_lambda': 0.063527226500667}\n\n[13:14:22] WARNING: C:/Users/Administrator/workspace/xgboost-win64_release_1.5.0/src/learner.cc:576: \n\nParameters: { \"early_stopping_round\" } might not be used.\n\n\n\n This could be a false alarm, with some parameters getting used by language bindings but\n\n then being mistakenly passed down to XGBoost core, or some parameter actually being used\n\n but getting flagged wrongly here. Please open an issue if you find any such cases.\n\n\n\n\n\n[13:14:23] WARNING: C:/Users/Administrator/workspace/xgboost-win64_release_1.5.0/src/learner.cc:576: \n\nParameters: { \"early_stopping_round\" } might not be used.\n\n\n\n This could be a false alarm, with some parameters getting used by language bindings but\n\n then being mistakenly passed down to XGBoost core, or some parameter actually being used\n\n but getting flagged wrongly here. Please open an issue if you find any such cases.\n\n\n\n\n\n| \u001b[0m 19 \u001b[0m | \u001b[0m-3.821 \u001b[0m | \u001b[0m 0.7648 \u001b[0m | \u001b[0m 0.1153 \u001b[0m | \u001b[0m 5.54 \u001b[0m | \u001b[0m 23.44 \u001b[0m | \u001b[0m 4.517 \u001b[0m | \u001b[0m 0.06353 \u001b[0m | \u001b[0m 0.9465 \u001b[0m |\n\n{'objective': 'reg:squarederror', 'early_stopping_round': 50, 'eval_metric': 'rmse', 'colsample_bytree': 0.7663684613824646, 'subsample': 0.681336879639247, 'min_child_weight': 29, 'max_depth': 11, 'eta': 0.0434838090691807, 'reg_alpha': 4.538172496852858, 'reg_lambda': 2.9134596476240175}\n\n[13:15:10] WARNING: C:/Users/Administrator/workspace/xgboost-win64_release_1.5.0/src/learner.cc:576: \n\nParameters: { \"early_stopping_round\" } might not be used.\n\n\n\n This could be a false alarm, with some parameters getting used by language bindings but\n\n then being mistakenly passed down to XGBoost core, or some parameter actually being used\n\n but getting flagged wrongly here. Please open an issue if you find any such cases.\n\n\n\n\n\n[13:15:11] WARNING: C:/Users/Administrator/workspace/xgboost-win64_release_1.5.0/src/learner.cc:576: \n\nParameters: { \"early_stopping_round\" } might not be used.\n\n\n\n This could be a false alarm, with some parameters getting used by language bindings but\n\n then being mistakenly passed down to XGBoost core, or some parameter actually being used\n\n but getting flagged wrongly here. Please open an issue if you find any such cases.\n\n\n\n\n\n| \u001b[0m 20 \u001b[0m | \u001b[0m-3.813 \u001b[0m | \u001b[0m 0.7664 \u001b[0m | \u001b[0m 0.04348 \u001b[0m | \u001b[0m 11.57 \u001b[0m | \u001b[0m 29.03 \u001b[0m | \u001b[0m 4.538 \u001b[0m | \u001b[0m 2.913 \u001b[0m | \u001b[0m 0.6813 \u001b[0m |\n\n{'objective': 'reg:squarederror', 'early_stopping_round': 50, 'eval_metric': 'rmse', 'colsample_bytree': 0.8349846959575062, 'subsample': 0.9790484991878241, 'min_child_weight': 7, 'max_depth': 5, 'eta': 0.11336872232404879, 'reg_alpha': 0.027603113711953675, 'reg_lambda': 3.4360173002023804}\n\n[13:17:46] WARNING: C:/Users/Administrator/workspace/xgboost-win64_release_1.5.0/src/learner.cc:576: \n\nParameters: { \"early_stopping_round\" } might not be used.\n\n\n\n This could be a false alarm, with some parameters getting used by language bindings but\n\n then being mistakenly passed down to XGBoost core, or some parameter actually being used\n\n but getting flagged wrongly here. Please open an issue if you find any such cases.\n\n\n\n\n\n[13:17:47] WARNING: C:/Users/Administrator/workspace/xgboost-win64_release_1.5.0/src/learner.cc:576: \n\nParameters: { \"early_stopping_round\" } might not be used.\n\n\n\n This could be a false alarm, with some parameters getting used by language bindings but\n\n then being mistakenly passed down to XGBoost core, or some parameter actually being used\n\n but getting flagged wrongly here. Please open an issue if you find any such cases.\n\n\n\n\n\n| \u001b[0m 21 \u001b[0m | \u001b[0m-3.83 \u001b[0m | \u001b[0m 0.835 \u001b[0m | \u001b[0m 0.1134 \u001b[0m | \u001b[0m 5.967 \u001b[0m | \u001b[0m 7.343 \u001b[0m | \u001b[0m 0.0276 \u001b[0m | \u001b[0m 3.436 \u001b[0m | \u001b[0m 0.979 \u001b[0m |\n\n{'objective': 'reg:squarederror', 'early_stopping_round': 50, 'eval_metric': 'rmse', 'colsample_bytree': 0.8207939072435217, 'subsample': 1.0, 'min_child_weight': 25, 'max_depth': 7, 'eta': 0.045052085309867346, 'reg_alpha': 4.63105080372857, 'reg_lambda': 1.2866888587445227}\n\n[13:18:46] WARNING: C:/Users/Administrator/workspace/xgboost-win64_release_1.5.0/src/learner.cc:576: \n\nParameters: { \"early_stopping_round\" } might not be used.\n\n\n\n This could be a false alarm, with some parameters getting used by language bindings but\n\n then being mistakenly passed down to XGBoost core, or some parameter actually being used\n\n but getting flagged wrongly here. Please open an issue if you find any such cases.\n\n\n\n\n\n[13:18:47] WARNING: C:/Users/Administrator/workspace/xgboost-win64_release_1.5.0/src/learner.cc:576: \n\nParameters: { \"early_stopping_round\" } might not be used.\n\n\n\n This could be a false alarm, with some parameters getting used by language bindings but\n\n then being mistakenly passed down to XGBoost core, or some parameter actually being used\n\n but getting flagged wrongly here. Please open an issue if you find any such cases.\n\n\n\n\n\n| \u001b[0m 22 \u001b[0m | \u001b[0m-3.814 \u001b[0m | \u001b[0m 0.8208 \u001b[0m | \u001b[0m 0.04505 \u001b[0m | \u001b[0m 7.272 \u001b[0m | \u001b[0m 25.61 \u001b[0m | \u001b[0m 4.631 \u001b[0m | \u001b[0m 1.287 \u001b[0m | \u001b[0m 1.0 \u001b[0m |\n\n{'objective': 'reg:squarederror', 'early_stopping_round': 50, 'eval_metric': 'rmse', 'colsample_bytree': 0.7415820237405875, 'subsample': 0.9875025924634507, 'min_child_weight': 27, 'max_depth': 9, 'eta': 0.02, 'reg_alpha': 4.575465304651051, 'reg_lambda': 3.3858651807679534}\n\n[13:21:07] WARNING: C:/Users/Administrator/workspace/xgboost-win64_release_1.5.0/src/learner.cc:576: \n\nParameters: { \"early_stopping_round\" } might not be used.\n\n\n\n This could be a false alarm, with some parameters getting used by language bindings but\n\n then being mistakenly passed down to XGBoost core, or some parameter actually being used\n\n but getting flagged wrongly here. Please open an issue if you find any such cases.\n\n\n\n\n\n[13:21:08] WARNING: C:/Users/Administrator/workspace/xgboost-win64_release_1.5.0/src/learner.cc:576: \n\nParameters: { \"early_stopping_round\" } might not be used.\n\n\n\n This could be a false alarm, with some parameters getting used by language bindings but\n\n then being mistakenly passed down to XGBoost core, or some parameter actually being used\n\n but getting flagged wrongly here. Please open an issue if you find any such cases.\n\n\n\n\n\n| \u001b[0m 23 \u001b[0m | \u001b[0m-3.817 \u001b[0m | \u001b[0m 0.7416 \u001b[0m | \u001b[0m 0.02 \u001b[0m | \u001b[0m 9.347 \u001b[0m | \u001b[0m 27.15 \u001b[0m | \u001b[0m 4.575 \u001b[0m | \u001b[0m 3.386 \u001b[0m | \u001b[0m 0.9875 \u001b[0m |\n\n{'objective': 'reg:squarederror', 'early_stopping_round': 50, 'eval_metric': 'rmse', 'colsample_bytree': 0.5, 'subsample': 0.733927508291536, 'min_child_weight': 27, 'max_depth': 11, 'eta': 0.02, 'reg_alpha': 3.32497669686325, 'reg_lambda': 4.9571016974842514}\n\n[13:25:00] WARNING: C:/Users/Administrator/workspace/xgboost-win64_release_1.5.0/src/learner.cc:576: \n\nParameters: { \"early_stopping_round\" } might not be used.\n\n\n\n This could be a false alarm, with some parameters getting used by language bindings but\n\n then being mistakenly passed down to XGBoost core, or some parameter actually being used\n\n but getting flagged wrongly here. Please open an issue if you find any such cases.\n\n\n\n\n\n[13:25:01] WARNING: C:/Users/Administrator/workspace/xgboost-win64_release_1.5.0/src/learner.cc:576: \n\nParameters: { \"early_stopping_round\" } might not be used.\n\n\n\n This could be a false alarm, with some parameters getting used by language bindings but\n\n then being mistakenly passed down to XGBoost core, or some parameter actually being used\n\n but getting flagged wrongly here. Please open an issue if you find any such cases.\n\n\n\n\n\n| \u001b[95m 24 \u001b[0m | \u001b[95m-3.803 \u001b[0m | \u001b[95m 0.5 \u001b[0m | \u001b[95m 0.02 \u001b[0m | \u001b[95m 11.71 \u001b[0m | \u001b[95m 27.6 \u001b[0m | \u001b[95m 3.325 \u001b[0m | \u001b[95m 4.957 \u001b[0m | \u001b[95m 0.7339 \u001b[0m |\n\n{'objective': 'reg:squarederror', 'early_stopping_round': 50, 'eval_metric': 'rmse', 'colsample_bytree': 0.5, 'subsample': 0.5, 'min_child_weight': 26, 'max_depth': 12, 'eta': 0.02, 'reg_alpha': 3.081921668614422, 'reg_lambda': 3.0163834184284712}\n\n[13:27:55] WARNING: C:/Users/Administrator/workspace/xgboost-win64_release_1.5.0/src/learner.cc:576: \n\nParameters: { \"early_stopping_round\" } might not be used.\n\n\n\n This could be a false alarm, with some parameters getting used by language bindings but\n\n then being mistakenly passed down to XGBoost core, or some parameter actually being used\n\n but getting flagged wrongly here. Please open an issue if you find any such cases.\n\n\n\n\n\n[13:27:56] WARNING: C:/Users/Administrator/workspace/xgboost-win64_release_1.5.0/src/learner.cc:576: \n\nParameters: { \"early_stopping_round\" } might not be used.\n\n\n\n This could be a false alarm, with some parameters getting used by language bindings but\n\n then being mistakenly passed down to XGBoost core, or some parameter actually being used\n\n but getting flagged wrongly here. Please open an issue if you find any such cases.\n\n\n\n\n\n| \u001b[0m 25 \u001b[0m | \u001b[0m-3.813 \u001b[0m | \u001b[0m 0.5 \u001b[0m | \u001b[0m 0.02 \u001b[0m | \u001b[0m 12.0 \u001b[0m | \u001b[0m 26.01 \u001b[0m | \u001b[0m 3.082 \u001b[0m | \u001b[0m 3.016 \u001b[0m | \u001b[0m 0.5 \u001b[0m |\n\n{'objective': 'reg:squarederror', 'early_stopping_round': 50, 'eval_metric': 'rmse', 'colsample_bytree': 1.0, 'subsample': 1.0, 'min_child_weight': 25, 'max_depth': 12, 'eta': 0.2, 'reg_alpha': 5.0, 'reg_lambda': 5.0}\n\n[13:30:32] WARNING: C:/Users/Administrator/workspace/xgboost-win64_release_1.5.0/src/learner.cc:576: \n\nParameters: { \"early_stopping_round\" } might not be used.\n\n\n\n This could be a false alarm, with some parameters getting used by language bindings but\n\n then being mistakenly passed down to XGBoost core, or some parameter actually being used\n\n but getting flagged wrongly here. Please open an issue if you find any such cases.\n\n\n\n\n\n[13:30:33] WARNING: C:/Users/Administrator/workspace/xgboost-win64_release_1.5.0/src/learner.cc:576: \n\nParameters: { \"early_stopping_round\" } might not be used.\n\n\n\n This could be a false alarm, with some parameters getting used by language bindings but\n\n then being mistakenly passed down to XGBoost core, or some parameter actually being used\n\n but getting flagged wrongly here. Please open an issue if you find any such cases.\n\n\n\n\n\n| \u001b[0m 26 \u001b[0m | \u001b[0m-3.864 \u001b[0m | \u001b[0m 1.0 \u001b[0m | \u001b[0m 0.2 \u001b[0m | \u001b[0m 12.0 \u001b[0m | \u001b[0m 25.92 \u001b[0m | \u001b[0m 5.0 \u001b[0m | \u001b[0m 5.0 \u001b[0m | \u001b[0m 1.0 \u001b[0m |\n\n=============================================================================================================\n\n-3.8032684999999997 {'colsample_bytree': 0.5, 'eta': 0.02, 'max_depth': 11.708480184158471, 'min_child_weight': 27.598200792362622, 'reg_alpha': 3.32497669686325, 'reg_lambda': 4.9571016974842514, 'subsample': 0.733927508291536}\n"}]},{"cell_type":"markdown","source":"接下来输出模型预测结果：","metadata":{}},{"cell_type":"code","source":"train_predict(train_x, test_x, best_clf)","metadata":{"tags":[],"collapsed":true,"jupyter":{"outputs_hidden":true}},"execution_count":17,"outputs":[{"name":"stderr","output_type":"stream","text":":87: DeprecationWarning: The default dtype for empty Series will be 'object' instead of 'float64' in a future version. Specify a dtype explicitly to silence this warning.\n\n prediction_train = pd.Series()\n"},{"name":"stdout","output_type":"stream","text":"[0]\ttrain-rmse:3.93743\teval-rmse:3.94887\n\n[50]\ttrain-rmse:3.49914\teval-rmse:3.73093\n\n[100]\ttrain-rmse:3.32144\teval-rmse:3.68839\n\n[150]\ttrain-rmse:3.23772\teval-rmse:3.67804\n\n[200]\ttrain-rmse:3.20197\teval-rmse:3.67385\n\n[250]\ttrain-rmse:3.17762\teval-rmse:3.67253\n\n[300]\ttrain-rmse:3.15443\teval-rmse:3.67234\n\n[302]\ttrain-rmse:3.15403\teval-rmse:3.67244\n\n[0]\ttrain-rmse:3.94448\teval-rmse:3.92137\n\n[50]\ttrain-rmse:3.50756\teval-rmse:3.69474\n\n[100]\ttrain-rmse:3.32726\teval-rmse:3.64895\n\n[150]\ttrain-rmse:3.24642\teval-rmse:3.63666\n\n[200]\ttrain-rmse:3.20761\teval-rmse:3.63409\n\n[250]\ttrain-rmse:3.18269\teval-rmse:3.63341\n\n[300]\ttrain-rmse:3.15935\teval-rmse:3.63319\n\n[350]\ttrain-rmse:3.13936\teval-rmse:3.63291\n\n[354]\ttrain-rmse:3.13787\teval-rmse:3.63293\n\n[0]\ttrain-rmse:3.93301\teval-rmse:3.96766\n\n[50]\ttrain-rmse:3.48744\teval-rmse:3.75187\n\n[100]\ttrain-rmse:3.30316\teval-rmse:3.71049\n\n[150]\ttrain-rmse:3.21876\teval-rmse:3.70025\n\n[200]\ttrain-rmse:3.18515\teval-rmse:3.69695\n\n[250]\ttrain-rmse:3.16190\teval-rmse:3.69627\n\n[300]\ttrain-rmse:3.14040\teval-rmse:3.69544\n\n[321]\ttrain-rmse:3.13221\teval-rmse:3.69595\n\n[0]\ttrain-rmse:3.91287\teval-rmse:4.04674\n\n[50]\ttrain-rmse:3.47627\teval-rmse:3.82653\n\n[100]\ttrain-rmse:3.28783\teval-rmse:3.78026\n\n[150]\ttrain-rmse:3.20957\teval-rmse:3.76870\n\n[200]\ttrain-rmse:3.17786\teval-rmse:3.76575\n\n[250]\ttrain-rmse:3.15028\teval-rmse:3.76448\n\n[300]\ttrain-rmse:3.12888\teval-rmse:3.76443\n\n[314]\ttrain-rmse:3.12480\teval-rmse:3.76418\n\n[0]\ttrain-rmse:3.96661\teval-rmse:3.83160\n\n[50]\ttrain-rmse:3.52079\teval-rmse:3.61955\n\n[100]\ttrain-rmse:3.33833\teval-rmse:3.57930\n\n[150]\ttrain-rmse:3.26075\teval-rmse:3.56961\n\n[200]\ttrain-rmse:3.22778\teval-rmse:3.56733\n\n[250]\ttrain-rmse:3.19642\teval-rmse:3.56590\n\n[289]\ttrain-rmse:3.17949\teval-rmse:3.56588\n\n[3.6724352915788456, 3.632935489439316, 3.6959392218600025, 3.7641771493981997, 3.5658763997083103] 3.6662727103969344\n"}]},{"cell_type":"markdown","source":"并将结果提交至竞赛主页","metadata":{}},{"cell_type":"markdown","source":"
\"image-20211209153243352\"","metadata":{}},{"cell_type":"markdown","source":"对比此前模型结果，能够发现目前XBG的模型结果较好，这其中差异极有可能是新增的NLP特征所导致的。","metadata":{}},{"cell_type":"markdown","source":"| 模型 | Private Score | Public Score |\n| ------ | ------ | ------ |\n| randomforest | 3.65455 | 3.74969 |\n| randomforest+validation | 3.65173 | 3.74954 |\n| LightGBM | 3.69723 | 3.80436 |\n| LightGBM+validation | 3.64403 | 3.73875 |\n| XGBoost | 3.62832 | 3.72358 |","metadata":{}},{"cell_type":"markdown","source":"  此外，课后同学们也可以进一步尝试在LightGBM和随机森林中带入NLP特征来进行计算，尝试是否能够进一步提分。当然，在各项单模型预测结束后，我们即可考虑进行模型融合了。","metadata":{}},{"cell_type":"markdown","source":"---","metadata":{}},{"cell_type":"markdown","source":"##
三、模型融合策略(一)\n##
**Voting融合**","metadata":{}},{"cell_type":"markdown","source":"  整体来看，常用模型融合的策略有两种，分别是Voting融合与Stacking融合，模型融合的目的和集成模型中的集成过程类似，都是希望能够尽可能借助不同模型的优势，最终输出一个更加可靠的结果。在Voting过程中，我们只需要对不同模型对测试集的预测结果进行加权汇总即可，而Stacking则相对复杂，需要借助此前不同模型的验证集预测结果和测试集预测结果再次进行模型训练，以验证集预测结果为训练集、训练集标签为标签构建新的训练集，在此模型上进行训练，然后以测试集预测结果作为新的预测集，并在新预测集上进行预测。","metadata":{}},{"cell_type":"markdown","source":"### 1.均值融合","metadata":{}},{"cell_type":"markdown","source":"  首先我们来看Voting融合过程。一般来说Voting融合也可以分为均值融合（多组预测结果求均值）、加权融合（根据某种方式赋予不同预测结果不同权重而后进行求和）以及Trick融合（根据某种特殊的规则赋予权重而后进行求和）三种，此处先介绍均值融合与加权融合的基本过程。","metadata":{}},{"cell_type":"markdown","source":"- 拼接不同模型预测结果","metadata":{}},{"cell_type":"markdown","source":"  首先，对随机森林、LightGBM和XGBoost模型预测结果进行读取，并简单查看三者相关系数：","metadata":{}},{"cell_type":"code","source":"data = pd.read_csv(\"result/submission_randomforest.csv\")\ndata['randomforest'] = data['target'].values\n\ntemp = pd.read_csv(\"result/submission_lightgbm.csv\")\ndata['lightgbm'] = temp['target'].values\n\n\ntemp = pd.read_csv(\"result/submission_xgboost.csv\")\ndata['xgboost'] = temp['target'].values\n\nprint(data.corr())","metadata":{},"execution_count":90,"outputs":[{"name":"stdout","output_type":"stream","text":" target randomforest lightgbm xgboost\n\ntarget 1.000000 1.000000 0.956251 0.947529\n\nrandomforest 1.000000 1.000000 0.956251 0.947529\n\nlightgbm 0.956251 0.956251 1.000000 0.951461\n\nxgboost 0.947529 0.947529 0.951461 1.000000\n"}]},{"cell_type":"code","source":"data.head(5)","metadata":{},"execution_count":91,"outputs":[{"execution_count":91,"output_type":"execute_result","data":{"text/html":["
\n","\n","\n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n","
card_idtargetrandomforestlightgbmxgboost
0C_ID_0ab67a22ab-3.528347-3.528347-3.576328-3.737840
1C_ID_130fd0cbdd-0.789489-0.789489-0.866846-0.540275
2C_ID_b709037bc5-0.380266-0.380266-0.370968-0.433453
3C_ID_d27d835a9f-0.269844-0.269844-0.121097-0.176081
4C_ID_2b5e3df5c2-1.038557-1.038557-1.047017-1.027519
\n","
"],"text/plain":[" card_id target randomforest lightgbm xgboost\n","0 C_ID_0ab67a22ab -3.528347 -3.528347 -3.576328 -3.737840\n","1 C_ID_130fd0cbdd -0.789489 -0.789489 -0.866846 -0.540275\n","2 C_ID_b709037bc5 -0.380266 -0.380266 -0.370968 -0.433453\n","3 C_ID_d27d835a9f -0.269844 -0.269844 -0.121097 -0.176081\n","4 C_ID_2b5e3df5c2 -1.038557 -1.038557 -1.047017 -1.027519"]},"metadata":{}}]},{"cell_type":"markdown","source":"然后计算不同模型预测结果的均值：","metadata":{}},{"cell_type":"code","source":"data['target'] = (data['randomforest'] + data['lightgbm'] + data['xgboost']) / 3","metadata":{},"execution_count":92,"outputs":[]},{"cell_type":"markdown","source":"写入结果文件并提交：","metadata":{}},{"cell_type":"code","source":"data[['card_id','target']].to_csv(\"result/voting_avr.csv\", index=False)","metadata":{},"execution_count":93,"outputs":[]},{"cell_type":"markdown","source":"
\"image-20211209184223461\"","metadata":{}},{"cell_type":"markdown","source":"| 模型 | Private Score | Public Score |\n| ------ | ------ | ------ |\n| randomforest | 3.65455 | 3.74969 |\n| randomforest+validation | 3.65173 | 3.74954 |\n| LightGBM | 3.69723 | 3.80436 |\n| LightGBM+validation | 3.64403 | 3.73875 |\n| XGBoost | 3.62832 | 3.72358 |\n| Voting_avr | 3.63650 | 3.73251 |","metadata":{}},{"cell_type":"markdown","source":"能够发现，简单的均值融合并不能有效果上的提升，接下来我们尝试加权融合。","metadata":{}},{"cell_type":"markdown","source":"### 2.加权融合","metadata":{}},{"cell_type":"markdown","source":"  加权融合的思路并不复杂，从客观计算流程上来看我们将赋予不同模型训练结果以不同权重，而具体权重的分配，我们可以根据三组模型在公榜上的评分决定，即假设模型A和B分别是2分和3分（分数越低越好的情况下），则在实际加权过程中，我们将赋予A模型结果3/5权重，B模型2/5权重，因此，加权融合过程如下：","metadata":{}},{"cell_type":"code","source":"data['target'] = data['randomforest']*0.2+data['lightgbm']*0.3 + data['xgboost']*0.5\ndata[['card_id','target']].to_csv(\"result/voting_wei1.csv\", index=False)","metadata":{},"execution_count":100,"outputs":[]},{"cell_type":"markdown","source":"输出结果如下：","metadata":{}},{"cell_type":"markdown","source":"
\"image-20211209185438900\"","metadata":{}},{"cell_type":"markdown","source":"| 模型 | Private Score | Public Score |\n| ------ | ------ | ------ |\n| randomforest | 3.65455 | 3.74969 |\n| randomforest+validation | 3.65173 | 3.74954 |\n| LightGBM | 3.69723 | 3.80436 |\n| LightGBM+validation | 3.64403 | 3.73875 |\n| XGBoost | 3.62832 | 3.72358 |\n| Voting_avr | 3.63650 | 3.73251 |\n| Voting_wei | 3.633307 | 3.72877 |","metadata":{}},{"cell_type":"markdown","source":"能够发现结果略微有所改善，但实际结果仍然不如单模型结果，此处预测极有可能是因额外i标签中存在异常值导致。接下来继续尝试Stacking融合。","metadata":{}},{"cell_type":"markdown","source":"---","metadata":{}},{"cell_type":"markdown","source":"##
四、模型融合策略（二）\n##
**Stacking融合**","metadata":{}},{"cell_type":"markdown","source":"  此处我们考虑手动进行Stacking融合，在此前的模型训练中，我们已经创建了predication_train和predication_test数据集，这两个数据集将作为训练集、测试集带入到下一轮的建模中，而本轮建模也被称为Stacking融合。","metadata":{}},{"cell_type":"markdown","source":"
\"image-20211208192640281\"","metadata":{}},{"cell_type":"markdown","source":"- 数据集校验","metadata":{}},{"cell_type":"markdown","source":"  首先快速查看此前创建的相关数据集，其中oof就是训练数据集的预测结果（多轮验证集预测结果拼接而来），而predictions则是此前单模型的预测结果：","metadata":{}},{"cell_type":"code","source":"oof_rf = pd.read_csv('./preprocess/train_randomforest.csv')\npredictions_rf = pd.read_csv('./preprocess/test_randomforest.csv')\n\noof_lgb = pd.read_csv('./preprocess/train_lightgbm.csv')\npredictions_lgb = pd.read_csv('./preprocess/test_lightgbm.csv')\n\noof_xgb = pd.read_csv('./preprocess/train_xgboost.csv')\npredictions_xgb = pd.read_csv('./preprocess/test_xgboost.csv')","metadata":{},"execution_count":81,"outputs":[]},{"cell_type":"code","source":"oof_rf.head(5)","metadata":{},"execution_count":101,"outputs":[{"execution_count":101,"output_type":"execute_result","data":{"text/html":["
\n","\n","\n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n","
0
0-0.426984
1-1.857115
20.402431
3-0.063488
4-0.278658
\n","
"],"text/plain":[" 0\n","0 -0.426984\n","1 -1.857115\n","2 0.402431\n","3 -0.063488\n","4 -0.278658"]},"metadata":{}}]},{"cell_type":"code","source":"predictions_lgb.head(5)","metadata":{},"execution_count":103,"outputs":[{"execution_count":103,"output_type":"execute_result","data":{"text/html":["
\n","\n","\n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n","
0
0-3.576328
1-0.866846
2-0.370968
3-0.121097
4-1.047017
\n","
"],"text/plain":[" 0\n","0 -3.576328\n","1 -0.866846\n","2 -0.370968\n","3 -0.121097\n","4 -1.047017"]},"metadata":{}}]},{"cell_type":"code","source":"oof_rf.shape, oof_lgb.shape","metadata":{},"execution_count":104,"outputs":[{"execution_count":104,"output_type":"execute_result","data":{"text/plain":["((201917, 1), (201917, 1))"]},"metadata":{}}]},{"cell_type":"code","source":"predictions_rf.shape, predictions_lgb.shape","metadata":{},"execution_count":105,"outputs":[{"execution_count":105,"output_type":"execute_result","data":{"text/plain":["((123623, 1), (123623, 1))"]},"metadata":{}}]},{"cell_type":"markdown","source":"  接下来进行Stacking模型融合，该过程本身并不复杂，重点需要清洗不同数据集的创建过程与调用过程，同时为了保证模型泛化能力，我们需要对其进行交叉验证：","metadata":{}},{"cell_type":"code","source":"def stack_model(oof_1, oof_2, oof_3, predictions_1, predictions_2, predictions_3, y):\n \n # Part 1.数据准备\n # 按行拼接列，拼接验证集所有预测结果\n # train_stack就是final model的训练数据\n train_stack = np.hstack([oof_1, oof_2, oof_3])\n # 按行拼接列，拼接测试集上所有预测结果\n # test_stack就是final model的测试数据\n test_stack = np.hstack([predictions_1, predictions_2, predictions_3])\n # 创建一个和验证集行数相同的全零数组\n # oof = np.zeros(train_stack.shape[0])\n # 创建一个和测试集行数相同的全零数组\n predictions = np.zeros(test_stack.shape[0])\n \n # Part 2.多轮交叉验证\n from sklearn.model_selection import RepeatedKFold\n folds = RepeatedKFold(n_splits=5, n_repeats=2, random_state=2020)\n \n # fold_为折数，trn_idx为每一折训练集index，val_idx为每一折验证集index\n for fold_, (trn_idx, val_idx) in enumerate(folds.split(train_stack, y)):\n # 打印折数信息\n print(\"fold n°{}\".format(fold_+1))\n # 训练集中划分为训练数据的特征和标签\n trn_data, trn_y = train_stack[trn_idx], y[trn_idx]\n # 训练集中划分为验证数据的特征和标签\n val_data, val_y = train_stack[val_idx], y[val_idx]\n # 开始训练时提示\n print(\"-\" * 10 + \"Stacking \" + str(fold_+1) + \"-\" * 10)\n # 采用贝叶斯回归作为结果融合的模型（final model）\n clf = BayesianRidge()\n # 在训练数据上进行训练\n clf.fit(trn_data, trn_y)\n # 在验证数据上进行预测，并将结果记录在oof对应位置\n # oof[val_idx] = clf.predict(val_data)\n # 对测试集数据进行预测，每一轮预测结果占比额外的1/10\n predictions += clf.predict(test_stack) / (5 * 2)\n \n # 返回测试集的预测结果\n return predictions","metadata":{},"execution_count":80,"outputs":[]},{"cell_type":"markdown","source":"接下来执行模型融合过程：","metadata":{}},{"cell_type":"code","source":"target = train['target'].values","metadata":{},"execution_count":82,"outputs":[]},{"cell_type":"code","source":"predictions_stack = stack_model(oof_rf, oof_lgb, oof_xgb, \n predictions_rf, predictions_lgb, predictions_xgb, target)","metadata":{},"execution_count":83,"outputs":[{"name":"stdout","output_type":"stream","text":"fold n°1\n\n----------Stacking 1----------\n\nfold n°2\n\n----------Stacking 2----------\n\nfold n°3\n\n----------Stacking 3----------\n\nfold n°4\n\n----------Stacking 4----------\n\nfold n°5\n\n----------Stacking 5----------\n\nfold n°6\n\n----------Stacking 6----------\n\nfold n°7\n\n----------Stacking 7----------\n\nfold n°8\n\n----------Stacking 8----------\n\nfold n°9\n\n----------Stacking 9----------\n\nfold n°10\n\n----------Stacking 10----------\n"}]},{"cell_type":"markdown","source":"查看输出结果，并将结果进行提交：","metadata":{}},{"cell_type":"code","source":"predictions_stack","metadata":{},"execution_count":78,"outputs":[{"execution_count":78,"output_type":"execute_result","data":{"text/plain":["array([-4.05848444, -0.68964305, -0.41582154, ..., 0.65614588,\n"," -2.35990886, 0.39282516])"]},"metadata":{}}]},{"cell_type":"code","source":"sub_df = pd.read_csv('data/sample_submission.csv')\nsub_df[\"target\"] = predictions_stack\nsub_df.to_csv('predictions_stack1.csv', index=False)","metadata":{},"execution_count":79,"outputs":[]},{"cell_type":"markdown","source":"能够看到最终结果如下：","metadata":{}},{"cell_type":"markdown","source":"
\"image-20211209190553412\"","metadata":{}},{"cell_type":"markdown","source":"| 模型 | Private Score | Public Score |\n| ------ | ------ | ------ |\n| randomforest | 3.65455 | 3.74969 |\n| randomforest+validation | 3.65173 | 3.74954 |\n| LightGBM | 3.69723 | 3.80436 |\n| LightGBM+validation | 3.64403 | 3.73875 |\n| XGBoost | 3.62832 | 3.72358 |\n| Voting_avr | 3.63650 | 3.73251 |\n| Voting_wei | 3.633307 | 3.72877 |\n| Stacking | 3.62798 | 3.72055 |","metadata":{}},{"cell_type":"markdown","source":"  截至目前，我们已经尝试了在当前数据情况下的所有有效方法，而通过这一些列尝试，我们也得到了目前最好结果，而整体的排名也从Baseline的前40%提高到20%左右。当然，当这些通用方法都进行了尝试之后，接下来我们需要进一步从更加具体的角度来看当前数据集的数据情况，并据此提出一些跟进一步的优化方法，并更进一步的提高最终预测结果。","metadata":{}}]}