{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# Elo world \n\nIn this kernel, I build a LGBM model that aggregates the `new_merchant_transactions.csv` and `historical_transactions.csv` tables to the main train table. New features are built by successive grouping on`card_id` and `month_lag`, in order to recover some information from the time serie.\n\nDuring the competition, I took into account the enlightments provided by others kernels, and included a few features that appeared to be important. In particular, I closely looked at the following kernels (ordered by release time):\n1.  [You're Going to Want More Categories [LB 3.737] by Peter Hurford](https://www.kaggle.com/peterhurford/you-re-going-to-want-more-categories-lb-3-737)\n2. [EloDA with Feature Engineering and Stacking by Bojan Tunguz](https://www.kaggle.com/tunguz/eloda-with-feature-engineering-and-stacking)\n3. [A Closer Look at Date Variables by Robin Denz](https://www.kaggle.com/denzo123/a-closer-look-at-date-variables)\n4. [LGB + FE (LB 3.707) by Konrad Banachewicz](https://www.kaggle.com/konradb/lgb-fe-lb-3-707)\n5. [My first kernel (3.699) by Chau Ngoc Huynh](https://www.kaggle.com/chauhuynh/my-first-kernel-3-699/)\n\n## Notebook  Content\n1. [Loading the data](#1)\n1. [Feature engineering](#2)\n1. [Training the model](#3)\n1. [Feature importance](#4)\n1. [Submission](#5)","metadata":{"_uuid":"332fa118684da9a9278d2caf565620686fd73a68","trusted":true}},{"cell_type":"code","source":"import numpy as np \nimport pandas as pd \nimport matplotlib.pyplot as plt\nimport seaborn as sns\nimport lightgbm as lgb\nimport xgboost as xgb\nfrom xgboost import XGBRegressor\nfrom sklearn import tree\nfrom sklearn.svm import SVR\nfrom sklearn.neighbors import KNeighborsRegressor\nfrom sklearn.impute import SimpleImputer\nfrom sklearn.ensemble import RandomForestRegressor\nfrom sklearn.linear_model import LogisticRegression, LinearRegression\nfrom sklearn.model_selection import KFold\nfrom tqdm import tqdm\nimport warnings\nimport time\nimport sys\nimport datetime\nimport matplotlib.pyplot as plt\nimport seaborn as sns\nfrom sklearn.metrics import mean_squared_error\nimport torch\nimport torch.nn as nn\nfrom torch.utils.data import DataLoader, Dataset\nimport gc\nwarnings.simplefilter(action='ignore', category=FutureWarning)\npd.set_option('display.max_columns', 500)","metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-11-21T10:25:38.832388Z","iopub.execute_input":"2022-11-21T10:25:38.832924Z","iopub.status.idle":"2022-11-21T10:25:38.843910Z","shell.execute_reply.started":"2022-11-21T10:25:38.832885Z","shell.execute_reply":"2022-11-21T10:25:38.842375Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# 1. 超参数设定","metadata":{}},{"cell_type":"code","source":"class CFG:\n    debug = False\n    batch_size = 256\n    num_epochs = 5\n\ncfg = CFG()\nif cfg.debug:\n    train = train.iloc[:200,]\n    test = test.iloc[:20,]\n    target = target[:200]\n    \n","metadata":{"execution":{"iopub.status.busy":"2022-11-21T10:25:38.849772Z","iopub.execute_input":"2022-11-21T10:25:38.851106Z","iopub.status.idle":"2022-11-21T10:25:38.861789Z","shell.execute_reply.started":"2022-11-21T10:25:38.851062Z","shell.execute_reply":"2022-11-21T10:25:38.860723Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"<a id=\"3\"></a> <br>\n## 2. 数据集加载和预处理\nWe now train the model with the features we previously defined. A first step consists in merging all the dataframes:","metadata":{"_uuid":"dcf0403c10b8ee817257a51e5edf8f1f81fcd593"}},{"cell_type":"code","source":"train = pd.read_csv('../input/elo-processed-data/train.csv')\ntest = pd.read_csv('../input/elo-processed-data/test.csv')\ntarget = train['target']\ndel train['target']","metadata":{"_uuid":"d27264c6c7f0af6af7ba141177bfd38f7a68dec3","execution":{"iopub.status.busy":"2022-11-21T10:25:38.868321Z","iopub.execute_input":"2022-11-21T10:25:38.868748Z","iopub.status.idle":"2022-11-21T10:25:46.884316Z","shell.execute_reply.started":"2022-11-21T10:25:38.868693Z","shell.execute_reply":"2022-11-21T10:25:46.882848Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"features = [c for c in train.columns if c not in ['card_id', 'first_active_month', 'Unnamed: 0']]\n#features = [f for f in features if f not in unimportant_features]\ncategorical_feats = ['feature_2', 'feature_3']","metadata":{"execution":{"iopub.status.busy":"2022-11-21T10:25:46.887086Z","iopub.execute_input":"2022-11-21T10:25:46.887583Z","iopub.status.idle":"2022-11-21T10:25:46.894124Z","shell.execute_reply.started":"2022-11-21T10:25:46.887534Z","shell.execute_reply":"2022-11-21T10:25:46.892818Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"We then set the hyperparameters of the LGBM model, these parameters are obtained by an [bayesian optimization done in another kernel](https://www.kaggle.com/fabiendaniel/hyperparameter-tuning/edit):","metadata":{"_uuid":"e67399b25b3886e306b274069ab240af7d060397"}},{"cell_type":"markdown","source":"We now train the model. Here, we use a standard KFold split of the dataset in order to validate the results and to stop the training. Interstingly, during the writing of this kernel, the model was enriched adding new features, which improved the CV score. **The variations observed on the CV were found to be quite similar to the variations on the LB**: it seems that the current competition won't give us headaches to define the correct validation scheme:","metadata":{"_uuid":"b7a7377cd7401f2cbd13ea707fbc2a2bebe229a6"}},{"cell_type":"code","source":"train[categorical_feats] = train[categorical_feats].astype('category')\n\ndef one_hot_encoder(df, categorical_columns = categorical_feats, nan_as_category = True):\n    original_columns = df.columns.tolist()\n\n    #categorical_columns = list(filter(lambda c: c in ['object'], df.dtypes))\n    df = pd.get_dummies(df, columns=categorical_columns, dummy_na=nan_as_category)\n\n    new_columns = list(filter(lambda c: c not in original_columns, df.columns))\n    return df, new_columns\n\ntrain_onehot, train_cols = one_hot_encoder(train[features], categorical_columns = categorical_feats, nan_as_category=False)\ntest_onehot, test_cols = one_hot_encoder(test[features], categorical_columns = categorical_feats, nan_as_category=False)\n\n# 缺失值采用中位数填补\nimp = SimpleImputer(add_indicator=False, strategy='median')\ncolnames = train_onehot.columns\ntrain_onehot = pd.DataFrame(imp.fit_transform(train_onehot))\ntrain_onehot.columns = colnames\n\ncolnames = test_onehot.columns\ntest_onehot = pd.DataFrame(imp.fit_transform(test_onehot))\ntest_onehot.columns = colnames","metadata":{"execution":{"iopub.status.busy":"2022-11-21T10:25:46.895779Z","iopub.execute_input":"2022-11-21T10:25:46.896604Z","iopub.status.idle":"2022-11-21T10:25:57.228190Z","shell.execute_reply.started":"2022-11-21T10:25:46.896554Z","shell.execute_reply":"2022-11-21T10:25:57.226843Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# 将数据变为torch.tensor，特征维度164，每个数据大小(batch, 164)\n# DataSet和DataLoader\n\nclass TrainDataset(Dataset):\n    def __init__(self, df_train, target):\n        self.train = torch.as_tensor(df_train.values)\n        self.labels = torch.as_tensor(np.array(target))\n\n    def __len__(self):\n        return self.train.shape[0]\n\n    def __getitem__(self, item):\n        return self.train[item], self.labels[item]\n    \nclass TestDataset(Dataset):\n    def __init__(self, df_train):\n        self.test = torch.as_tensor(df_train.values)\n\n    def __len__(self):\n        return self.test.shape[0]\n\n    def __getitem__(self, item):\n        return self.test[item]\n    \ntrain_dataset = TrainDataset(train_onehot, target)\ntest_dataset = TestDataset(train_onehot)\n\ntrain_loader = DataLoader(train_dataset,\n                          batch_size=24,\n                          shuffle=True,\n                          num_workers=0, pin_memory=True, drop_last=True)\n# valid_loader = DataLoader(valid_dataset,\n#                           batch_size=1024,\n#                           shuffle=False,\n#                           num_workers=0, pin_memory=True, drop_last=False)","metadata":{"execution":{"iopub.status.busy":"2022-11-21T10:25:57.230995Z","iopub.execute_input":"2022-11-21T10:25:57.231385Z","iopub.status.idle":"2022-11-21T10:25:57.242428Z","shell.execute_reply.started":"2022-11-21T10:25:57.231351Z","shell.execute_reply":"2022-11-21T10:25:57.241458Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# 神经网络相关构建\n\nnet = nn.Sequential(nn.Flatten(),\n                    nn.Linear(164, 64),\n                    nn.ReLU(),\n                    nn.Linear(64, 1)).double()\n\nloss = nn.MSELoss()\n\ntrainer = torch.optim.SGD(net.parameters(), lr=0.1)","metadata":{"execution":{"iopub.status.busy":"2022-11-21T10:25:57.243715Z","iopub.execute_input":"2022-11-21T10:25:57.244489Z","iopub.status.idle":"2022-11-21T10:25:57.261916Z","shell.execute_reply.started":"2022-11-21T10:25:57.244449Z","shell.execute_reply":"2022-11-21T10:25:57.260476Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"class AverageMeter(object):\n    \"\"\"Computes and stores the average and current value\"\"\"\n    def __init__(self):\n        self.reset()\n\n    def reset(self):\n        self.val = 0\n        self.avg = 0\n        self.sum = 0\n        self.count = 0\n\n    def update(self, val, n=1):\n        self.val = val\n        self.sum += val * n\n        self.count += n\n        self.avg = self.sum / self.count","metadata":{"execution":{"iopub.status.busy":"2022-11-21T10:25:57.263191Z","iopub.execute_input":"2022-11-21T10:25:57.263597Z","iopub.status.idle":"2022-11-21T10:25:57.274687Z","shell.execute_reply.started":"2022-11-21T10:25:57.263565Z","shell.execute_reply":"2022-11-21T10:25:57.273631Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#5fold 交叉验证\nfolds = KFold(n_splits=5, shuffle=True, random_state=15)\noof = np.zeros(len(train))\npredictions = np.zeros(len(test))\nstart = time.time()\nfeature_importance_df = pd.DataFrame()\noof_preds = np.zeros(train_onehot.shape[0])\nclfs = list()\n\ntrain_loss = []\nvalid_losses = []\nfor fold_, (trn_idx, val_idx) in enumerate(folds.split(train_onehot.values, target.values)):\n    print(\"=======fold {}=======\".format(fold_))\n    X_train, y_train = train_onehot.iloc[trn_idx], target.iloc[trn_idx]\n    X_valid, y_valid = train_onehot.iloc[val_idx], target.iloc[val_idx]\n\n    train_dataset = TrainDataset(X_train, y_train)\n    valid_dataset = TrainDataset(X_valid, y_valid)\n    \n    train_loader = DataLoader(train_dataset,\n                          batch_size=cfg.batch_size,\n                          shuffle=True,\n                          num_workers=0, pin_memory=True, drop_last=True)\n\n    valid_loader = DataLoader(valid_dataset,\n                          batch_size=cfg.batch_size,\n                          shuffle=True,\n                          num_workers=0, pin_memory=True, drop_last=True)\n\n    for epoch in range(cfg.num_epochs):\n        print(f\"=====epoch {epoch} starting=====\")\n        losses = AverageMeter()\n        for step, (X, y) in tqdm(enumerate(train_loader)):\n            y_preds = net(X)\n            l = loss(y_preds, y)  # X和y的小批量损失\n            l = torch.sqrt(l) #RMSE\n            # 因为l形状是(batch_size,1)，而不是一个标量。l中的所有元素被加到一起，并以此计算的梯度\n            losses.update(l.item(), cfg.batch_size)\n            trainer.zero_grad()\n            l.sum().backward()\n            trainer.step()\n        train_loss.append(losses.avg)\n        print(f'epoch {epoch + 1}, loss {train_loss[-1]:f}')\n    \n    #validation\n    for step, (X, y) in tqdm(enumerate(valid_loader)):\n        valid_loss = AverageMeter()\n        with torch.no_grad():\n            y_preds = net(X)\n            l = loss(y_preds, y)\n            l = torch.sqrt(l)\n            valid_loss.update(l.item(), cfg.batch_size)\n    valid_losses.append(valid_loss.avg)\n    print(f'valid loss :{valid_loss.avg:f}')\n            \n\n\nprint(\"CV score: {:<8.5f}\".format(np.mean(valid_losses)))","metadata":{"execution":{"iopub.status.busy":"2022-11-21T10:28:32.979194Z","iopub.execute_input":"2022-11-21T10:28:32.979639Z","iopub.status.idle":"2022-11-21T10:30:40.807871Z","shell.execute_reply.started":"2022-11-21T10:28:32.979605Z","shell.execute_reply":"2022-11-21T10:30:40.806685Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"<a id=\"5\"></a> <br>\n## 5. 测试集结果--暂时没做\nNow, we just need to prepare the submission file:","metadata":{"_uuid":"003ae1b1bd522b1b0d992ff220ed98d2a6d7477a"}},{"cell_type":"code","source":"# sub_df = pd.DataFrame({\"card_id\":test[\"card_id\"].values})\n# sub_df[\"target\"] = predictions\n# sub_df.to_csv(\"submit.csv\", index=False)","metadata":{"_uuid":"82d5ac08a13603b2a66c59d98584c4b709daee2d","execution":{"iopub.status.busy":"2022-11-21T10:26:32.207596Z","iopub.status.idle":"2022-11-21T10:26:32.209010Z","shell.execute_reply.started":"2022-11-21T10:26:32.208589Z","shell.execute_reply":"2022-11-21T10:26:32.208630Z"},"trusted":true},"execution_count":null,"outputs":[]}]}