{"cells":[{"metadata":{},"cell_type":"markdown","source":"## Introduction\nIt's just some minor changes to the great kernel [libffm-model](https://www.kaggle.com/ogrellier/libffm-model) with predicting test data in each fold. Please Upvote the original kernel. Last part is a blend of some of best scoring public kernels with libffm predictions.\n\n[Field-Aware Factorization](https://www.csie.ntu.edu.tw/~cjlin/libffm) is a powerful representation learning.\n\n[Github here.](https://github.com/ycjuan/libffm)\n "},{"metadata":{"_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","trusted":true},"cell_type":"code","source":"import numpy as np\nimport pandas as pd \nimport os\nimport seaborn as sns\nimport matplotlib.pyplot as plt\nfrom scipy.stats import rankdata\n\nfor dirname, _, filenames in os.walk('/kaggle/input'):\n    for filename in filenames:\n        print(os.path.join(dirname, filename))\n\nN_Splits = 25\nSEED = 2020","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Read the data"},{"metadata":{"trusted":false},"cell_type":"code","source":"train = pd.read_csv('/kaggle/input/cat-in-the-dat-ii/train.csv')\ntest = pd.read_csv('/kaggle/input/cat-in-the-dat-ii/test.csv')\ntest.insert(1, 'target', 0)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Label Encode to ease creation of libffm format"},{"metadata":{"trusted":false},"cell_type":"code","source":"features = [_f for _f in train if _f not in ['id', 'target']]\n\ndef factor_encoding(train, test):\n    \n    assert sorted(train.columns) == sorted(test.columns)\n    \n    full = pd.concat([train, test], axis=0, sort=False)\n    # Factorize everything\n    for f in full:\n        full[f], _ = pd.factorize(full[f])\n        full[f] += 1  # make sure no negative\n        \n    return full.iloc[:train.shape[0]], full.iloc[train.shape[0]:]\n\ntrain_f, test_f = factor_encoding(train[features], test[features])","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Create LibFFM files\n\n\nThe data format of LIBFFM has a very special format (taken from [libffm page](https://github.com/ycjuan/libffm)):\n```\n<label> <field1>:<feature1>:<value1> <field2>:<feature2>:<value2> ...\n.\n.\n.\n```\n\n`field` and `feature` should be non-negative integers.\n\nIt is important to understand the difference between `field` and `feature`. For example, if we have a raw data like this:\n\n| Click | Advertiser | Publisher |\n|:-----:|:----------:|:---------:|\n|    0 |       Nike |       CNN |\n|    1 |       ESPN |       BBC |\n\nHere, we have \n \n - 2 fields: Advertiser and Publisher\n - 4 features: Advertiser-Nike, Advertiser-ESPN, Publisher-CNN, Publisher-BBC\n\nUsually you will need to build two dictionares, one for field and one for features, like this:\n    \n    DictField[Advertiser] -> 0\n    DictField[Publisher]  -> 1\n    \n    DictFeature[Advertiser-Nike] -> 0\n    DictFeature[Publisher-CNN]   -> 1\n    DictFeature[Advertiser-ESPN] -> 2\n    DictFeature[Publisher-BBC]   -> 3\n\nThen, you can generate FFM format data:\n\n    0 0:0:1 1:1:1\n    1 0:2:1 1:3:1\n\nNote that because these features are categorical, the values here are all ones.\n\nThe class defined below go through all features and rows and update a python dicts as new values are encountered."},{"metadata":{"trusted":false},"cell_type":"code","source":"class LibFFMEncoder(object):\n    def __init__(self):\n        self.encoder = 1\n        self.encoding = {}\n\n    def encode_for_libffm(self, row):\n        txt = f\"{row[0]}\"\n        for i, r in enumerate(row[1:]):\n            try:\n                txt += f' {i+1}:{self.encoding[(i, r)]}:1'\n            except KeyError:\n                self.encoding[(i, r)] = self.encoder\n                self.encoder += 1\n                txt += f' {i+1}:{self.encoding[(i, r)]}:1'\n\n        return txt\n\n# Create files for testing and OOF\nfrom sklearn.model_selection import KFold, StratifiedKFold\nfold_ids = [\n    [trn_, val_] for (trn_, val_) in StratifiedKFold(N_Splits,True,SEED).split(train, train['target'])\n]\nfor fold_, (trn_, val_) in enumerate(fold_ids):\n\n    # Fit the encoder\n    encoder = LibFFMEncoder()\n    libffm_format_trn = pd.concat([train['target'].iloc[trn_], train_f.iloc[trn_]], axis=1).apply(\n        lambda row: encoder.encode_for_libffm(row), raw=True, axis=1\n    )\n    # Encode validation set\n    libffm_format_val = pd.concat([train['target'].iloc[val_], train_f.iloc[val_]], axis=1).apply(\n        lambda row: encoder.encode_for_libffm(row), raw=True, axis=1\n    )\n    libffm_format_tst = pd.concat([test['target'], test_f], axis=1).apply(\n        lambda row: encoder.encode_for_libffm(row), raw=True, axis=1\n    )\n    print(train['target'].iloc[trn_].shape, train['target'].iloc[val_].shape, libffm_format_tst.shape)\n    \n    libffm_format_trn.to_csv(f'libffm_trn_fold_{fold_+1}.txt', index=False, header=False)\n    libffm_format_val.to_csv(f'libffm_val_fold_{fold_+1}.txt', index=False, header=False)\n    libffm_format_tst.to_csv(f'libffm_tst_fold_{fold_+1}.txt', index=False, header=False)\n\n    \n\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Make ffm-train and ffm-predict excutable"},{"metadata":{"trusted":false},"cell_type":"code","source":"!cp /kaggle/input/libffm-binaries/ffm-train .\n!cp /kaggle/input/libffm-binaries/ffm-predict .\n!chmod u+x ffm-train\n!chmod u+x ffm-predict","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Run OOF"},{"metadata":{"trusted":false},"cell_type":"code","source":"from sklearn.metrics import log_loss, roc_auc_score\n\n!./ffm-train -p libffm_val_fold_1.txt -r 0.05 -l 0.0002 -k 50 --auto-stop libffm_trn_fold_1.txt libffm_fold_1_model\n!./ffm-predict libffm_val_fold_1.txt libffm_fold_1_model val_preds_fold_1.txt\n!./ffm-predict libffm_tst_fold_1.txt libffm_fold_1_model tst_preds_fold_1.txt\nos.remove('libffm_val_fold_1.txt')\nos.remove('libffm_trn_fold_1.txt')\nos.remove('libffm_fold_1_model')\nos.remove('libffm_tst_fold_1.txt')\n\n(\n    log_loss(train['target'].iloc[fold_ids[0][1]], pd.read_csv('val_preds_fold_1.txt', header=None).values[:,0]),\n    roc_auc_score(train['target'].iloc[fold_ids[0][1]], pd.read_csv('val_preds_fold_1.txt', header=None).values[:,0])\n)\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":false},"cell_type":"code","source":"!./ffm-train -p libffm_val_fold_2.txt -r 0.05 -l 0.00002 -k 50  --auto-stop libffm_trn_fold_2.txt libffm_fold_2_model\n!./ffm-predict libffm_val_fold_2.txt libffm_fold_2_model val_preds_fold_2.txt\n!./ffm-predict libffm_tst_fold_2.txt libffm_fold_2_model tst_preds_fold_2.txt\nos.remove('libffm_val_fold_2.txt')\nos.remove('libffm_trn_fold_2.txt')\nos.remove('libffm_fold_2_model')\nos.remove('libffm_tst_fold_2.txt')\n(\n    log_loss(train['target'].iloc[fold_ids[1][1]], pd.read_csv('val_preds_fold_2.txt', header=None).values[:,0]),\n    roc_auc_score(train['target'].iloc[fold_ids[1][1]], pd.read_csv('val_preds_fold_2.txt', header=None).values[:,0])\n)\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"0.7839462564075135"},{"metadata":{"trusted":false},"cell_type":"code","source":"!./ffm-train -p libffm_val_fold_3.txt -r 0.05 -l 0.00002 -k 50 --auto-stop libffm_trn_fold_3.txt libffm_fold_3_model\n!./ffm-predict libffm_val_fold_3.txt libffm_fold_3_model val_preds_fold_3.txt\n!./ffm-predict libffm_tst_fold_3.txt libffm_fold_3_model tst_preds_fold_3.txt\nos.remove('libffm_val_fold_3.txt')\nos.remove('libffm_trn_fold_3.txt')\nos.remove('libffm_fold_3_model')\nos.remove('libffm_tst_fold_3.txt')\n(\n    log_loss(train['target'].iloc[fold_ids[2][1]], pd.read_csv('val_preds_fold_3.txt', header=None).values[:,0]),\n    roc_auc_score(train['target'].iloc[fold_ids[2][1]], pd.read_csv('val_preds_fold_3.txt', header=None).values[:,0])\n)\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":false},"cell_type":"code","source":"!./ffm-train -p libffm_val_fold_4.txt -r 0.05 -l 0.00002 -k 50 --auto-stop libffm_trn_fold_4.txt libffm_fold_4_model\n!./ffm-predict libffm_val_fold_4.txt libffm_fold_4_model val_preds_fold_4.txt\n!./ffm-predict libffm_tst_fold_4.txt libffm_fold_4_model tst_preds_fold_4.txt\nos.remove('libffm_val_fold_4.txt')\nos.remove('libffm_trn_fold_4.txt')\nos.remove('libffm_fold_4_model')\nos.remove('libffm_tst_fold_4.txt')\n(\n    log_loss(train['target'].iloc[fold_ids[3][1]], pd.read_csv('val_preds_fold_4.txt', header=None).values[:,0]),\n    roc_auc_score(train['target'].iloc[fold_ids[3][1]], pd.read_csv('val_preds_fold_4.txt', header=None).values[:,0])\n)\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":false},"cell_type":"code","source":"!./ffm-train -p libffm_val_fold_5.txt -r 0.05 -l 0.00002 -k 50 --auto-stop libffm_trn_fold_5.txt libffm_fold_5_model\n!./ffm-predict libffm_val_fold_5.txt libffm_fold_5_model val_preds_fold_5.txt\n!./ffm-predict libffm_tst_fold_5.txt libffm_fold_5_model tst_preds_fold_5.txt\nos.remove('libffm_val_fold_5.txt')\nos.remove('libffm_trn_fold_5.txt')\nos.remove('libffm_fold_5_model')\nos.remove('libffm_tst_fold_5.txt')\n(\n    log_loss(train['target'].iloc[fold_ids[4][1]], pd.read_csv('val_preds_fold_5.txt', header=None).values[:,0]),\n    roc_auc_score(train['target'].iloc[fold_ids[4][1]], pd.read_csv('val_preds_fold_5.txt', header=None).values[:,0])\n)\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":false},"cell_type":"code","source":"!./ffm-train -p libffm_val_fold_6.txt -r 0.05 -l 0.00002 -k 50 --auto-stop libffm_trn_fold_6.txt libffm_fold_6_model\n!./ffm-predict libffm_val_fold_6.txt libffm_fold_6_model val_preds_fold_6.txt\n!./ffm-predict libffm_tst_fold_6.txt libffm_fold_6_model tst_preds_fold_6.txt\nos.remove('libffm_val_fold_6.txt')\nos.remove('libffm_trn_fold_6.txt')\nos.remove('libffm_fold_6_model')\nos.remove('libffm_tst_fold_6.txt')\n(\n    log_loss(train['target'].iloc[fold_ids[5][1]], pd.read_csv('val_preds_fold_6.txt', header=None).values[:,0]),\n    roc_auc_score(train['target'].iloc[fold_ids[5][1]], pd.read_csv('val_preds_fold_6.txt', header=None).values[:,0])\n)\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":false},"cell_type":"code","source":"!./ffm-train -p libffm_val_fold_7.txt -r 0.05 -l 0.00002 -k 50 --auto-stop libffm_trn_fold_7.txt libffm_fold_7_model\n!./ffm-predict libffm_val_fold_7.txt libffm_fold_7_model val_preds_fold_7.txt\n!./ffm-predict libffm_tst_fold_7.txt libffm_fold_7_model tst_preds_fold_7.txt\nos.remove('libffm_val_fold_7.txt')\nos.remove('libffm_trn_fold_7.txt')\nos.remove('libffm_fold_7_model')\nos.remove('libffm_tst_fold_7.txt')\n(\n    log_loss(train['target'].iloc[fold_ids[6][1]], pd.read_csv('val_preds_fold_7.txt', header=None).values[:,0]),\n    roc_auc_score(train['target'].iloc[fold_ids[6][1]], pd.read_csv('val_preds_fold_7.txt', header=None).values[:,0])\n)\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":false},"cell_type":"code","source":"!./ffm-train -p libffm_val_fold_8.txt -r 0.05 -l 0.00002 -k 50 --auto-stop libffm_trn_fold_8.txt libffm_fold_8_model\n!./ffm-predict libffm_val_fold_8.txt libffm_fold_8_model val_preds_fold_8.txt\n!./ffm-predict libffm_tst_fold_8.txt libffm_fold_8_model tst_preds_fold_8.txt\nos.remove('libffm_val_fold_8.txt')\nos.remove('libffm_trn_fold_8.txt')\nos.remove('libffm_fold_8_model')\nos.remove('libffm_tst_fold_8.txt')\n(\n    log_loss(train['target'].iloc[fold_ids[7][1]], pd.read_csv('val_preds_fold_8.txt', header=None).values[:,0]),\n    roc_auc_score(train['target'].iloc[fold_ids[7][1]], pd.read_csv('val_preds_fold_8.txt', header=None).values[:,0])\n)\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":false},"cell_type":"code","source":"!./ffm-train -p libffm_val_fold_9.txt -r 0.05 -l 0.00002 -k 50 --auto-stop libffm_trn_fold_9.txt libffm_fold_9_model\n!./ffm-predict libffm_val_fold_9.txt libffm_fold_9_model val_preds_fold_9.txt\n!./ffm-predict libffm_tst_fold_9.txt libffm_fold_9_model tst_preds_fold_9.txt\nos.remove('libffm_val_fold_9.txt')\nos.remove('libffm_trn_fold_9.txt')\nos.remove('libffm_fold_9_model')\nos.remove('libffm_tst_fold_9.txt')\n(\n    log_loss(train['target'].iloc[fold_ids[8][1]], pd.read_csv('val_preds_fold_9.txt', header=None).values[:,0]),\n    roc_auc_score(train['target'].iloc[fold_ids[8][1]], pd.read_csv('val_preds_fold_9.txt', header=None).values[:,0])\n)\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":false},"cell_type":"code","source":"!./ffm-train -p libffm_val_fold_10.txt -r 0.05 -l 0.00002 -k 50 --auto-stop libffm_trn_fold_10.txt libffm_fold_10_model\n!./ffm-predict libffm_val_fold_10.txt libffm_fold_10_model val_preds_fold_10.txt\n!./ffm-predict libffm_tst_fold_10.txt libffm_fold_10_model tst_preds_fold_10.txt\nos.remove('libffm_val_fold_10.txt')\nos.remove('libffm_trn_fold_10.txt')\nos.remove('libffm_fold_10_model')\nos.remove('libffm_tst_fold_10.txt')\n(\n    log_loss(train['target'].iloc[fold_ids[9][1]], pd.read_csv('val_preds_fold_10.txt', header=None).values[:,0]),\n    roc_auc_score(train['target'].iloc[fold_ids[9][1]], pd.read_csv('val_preds_fold_10.txt', header=None).values[:,0])\n)\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":false},"cell_type":"code","source":"!./ffm-train -p libffm_val_fold_11.txt -r 0.05 -l 0.00002 -k 50 --auto-stop libffm_trn_fold_11.txt libffm_fold_11_model\n!./ffm-predict libffm_val_fold_11.txt libffm_fold_11_model val_preds_fold_11.txt\n!./ffm-predict libffm_tst_fold_11.txt libffm_fold_11_model tst_preds_fold_11.txt\nos.remove('libffm_val_fold_11.txt')\nos.remove('libffm_trn_fold_11.txt')\nos.remove('libffm_fold_11_model')\nos.remove('libffm_tst_fold_11.txt')\n(\n    log_loss(train['target'].iloc[fold_ids[10][1]], pd.read_csv('val_preds_fold_11.txt', header=None).values[:,0]),\n    roc_auc_score(train['target'].iloc[fold_ids[10][1]], pd.read_csv('val_preds_fold_11.txt', header=None).values[:,0])\n)\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":false},"cell_type":"code","source":"!./ffm-train -p libffm_val_fold_12.txt -r 0.05 -l 0.00002 -k 50 --auto-stop libffm_trn_fold_12.txt libffm_fold_12_model\n!./ffm-predict libffm_val_fold_12.txt libffm_fold_12_model val_preds_fold_12.txt\n!./ffm-predict libffm_tst_fold_12.txt libffm_fold_12_model tst_preds_fold_12.txt\nos.remove('libffm_val_fold_12.txt')\nos.remove('libffm_trn_fold_12.txt')\nos.remove('libffm_fold_12_model')\nos.remove('libffm_tst_fold_12.txt')\n(\n    log_loss(train['target'].iloc[fold_ids[11][1]], pd.read_csv('val_preds_fold_12.txt', header=None).values[:,0]),\n    roc_auc_score(train['target'].iloc[fold_ids[11][1]], pd.read_csv('val_preds_fold_12.txt', header=None).values[:,0])\n)\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":false},"cell_type":"code","source":"!./ffm-train -p libffm_val_fold_13.txt -r 0.05 -l 0.00002 -k 50 --auto-stop libffm_trn_fold_13.txt libffm_fold_13_model\n!./ffm-predict libffm_val_fold_13.txt libffm_fold_13_model val_preds_fold_13.txt\n!./ffm-predict libffm_tst_fold_13.txt libffm_fold_13_model tst_preds_fold_13.txt\nos.remove('libffm_val_fold_13.txt')\nos.remove('libffm_trn_fold_13.txt')\nos.remove('libffm_fold_13_model')\nos.remove('libffm_tst_fold_13.txt')\n(\n    log_loss(train['target'].iloc[fold_ids[12][1]], pd.read_csv('val_preds_fold_13.txt', header=None).values[:,0]),\n    roc_auc_score(train['target'].iloc[fold_ids[12][1]], pd.read_csv('val_preds_fold_13.txt', header=None).values[:,0])\n)\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":false},"cell_type":"code","source":"!./ffm-train -p libffm_val_fold_14.txt -r 0.05 -l 0.00002 -k 50 --auto-stop libffm_trn_fold_14.txt libffm_fold_14_model\n!./ffm-predict libffm_val_fold_14.txt libffm_fold_14_model val_preds_fold_14.txt\n!./ffm-predict libffm_tst_fold_14.txt libffm_fold_14_model tst_preds_fold_14.txt\nos.remove('libffm_val_fold_14.txt')\nos.remove('libffm_trn_fold_14.txt')\nos.remove('libffm_fold_14_model')\nos.remove('libffm_tst_fold_14.txt')\n(\n    log_loss(train['target'].iloc[fold_ids[13][1]], pd.read_csv('val_preds_fold_14.txt', header=None).values[:,0]),\n    roc_auc_score(train['target'].iloc[fold_ids[13][1]], pd.read_csv('val_preds_fold_14.txt', header=None).values[:,0])\n)\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":false},"cell_type":"code","source":"!./ffm-train -p libffm_val_fold_15.txt -r 0.05 -l 0.00002 -k 50 --auto-stop libffm_trn_fold_15.txt libffm_fold_15_model\n!./ffm-predict libffm_val_fold_15.txt libffm_fold_15_model val_preds_fold_15.txt\n!./ffm-predict libffm_tst_fold_15.txt libffm_fold_15_model tst_preds_fold_15.txt\nos.remove('libffm_val_fold_15.txt')\nos.remove('libffm_trn_fold_15.txt')\nos.remove('libffm_fold_15_model')\nos.remove('libffm_tst_fold_15.txt')\n(\n    log_loss(train['target'].iloc[fold_ids[14][1]], pd.read_csv('val_preds_fold_15.txt', header=None).values[:,0]),\n    roc_auc_score(train['target'].iloc[fold_ids[14][1]], pd.read_csv('val_preds_fold_15.txt', header=None).values[:,0])\n)\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":false},"cell_type":"code","source":"!./ffm-train -p libffm_val_fold_16.txt -r 0.05 -l 0.00002 -k 50 --auto-stop libffm_trn_fold_16.txt libffm_fold_16_model\n!./ffm-predict libffm_val_fold_16.txt libffm_fold_16_model val_preds_fold_16.txt\n!./ffm-predict libffm_tst_fold_16.txt libffm_fold_16_model tst_preds_fold_16.txt\nos.remove('libffm_val_fold_16.txt')\nos.remove('libffm_trn_fold_16.txt')\nos.remove('libffm_fold_16_model')\nos.remove('libffm_tst_fold_16.txt')\n(\n    log_loss(train['target'].iloc[fold_ids[15][1]], pd.read_csv('val_preds_fold_16.txt', header=None).values[:,0]),\n    roc_auc_score(train['target'].iloc[fold_ids[15][1]], pd.read_csv('val_preds_fold_16.txt', header=None).values[:,0])\n)\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":false},"cell_type":"code","source":"!./ffm-train -p libffm_val_fold_17.txt -r 0.05 -l 0.00002 -k 50 --auto-stop libffm_trn_fold_17.txt libffm_fold_17_model\n!./ffm-predict libffm_val_fold_17.txt libffm_fold_17_model val_preds_fold_17.txt\n!./ffm-predict libffm_tst_fold_17.txt libffm_fold_17_model tst_preds_fold_17.txt\nos.remove('libffm_val_fold_17.txt')\nos.remove('libffm_trn_fold_17.txt')\nos.remove('libffm_fold_17_model')\nos.remove('libffm_tst_fold_17.txt')\n(\n    log_loss(train['target'].iloc[fold_ids[16][1]], pd.read_csv('val_preds_fold_17.txt', header=None).values[:,0]),\n    roc_auc_score(train['target'].iloc[fold_ids[16][1]], pd.read_csv('val_preds_fold_17.txt', header=None).values[:,0])\n)\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":false},"cell_type":"code","source":"!./ffm-train -p libffm_val_fold_18.txt -r 0.05 -l 0.00002 -k 50 --auto-stop libffm_trn_fold_18.txt libffm_fold_18_model\n!./ffm-predict libffm_val_fold_18.txt libffm_fold_18_model val_preds_fold_18.txt\n!./ffm-predict libffm_tst_fold_18.txt libffm_fold_18_model tst_preds_fold_18.txt\nos.remove('libffm_val_fold_18.txt')\nos.remove('libffm_trn_fold_18.txt')\nos.remove('libffm_fold_18_model')\nos.remove('libffm_tst_fold_18.txt')\n(\n    log_loss(train['target'].iloc[fold_ids[17][1]], pd.read_csv('val_preds_fold_18.txt', header=None).values[:,0]),\n    roc_auc_score(train['target'].iloc[fold_ids[17][1]], pd.read_csv('val_preds_fold_18.txt', header=None).values[:,0])\n)\n\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":false},"cell_type":"code","source":"!./ffm-train -p libffm_val_fold_19.txt -r 0.05 -l 0.00002 -k 50 --auto-stop libffm_trn_fold_19.txt libffm_fold_19_model\n!./ffm-predict libffm_val_fold_19.txt libffm_fold_19_model val_preds_fold_19.txt\n!./ffm-predict libffm_tst_fold_19.txt libffm_fold_19_model tst_preds_fold_19.txt\nos.remove('libffm_val_fold_19.txt')\nos.remove('libffm_trn_fold_19.txt')\nos.remove('libffm_fold_19_model')\nos.remove('libffm_tst_fold_19.txt')\n(\n    log_loss(train['target'].iloc[fold_ids[18][1]], pd.read_csv('val_preds_fold_19.txt', header=None).values[:,0]),\n    roc_auc_score(train['target'].iloc[fold_ids[18][1]], pd.read_csv('val_preds_fold_19.txt', header=None).values[:,0])\n)\n\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":false},"cell_type":"code","source":"!./ffm-train -p libffm_val_fold_20.txt -r 0.05 -l 0.00002 -k 50 --auto-stop libffm_trn_fold_20.txt libffm_fold_20_model\n!./ffm-predict libffm_val_fold_20.txt libffm_fold_20_model val_preds_fold_20.txt\n!./ffm-predict libffm_tst_fold_20.txt libffm_fold_20_model tst_preds_fold_20.txt\nos.remove('libffm_val_fold_20.txt')\nos.remove('libffm_trn_fold_20.txt')\nos.remove('libffm_fold_20_model')\nos.remove('libffm_tst_fold_20.txt')\n(\n    log_loss(train['target'].iloc[fold_ids[19][1]], pd.read_csv('val_preds_fold_20.txt', header=None).values[:,0]),\n    roc_auc_score(train['target'].iloc[fold_ids[19][1]], pd.read_csv('val_preds_fold_20.txt', header=None).values[:,0])\n)\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":false},"cell_type":"code","source":"!./ffm-train -p libffm_val_fold_21.txt -r 0.05 -l 0.00002 -k 50 --auto-stop libffm_trn_fold_21.txt libffm_fold_21_model\n!./ffm-predict libffm_val_fold_21.txt libffm_fold_21_model val_preds_fold_21.txt\n!./ffm-predict libffm_tst_fold_21.txt libffm_fold_21_model tst_preds_fold_21.txt\nos.remove('libffm_val_fold_21.txt')\nos.remove('libffm_trn_fold_21.txt')\nos.remove('libffm_fold_21_model')\nos.remove('libffm_tst_fold_21.txt')\n(\n    log_loss(train['target'].iloc[fold_ids[20][1]], pd.read_csv('val_preds_fold_21.txt', header=None).values[:,0]),\n    roc_auc_score(train['target'].iloc[fold_ids[20][1]], pd.read_csv('val_preds_fold_21.txt', header=None).values[:,0])\n)\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":false},"cell_type":"code","source":"!./ffm-train -p libffm_val_fold_22.txt -r 0.05 -l 0.00002 -k 50 --auto-stop libffm_trn_fold_22.txt libffm_fold_22_model\n!./ffm-predict libffm_val_fold_22.txt libffm_fold_22_model val_preds_fold_22.txt\n!./ffm-predict libffm_tst_fold_22.txt libffm_fold_22_model tst_preds_fold_22.txt\nos.remove('libffm_val_fold_22.txt')\nos.remove('libffm_trn_fold_22.txt')\nos.remove('libffm_fold_22_model')\nos.remove('libffm_tst_fold_22.txt')\n(\n    log_loss(train['target'].iloc[fold_ids[21][1]], pd.read_csv('val_preds_fold_22.txt', header=None).values[:,0]),\n    roc_auc_score(train['target'].iloc[fold_ids[21][1]], pd.read_csv('val_preds_fold_22.txt', header=None).values[:,0])\n)\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":false},"cell_type":"code","source":"!./ffm-train -p libffm_val_fold_23.txt -r 0.05 -l 0.00002 -k 50 --auto-stop libffm_trn_fold_23.txt libffm_fold_23_model\n!./ffm-predict libffm_val_fold_23.txt libffm_fold_23_model val_preds_fold_23.txt\n!./ffm-predict libffm_tst_fold_23.txt libffm_fold_23_model tst_preds_fold_23.txt\nos.remove('libffm_val_fold_23.txt')\nos.remove('libffm_trn_fold_23.txt')\nos.remove('libffm_fold_23_model')\nos.remove('libffm_tst_fold_23.txt')\n(\n    log_loss(train['target'].iloc[fold_ids[22][1]], pd.read_csv('val_preds_fold_23.txt', header=None).values[:,0]),\n    roc_auc_score(train['target'].iloc[fold_ids[22][1]], pd.read_csv('val_preds_fold_23.txt', header=None).values[:,0])\n)\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":false},"cell_type":"code","source":"!./ffm-train -p libffm_val_fold_24.txt -r 0.05 -l 0.00002 -k 50 --auto-stop libffm_trn_fold_24.txt libffm_fold_24_model\n!./ffm-predict libffm_val_fold_24.txt libffm_fold_24_model val_preds_fold_24.txt\n!./ffm-predict libffm_tst_fold_24.txt libffm_fold_24_model tst_preds_fold_24.txt\nos.remove('libffm_val_fold_24.txt')\nos.remove('libffm_trn_fold_24.txt')\nos.remove('libffm_fold_24_model')\nos.remove('libffm_tst_fold_24.txt')\n(\n    log_loss(train['target'].iloc[fold_ids[23][1]], pd.read_csv('val_preds_fold_24.txt', header=None).values[:,0]),\n    roc_auc_score(train['target'].iloc[fold_ids[23][1]], pd.read_csv('val_preds_fold_24.txt', header=None).values[:,0])\n)\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":false},"cell_type":"code","source":"!./ffm-train -p libffm_val_fold_25.txt -r 0.05 -l 0.00002 -k 50 --auto-stop libffm_trn_fold_25.txt libffm_fold_25_model\n!./ffm-predict libffm_val_fold_25.txt libffm_fold_25_model val_preds_fold_25.txt\n!./ffm-predict libffm_tst_fold_25.txt libffm_fold_25_model tst_preds_fold_25.txt\nos.remove('libffm_val_fold_25.txt')\nos.remove('libffm_trn_fold_25.txt')\nos.remove('libffm_fold_25_model')\nos.remove('libffm_tst_fold_25.txt')\n(\n    log_loss(train['target'].iloc[fold_ids[24][1]], pd.read_csv('val_preds_fold_25.txt', header=None).values[:,0]),\n    roc_auc_score(train['target'].iloc[fold_ids[24][1]], pd.read_csv('val_preds_fold_25.txt', header=None).values[:,0])\n)\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":false},"cell_type":"code","source":"","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Compute OOF score"},{"metadata":{"trusted":false},"cell_type":"code","source":"oof_preds = np.zeros(train.shape[0])\nfor fold_, (_, val_) in enumerate(fold_ids):\n    oof_preds[val_] = pd.read_csv(f'val_preds_fold_{fold_+1}.txt', header=None).values[:, 0]\noof_score = roc_auc_score(train['target'], oof_preds)\nprint(oof_score)","execution_count":null,"outputs":[]},{"metadata":{"trusted":false},"cell_type":"code","source":"test_preds = np.zeros((test.shape[0], N_Splits))\nfor fold_ in range(N_Splits):\n    test_preds[:, fold_] = pd.read_csv(f'tst_preds_fold_{fold_+1}.txt', header=None).values[:, 0]\n\ntest_preds_avg = test_preds.mean(axis=1)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Prepare submission"},{"metadata":{"trusted":false},"cell_type":"code","source":"submission = test[['id']].copy()\nsubmission['target'] = test_preds_avg\nsubmission.to_csv('libffm_sub_531.csv', index=False)","execution_count":null,"outputs":[]},{"metadata":{"trusted":false},"cell_type":"code","source":"np.save('test_preds_libffm.npy', test_preds_avg)\nnp.save('oof_preds_libffm.npy', oof_preds)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"**Blend-Part**"},{"metadata":{"trusted":false},"cell_type":"code","source":"subs = [\n    '/kaggle/input/bestpublicscores/sub_623.csv',\n    '/kaggle/input/bestpublicscores/sub_634.csv',\n    '/kaggle/input/bestpublicscores/sub_626.csv',\n    '/kaggle/input/bestpublicscores/sub_600.csv',\n    '/kaggle/input/bestpublicscores/sub_590.csv',\n    '/kaggle/input/otherbestpublicscores/sub_659.csv',\n    '/kaggle/input/otherbestpublicscores/sub_606.csv',\n    '/kaggle/input/otherbestpublicscores/sub_563.csv',\n    '/kaggle/input/otherbestpublicscores/sub_620.csv',\n    '/kaggle/input/bestpublicscores3/sub_589.csv',\n    'libffm_sub_531.csv'\n       ]\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":false},"cell_type":"code","source":"predictions = pd.concat([pd.read_csv(sub, index_col='id') for sub in subs], axis=1).reset_index(drop=True)\npredictions.columns = ['sub_'+str(i) for i in range(11)]\npredictions","execution_count":null,"outputs":[]},{"metadata":{"trusted":false},"cell_type":"code","source":"for col in predictions.columns:\n    predictions[col]=predictions[col].rank()/predictions.shape[0]","execution_count":null,"outputs":[]},{"metadata":{"trusted":false},"cell_type":"code","source":"corr = predictions.corr()\nmask = np.zeros_like(corr, dtype=np.bool)\nmask[np.triu_indices_from(mask)] = True\nf, ax = plt.subplots(figsize=(12, 12))\nsns.heatmap(corr, mask=mask, cmap='Blues', vmin=0.95, center=0, linewidths=1, annot=True, fmt='.4f')\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":false},"cell_type":"code","source":"coefs = [0.1, 0.075, 0.1, 0.05, 0.025, 0.375, 0.1, 0.05, 0.05, 0.05, 0.025]\ndef blend_subs(df, coefs=coefs):\n    blend = np.zeros(df.shape[0])\n    for idx, column in enumerate(df.columns):\n        blend += coefs[idx] * (df[column].values)\n    return blend\n\nblend = blend_subs(predictions)","execution_count":null,"outputs":[]},{"metadata":{"trusted":false},"cell_type":"code","source":"blend","execution_count":null,"outputs":[]},{"metadata":{"trusted":false},"cell_type":"code","source":"submission['target'] = blend\nsubmission.to_csv('TopPublicBlend.csv',index=False)\nsubmission.head()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"The Public submission files are from following kernels:\n\n[deepfm-model](https://www.kaggle.com/siavrez/deepfm-model)\n\n[keras-r-embeddings-baseline](https://www.kaggle.com/springmanndaniel/keras-r-embeddings-baseline)\n\n[same-old-entity-embeddings](https://www.kaggle.com/abhishek/same-old-entity-embeddings)\n\n[catboost-in-action-with-dnn](https://www.kaggle.com/lucamassaron/catboost-in-action-with-dnn)\n\n[oh-my-plain-logreg](https://www.kaggle.com/superant/oh-my-plain-logreg)\n\n[complicated](https://www.kaggle.com/scirpus/complicated)\n\n[let-s-overfit-some](https://www.kaggle.com/ccccat/let-s-overfit-some)"},{"metadata":{"trusted":false},"cell_type":"code","source":"","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"codemirror_mode":{"name":"ipython","version":3},"file_extension":".py","mimetype":"text/x-python","name":"python","nbconvert_exporter":"python","pygments_lexer":"ipython3","version":"3.6.6"}},"nbformat":4,"nbformat_minor":1}