{"cells":[{"metadata":{},"cell_type":"markdown","source":"## Introduction\n\n[Field-Aware Factorization](https://www.csie.ntu.edu.tw/~cjlin/libffm) is a powerful representation learning.\n\n[Github here.](https://github.com/ycjuan/libffm)\n\nThis notebook demonstrates a way to use libffm binaries into a Kaggle kernel.\n\nRelease Notes :\n - V4 : New version with Out-of-Fold\n - V6 : fixed the encoder, previous version was kind of a regularizer :) \n "},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"code","source":"import numpy as np\nimport pandas as pd \nimport os\nfor dirname, _, filenames in os.walk('/kaggle/input'):\n    for filename in filenames:\n        print(os.path.join(dirname, filename))\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Read the data"},{"metadata":{"trusted":true},"cell_type":"code","source":"train = pd.read_csv('/kaggle/input/cat-in-the-dat-ii/train.csv')\ntest = pd.read_csv('/kaggle/input/cat-in-the-dat-ii/test.csv')\ntest.insert(1, 'target', 0)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Label Encode to ease creation of libffm format"},{"metadata":{"trusted":true},"cell_type":"code","source":"features = [_f for _f in train if _f not in ['id', 'target']]\n\ndef factor_encoding(train, test):\n    \n    assert sorted(train.columns) == sorted(test.columns)\n    \n    full = pd.concat([train, test], axis=0, sort=False)\n    # Factorize everything\n    for f in full:\n        full[f], _ = pd.factorize(full[f])\n        full[f] += 1  # make sure no negative\n        \n    return full.iloc[:train.shape[0]], full.iloc[train.shape[0]:]\n\ntrain_f, test_f = factor_encoding(train[features], test[features])","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Create LibFFM files\n\n\nThe data format of LIBFFM has a very special format (taken from [libffm page](https://github.com/ycjuan/libffm)):\n```\n<label> <field1>:<feature1>:<value1> <field2>:<feature2>:<value2> ...\n.\n.\n.\n```\n\n`field` and `feature` should be non-negative integers.\n\nIt is important to understand the difference between `field` and `feature`. For example, if we have a raw data like this:\n\n| Click | Advertiser | Publisher |\n|:-----:|:----------:|:---------:|\n|    0 |       Nike |       CNN |\n|    1 |       ESPN |       BBC |\n\nHere, we have \n \n - 2 fields: Advertiser and Publisher\n - 4 features: Advertiser-Nike, Advertiser-ESPN, Publisher-CNN, Publisher-BBC\n\nUsually you will need to build two dictionares, one for field and one for features, like this:\n    \n    DictField[Advertiser] -> 0\n    DictField[Publisher]  -> 1\n    \n    DictFeature[Advertiser-Nike] -> 0\n    DictFeature[Publisher-CNN]   -> 1\n    DictFeature[Advertiser-ESPN] -> 2\n    DictFeature[Publisher-BBC]   -> 3\n\nThen, you can generate FFM format data:\n\n    0 0:0:1 1:1:1\n    1 0:2:1 1:3:1\n\nNote that because these features are categorical, the values here are all ones.\n\nThe class defined below go through all features and rows and update a python dicts as new values are encountered."},{"metadata":{"trusted":true},"cell_type":"code","source":"class LibFFMEncoder(object):\n    def __init__(self):\n        self.encoder = 1\n        self.encoding = {}\n\n    def encode_for_libffm(self, row):\n        txt = f\"{row[0]}\"\n        for i, r in enumerate(row[1:]):\n            try:\n                txt += f' {i+1}:{self.encoding[(i, r)]}:1'\n            except KeyError:\n                self.encoding[(i, r)] = self.encoder\n                self.encoder += 1\n                txt += f' {i+1}:{self.encoding[(i, r)]}:1'\n\n        return txt\n\n# Create files for testing and OOF\nfrom sklearn.model_selection import KFold\nfold_ids = [\n    [trn_, val_] for (trn_, val_) in KFold(5,True,1).split(train)\n]\nfor fold_, (trn_, val_) in enumerate(fold_ids):\n    # Fit the encoder\n    encoder = LibFFMEncoder()\n    libffm_format_trn = pd.concat([train['target'].iloc[trn_], train_f.iloc[trn_]], axis=1).apply(\n        lambda row: encoder.encode_for_libffm(row), raw=True, axis=1\n    )\n    # Encode validation set\n    libffm_format_val = pd.concat([train['target'].iloc[val_], train_f.iloc[val_]], axis=1).apply(\n        lambda row: encoder.encode_for_libffm(row), raw=True, axis=1\n    )\n    \n    print(train['target'].iloc[trn_].shape, train['target'].iloc[val_].shape, libffm_format_val.shape)\n    \n    libffm_format_trn.to_csv(f'libffm_trn_fold_{fold_+1}.txt', index=False, header=False)\n    libffm_format_val.to_csv(f'libffm_val_fold_{fold_+1}.txt', index=False, header=False)\n    \n    \n# Create files for final model\nencoder = LibFFMEncoder()\nlibffm_format_trn = pd.concat([train['target'], train_f], axis=1).apply(\n        lambda row: encoder.encode_for_libffm(row), raw=True, axis=1\n)\nlibffm_format_tst = pd.concat([test['target'], test_f], axis=1).apply(\n    lambda row: encoder.encode_for_libffm(row), raw=True, axis=1\n)\n\nlibffm_format_trn.to_csv(f'libffm_trn.txt', index=False, header=False)\nlibffm_format_tst.to_csv(f'libffm_tst.txt', index=False, header=False)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Make ffm-train and ffm-predict excutable"},{"metadata":{"trusted":true},"cell_type":"code","source":"!cp /kaggle/input/libffm-binaries/ffm-train .\n!cp /kaggle/input/libffm-binaries/ffm-predict .\n!chmod u+x ffm-train\n!chmod u+x ffm-predict","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Run OOF"},{"metadata":{"trusted":true},"cell_type":"code","source":"from sklearn.metrics import log_loss, roc_auc_score\n\n!./ffm-train -p libffm_val_fold_1.txt -r 0.05 -l 0.00001 -k 50 -t 7 libffm_trn_fold_1.txt libffm_fold_1_model\n!./ffm-predict libffm_val_fold_1.txt libffm_fold_1_model val_preds_fold_1.txt\n(\n    log_loss(train['target'].iloc[fold_ids[0][1]], pd.read_csv('val_preds_fold_1.txt', header=None).values[:,0]),\n    roc_auc_score(train['target'].iloc[fold_ids[0][1]], pd.read_csv('val_preds_fold_1.txt', header=None).values[:,0])\n)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"!./ffm-train -p libffm_val_fold_2.txt -r 0.05 -l 0.00001 -k 50 -t 7 libffm_trn_fold_2.txt libffm_fold_2_model\n!./ffm-predict libffm_val_fold_2.txt libffm_fold_2_model val_preds_fold_2.txt\n(\n    log_loss(train['target'].iloc[fold_ids[1][1]], pd.read_csv('val_preds_fold_2.txt', header=None).values[:,0]),\n    roc_auc_score(train['target'].iloc[fold_ids[1][1]], pd.read_csv('val_preds_fold_2.txt', header=None).values[:,0])\n)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"!./ffm-train -p libffm_val_fold_3.txt -r 0.05 -l 0.00001 -k 50 -t 7 libffm_trn_fold_3.txt libffm_fold_3_model\n!./ffm-predict libffm_val_fold_3.txt libffm_fold_3_model val_preds_fold_3.txt\n(\n    log_loss(train['target'].iloc[fold_ids[2][1]], pd.read_csv('val_preds_fold_3.txt', header=None).values[:,0]),\n    roc_auc_score(train['target'].iloc[fold_ids[2][1]], pd.read_csv('val_preds_fold_3.txt', header=None).values[:,0])\n)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"!./ffm-train -p libffm_val_fold_4.txt -r 0.05 -l 0.00001 -k 50 -t 7 libffm_trn_fold_4.txt libffm_fold_4_model\n!./ffm-predict libffm_val_fold_4.txt libffm_fold_4_model val_preds_fold_4.txt\n(\n    log_loss(train['target'].iloc[fold_ids[3][1]], pd.read_csv('val_preds_fold_4.txt', header=None).values[:,0]),\n    roc_auc_score(train['target'].iloc[fold_ids[3][1]], pd.read_csv('val_preds_fold_4.txt', header=None).values[:,0])\n)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"!./ffm-train -p libffm_val_fold_5.txt -r 0.05 -l 0.00001 -k 50 -t 7 libffm_trn_fold_5.txt libffm_fold_5_model\n!./ffm-predict libffm_val_fold_5.txt libffm_fold_5_model val_preds_fold_5.txt\n(\n    log_loss(train['target'].iloc[fold_ids[4][1]], pd.read_csv('val_preds_fold_5.txt', header=None).values[:,0]),\n    roc_auc_score(train['target'].iloc[fold_ids[4][1]], pd.read_csv('val_preds_fold_5.txt', header=None).values[:,0])\n)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Compute OOF score"},{"metadata":{"trusted":true},"cell_type":"code","source":"oof_preds = np.zeros(train.shape[0])\nfor fold_, (_, val_) in enumerate(fold_ids):\n    oof_preds[val_] = pd.read_csv(f'val_preds_fold_{fold_+1}.txt', header=None).values[:, 0]\noof_score = roc_auc_score(train['target'], oof_preds)\nprint(oof_score)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Train a libffm model"},{"metadata":{"trusted":true},"cell_type":"code","source":"!./ffm-train -r 0.05 -l 0.00001 -k 50 -t 7 libffm_trn.txt libffm_model","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Predict for test set"},{"metadata":{"trusted":true},"cell_type":"code","source":"!./ffm-predict libffm_tst.txt libffm_model tst_preds.txt","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Prepare submission"},{"metadata":{"trusted":true},"cell_type":"code","source":"submission = test[['id']].copy()\nsubmission['target'] = pd.read_csv('tst_preds.txt', header=None).values[:,0]\nsubmission.to_csv('libffm_prediction.csv', index=False)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":1}