{"cells":[{"metadata":{},"cell_type":"markdown","source":"# Introduction\n\nThis notebook consists of seven main sections:\n* 0) Setup\n    * Basic notebook setup.\n* 1) A refactoring/collection of other notebooks\n    * We tried to make the code more understandable and extracted many feature-engineered attributes and useful functions from this notebook.\n* 2) SVD-Based classification and experiments\n    * Here we do some more feature engineering and make a simple classifier based on a few of the extracted vectors. The result of this classifier will be later passed as an attribute to the final classifier.\n* 3) Cascading classifiers - Original features\n    * Here we combine the attributes extracted from the first notebook and the results from the SVD classifier.\n* 4) Cascading classifiers - SVD + Atemporal features + Temporal features\n    * Here we experiment with separating the classifiers even further (we called this approach \"divide et impera\"), fitting models on features directly related to time (called \"temporal\" from now on) and those not directly related to time (called \"atemporal\" from now on) separately and then combining them in a cascading classifier. This test is mostly exploratory, but we expect it to allow each classifier to pick up on subtler patterns within their domains.\n* 5) Cascading classifiers - Frankenstein's monster\n    * After not achieving much better results, we decided to make a final classifier that has access to every feature we generated so far (SVD vectors, SVD classifier result, Temporal classifier result, Atemporal classifier result and our original features). Once again, this test is mostly exploratory.\n* 6) Conclusion and analysis","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"# Section 0 - Setup\n\nHere we import the needed libraries and load our datasets.","execution_count":null},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"code","source":"# This Python 3 environment comes with many helpful analytics libraries installed\n# It is defined by the kaggle/python Docker image: https://github.com/kaggle/docker-python\n# For example, here's several helpful packages to load\n\nimport numpy as np # linear algebra\nimport pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\nimport datetime\nimport gc\nfrom pathlib import Path\n# Input data files are available in the read-only \"../input/\" directory\n# For example, running this (by clicking run or pressing Shift+Enter) will list all files under the input directory\n\nimport os\n# for dirname, _, filenames in os.walk('../'):\n#     for filename in filenames:\n#         print(os.path.join(dirname, filename))\n\n# You can write up to 5GB to the current directory (/kaggle/working/) that gets preserved as output when you create a version using \"Save & Run All\" \n# You can also write temporary files to /kaggle/temp/, but they won't be saved outside of the current session\n\n# copy-paste\nimport numpy as np\nimport pandas as pd\nimport datetime\nimport gc\nimport matplotlib.pyplot as plt\nimport seaborn as sns\nimport lightgbm as lgb\nfrom sklearn.model_selection import StratifiedKFold\nfrom sklearn.metrics import mean_squared_error\nfrom sklearn.decomposition import TruncatedSVD\nimport warnings\nimport math\nwarnings.filterwarnings('ignore')\nnp.random.seed(4590)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"%%time\nPATH = Path('../input/elo-merchant-category-recommendation')\ndf_train = pd.read_csv(PATH/'train.csv', parse_dates=['first_active_month']);\ndf_test = pd.read_csv(PATH/'test.csv', parse_dates=['first_active_month']);","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"%%time\ndf_hist_trans = pd.read_csv(PATH/'historical_transactions.csv', parse_dates=['purchase_date']);\ndf_new_merch_trans = pd.read_csv(PATH/'new_merchant_transactions.csv', parse_dates=['purchase_date']);","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Section I - Refactoring\n\nWe analysed some existing notebooks to understand how people were approaching the problem so far.\n\nThis section consists in a collection of interesting and useful snippets taken from those notebooks.\n\nThese cells were extracted from the following sources:\n\n1) [my-first-kernel](https://www.kaggle.com/chauhuynh/my-first-kernel-3-699) by [chauhuynh](https://www.kaggle.com/chauhuynh).\n\n2) [elo-world](https://www.kaggle.com/fabiendaniel/elo-world) by [fabiendaniel](https://www.kaggle.com/fabiendaniel)\n\n## Memory-saving function\n\nThis function scans datasets for potential type changes that could save memory. This considerably reduces our dataset's memory consumption.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"# https://www.kaggle.com/fabiendaniel/elo-world\ndef reduce_mem_usage(df, verbose=True):\n    prefixes = ['int', 'float']\n    start_mem = df.memory_usage().sum() / 1024**2    \n    for col in df.columns:\n        col_type = str(df[col].dtype)\n        if not col_type.startswith('int') and not col_type.startswith('float'):\n#             print('col_type:', col_type, 'not compressed')\n            continue\n        c_min = df[col].min()\n        c_max = df[col].max()\n        if col_type.startswith('int'):\n            if c_min > np.iinfo(np.int8).min and c_max < np.iinfo(np.int8).max:\n                df[col] = df[col].astype(np.int8)\n            elif c_min > np.iinfo(np.int16).min and c_max < np.iinfo(np.int16).max:\n                df[col] = df[col].astype(np.int16)\n            elif c_min > np.iinfo(np.int32).min and c_max < np.iinfo(np.int32).max:\n                df[col] = df[col].astype(np.int32)\n            elif c_min > np.iinfo(np.int64).min and c_max < np.iinfo(np.int64).max:\n                df[col] = df[col].astype(np.int64)  \n        elif col_type.startswith('float'):\n            if c_min > np.finfo(np.float16).min and c_max < np.finfo(np.float16).max:\n                df[col] = df[col].astype(np.float16)\n            elif c_min > np.finfo(np.float32).min and c_max < np.finfo(np.float32).max:\n                df[col] = df[col].astype(np.float32)\n            else:\n                df[col] = df[col].astype(np.float64)    \n    if verbose:\n        end_mem = df.memory_usage().sum() / 1024**2\n        print('Mem. usage decreased to {:5.2f} Mb ({:.1f}% reduction)'.format(end_mem, 100 * (start_mem - end_mem) / start_mem))\n    return df\n\n# https://www.kaggle.com/chauhuynh/my-first-kernel-3-699/\n# Same logic\ndef fillna_mode(df, cols, pipeline=False):\n    for c in cols:\n        df[c].fillna(df[c].mode()[0], inplace=True)\n    return df if pipeline else None\n\n\ndef get_nan_col_names(df: pd.DataFrame, df_name='<not named>'):\n    total = df.shape[0]\n    missing_cols = []\n    for c in df.columns:\n        quo = (total - pd.notna(df[c]).sum())/total\n        if quo != 0:\n            missing_cols.append(c)\n    print(df_name, 'MISSING COLS:', missing_cols)\n    return missing_cols","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"reduce_mem_usage(df_train);\nreduce_mem_usage(df_test);\nreduce_mem_usage(df_hist_trans);\nreduce_mem_usage(df_new_merch_trans);","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Time and date-related feature functions\n\nThese functions will be used to generate extra features based on the date data we have.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"#https://www.kaggle.com/c/elo-merchant-category-recommendation/discussion/73244\ndef add_month_diff(df, pipeline=False):\n    df['month_diff'] = ((datetime.datetime.today() - df['purchase_date']).dt.days) // 30\n    df['month_diff'] += df['month_lag']\n    return df if pipeline else None\n\n\n# https://www.kaggle.com/chauhuynh/my-first-kernel-3-699\ndef pre_process_trans(df, date_col, date_formated=False, add_month=True):\n    fillna_mode(df, cols=get_nan_col_names(df))\n    df['authorized_flag'] = df['authorized_flag'].map({'Y':1, 'N':0}).astype(np.int8)\n    df['category_1'] = df['category_1'].map({'Y': 1, 'N': 0}).astype(np.int8)    \n    df['category_2'] = df['category_2'].astype(np.int8)\n\n    if not date_formated:\n        df[date_col] = pd.to_datetime(df[date_col])\n    df['year'] = df[date_col].dt.year\n    df['weekofyear'] = df[date_col].dt.weekofyear\n    df['month'] = df[date_col].dt.month\n    df['dayofweek'] = df[date_col].dt.dayofweek\n    df['weekend'] = (df[date_col].dt.weekday >= 5).astype(np.int8)\n    df['hour'] = df[date_col].dt.hour\n    if add_month:\n        add_month_diff(df)\n\n\n# https://www.kaggle.com/chauhuynh/my-first-kernel-3-699/\ndef get_new_columns(name,aggs):\n    return [name + '_' + k + '_' + agg for k in aggs.keys() for agg in aggs[k]]\n\n\n# Refactor of https://www.kaggle.com/chauhuynh/my-first-kernel-3-699/\ndef custom_group_by(df: pd.DataFrame, df_name: str, agg_by: dict):\n    for col in ['category_2','category_3']:\n        df[col+'_mean'] = df.groupby([col])['purchase_amount'].transform('mean')\n        aggs[col+'_mean'] = ['mean']    \n    new_columns = get_new_columns(df_name, agg_by)\n#     print(\"new_columns: \\n\", *new_columns, sep='\\t\\n')\n    df_group = df.groupby('card_id').agg(agg_by)\n    df_group.columns = new_columns\n    df_group.reset_index(drop=False, inplace=True)\n    df_group[df_name + '_purchase_date_diff'] = (\n        df_group[df_name + '_purchase_date_max']\n            - df_group[df_name + '_purchase_date_min']\n        ).dt.days\n    df_group[df_name + '_purchase_date_average'] = (\n        df_group[df_name + '_purchase_date_diff']\n            / df_group[df_name + '_card_id_size']\n        )\n    df_group[df_name + '_purchase_date_uptonow'] = (\n        datetime.datetime.today()\n        - df_group[df_name + '_purchase_date_max']\n        ).dt.days\n\n    return df_group","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Feature-engineered attributes\nThese attributes were extracted from the following source:\n1. [my-first-kernel][mfk]\n\n[mfk]: https://www.kaggle.com/chauhuynh/my-first-kernel-3-699/comments","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"aggs = {}\n\n# Count number of unique values at each column.\nfor col in ['month','hour','weekofyear','dayofweek','year','subsector_id','merchant_id','merchant_category_id']:\n    aggs[col] = ('nunique',)\n\n\naggs['purchase_amount'] = ('sum','max','min','mean','var')\naggs['installments'] = ('sum','max','min','mean','var')\n\n# purchase range\naggs['purchase_date'] = ('max','min')\n\naggs['month_lag'] = ('max','min','mean','var')\naggs['month_diff'] = ('mean',)\n\n# How many transactions:\n# - were on the weekend, and the percentage\n# - has category_1 as 1 (binary feature), and the percentage\n# - were authorized, and its percentage\n\naggs['weekend'] = ('sum', 'mean')\naggs['category_1'] = ('sum', 'mean')\naggs['authorized_flag'] = ('sum', 'mean')\n\n# How many purchases each card did?\naggs['card_id'] = ('size', )","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"_old_train = df_train.copy()\n_odl_test = df_test.copy()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"%%time\ngps = []\nfor name, df in [('hist', df_hist_trans), ('new_hist', df_new_merch_trans)]:    \n    pre_process_trans(df, date_col='purchase_date')\n    df_group = custom_group_by(df, name, aggs.copy())\n    gps.append(df_group)\n    df_train = df_train.merge(df_group, on='card_id', how='left')\n    df_test = df_test.merge(df_group, on='card_id', how='left')\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"prob = [\n    'hist_purchase_date_max',\n    'hist_purchase_date_min',\n    'new_hist_purchase_date_max',\n    'new_hist_purchase_date_min',\n]\n\ndef post_process_df(df):\n    global prob\n    \n    df['first_active_month'] = pd.to_datetime(df['first_active_month'])\n    df['fam_dayofweek'] = df['first_active_month'].dt.dayofweek\n    df['fam_weekofyear'] = df['first_active_month'].dt.weekofyear\n    df['fam_month'] = df['first_active_month'].dt.month\n    df['fam_elapsed_time'] = (datetime.datetime.today() - df['first_active_month']).dt.days\n    df['hist_first_buy'] = (df['hist_purchase_date_min'] - df['first_active_month']).dt.days\n    df['new_hist_first_buy'] = (df['new_hist_purchase_date_min'] - df['first_active_month']).dt.days\n\n    df[prob[:2]] = df[prob[:2]].astype(np.int64) * 1e-9\n\n    \n    df['transactions_total'] = df['new_hist_card_id_size']+df['hist_card_id_size']\n    df['purchase_amount_total'] = df['new_hist_purchase_amount_sum']+df['hist_purchase_amount_sum']\n\n    for f in ['feature_1','feature_2','feature_3']:\n        order_label = df_train.groupby([f])['outliers'].mean()\n        df_train[f] = df_train[f].map(order_label)\n        df_test[f] = df_test[f].map(order_label)\n    ","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"print(df_train.hist_purchase_date_max.notna().sum())\nprint(df_train.hist_purchase_date_min.notna().sum())\nprint(df_train.new_hist_purchase_date_max.notna().sum())\nprint(df_train.new_hist_purchase_date_min.notna().sum())","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"df_train['outliers'] = (df_train.target < -30).astype(np.int8)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"post_process_df(df_train)\npost_process_df(df_test)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"df_test.drop(['new_hist_purchase_date_max', 'new_hist_purchase_date_min'], axis=1,inplace=True)\ndf_train.drop(['new_hist_purchase_date_max', 'new_hist_purchase_date_min'], axis=1,inplace=True)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Meta-paremeters for the models\n\nThe following function fits an LGB model to a dataset `ds` with target `ds_ta` and test dataset `ds_te` with the num_leaves parameter set to `num_leaves`. The metaparameters were compiled from the notebooks and the two values for `num_leaves` uses in this notebook were found via trial-and-error.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"def LGB_Train(df_train,df_test,target,num_leaves):\n    param = {'num_leaves': num_leaves,\n             'min_data_in_leaf': 30, \n             'objective':'regression',\n             'max_depth': -1,\n             'learning_rate': 0.01,\n             \"min_child_samples\": 20,\n             \"boosting\": \"gbdt\",\n             \"feature_fraction\": 0.9,\n             \"bagging_freq\": 1,\n             \"bagging_fraction\": 0.9 ,\n             \"bagging_seed\": 11,\n             \"metric\": 'rmse',\n             \"lambda_l1\": 0.1,\n             \"verbosity\": -1,\n             \"nthread\": 4,\n             \"random_state\": 4590}\n    folds = StratifiedKFold(n_splits=5, shuffle=True, random_state=4590)\n    oof = np.zeros(len(df_train))\n    predictions_train = np.zeros(len(df_train)) \n    predictions_test = np.zeros(len(df_test))\n    feature_importance_df = pd.DataFrame()\n    df_train_columns = [c for c in df_train.columns if c not in ['card_id', 'first_active_month', 'outliers', 'target']]\n\n    for fold_, (trn_idx, val_idx) in enumerate(folds.split(df_train,df_train['outliers'].values)):\n        print(\"fold {}\".format(fold_))\n        trn_data = lgb.Dataset(df_train.iloc[trn_idx][df_train_columns], label=target.iloc[trn_idx])#, categorical_feature=categorical_feats)\n        val_data = lgb.Dataset(df_train.iloc[val_idx][df_train_columns], label=target.iloc[val_idx])#, categorical_feature=categorical_feats)\n\n        num_round = 10000\n        clf = lgb.train(param, trn_data, num_round, valid_sets = [trn_data, val_data], verbose_eval=100, early_stopping_rounds = 100)\n        oof[val_idx] = clf.predict(df_train.iloc[val_idx][df_train_columns], num_iteration=clf.best_iteration)\n\n        fold_importance_df = pd.DataFrame()\n        fold_importance_df[\"Feature\"] = df_train_columns\n        fold_importance_df[\"importance\"] = clf.feature_importance()\n        fold_importance_df[\"fold\"] = fold_ + 1\n        feature_importance_df = pd.concat([feature_importance_df, fold_importance_df], axis=0)\n\n        predictions_test += clf.predict(df_test[df_train_columns], num_iteration=clf.best_iteration) / folds.n_splits\n        predictions_train += clf.predict(df_train[df_train_columns], num_iteration=clf.best_iteration) / folds.n_splits\n\n    return {'predictions_test':predictions_test,'predictions_train':predictions_train,'feature_importances':feature_importance_df}\n    ","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Section II - SVD\n\nIn this section we'll be using Singular Value Decomposition for both assisting the classifiers and trying to better understand the relationships between some of the numerical attributes.\n\n## Calculation and analysis\n\nFirstly, we compute the decomposition and try analysing some of its results.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"#A dataframe containing only numerical values\ndf_num=df_train[[col for col in df_train.columns if not col in ['target','outliers']]].select_dtypes(include='number')\n\n#NaN/Missing values will be replaced with zeroes\n# df_num.fillna(0,inplace=True)\nfillna_mode(df_num, df_num.columns)\n\n\ndecomp = TruncatedSVD(n_components=len(df_num.columns.values)-1,random_state=1)\ndecomp.fit(df_num)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"We intend to select the most descriptive vectors, that is, the ones across which the data points spreads the most.\n\nLet's plot their eigenvalues.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"plot=sns.lineplot(x=range(10),y=decomp.singular_values_[:10])\nplot.set_title(\"Eigenvalues in descending order\")\nplot.set(ylabel=\"Eigenvalue\",xlabel=\"Vector\")\nplt.show()\nplot=sns.lineplot(x=range(3,len(decomp.singular_values_)),y=decomp.singular_values_[3:])\nplot.set_title(\"Eigenvalues [from 3rd on] in descending order\")\nplot.set(ylabel=\"Eigenvalue\",xlabel=\"Vector\")","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"The first few vectors clearly represent much of the information contained here. This is a good sign.\n\nThe following plot illustrates the squared weights of each numerical feature for projecting onto the first 10 component vectors (for estimating the importance of each attribute on the projection).","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"sns.heatmap([[x**2 for x in component] for component in decomp.components_[:10]])","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Some attributes consistently have more weight than others. Let's figure out which exactly are those.\n\nThe following graphs show the squared weight of each of the 3 most important attributes for the first 3 vectors.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"for component in decomp.components_[:3]:\n    weights=sorted(zip([x**2 for x in component],df_num.columns.values),reverse=True)[:3]\n    sns.barplot(data=pd.DataFrame(weights,columns=['Squared Weight','Feature Name']),y='Feature Name',x='Squared Weight')\n    plt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"It appears that the most descriptive features are consistently the **purchase date** and **expenditure**. This might make sense intuitively, but let's try not to let it bias our interpretation.","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"Let's figure out how many of the resulting vectors we should pass to the classifier.\n\nThis analysis is mostly just exploratory. We don't have much basis for actually justifying these specific choices and experiments beforehand (beyond quasi-scientifical intuition).\n\nWe arbitrarily chose to 'keep 80% of the information' (keep the vectors whose eigenvalues sum up to 80% of the total sum). This is a standard limit used when applying this kind of decomposition in images and its  effectiveness is visible (pun unintended) on those cases. We hope this concept translates well to this specific situation.\n\nFor achieving that, we'll be keeping the following number of vectors:","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"for num_vectors in range(1,len(decomp.singular_values_)):\n    acc_ratio=sum(decomp.singular_values_[:num_vectors])/sum(decomp.singular_values_)\n    if (sum(decomp.singular_values_[:num_vectors])/sum(decomp.singular_values_))>=0.80:\n            break;\nprint(str(num_vectors)+\" (\"+str(round(acc_ratio*10000)/100)+\"%)\")","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"> Sidenote: the first two vectors account for 99.92% of the accumulated eigenvalues. This looks promising.\n\nFor the record, here's a plot of the accumulated squared weights of each attribute for the vectors that will be used.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"sq_eigenval=dict(zip(df_num.columns.values,[0 for i in range(len(df_num.columns.values))]))\nfor component in decomp.components_[:num_vectors]:\n    weights=zip([x**2 for x in component],df_num.columns.values)\n    for line in weights:\n        sq_eigenval[line[1]]+=line[0] if not np.isnan(line[0]) else 0\n\n#Normalization\nsq_eigenval=pd.DataFrame(sq_eigenval.items(),columns=['Feature','Sq. Weight']).sort_values(by='Sq. Weight')\nsq_eigenval['Norm. Sq. W.']=(sq_eigenval['Sq. Weight']-sq_eigenval['Sq. Weight'].min())/(sq_eigenval['Sq. Weight'].max()-sq_eigenval['Sq. Weight'].min())\n\nplt.figure(figsize=(14,25))\nsns.barplot(\n            y='Feature',\n            x='Norm. Sq. W.',\n            data=sq_eigenval\n)\nplt.title('Normalized Sq. Weights of Features (SVD)')\nplt.tight_layout()\nplt.savefig('svd_importances.png')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"We should now generate our `training` and `testing` sets for the SVD classifier.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"df_num=df_train[[col for col in df_train.columns if not col in ['target','outliers']]].select_dtypes(include='number')\n# df_num.fillna(0,inplace=True)\nfillna_mode(df_num, df_num.columns)\n\ndf_svd_train=pd.DataFrame([x[:num_vectors]for x in decomp.transform(df_num)],columns=['svd_'+str(i) for i in range(num_vectors)])\n#df_train_svd=df_train.join(df_svd,lsuffix='_caller', rsuffix='_other')\ndf_svd_train['target']=df_train['target']\ndf_svd_train['outliers']=df_train['outliers']\n\ndf_num=pd.DataFrame(data=df_test,columns=df_num.columns.values)\n# df_num.fillna(0,inplace=True)\nfillna_mode(df_num, df_num.columns)\n\n\ndf_svd_test=pd.DataFrame([x[:num_vectors]for x in decomp.transform(df_num)],columns=['svd_'+str(i) for i in range(num_vectors)])\n#df_test_svd=df_test.join(df_svd,lsuffix='_caller', rsuffix='_other')\nprint(\"Datasets generated.\")","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Before trying to do any classification, let's project our data using the first two vectors and see if any simple visible pattern arises.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"sns.scatterplot(data=df_svd_train,x='svd_1',y='svd_0',hue='target')\nplt.show()\nsns.scatterplot(data=df_svd_train.query('svd_1>0'),x='svd_1',y='svd_0',hue='target')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"This looks nothing like we'd expect from an SVD. The vectors still have huge eigenvalues, though, so we must be onto something.\n\nLet's just give that to the classifiers.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"target=df_svd_train['target']\ndel df_svd_train['target']","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"%%time\nresults=LGB_Train(df_svd_train,df_svd_test,target,5)\npredictions_train_svd=results['predictions_train']\npredictions_test_svd=results['predictions_test']","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Ok. Now that we've finished classifying, we may use the predictions as an attribute for our next classifier.","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"# Section III - Cascading Classifiers (Original features)\n\nWe'll first integrate the results of our classifiers into our training and testing sets.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"df_train['svd_prediction']=predictions_train_svd\ndf_test['svd_prediction']=predictions_test_svd\n\ndf_train_columns = [c for c in df_train.columns if c not in ['card_id', 'first_active_month', 'outliers', 'target']]\ntarget = df_train['target']\ndel df_train['target']","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Then we fit another Light Gradient Boosting model to our new training set.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"%%time\nresults=LGB_Train(df_train,df_test,target,31)\nfeature_importance_df=results['feature_importances']\npredictions_brute=results['predictions_test']","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"After fitting, we can compare the importance of each feature with a bar plot.\n\n> Sidenote: The svd_classifier feature seems to have a high importance, providing us with a good justification for using this cascading method.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"cols = (feature_importance_df[[\"Feature\", \"importance\"]]\n        .groupby(\"Feature\")\n        .mean()\n        .sort_values(by=\"importance\", ascending=False)[:1000].index)\n\nbest_features = feature_importance_df.loc[feature_importance_df.Feature.isin(cols)]\n\nplt.figure(figsize=(14,25))\nsns.barplot(x=\"importance\",\n            y=\"Feature\",\n            data=best_features.sort_values(by=\"importance\",\n                                           ascending=False))\nplt.title('LightGBM Features (avg over folds)')\nplt.tight_layout()\nplt.savefig('lgbm_importances.png')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Section IV - Cascading classifiers (Divide et impera)\n\nThis approach will be slightly more complex than what we have tried so far. We'll be generating 4 different classifiers, divided in two layers:\n* Layer 1 - Three LGB classifiers, each based on either SVD, \"temporal\" or \"atemporal\" features only.\n* Layer 2 - One LGB classifier that receives the predictions of the previous layer and generates its own prediction, which will be our final guess.\n\nThe following illustration may help understanding it more clearly:\n\n![Illustration](https://i.imgur.com/O6MOofk.png)\n\nFirstly, we'll separate the training set into two separate sets: `temporal` and `atemporal` (the SVD-based prediction has already been made and is stored in the 'svd_prediction' column).","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"temporal_features=[c for c in df_train.columns if ((\"week\" in c) or (\"day\" in c) or (\"date\" in c) or (\"year\" in c))]\ndf_train_temporal = df_train[temporal_features]\ndf_test_temporal = df_test[temporal_features]\ndf_train_temporal['outliers']=df_train['outliers']\ndf_train_atemporal = df_train[[c for c in df_train.columns if c not in temporal_features]]\ndf_test_atemporal = df_test[[c for c in df_test.columns if c not in temporal_features]]\ndf_train_atemporal['outliers']=df_train['outliers']","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Then we fit each model separately.\n\n## 'Atemporal' Features","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"%%time\nresults=LGB_Train(df_train_atemporal,df_test_atemporal,target,31)\npredictions_test_atemporal=results['predictions_test']\npredictions_train_atemporal=results['predictions_train']","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## 'Temporal' Features","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"%%time\nresults=LGB_Train(df_train_temporal,df_test_temporal,target,31)\npredictions_test_temporal=results['predictions_test']\npredictions_train_temporal=results['predictions_train']","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Cascading the results into the final classifier\n\nWe then generate a final training set with 3 features: `svd_prediction`, `temporal_prediction` and `atemporal_prediction` and fit another classifier.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"df_train_impera = df_train[['svd_prediction','outliers']]\ndf_train_impera['temporal_prediction']=predictions_train_temporal\ndf_train_impera['atemporal_prediction']=predictions_train_atemporal\ndf_test_impera=df_test[['svd_prediction']]\ndf_test_impera['temporal_prediction']=predictions_test_temporal\ndf_test_impera['atemporal_prediction']=predictions_test_atemporal","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"%%time\nresults=LGB_Train(df_train_impera,df_test_impera,target,31)\nfeature_importance_df=results['feature_importances']\npredictions_impera=results['predictions_test']","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Now we list the feature importances for the final classifier.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"cols = (feature_importance_df[[\"Feature\", \"importance\"]]\n        .groupby(\"Feature\")\n        .mean()\n        .sort_values(by=\"importance\", ascending=False)[:1000].index)\n\nbest_features = feature_importance_df.loc[feature_importance_df.Feature.isin(cols)]\n\nplt.figure(figsize=(7,3))\nsns.barplot(x=\"importance\",\n            y=\"Feature\",\n            data=best_features.sort_values(by=\"importance\",\n                                           ascending=False))\nplt.title('LightGBM Features (avg over folds) for Divide-et-Impera method')\nplt.tight_layout()\nplt.savefig('lgbm_dei_importances.png')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Section V - Cascading Classifiers (Frankenstein's monster)\n\nIn this section, we'll simply combine everything we have done so far into a single classifier and evaluate its performance. This classifier will have as its input the set of all the features we obtained plus the predictions of the first layer of the \"Divide et Impera\" model and the projections on the two first SVD vectors.\n\nFirstly, we update our training dataset with the new columns we generated.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"df_train['temporal_prediction']=predictions_train_temporal\ndf_train['atemporal_prediction']=predictions_train_atemporal\ndf_test['temporal_prediction']=predictions_test_temporal\ndf_test['atemporal_prediction']=predictions_test_atemporal\ndf_train=df_train.join(df_svd_train[[c for c in df_svd_train.columns if \"svd\" in c]], lsuffix='_caller', rsuffix='_other')\ndf_test=df_test.join(df_svd_test[[c for c in df_svd_train.columns if \"svd\" in c]], lsuffix='_caller', rsuffix='_other')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"And then we fit our new model.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"%%time\nresults=LGB_Train(df_train,df_test,target,31)\nfeature_importance_df=results['feature_importances']\npredictions_frankie=results['predictions_test']","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"And finally, we plot \"Frankenstein's monster\"'s feature importances.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"cols = (feature_importance_df[[\"Feature\", \"importance\"]]\n        .groupby(\"Feature\")\n        .mean()\n        .sort_values(by=\"importance\", ascending=False)[:1000].index)\n\nbest_features = feature_importance_df.loc[feature_importance_df.Feature.isin(cols)]\n\nplt.figure(figsize=(14,25))\nsns.barplot(x=\"importance\",\n            y=\"Feature\",\n            data=best_features.sort_values(by=\"importance\",\n                                           ascending=False))\nplt.title('LightGBM Features (avg over folds) for Frankenstein\\'s method')\nplt.tight_layout()\nplt.savefig('lgbm_frankie_importances.png')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"We'll submit the classifiers' results for evaluating this notebook's performance.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"sub_df = pd.DataFrame({\"card_id\":df_test[\"card_id\"].values})\nsub_df[\"target\"] = predictions_frankie\nsub_df.to_csv(\"submission_frankie.csv\", index=False)\n\nsub_df = pd.DataFrame({\"card_id\":df_test[\"card_id\"].values})\nsub_df[\"target\"] = predictions_impera\nsub_df.to_csv(\"submission_impera.csv\", index=False)\n\nsub_df = pd.DataFrame({\"card_id\":df_test[\"card_id\"].values})\nsub_df[\"target\"] = predictions_brute\nsub_df.to_csv(\"submission_brutus.csv\", index=False)\n\nsub_df = pd.DataFrame({\"card_id\":df_test[\"card_id\"].values})\nsub_df[\"target\"] = predictions_test_atemporal\nsub_df.to_csv(\"submission_atemporal.csv\", index=False)\n\nsub_df = pd.DataFrame({\"card_id\":df_test[\"card_id\"].values})\nsub_df[\"target\"] = predictions_test_svd\nsub_df.to_csv(\"submission_naive_svd.csv\", index=False)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Conclusion\n\n## Light Gradient Boosting\n\nWe compared a few pre-processing methods and classifier structures/hierarchies.\n\n* By using a simple model that simply considers all the original features, we obtained our baseline results.\n* It appears that adding the SVD vectors to such model does not significantly improve its performance.\n* A pure SVD-feature-based classifier also had a rather impressive performance alone (by using only two features it achieved a RMSE close to the baseline), but still worse than the baseline.\n* Cascading the results of a classifier that works exclusively on those vectors into another with all features also had little impact on its results on the final results (in fact, it was ever-so-slightly worse, even though it performed better on the training and validation sets).\n* Separating the classifiers into three groups (SVD features, Temporal features and Atemporal features) and cascading their results into a final classifier seems to have significantly reduced the RMSE for both the training and validation sets. The error reduction was considerably higher than the previous methods, but the public score lowered (higher error), indicating that the model was probably overfit to our training data.\n* The Frankenstein model (a combination of all of the methods above) seems to have a remarkably low error on the training and validation sets. Its results on the test set were the worst of all methods, though.\n\n## Last Comments\n\nAll the methods tested and designed here successfully reduced (sometimes inredibly well) the error in our training and validation sets. Unfortunately, that reduction was not reflected in the test scores. In fact, the best models in our validation sets had the worst performance in the testing sets.\n\nWe believe this notebook lists an ordered set of increasingly overfitting-prone methods, and that might have some educational value.\n\nThis notebook might be useful as a tool for visualizing cascading classifiers, SVDs and for providing an example of a failed attempt at approaching this problem","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"# Resources\n\n0. [base kernel][base]\n1. [Why LGBM][lgbm]\n2. [LGBM x XGB][lgbm-xgb]\n3. [Parameter Tuning (TODO)][param-tuning]\n4. Álgebra Linear com Aplicações - 10ª Edição, Howard Anton e Chris Rorres\n5. [Eigenfaces (para intuição da SVD)][eigenfaces]\n\n\n[base]: https://www.kaggle.com/chauhuynh/my-first-kernel-3-699/comments\n[lgbm]: https://www.kaggle.com/c/elo-merchant-category-recommendation/discussion/73762\n[lgbm-xgb]: https://www.kaggle.com/c/elo-merchant-category-recommendation/discussion/80484\n[param-tuning]: https://lightgbm.readthedocs.io/en/latest/Parameters-Tuning.html\n[eigenfaces]: https://docs.opencv.org/2.4/modules/contrib/doc/facerec/facerec_tutorial.html#eigenfaces","execution_count":null}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}