{"cells":[{"metadata":{},"cell_type":"markdown","source":"# Introduction\n\nThis notebook is a continuation of [this one](https://www.kaggle.com/objectobject/submiss-o-grupo-boosting). In the previous notebook, we compared various methods for organizing Light Gradient Boosting classifiers in cascades. We'll be doing the same comparations with Random Forests now.\n\nIf you have already read the descriptions for each method, you may skip to section VI (Comparations and conclusions) directly.\n\nThis notebook consists of seven main sections:\n* 0) Setup\n    * Basic notebook setup.\n* 1) A refactoring/collection of other notebooks\n    * We tried to make the code more understandable and extracted many feature-engineered attributes and useful functions from this notebook.\n* 2) SVD-Based classification and experiments\n    * Here we do some more feature engineering and make a simple classifier based on a few of the extracted vectors. The result of this classifier will be later passed as an attribute to the final classifier.\n* 3) Cascading classifiers - Original features\n    * Here we combine the attributes extracted from the first notebook and the results from the SVD classifier.\n* 4) Cascading classifiers - SVD + Atemporal features + Temporal features\n    * Here we experiment with separating the classifiers even further (we called this approach \"divide et impera\"), fitting models on features directly related to time (called \"temporal\" from now on) and those not directly related to time (called \"atemporal\" from now on) separately and then combining them in a cascading classifier. This test is mostly exploratory, but we expect it to allow each classifier to pick up on subtler patterns within their domains.\n* 5) Cascading classifiers - Frankenstein's monster\n    * After not achieving much better results, we decided to make a final classifier that has access to every feature we generated so far (SVD vectors, SVD classifier result, Temporal classifier result, Atemporal classifier result and our original features). Once again, this test is mostly exploratory.\n* 6) Conclusions and comparations between the two models","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"# Section 0 - Setup\n\nHere we import the needed libraries and load our datasets.","execution_count":null},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"code","source":"# This Python 3 environment comes with many helpful analytics libraries installed\n# It is defined by the kaggle/python Docker image: https://github.com/kaggle/docker-python\n# For example, here's several helpful packages to load\n\nimport numpy as np # linear algebra\nimport pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\nimport datetime\nimport gc\nfrom pathlib import Path\n# Input data files are available in the read-only \"../input/\" directory\n# For example, running this (by clicking run or pressing Shift+Enter) will list all files under the input directory\n\nimport os\n# for dirname, _, filenames in os.walk('../'):\n#     for filename in filenames:\n#         print(os.path.join(dirname, filename))\n\n# You can write up to 5GB to the current directory (/kaggle/working/) that gets preserved as output when you create a version using \"Save & Run All\" \n# You can also write temporary files to /kaggle/temp/, but they won't be saved outside of the current session\n\n# copy-paste\nimport numpy as np\nimport pandas as pd\nimport datetime\nimport gc\nimport matplotlib.pyplot as plt\nimport seaborn as sns\nfrom sklearn.ensemble import RandomForestRegressor\nfrom sklearn.model_selection import StratifiedKFold\nfrom sklearn.metrics import mean_squared_error\nfrom sklearn.decomposition import TruncatedSVD\nimport warnings\nimport math\nwarnings.filterwarnings('ignore')\nnp.random.seed(4590)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"%%time\nPATH = Path('../input/elo-merchant-category-recommendation')\ndf_train = pd.read_csv(PATH/'train.csv', parse_dates=['first_active_month']);\ndf_test = pd.read_csv(PATH/'test.csv', parse_dates=['first_active_month']);","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"%%time\ndf_hist_trans = pd.read_csv(PATH/'historical_transactions.csv', parse_dates=['purchase_date']);\ndf_new_merch_trans = pd.read_csv(PATH/'new_merchant_transactions.csv', parse_dates=['purchase_date']);","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Section I - Refactoring\n\nWe analysed some existing notebooks to understand how people were approaching the problem so far.\n\nThis section consists in a collection of interesting and useful snippets taken from those notebooks.\n\nThese cells were extracted from the following sources:\n\n1) [my-first-kernel](https://www.kaggle.com/chauhuynh/my-first-kernel-3-699) by [chauhuynh](https://www.kaggle.com/chauhuynh).\n\n2) [elo-world](https://www.kaggle.com/fabiendaniel/elo-world) by [fabiendaniel](https://www.kaggle.com/fabiendaniel)\n\n## Memory-saving function\n\nThis function scans datasets for potential type changes that could save memory. This considerably reduces our dataset's memory consumption.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"# https://www.kaggle.com/fabiendaniel/elo-world\ndef reduce_mem_usage(df, verbose=True):\n    prefixes = ['int', 'float']\n    start_mem = df.memory_usage().sum() / 1024**2    \n    for col in df.columns:\n        col_type = str(df[col].dtype)\n        if not col_type.startswith('int') and not col_type.startswith('float'):\n#             print('col_type:', col_type, 'not compressed')\n            continue\n        c_min = df[col].min()\n        c_max = df[col].max()\n        if col_type.startswith('int'):\n            if c_min > np.iinfo(np.int8).min and c_max < np.iinfo(np.int8).max:\n                df[col] = df[col].astype(np.int8)\n            elif c_min > np.iinfo(np.int16).min and c_max < np.iinfo(np.int16).max:\n                df[col] = df[col].astype(np.int16)\n            elif c_min > np.iinfo(np.int32).min and c_max < np.iinfo(np.int32).max:\n                df[col] = df[col].astype(np.int32)\n            elif c_min > np.iinfo(np.int64).min and c_max < np.iinfo(np.int64).max:\n                df[col] = df[col].astype(np.int64)  \n        elif col_type.startswith('float'):\n            if c_min > np.finfo(np.float16).min and c_max < np.finfo(np.float16).max:\n                df[col] = df[col].astype(np.float16)\n            elif c_min > np.finfo(np.float32).min and c_max < np.finfo(np.float32).max:\n                df[col] = df[col].astype(np.float32)\n            else:\n                df[col] = df[col].astype(np.float64)    \n    if verbose:\n        end_mem = df.memory_usage().sum() / 1024**2\n        print('Mem. usage decreased to {:5.2f} Mb ({:.1f}% reduction)'.format(end_mem, 100 * (start_mem - end_mem) / start_mem))\n    return df\n\n# https://www.kaggle.com/chauhuynh/my-first-kernel-3-699/\n# Same logic\ndef fillna_mode(df, cols, pipeline=False):\n    for c in cols:\n        df[c].fillna(df[c].mode()[0], inplace=True)\n    return df if pipeline else None\n\n\ndef get_nan_col_names(df: pd.DataFrame, df_name='<not named>'):\n    total = df.shape[0]\n    missing_cols = []\n    for c in df.columns:\n        quo = (total - pd.notna(df[c]).sum())/total\n        if quo != 0:\n            missing_cols.append(c)\n    print(df_name, 'MISSING COLS:', missing_cols)\n    return missing_cols","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"reduce_mem_usage(df_train);\nreduce_mem_usage(df_test);\nreduce_mem_usage(df_hist_trans);\nreduce_mem_usage(df_new_merch_trans);","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Time and date-related feature functions\n\nThese functions will be used to generate extra features based on the date data we have.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"#https://www.kaggle.com/c/elo-merchant-category-recommendation/discussion/73244\ndef add_month_diff(df, pipeline=False):\n    df['month_diff'] = ((datetime.datetime.today() - df['purchase_date']).dt.days) // 30\n    df['month_diff'] += df['month_lag']\n    return df if pipeline else None\n\n\n# https://www.kaggle.com/chauhuynh/my-first-kernel-3-699\ndef pre_process_trans(df, date_col, date_formated=False, add_month=True):\n    fillna_mode(df, cols=get_nan_col_names(df))\n    df['authorized_flag'] = df['authorized_flag'].map({'Y':1, 'N':0}).astype(np.int8)\n    df['category_1'] = df['category_1'].map({'Y': 1, 'N': 0}).astype(np.int8)    \n    df['category_2'] = df['category_2'].astype(np.int8)\n\n    if not date_formated:\n        df[date_col] = pd.to_datetime(df[date_col])\n    df['year'] = df[date_col].dt.year\n    df['weekofyear'] = df[date_col].dt.weekofyear\n    df['month'] = df[date_col].dt.month\n    df['dayofweek'] = df[date_col].dt.dayofweek\n    df['weekend'] = (df[date_col].dt.weekday >= 5).astype(np.int8)\n    df['hour'] = df[date_col].dt.hour\n    if add_month:\n        add_month_diff(df)\n\n\n# https://www.kaggle.com/chauhuynh/my-first-kernel-3-699/\ndef get_new_columns(name,aggs):\n    return [name + '_' + k + '_' + agg for k in aggs.keys() for agg in aggs[k]]\n\n\n# Refactor of https://www.kaggle.com/chauhuynh/my-first-kernel-3-699/\ndef custom_group_by(df: pd.DataFrame, df_name: str, agg_by: dict):\n    for col in ['category_2','category_3']:\n        df[col+'_mean'] = df.groupby([col])['purchase_amount'].transform('mean')\n        aggs[col+'_mean'] = ['mean']    \n    new_columns = get_new_columns(df_name, agg_by)\n#     print(\"new_columns: \\n\", *new_columns, sep='\\t\\n')\n    df_group = df.groupby('card_id').agg(agg_by)\n    df_group.columns = new_columns\n    df_group.reset_index(drop=False, inplace=True)\n    df_group[df_name + '_purchase_date_diff'] = (\n        df_group[df_name + '_purchase_date_max']\n            - df_group[df_name + '_purchase_date_min']\n        ).dt.days\n    df_group[df_name + '_purchase_date_average'] = (\n        df_group[df_name + '_purchase_date_diff']\n            / df_group[df_name + '_card_id_size']\n        )\n    df_group[df_name + '_purchase_date_uptonow'] = (\n        datetime.datetime.today()\n        - df_group[df_name + '_purchase_date_max']\n        ).dt.days\n\n    return df_group","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Feature-engineered attributes\nThese attributes were extracted from the following source:\n1. [my-first-kernel][mfk]\n\n[mfk]: https://www.kaggle.com/chauhuynh/my-first-kernel-3-699/comments","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"aggs = {}\n\n# Count number of unique values at each column.\nfor col in ['month','hour','weekofyear','dayofweek','year','subsector_id','merchant_id','merchant_category_id']:\n    aggs[col] = ('nunique',)\n\n\naggs['purchase_amount'] = ('sum','max','min','mean','var')\naggs['installments'] = ('sum','max','min','mean','var')\n\n# purchase range\naggs['purchase_date'] = ('max','min')\n\naggs['month_lag'] = ('max','min','mean','var')\naggs['month_diff'] = ('mean',)\n\n# How many transactions:\n# - were on the weekend, and the percentage\n# - has category_1 as 1 (binary feature), and the percentage\n# - were authorized, and its percentage\n\naggs['weekend'] = ('sum', 'mean')\naggs['category_1'] = ('sum', 'mean')\naggs['authorized_flag'] = ('sum', 'mean')\n\n# How many purchases each card did?\naggs['card_id'] = ('size', )","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"_old_train = df_train.copy()\n_odl_test = df_test.copy()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"%%time\ngps = []\nfor name, df in [('hist', df_hist_trans), ('new_hist', df_new_merch_trans)]:    \n    pre_process_trans(df, date_col='purchase_date')\n    df_group = custom_group_by(df, name, aggs.copy())\n    gps.append(df_group)\n    df_train = df_train.merge(df_group, on='card_id', how='left')\n    df_test = df_test.merge(df_group, on='card_id', how='left')\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"prob = [\n    'hist_purchase_date_max',\n    'hist_purchase_date_min',\n    'new_hist_purchase_date_max',\n    'new_hist_purchase_date_min',\n]\n\ndef post_process_df(df):\n    global prob\n    \n    df['first_active_month'] = pd.to_datetime(df['first_active_month'])\n    df['fam_dayofweek'] = df['first_active_month'].dt.dayofweek\n    df['fam_weekofyear'] = df['first_active_month'].dt.weekofyear\n    df['fam_month'] = df['first_active_month'].dt.month\n    df['fam_elapsed_time'] = (datetime.datetime.today() - df['first_active_month']).dt.days\n    df['hist_first_buy'] = (df['hist_purchase_date_min'] - df['first_active_month']).dt.days\n    df['new_hist_first_buy'] = (df['new_hist_purchase_date_min'] - df['first_active_month']).dt.days\n\n    df[prob[:2]] = df[prob[:2]].astype(np.int64) * 1e-9\n\n    \n    df['transactions_total'] = df['new_hist_card_id_size']+df['hist_card_id_size']\n    df['purchase_amount_total'] = df['new_hist_purchase_amount_sum']+df['hist_purchase_amount_sum']\n\n    for f in ['feature_1','feature_2','feature_3']:\n        order_label = df_train.groupby([f])['outliers'].mean()\n        df_train[f] = df_train[f].map(order_label)\n        df_test[f] = df_test[f].map(order_label)\n    ","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"print(df_train.hist_purchase_date_max.notna().sum())\nprint(df_train.hist_purchase_date_min.notna().sum())\nprint(df_train.new_hist_purchase_date_max.notna().sum())\nprint(df_train.new_hist_purchase_date_min.notna().sum())","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"df_train['outliers'] = (df_train.target < -30).astype(np.int8)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"post_process_df(df_train)\npost_process_df(df_test)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"df_test.drop(['new_hist_purchase_date_max', 'new_hist_purchase_date_min'], axis=1,inplace=True)\ndf_train.drop(['new_hist_purchase_date_max', 'new_hist_purchase_date_min'], axis=1,inplace=True)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Meta-paremeters for the models\n\nThe following function fits a Random Forest model to a dataset `ds` with target `ds_ta` and test dataset `ds_te` with the num_leaves parameter set to `num_leaves`. The metaparameters were compiled from the notebooks and the two values for `num_leaves` uses in this notebook were found via trial-and-error.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"def Tree_Train(df_train,df_test,target,num_leaves):\n    folds = StratifiedKFold(n_splits=5, shuffle=True, random_state=4590)\n    oof = np.zeros(len(df_train))\n    predictions_train = np.zeros(len(df_train)) \n    predictions_test = np.zeros(len(df_test))\n    feature_importance_df = pd.DataFrame()\n    df_train_columns = [c for c in df_train.columns if c not in ['card_id', 'first_active_month', 'outliers', 'target']]\n    clf= RandomForestRegressor(max_leaf_nodes=num_leaves,random_state=3101,criterion='mse',bootstrap=True,oob_score=True)\n    print(\"Fitting Random Forests\")\n    clf.fit(df_train[df_train_columns].fillna(0),target)\n    print(\"Error (rmse oob_score): \"+str(math.sqrt(clf.oob_score_)))\n    feature_importance_df = pd.DataFrame(data=zip(df_train_columns,clf.feature_importances_),columns=['Feature','importance'])\n    predictions_test = clf.predict(df_test[df_train_columns].fillna(0))\n    predictions_train = clf.predict(df_train[df_train_columns].fillna(0))\n    rmse=0\n    for i,item in enumerate(target):\n        rmse+=(item-predictions_train[i])**2\n    print(\"Error (rmse over training set): \"+str(math.sqrt(rmse/len(target))))\n\n    return {'predictions_test':predictions_test,'predictions_train':predictions_train,'feature_importances':feature_importance_df}\n    ","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Section II - SVD\n\nIn this section we'll be using Singular Value Decomposition for both assisting the classifiers and trying to better understand the relationships between some of the numerical attributes.\n\n## Calculation and analysis\n\nFirstly, we compute the decomposition and try analysing some of its results.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"#A dataframe containing only numerical values\ndf_num=df_train[[col for col in df_train.columns if not col in ['target','outliers']]].select_dtypes(include='number')\n\n#NaN/Missing values will be replaced with zeroes\ndf_num.fillna(0,inplace=True)\n\ndecomp = TruncatedSVD(n_components=len(df_num.columns.values)-1,random_state=1)\ndecomp.fit(df_num)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"We intend to select the most descriptive vectors, that is, the ones across which the data points spreads the most.\n\nLet's plot their eigenvalues.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"plot=sns.lineplot(x=range(10),y=decomp.singular_values_[:10])\nplot.set_title(\"Eigenvalues in descending order\")\nplot.set(ylabel=\"Eigenvalue\",xlabel=\"Vector\")\nplt.show()\nplot=sns.lineplot(x=range(3,len(decomp.singular_values_)),y=decomp.singular_values_[3:])\nplot.set_title(\"Eigenvalues [from 3rd on] in descending order\")\nplot.set(ylabel=\"Eigenvalue\",xlabel=\"Vector\")","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"The first few vectors clearly represent much of the information contained here. This is a good sign.\n\nThe following plot illustrates the squared weights of each numerical feature for projecting onto the first 10 component vectors (for estimating the importance of each attribute on the projection).","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"sns.heatmap([[x**2 for x in component] for component in decomp.components_[:10]])","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Some attributes consistently have more weight than others. Let's figure out which exactly are those.\n\nThe following graphs show the squared weight of each of the 3 most important attributes for the first 3 vectors.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"for component in decomp.components_[:3]:\n    weights=sorted(zip([x**2 for x in component],df_num.columns.values),reverse=True)[:3]\n    sns.barplot(data=pd.DataFrame(weights,columns=['Squared Weight','Feature Name']),y='Feature Name',x='Squared Weight')\n    plt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"It appears that the most descriptive features are consistently the **purchase date** and **expenditure**. This might make sense intuitively, but let's try not to let it bias our interpretation.","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"Let's figure out how many of the resulting vectors we should pass to the classifier.\n\nThis analysis is mostly just exploratory. We don't have much basis for actually justifying these specific choices and experiments beforehand (beyond quasi-scientifical intuition).\n\nWe arbitrarily chose to 'keep 80% of the information' (keep the vectors whose eigenvalues sum up to 80% of the total sum). This is a standard limit used when applying this kind of decomposition in images and its  effectiveness is visible (pun unintended) on those cases. We hope this concept translates well to this specific situation.\n\nFor achieving that, we'll be keeping the following number of vectors:","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"for num_vectors in range(1,len(decomp.singular_values_)):\n    acc_ratio=sum(decomp.singular_values_[:num_vectors])/sum(decomp.singular_values_)\n    if (sum(decomp.singular_values_[:num_vectors])/sum(decomp.singular_values_))>=0.80:\n            break;\nprint(str(num_vectors)+\" (\"+str(round(acc_ratio*10000)/100)+\"%)\")","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"> Sidenote: the first two vectors account for 99.92% of the accumulated eigenvalues. This looks promising.\n\nFor the record, here's a plot of the accumulated squared weights of each attribute for the vectors that will be used.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"sq_eigenval=dict(zip(df_num.columns.values,[0 for i in range(len(df_num.columns.values))]))\nfor component in decomp.components_[:num_vectors]:\n    weights=zip([x**2 for x in component],df_num.columns.values)\n    for line in weights:\n        sq_eigenval[line[1]]+=line[0] if not np.isnan(line[0]) else 0\n\n#Normalization\nsq_eigenval=pd.DataFrame(sq_eigenval.items(),columns=['Feature','Sq. Weight']).sort_values(by='Sq. Weight')\nsq_eigenval['Norm. Sq. W.']=(sq_eigenval['Sq. Weight']-sq_eigenval['Sq. Weight'].min())/(sq_eigenval['Sq. Weight'].max()-sq_eigenval['Sq. Weight'].min())\n\nplt.figure(figsize=(14,25))\nsns.barplot(\n            y='Feature',\n            x='Norm. Sq. W.',\n            data=sq_eigenval\n)\nplt.title('Normalized Sq. Weights of Features (SVD)')\nplt.tight_layout()\nplt.savefig('svd_importances.png')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"We should now generate our `training` and `testing` sets for the SVD classifier.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"df_num=df_train[[col for col in df_train.columns if not col in ['target','outliers']]].select_dtypes(include='number')\ndf_num.fillna(0,inplace=True)\ndf_svd_train=pd.DataFrame([x[:num_vectors]for x in decomp.transform(df_num)],columns=['svd_'+str(i) for i in range(num_vectors)])\n#df_train_svd=df_train.join(df_svd,lsuffix='_caller', rsuffix='_other')\ndf_svd_train['target']=df_train['target']\ndf_svd_train['outliers']=df_train['outliers']\ndf_num=pd.DataFrame(data=df_test,columns=df_num.columns.values)\ndf_num.fillna(0,inplace=True)\ndf_svd_test=pd.DataFrame([x[:num_vectors]for x in decomp.transform(df_num)],columns=['svd_'+str(i) for i in range(num_vectors)])\n#df_test_svd=df_test.join(df_svd,lsuffix='_caller', rsuffix='_other')\nprint(\"Datasets generated.\")","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Before trying to do any classification, let's project our data using the first two vectors and see if any simple visible pattern arises.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"sns.scatterplot(data=df_svd_train,x='svd_1',y='svd_0',hue='target')\nplt.show()\nsns.scatterplot(data=df_svd_train.query('svd_1>0'),x='svd_1',y='svd_0',hue='target')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"This looks nothing like we'd expect from an SVD. The vectors still have huge eigenvalues, though, so we must be onto something.\n\nLet's just give that to the classifiers.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"target=df_svd_train['target']\ndel df_svd_train['target']","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"%%time\nresults=Tree_Train(df_svd_train,df_svd_test,target,5)\npredictions_train_svd=results['predictions_train']\npredictions_test_svd=results['predictions_test']","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Ok. Now that we've finished classifying, we may use the predictions as an attribute for our next classifier.","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"# Section III - Cascading Classifiers (Original features)\n\nWe'll first integrate the results of our classifiers into our training and testing sets.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"df_train['svd_prediction']=predictions_train_svd\ndf_test['svd_prediction']=predictions_test_svd\n\ndf_train_columns = [c for c in df_train.columns if c not in ['card_id', 'first_active_month', 'outliers', 'target']]\ntarget = df_train['target']\ndel df_train['target']","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Then we fit another model to our new training set.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"%%time\nresults=Tree_Train(df_train,df_test,target,31)\nfeature_importance_df=results['feature_importances']\npredictions_brute=results['predictions_test']","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"After fitting, we can compare the importance of each feature with a bar plot.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"cols = (feature_importance_df[[\"Feature\", \"importance\"]]\n        .groupby(\"Feature\")\n        .mean()\n        .sort_values(by=\"importance\", ascending=False)[:1000].index)\n\nbest_features = feature_importance_df.loc[feature_importance_df.Feature.isin(cols)]\n\nplt.figure(figsize=(14,25))\nsns.barplot(x=\"importance\",\n            y=\"Feature\",\n            data=best_features.sort_values(by=\"importance\",\n                                           ascending=False))\nplt.title('Random Tree Feature Importances')\nplt.tight_layout()\nplt.savefig('tree_importances.png')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Section IV - Cascading classifiers (Divide et impera)\n\nThis approach will be slightly more complex than what we have tried so far. We'll be generating 4 different classifiers, divided in two layers:\n* Layer 1 - Three Random Forest classifiers, each based on either SVD, \"temporal\" or \"atemporal\" features only.\n* Layer 2 - One Random Forest classifier that receives the predictions of the previous layer and generates its own prediction, which will be our final guess.\n\nFirstly, we'll separate the training set into two separate sets: `temporal` and `atemporal` (the SVD-based prediction has already been made and is stored in the 'svd_prediction' column).","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"temporal_features=[c for c in df_train.columns if ((\"week\" in c) or (\"day\" in c) or (\"date\" in c) or (\"year\" in c))]\ndf_train_temporal = df_train[temporal_features]\ndf_test_temporal = df_test[temporal_features]\ndf_train_temporal['outliers']=df_train['outliers']\ndf_train_atemporal = df_train[[c for c in df_train.columns if c not in temporal_features]]\ndf_test_atemporal = df_test[[c for c in df_test.columns if c not in temporal_features]]\ndf_train_atemporal['outliers']=df_train['outliers']","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Then we fit each model separately.\n\n## 'Atemporal' Features","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"%%time\nresults=Tree_Train(df_train_atemporal,df_test_atemporal,target,31)\npredictions_test_atemporal=results['predictions_test']\npredictions_train_atemporal=results['predictions_train']","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## 'Temporal' Features","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"%%time\nresults=Tree_Train(df_train_temporal,df_test_temporal,target,31)\npredictions_test_temporal=results['predictions_test']\npredictions_train_temporal=results['predictions_train']","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Cascading the results into the final classifier\n\nWe then generate a final training set with 3 features: `svd_prediction`, `temporal_prediction` and `atemporal_prediction` and fit another classifier.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"df_train_impera = df_train[['svd_prediction','outliers']]\ndf_train_impera['temporal_prediction']=predictions_train_temporal\ndf_train_impera['atemporal_prediction']=predictions_train_atemporal\ndf_test_impera=df_test[['svd_prediction']]\ndf_test_impera['temporal_prediction']=predictions_test_temporal\ndf_test_impera['atemporal_prediction']=predictions_test_atemporal","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"%%time\nresults=Tree_Train(df_train_impera,df_test_impera,target,31)\nfeature_importance_df=results['feature_importances']\npredictions_impera=results['predictions_test']","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Now we list the feature importances for the final classifier.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"cols = (feature_importance_df[[\"Feature\", \"importance\"]]\n        .groupby(\"Feature\")\n        .mean()\n        .sort_values(by=\"importance\", ascending=False)[:1000].index)\n\nbest_features = feature_importance_df.loc[feature_importance_df.Feature.isin(cols)]\n\nplt.figure(figsize=(7,3))\nsns.barplot(x=\"importance\",\n            y=\"Feature\",\n            data=best_features.sort_values(by=\"importance\",\n                                           ascending=False))\nplt.title('Random Tree Feature Importances for Divide-et-Impera method')\nplt.tight_layout()\nplt.savefig('tree_dei_importances.png')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Section V - Cascading Classifiers (Frankenstein's monster)\n\nIn this section, we'll simply combine everything we have done so far into a single classifier and evaluate its performance. This classifier will have as its input the set of all the features we obtained plus the predictions of the first layer of the \"Divide et Impera\" model and the projections on the two first SVD vectors.\n\nFirstly, we update our training dataset with the new columns we generated.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"df_train['temporal_prediction']=predictions_train_temporal\ndf_train['atemporal_prediction']=predictions_train_atemporal\ndf_test['temporal_prediction']=predictions_test_temporal\ndf_test['atemporal_prediction']=predictions_test_atemporal\ndf_train=df_train.join(df_svd_train[[c for c in df_svd_train.columns if \"svd\" in c]], lsuffix='_caller', rsuffix='_other')\ndf_test=df_test.join(df_svd_test[[c for c in df_svd_train.columns if \"svd\" in c]], lsuffix='_caller', rsuffix='_other')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"And then we fit our new model.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"%%time\nresults=Tree_Train(df_train,df_test,target,31)\nfeature_importance_df=results['feature_importances']\npredictions_frankie=results['predictions_test']","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"And finally, we plot \"Frankenstein's monster\"'s feature importances.","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"We'll submit the classifiers' results for evaluating this notebook's performance.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"cols = (feature_importance_df[[\"Feature\", \"importance\"]]\n        .groupby(\"Feature\")\n        .mean()\n        .sort_values(by=\"importance\", ascending=False)[:1000].index)\n\nbest_features = feature_importance_df.loc[feature_importance_df.Feature.isin(cols)]\n\nplt.figure(figsize=(14,25))\nsns.barplot(x=\"importance\",\n            y=\"Feature\",\n            data=best_features.sort_values(by=\"importance\",\n                                           ascending=False))\nplt.title('Random Tree Feature Importances for Frankenstein\\'s method')\nplt.tight_layout()\nplt.savefig('tree_frankie_importances.png')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"sub_df = pd.DataFrame({\"card_id\":df_test[\"card_id\"].values})\nsub_df[\"target\"] = predictions_frankie\nsub_df.to_csv(\"submission_frankie.csv\", index=False)\n\nsub_df = pd.DataFrame({\"card_id\":df_test[\"card_id\"].values})\nsub_df[\"target\"] = predictions_impera\nsub_df.to_csv(\"submission_impera.csv\", index=False)\n\nsub_df = pd.DataFrame({\"card_id\":df_test[\"card_id\"].values})\nsub_df[\"target\"] = predictions_brute\nsub_df.to_csv(\"submission_brutus.csv\", index=False)\n\nsub_df = pd.DataFrame({\"card_id\":df_test[\"card_id\"].values})\nsub_df[\"target\"] = predictions_test_atemporal\nsub_df.to_csv(\"submission_atemporal.csv\", index=False)\n\nsub_df = pd.DataFrame({\"card_id\":df_test[\"card_id\"].values})\nsub_df[\"target\"] = predictions_test_svd\nsub_df.to_csv(\"submission_naive_svd.csv\", index=False)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Conclusions and Comparations between the models\n\nBoth models followed the same patterns: our increasingly complex structures mostly just made the models get increasingly overfit. For some reason, this relationship seems to have a linear impact on the RMSE measurements (see graphs below).\n\n## Error plots\n\nBelow are some plots illustrating the performance of each structure for each model (they coordinates are `validation error` - the RMSE we get from testing with part of the data we used for training - and `test error` - the RMSE we get from our public results for the kaggle competition -).\n\nThese data were taken from the [boosting notebook](https://www.kaggle.com/objectobject/submiss-o-grupo-boosting)'s 7th version and this notebook's 1st.\n\n### Light Gradient Boosting","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"lgbm=pd.DataFrame(data=[\n[\"Only SVD projections\",3.81,3.89],\n[\"SVD prediction + Dataset Features Cascade\",3.63,3.70],\n[\"Divide-et-Impera Cascade\",3.33,3.80],\n[\"Non-time-related Features\",3.64,3.71],\n[\"Frankie Cascade\",2.87,3.87]],columns=['Feature Set','Validation Error','Test Error'])\nplt.title('Error plot for LGB-based models', color='black')\nplot=sns.scatterplot(data=lgbm,x=\"Validation Error\",y=\"Test Error\",hue='Feature Set')\nplt.legend(loc='center left', bbox_to_anchor=(1.0, 0.5))\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Random Forests","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"forestsm=pd.DataFrame(data=[\n[\"Only SVD projections\",0.008497750292186312,3.9],\n[\"SVD prediction + Dataset Features Cascade\",0.00811753831695943,3.76],\n[\"Divide-et-Impera Cascade\",0.007901780981265769,3.79],\n[\"Non-time-related Features\",0.008128093345136984,3.76],\n[\"Frankie Cascade\",0.007733020898929636,3.79]],columns=['Feature Set','Validation Error','Test Error'])\n#This correction had to be made because we miscalculated the RMSE in the original version.\nforestsm['Validation Error']=np.sqrt(((forestsm['Validation Error']*123623)**2)/123623)\nplt.title('Error plot for Random Forest-based models', color='black')\nplot=sns.scatterplot(data=forestsm,x=\"Validation Error\",y=\"Test Error\",hue='Feature Set')\nplt.legend(loc='center left', bbox_to_anchor=(1.0, 0.5))\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Trendlines for both sets of main models (all excluding pure SVD-based classifiers)","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"forestsm['Type']='Random Forests'\nlgbm['Type']='LGB'\nplot=sns.lmplot(data=pd.concat([forestsm,lgbm]).query(expr='`Feature Set`!=\"Only SVD projections\"'),x=\"Validation Error\",y=\"Test Error\",hue='Type')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Comparing the models\n\nWe'll be dividing the analysis in two groups:\n1. Main models (all the structures except the pure SVD-based one - the outlier in all plots -)\n2. SVD models\n\n### Main Models\n\nWhen we exclude the pure SVD-based models, the relationship between  the `Validation Error` and `Test Error` seems to be linear, that is, the more accurate our models were in the validation stage, the more - in a predictable, almost linear way - inaccurate they were in the testing stage. Interestingly, the trend lines between both approaches seem to be nearly parallel.\n\nThe best Boosting-based models achieved better results than the Random Forests, but the four main Random forest models were better than the Boosting-based Frankenstein and Divide-et-Impera cascades.\n\nWe mainly attribute this difference to the fact that Boosting algorithms are generally better adapted for regression problems and have better configurable meta-paremeters for avoiding overfitting. The Forest-Based models, with less effective overfitting-avoiding mechanisms (in this situation), all had a better score in their training/validation sets, but their overall performance in the test sets was worse.\n \nThe Boosting models took 2856 seconds to run (47 minutes and 36 seconds), while the Random Forest models took 1121 seconds (18 minutes and 48 seconds). This is a significant difference, and its magnitude was not reflected in the score difference. Some of the Forest-Based (faster) models actually had a better performance than their slower Boosting-Based counterparts, as we showed above.\n\n### SVD models\n\nThese models were actually only supposed to exist as a part of the cascading models, but they achieved a low enough RMSE alone to be included in the final graphs. The difference in overfitting between the Boosting and Forest-based models was also reflected in their results.","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"# Resources\n\n0. [base kernel][base]\n1. [Random Forests](https://scikit-learn.org/stable/modules/generated/sklearn.ensemble.RandomForestRegressor.html)\n2. Álgebra Linear com Aplicações - 10ª Edição, Howard Anton e Chris Rorres\n3. [Eigenfaces (para intuição da SVD)][eigenfaces]\n\n\n[base]: https://www.kaggle.com/chauhuynh/my-first-kernel-3-699/comments\n[lgbm]: https://www.kaggle.com/c/elo-merchant-category-recommendation/discussion/73762\n[lgbm-xgb]: https://www.kaggle.com/c/elo-merchant-category-recommendation/discussion/80484\n[param-tuning]: https://lightgbm.readthedocs.io/en/latest/Parameters-Tuning.html\n[eigenfaces]: https://docs.opencv.org/2.4/modules/contrib/doc/facerec/facerec_tutorial.html#eigenfaces","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}