{"cells":[{"metadata":{"_uuid":"02732593cfb0bb1cf40ac4669d2fac49cef1b8e8"},"cell_type":"markdown","source":"## Goal of the Notebook\nAfter looking at the various kernels by fellow Kagglers I came to realize that incresign complexity was only marginally improving RMSE (at least from available kernels). So I thought of running a PCA to perform a dimensionality check. I soon came to realize that most of the available data is just noise: the three first components of PCA account for 99.99% of the variance as we will see below. I also used these three components as added features and re-run my model but to no avail. I hope you find this analysis useful and you come to new ideas of why this might be happening. "},{"metadata":{"trusted":true,"_uuid":"88480f11f65822187ad2f781d57734f1d5dcea5f"},"cell_type":"code","source":"import pandas as pd\nimport numpy as np\nimport datetime\nimport time\nfrom sklearn.ensemble import RandomForestClassifier\nfrom sklearn.model_selection import KFold, train_test_split\nfrom sklearn.metrics import mean_squared_error\nfrom sklearn.decomposition import PCA\nimport lightgbm as lgb\nimport matplotlib.pyplot as plt\n%matplotlib inline","execution_count":null,"outputs":[]},{"metadata":{"_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","trusted":true},"cell_type":"code","source":"#Import data\nprint('Importing data...')\ndf_train = pd.read_csv('../input/train.csv')\ndf_history = pd.read_csv(\"../input/historical_transactions.csv\")","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"4665e7866e04ec45b2ce77796698a2aa4d2baec8"},"cell_type":"code","source":"#Preprocess transactions\nprint('Preprocessing historical transactions...')\ndf_history['authorized_flag'] = df_history['authorized_flag'].map({'Y':1, 'N':0})\ndf_history['category_1'] = df_history['category_1'].map({'Y':1, 'N':0})\ndf_history['purchase_date'] = pd.to_datetime(df_history['purchase_date'])\nlast_date_hist = datetime.datetime(2018, 2, 28)\ndf_history['time_since_purchase_date'] = ((last_date_hist - df_history['purchase_date']).dt.days)\ndf_history.loc[:, 'purchase_date'] = pd.DatetimeIndex(df_history['purchase_date']).\\\n                                      astype(np.int64) * 1e-9\n\ndf_history['installments'] = df_history['installments'].replace(999,-1)\ncols_with_nulls = ['city_id', 'state_id', 'subsector_id', 'installments']\nfor col in cols_with_nulls:\n    df_history[col] = df_history[col].replace(-1, np.nan)\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"24b7e3440aba1514357edb381ccefb653ce26915"},"cell_type":"code","source":"#Perform aggregations by card ID\nprint('Aggregating historical transactions...')\n\nagg_func = {\n        'authorized_flag': ['mean'],\n        'city_id': ['nunique'], \n        'category_1': ['sum', 'mean'],\n        'installments': ['median', 'max'],\n        'category_3': ['nunique'],\n        'merchant_category_id': ['nunique'], \n        'merchant_id': ['nunique'],\n        'month_lag': ['min', 'max'],\n        'purchase_amount': ['sum', 'median', 'max', 'min'],\n        'purchase_date': ['min', 'max'],\n        'time_since_purchase_date': ['min', 'max', 'mean'],\n        'category_2': ['nunique'], \n        'state_id': ['nunique'], \n        'subsector_id': ['nunique']\n        }\n\n\nagg_history = df_history.groupby(['card_id']).agg(agg_func)\nagg_history.columns = ['hist_' + '_'.join(col).strip() for col in agg_history.columns.values]\nagg_history.reset_index(inplace=True)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"b8c2f266b220c45be8b4cbcf552ac67d14cf9972"},"cell_type":"code","source":"#Merge with train and test\nprint('Merging all data...')\ndf_train_all = pd.merge(df_train, agg_history, on='card_id', how='left')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"89a55ecfc94d774868acfa53402ad1622124cc6f"},"cell_type":"code","source":"#Split initial train set into new train and test sets\ny_label_regr = df_train_all['target']\n\ndf_train_all = df_train_all.drop(['target',\n                                    'first_active_month', \n                                    'card_id'\n                                    ],\n                                     axis = 1)\n\ntrain_x, test_x, train_y, test_y = train_test_split(df_train_all, y_label_regr, test_size=0.7, random_state=42)\n\ntrain_x.reset_index(inplace=True, drop = True)\ntest_x.reset_index(inplace=True, drop = True)\ntrain_y.reset_index(inplace=True, drop = True)\ntest_y.reset_index(inplace=True, drop = True)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"98ec17847918bbc7c6b0cc008ef035a9251955dc"},"cell_type":"markdown","source":"First lets train a model to check its performance using the original data."},{"metadata":{"trusted":true,"_uuid":"a6b11d75d66691a4ab012cff112fbd3eb8978cd1"},"cell_type":"code","source":"#Train LightGBM model on original data\nparam = {'num_leaves': 111,\n         'min_data_in_leaf': 149,\n         'objective':'regression',\n         'max_depth': 9,\n         'learning_rate': 0.005,\n         \"boosting\": \"gbdt\",\n         \"feature_fraction\": 0.7522,\n         \"bagging_freq\": 1,\n         \"bagging_fraction\": 0.7083 ,\n         \"bagging_seed\": 11,\n         \"metric\": 'rmse',\n         \"lambda_l1\": 0.3134,\n         \"random_state\": 133,\n         \"verbosity\": -1}\n\nfeatures = train_x.columns\nfolds = KFold(n_splits=5, shuffle=True, random_state=15)\noof_normal = np.zeros(len(df_train_all))\npredictions_normal = np.zeros(len(test_x))\nstart = time.time()\nfeature_importance_df = pd.DataFrame()\n\nfor fold_, (trn_idx, val_idx) in enumerate(folds.split(train_x.values, train_y)):\n    print(\"fold n°{}\".format(fold_))\n    trn_data = lgb.Dataset(train_x.iloc[trn_idx][features],\n                           label=train_y[trn_idx],\n                           #categorical_feature=cat_feats\n                           )\n    val_data = lgb.Dataset(train_x.iloc[val_idx][features],\n                           label=train_y[val_idx],\n                           #categorical_feature=cat_feats\n                           )\n\n    num_round = 10000\n    clf = lgb.train(param,\n                    trn_data,\n                    num_round,\n                    valid_sets=[trn_data, val_data],\n                    verbose_eval=100,\n                    early_stopping_rounds=200)\n\n    oof_normal[val_idx] = clf.predict(train_x.iloc[val_idx][features], num_iteration=clf.best_iteration)\n\n    fold_importance_df = pd.DataFrame()\n    fold_importance_df[\"feature\"] = features\n    fold_importance_df[\"importance\"] = clf.feature_importance()\n    fold_importance_df[\"fold\"] = fold_ + 1\n    feature_importance_df = pd.concat([feature_importance_df, fold_importance_df], axis=0)\n\n    predictions_normal += clf.predict(test_x[features], num_iteration=clf.best_iteration) / folds.n_splits\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"22ef87902c3f28c19d40e430e66bde3228ba8db5"},"cell_type":"code","source":"#Perform Principal Component Analysis  \npca = PCA()\npca.fit(train_x)\npca.transform(train_x)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"6eeb0245c521ca1b5571ed2c1f64ca97fc35dac8"},"cell_type":"code","source":"#Visualize Scree plot\nfig,ax=plt.subplots(1,2,figsize=(12,6))\npc_total=np.arange(1,pca.n_components_+1)\nax[0].plot(pc_total,np.cumsum(pca.explained_variance_ratio_))\nax[0].set_xticks(pc_total)\nax[0].set_xlabel('Principal Components')\nax[0].set_ylabel('Cumulative explained variance')\n###############################################################\nax[1].plot(pc_total,pca.explained_variance_)\nax[1].set_xticks(pc_total)\nax[1].set_xlabel('Principal Components')\nax[1].set_ylabel('Explained Variance Ratio')\nfig.suptitle('SCREE PLOT')\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"a2ff27c9bf3b090be2d05ce65c782840f35e27b5"},"cell_type":"code","source":"var_exp_3 = sum(pca.explained_variance_ratio_[:3])\nprint('Variance explained by the first 3 PCA components:', var_exp_3)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"f91eb0ec3733fe6c8425ff2be5fb987317f23cf6"},"cell_type":"code","source":"#Visualize Biplot\n#Unfortunately this throws a size error here but not in my local notebook. I am just leaving the \n#code here commented out so you can use it on your own\n\n# y=pca.fit_transform(train_x)\n\n# plt.figure(figsize = (12,10))\n\n# xvector = pca.components_[0] \n# yvector = pca.components_[1]\n\n# xs = y[:,0]\n# ys = y[:,1]\n\n# ## visualize projections\n# for i in range(len(xvector)):\n#     plt.arrow(0, 0, xvector[i]*max(xs), yvector[i]*max(ys),\n#               color='darkred', width=0.2, head_width=0.5)\n#     plt.text(xvector[i]*max(xs)*1.2, yvector[i]*max(ys)*1.2,\n#              list(train_x.columns)[i], color='darkred', fontsize=6)\n\n# plt.scatter(xs,ys,c='b',alpha=0.02)\n# plt.axhline(0, color='black',alpha=0.8)\n# plt.axvline(0, color='black',alpha=0.8)\n# plt.xlim(-300,300)\n# plt.ylim(-300,300)\n# plt.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"90f0ab8233f5a2339baac37f8a0277a1dd775778"},"cell_type":"code","source":"#Create train and test sets from the first 3 PCA components\npca_train_x= pca.transform(train_x)\npca_train_x = pca_train_x[:,:3]\npca_train_x = pd.DataFrame(pca_train_x, columns=['comp1', 'comp2', 'comp3'])\n\npca_test_x= pca.transform(test_x)\npca_test_x = pca_test_x[:,:3]\npca_test_x = pd.DataFrame(pca_test_x, columns=['comp1', 'comp2', 'comp3'])","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"9d793f53948b50f8131ee5174d3144d2ff3ac9cc"},"cell_type":"markdown","source":"Now let's train a LightGBM model using only these 3 components."},{"metadata":{"trusted":true,"_uuid":"f7cd63d69d67463b0f0dc33ca632b40d3befb60a"},"cell_type":"code","source":"#Train LightGBM model on these 3 PCA components only\nparam = {#'num_leaves': 21,\n         #'min_data_in_leaf': 49,\n         'objective':'regression',\n         'max_depth': 8,\n         'learning_rate': 0.001,\n         #\"boosting\": \"gbdt\",\n         #\"feature_fraction\": 0.5,\n         #\"bagging_freq\": 1,\n         #\"bagging_fraction\": 0.5 ,\n         #\"bagging_seed\": 11,\n         \"metric\": 'rmse',\n         #\"lambda_l1\": 0.3134,\n         \"random_state\": 133,\n         #\"is_unbalance\": True,\n         \"verbosity\": -1}\n\nfeatures = pca_train_x.columns\nfolds = KFold(n_splits=5, shuffle=True, random_state=15)\noof_pca = np.zeros(len(pca_train_x))\npredictions_pca = np.zeros(len(pca_test_x))\nstart = time.time()\nfeature_importance_df = pd.DataFrame()\n\nfor fold_, (trn_idx, val_idx) in enumerate(folds.split(pca_train_x.values, train_y)):\n    print(\"fold n°{}\".format(fold_))\n    trn_data = lgb.Dataset(pca_train_x.iloc[trn_idx][features],\n                           label=train_y[trn_idx],\n                           #categorical_feature=cat_feats\n                           )\n    val_data = lgb.Dataset(pca_train_x.iloc[val_idx][features],\n                           label=train_y[val_idx],\n                           #categorical_feature=cat_feats\n                           )\n\n    num_round = 10000\n    clf = lgb.train(param,\n                    trn_data,\n                    num_round,\n                    valid_sets=[trn_data, val_data],\n                    verbose_eval=100,\n                    early_stopping_rounds=200)\n\n    oof_pca[val_idx] = clf.predict(pca_train_x.iloc[val_idx][features], num_iteration=clf.best_iteration)\n\n    fold_importance_df = pd.DataFrame()\n    fold_importance_df[\"feature\"] = features\n    fold_importance_df[\"importance\"] = clf.feature_importance()\n    fold_importance_df[\"fold\"] = fold_ + 1\n    feature_importance_df = pd.concat([feature_importance_df, fold_importance_df], axis=0)\n\n    predictions_pca += clf.predict(pca_test_x[features], num_iteration=clf.best_iteration) / folds.n_splits\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"d195c434237b6eef1fe5993086fe546f13f0e83a"},"cell_type":"code","source":"#Compare performance\nprint(\"RMSE test normal: {:<8.5f}\".format(mean_squared_error(predictions_normal, test_y) ** 0.5))\nprint(\"RMSE test PCA: {:<8.5f}\".format(mean_squared_error(predictions_pca, test_y) ** 0.5))","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"834df9a2de9f59e311a9387e38caaca5bd2b6911"},"cell_type":"markdown","source":"We see that the loss is minimal. Maybe not minimal for a Kaggle competition but a) in real life situations we would indeed opt for the three new features vs original 29 and b) the 3-feature model is not fine-tuned thus the difference may well be even smaller than that."},{"metadata":{"_uuid":"f57c9b74736019ab46d633061ee092d4b3b7c645"},"cell_type":"markdown","source":"Let's now re-run our initial model after adding the three PCA components as features and see what we get."},{"metadata":{"trusted":true,"_uuid":"4b1392abd9bdabcc9230aff9a60b0a0c36e1b201"},"cell_type":"code","source":"#Add these 3 PCA components as features on original data and re-run model\ntrain_x_2 = pd.concat([train_x, pca_train_x], axis = 1)\ntest_x_2 = pd.concat([test_x, pca_test_x], axis = 1)\n\ndel train_x\ndel test_x\ndel pca_train_x\ndel pca_test_x\n\n#Train LightGBM model\nparam = {'num_leaves': 111,\n         'min_data_in_leaf': 149,\n         'objective':'regression',\n         'max_depth': 9,\n         'learning_rate': 0.005,\n         \"boosting\": \"gbdt\",\n         \"feature_fraction\": 0.7522,\n         \"bagging_freq\": 1,\n         \"bagging_fraction\": 0.7083 ,\n         \"bagging_seed\": 11,\n         \"metric\": 'rmse',\n         \"lambda_l1\": 0.3134,\n         \"random_state\": 133,\n         \"verbosity\": -1}\n\nfeatures = train_x_2.columns\nfolds = KFold(n_splits=5, shuffle=True, random_state=15)\noof_pca = np.zeros(len(train_x_2))\npredictions_all = np.zeros(len(test_x_2))\nstart = time.time()\nfeature_importance_df = pd.DataFrame()\n\nfor fold_, (trn_idx, val_idx) in enumerate(folds.split(train_x_2.values, train_y)):\n    print(\"fold n°{}\".format(fold_))\n    trn_data = lgb.Dataset(train_x_2.iloc[trn_idx][features],\n                           label=train_y[trn_idx],\n                           #categorical_feature=cat_feats\n                           )\n    val_data = lgb.Dataset(train_x_2.iloc[val_idx][features],\n                           label=train_y[val_idx],\n                           #categorical_feature=cat_feats\n                           )\n\n    num_round = 10000\n    clf = lgb.train(param,\n                    trn_data,\n                    num_round,\n                    valid_sets=[trn_data, val_data],\n                    verbose_eval=100,\n                    early_stopping_rounds=200)\n\n    oof_pca[val_idx] = clf.predict(train_x_2.iloc[val_idx][features], num_iteration=clf.best_iteration)\n\n    fold_importance_df = pd.DataFrame()\n    fold_importance_df[\"feature\"] = features\n    fold_importance_df[\"importance\"] = clf.feature_importance()\n    fold_importance_df[\"fold\"] = fold_ + 1\n    feature_importance_df = pd.concat([feature_importance_df, fold_importance_df], axis=0)\n\n    predictions_all += clf.predict(test_x_2[features], num_iteration=clf.best_iteration) / folds.n_splits\n\nprint(\"RMSE test PCA: {:<8.5f}\".format(mean_squared_error(predictions_all, test_y) ** 0.5))","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"8a16b3772e4e6a4b986cf015f85653dee25b418c"},"cell_type":"markdown","source":"We see that using the three components as new features produces around the same score with our initial model. Maybe more or less components can actually improve our model. I leave that for your experimentation...Hope you enjoyed it!"},{"metadata":{"trusted":true,"_uuid":"3d22128530b2a3cec8a2e90b603604f871b721ed"},"cell_type":"code","source":"","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"45596b1696c1303c548a4595f3e16f1f84c3725c"},"cell_type":"code","source":"","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"name":"python","version":"3.6.6","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"}},"nbformat":4,"nbformat_minor":1}