{"cells":[{"cell_type":"markdown","metadata":{"_cell_guid":"a5d6d250-d7fc-7a2c-5f63-f95f7143fdc4"},"source":"#EDA of important features#"},{"cell_type":"markdown","metadata":{"_cell_guid":"a54246b8-3f34-fb61-4f77-219eae8a1cf4"},"source":"An XGBoost fitted on approximately 700 columns achieves already a LB-Score of ~0.25 without any feature engineering. What can we learn from the 20 most essential numeric features suggested by that model?\n\nThe analysis in this notebook reveals that the positive and negative samples differ significantly  both in their missing-value counts as well as in their missing-value correlation structure. On the other hand, the behavior on non-missing samples is surprisingly similar for positive and negative samples."},{"cell_type":"markdown","metadata":{"_cell_guid":"1e0a334d-444b-62f5-45e3-44c289d71196"},"source":"#Data preparation#"},{"cell_type":"markdown","metadata":{"_cell_guid":"bae71786-56e0-209f-9023-0e981c037815"},"source":"In the first step, we import standard libraries and fix the most essential features as suggested by an XGB oracle."},{"cell_type":"code","execution_count":null,"metadata":{"_cell_guid":"7bc5ef86-1271-0c34-efa3-e0fd1a866aba"},"outputs":[],"source":"%matplotlib inline\nimport matplotlib.pyplot as plt\nimport numpy as np\nimport pandas as pd\nimport seaborn as sns\n\n\nfeature_names = ['L3_S38_F3960', 'L3_S33_F3865', 'L3_S38_F3956', 'L3_S33_F3857',\n       'L3_S29_F3321', 'L1_S24_F1846', 'L3_S32_F3850', 'L3_S29_F3354',\n       'L3_S29_F3324', 'L3_S35_F3889', 'L0_S1_F28', 'L1_S24_F1844',\n       'L3_S29_F3376', 'L0_S0_F22', 'L3_S33_F3859', 'L3_S38_F3952', \n       'L3_S30_F3754', 'L2_S26_F3113', 'L3_S30_F3759', 'L0_S5_F114']"},{"cell_type":"markdown","metadata":{"_cell_guid":"71087ac7-b896-413a-ed8e-7949a7bc7c11"},"source":"We determine the indices of the most important features. After that the training data is loaded"},{"cell_type":"code","execution_count":null,"metadata":{"_cell_guid":"1608d250-9359-6450-2934-b16e53560e10"},"outputs":[],"source":"numeric_cols = pd.read_csv(\"../input/train_numeric.csv\", nrows = 1).columns.values\nimp_idxs = [np.argwhere(feature_name == numeric_cols)[0][0] for feature_name in feature_names]\ntrain = pd.read_csv(\"../input/train_numeric.csv\", \n                index_col = 0, header = 0, usecols = [0, len(numeric_cols) - 1] + imp_idxs)\ntrain = train[feature_names + ['Response']]"},{"cell_type":"markdown","metadata":{"_cell_guid":"00312ea2-d10a-59c6-00c3-2faf3ebbf459"},"source":"The data is split into positive and negative samples."},{"cell_type":"code","execution_count":null,"metadata":{"_cell_guid":"79c3998f-370b-b7c0-54e6-d2136822c4c3"},"outputs":[],"source":" X_neg, X_pos = train[train['Response'] == 0].iloc[:, :-1], train[train['Response']==1].iloc[:, :-1]"},{"cell_type":"markdown","metadata":{"_cell_guid":"baf610f1-c77e-8823-8d7a-cbecb2225010"},"source":"#Univariate characteristics#"},{"cell_type":"markdown","metadata":{"_cell_guid":"5f78f041-eec4-16ff-e627-406dacddc524"},"source":"In order to understand better the predictive power of single features, we compare the univariate distributions of the most important features. First, we divide the train data into batches column-wise to prepare the data for plotting."},{"cell_type":"code","execution_count":null,"metadata":{"_cell_guid":"88aebbf1-467e-4c7f-bec5-a0084b2fc854"},"outputs":[],"source":"BATCH_SIZE = 5\ntrain_batch =[pd.melt(train[train.columns[batch: batch + BATCH_SIZE].append(np.array(['Response']))], \n                      id_vars = 'Response', value_vars = feature_names[batch: batch + BATCH_SIZE])\n              for batch in list(range(0, train.shape[1] - 1, BATCH_SIZE))]"},{"cell_type":"markdown","metadata":{"_cell_guid":"ad598f61-a089-f518-0099-01118c14ff37"},"source":"After this split, we can now draw violin plots. Due to memory reasons, we have to split the presentation into several cells. For many of the distributions there is no clear difference between the positive and negative samples."},{"cell_type":"code","execution_count":null,"metadata":{"_cell_guid":"4a91e953-d241-4cb2-b585-6d83c15cd609"},"outputs":[],"source":"FIGSIZE = (12,16)\n_, axs = plt.subplots(len(train_batch), figsize = FIGSIZE)\nplt.suptitle('Univariate distributions')\nfor data, ax in zip(train_batch, axs):\n    sns.violinplot(x = 'variable',  y = 'value', hue = 'Response', data = data, ax = ax, split =True)"},{"cell_type":"markdown","metadata":{"_cell_guid":"0fe9379a-21cc-b1e6-ea6c-f0c67033c67f"},"source":"The data set is characterized by a large proportion of missing values. When comparing the missing values between positive and negative samples, we see striking differences."},{"cell_type":"code","execution_count":null,"metadata":{"_cell_guid":"d5dfca85-a879-f0e7-1a1a-3968c9ac3ce1"},"outputs":[],"source":"non_missing = pd.DataFrame(pd.concat([(X_neg.count()/X_neg.shape[0]).to_frame('negative samples'),\n                                      (X_pos.count()/X_pos.shape[0]).to_frame('positive samples'),  \n                                      ], \n                       axis = 1))\nnon_missing_sort = non_missing.sort_values(['negative samples'])\nnon_missing_sort.plot.barh(title = 'Proportion of non-missing values', figsize = FIGSIZE)\nplt.gca().invert_yaxis()"},{"cell_type":"markdown","metadata":{"_cell_guid":"3b846266-9294-f44b-ed08-134ad837afdd"},"source":"#Correlation structure#"},{"cell_type":"markdown","metadata":{"_cell_guid":"565f3afa-ecb6-5426-6754-59d904bc7b70"},"source":"In the previous section we have seen differences between negative and positive samples for univariate characteristics. We go down the rabbit hole a little further and analyze covariances for the negative and positive samples separately. "},{"cell_type":"code","execution_count":null,"metadata":{"_cell_guid":"5364b4a1-5f16-1e0e-6974-83450ebbbc10"},"outputs":[],"source":"FIGSIZE = (13,4)\n_, (ax1, ax2) = plt.subplots(1,2, figsize = FIGSIZE)\nMIN_PERIODS = 100\n\ntriang_mask = np.zeros((X_pos.shape[1], X_pos.shape[1]))\ntriang_mask[np.triu_indices_from(triang_mask)] = True\n\nax1.set_title('Negative Class')\nsns.heatmap(X_neg.corr(min_periods = MIN_PERIODS), mask = triang_mask, square=True,  ax = ax1)\n\nax2.set_title('Positive Class')\nsns.heatmap(X_pos.corr(min_periods = MIN_PERIODS), mask = triang_mask, square=True,  ax = ax2)"},{"cell_type":"markdown","metadata":{"_cell_guid":"f5b2441d-abb6-9e67-e5bd-22e9c2f7b5ba"},"source":"The difference between the two matrices is sparse except for three specific feature combinations."},{"cell_type":"code","execution_count":null,"metadata":{"_cell_guid":"3575259a-8e9a-4035-a842-4a00f2475cf2"},"outputs":[],"source":"sns.heatmap(X_pos.corr(min_periods = MIN_PERIODS) -X_neg.corr(min_periods = MIN_PERIODS), \n             mask = triang_mask, square=True)"},{"cell_type":"markdown","metadata":{"_cell_guid":"9ac5dd1e-1cb7-b4bd-269d-bc57fb7682ca"},"source":"Finally, as in the univariate case, we analyze correlations between missing values in  different features."},{"cell_type":"code","execution_count":null,"metadata":{"_cell_guid":"8c417b52-0487-87e0-ebb3-913c5c2b0432"},"outputs":[],"source":"nan_pos, nan_neg = np.isnan(X_pos), np.isnan(X_neg)\n\ntriang_mask = np.zeros((X_pos.shape[1], X_pos.shape[1]))\ntriang_mask[np.triu_indices_from(triang_mask)] = True\n\nFIGSIZE = (13,4)\n_, (ax1, ax2) = plt.subplots(1,2, figsize = FIGSIZE)\nMIN_PERIODS = 100\n\nax1.set_title('Negative Class')\nsns.heatmap(nan_neg.corr(),   square=True, mask = triang_mask, ax = ax1)\n\nax2.set_title('Positive Class')\nsns.heatmap(nan_pos.corr(), square=True, mask = triang_mask,  ax = ax2)"},{"cell_type":"markdown","metadata":{"_cell_guid":"41be60dc-69f4-6572-de5f-4b5b3866b0a5"},"source":"For the difference of the missing-value correlation matrices, a striking pattern emerges. A further and more systematic analysis of such missing-value patterns has the potential to beget powerful features."},{"cell_type":"code","execution_count":null,"metadata":{"_cell_guid":"6eb3a4be-8b23-298c-3986-c877c3321b36"},"outputs":[],"source":"sns.heatmap(nan_neg.corr() - nan_pos.corr(), mask = triang_mask, square=True)"}],"metadata":{"_change_revision":0,"_is_fork":false,"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"codemirror_mode":{"name":"ipython","version":3},"file_extension":".py","mimetype":"text/x-python","name":"python","nbconvert_exporter":"python","pygments_lexer":"ipython3","version":"3.5.2"}},"nbformat":4,"nbformat_minor":0}