{"cells":[{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"code","source":"# This Python 3 environment comes with many helpful analytics libraries installed\n# It is defined by the kaggle/python docker image: https://github.com/kaggle/docker-python\n# For example, here's several helpful packages to load in \n\nimport numpy as np # linear algebra\nimport pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\nimport seaborn as sns\nimport matplotlib.pyplot as plt\n\n\n# Input data files are available in the \"../input/\" directory.\n# For example, running this (by clicking run or pressing Shift+Enter) will list the files in the input directory\n\nimport os\nprint(os.listdir(\"../input\"))\n\n# Any results you write to the current directory are saved as output.","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"6d0ad859b500070af88b3e82f99472f08c640076"},"cell_type":"markdown","source":"## A glance at the target variable\n\nFirst step is to really understand a bit more about the target variable:\n\n1. How is the variable distributed\n2. To which variables it  correlates the most.\n3. What's the target variable distribution given other indipedent variables."},{"metadata":{"_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","trusted":true},"cell_type":"code","source":"#Loading Train and Test Data\ntrain = pd.read_csv(\"../input/train.csv\", parse_dates=[\"first_active_month\"])\ntest = pd.read_csv(\"../input/test.csv\", parse_dates=[\"first_active_month\"])\n\nprint(f\"Train set contains {train.shape[0]} observations and {train.shape[1]} variables\")\nprint(f\"Test set contains {test.shape[0]} observations and {test.shape[1]} variables\")\ntrain.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"82ac3585debab65a3f89e13dbc7433d878468963"},"cell_type":"code","source":"dupcheck = len(train) - len(train['card_id'].unique())\nprint(f\"The data contains {dupcheck} duplicate card ids\")","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"01f6320c662d49c314d8496a180bcb18cd9053a4"},"cell_type":"markdown","source":"The target variable seems to be pretty much normally distributed with some outliers on both sides. We will look at those at a later stage. The boxplot will help use realize that the distribution has the median at 0 and is  pretty much symmetric. \n\nIt's possible tha target variable - the loyalty score - has been standardized by subtracting the mean and dividing by the standard deviation. It's hard to say by just looking at the summary statistics, since we only see the train set."},{"metadata":{"trusted":true,"_uuid":"e688edc40261e09deb9bf7dbb571ecb3d0261709"},"cell_type":"code","source":"plt.figure(figsize = (16, 6))\n\nplt.subplot(121)\nplt.hist(train[\"target\"], bins = 25)\nplt.title(\"Target Variable Histogram\")\n\nplt.subplot(122)\nsns.boxplot(data = train, y = \"target\")\nplt.title(\"Target Variable Boxplot\")\n\nplt.show()\n\ntrain[\"target\"].describe()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"f59e7a55a0fd8cd7f9b0034f391f3f79bf8c6cfc"},"cell_type":"markdown","source":"If we look at the first active month, the distribution of cards activated in any given period is identical across train and test set. Which means that the two sets have been properly stratified. There's a peak towards the end of 2017 and a sudden dip in the first two months of 2018."},{"metadata":{"trusted":true,"_uuid":"4250ece2960dcd142256e62111f0327597cadda9"},"cell_type":"code","source":"train['first_active_month'].value_counts(normalize = True).sort_index().plot(figsize = (16,4), label = 'Train')\ntest['first_active_month'].value_counts(normalize = True).sort_index().plot(figsize = (16,4), label = 'Test')\nplt.legend()\nplt.title('Frequency of cards by first active month - Train vs Test Set')\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"43f57720a5aa794f92e563c8c4a4eda6be486122"},"cell_type":"markdown","source":"If we look at the loyalty score by the first month the card was active we may notice that this score tends to go up with time, with an increasing trend especially in the last months.\n\n> This means that the cards activated towards mid-end 2017 tend to have  a **higher target value**\n"},{"metadata":{"trusted":true,"_uuid":"f00747458534333019d8dde588913dd24b6d8821"},"cell_type":"code","source":"train[['first_active_month', 'target']].groupby('first_active_month').agg(np.mean).plot(figsize = (16,4))\nplt.title(\"Average Target Value by Card First Active Month\")\n\n\ntrain[['first_active_month', 'target']].groupby('first_active_month'\n                                               ).agg(np.mean).rolling(5).mean().plot(figsize = (16, 4))\nplt.title(\"5 Periods Average Target Value by Card First Active Month\")","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"fad06dfb81bd7bf04bb5898de2a1173eaa46a0dc"},"cell_type":"markdown","source":"If we look at the other three features across both the train and test set we can see that the two sets are well stratified. **Feature_1** has 5 level with a majority at level 3. **Feature_2** has 3 levels, while **Feature_3** is a binomial one. Haven't read all the discussion, but hard to say what those features are. "},{"metadata":{"trusted":true,"_uuid":"cb98d76266de175cdf8f7ce843b969f98706b91a"},"cell_type":"code","source":"feats = np.arange(1,4)\n\nnrows = len(feats)\nncols = 2\n\nplt.figure(figsize = (15,4.5*len(feats)))\nidxs = np.arange(1,7).reshape(nrows, ncols)\n\nfor idx, i in enumerate(feats):\n    \n    f_name = f\"feature_{i}\"\n    \n    plt.subplot(len(feats), ncols, idxs[idx][0])\n    train[f_name].value_counts(normalize=True).plot(kind = 'bar', color = 'blue', alpha = 0.5)\n    plt.title(f\"Train Set {f_name}\")\n    \n    plt.subplot(len(feats), ncols, idxs[idx][1])\n    train[f_name].value_counts(normalize=True).plot(kind = 'bar', color = 'green', alpha = 0.5)\n    plt.title(f\"Test Set {f_name}\")\n\nplt.suptitle('Features Distribution Across Train and Test Set')","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"33c70c67e29ce7fc077fa472d1c6ac06851aace9"},"cell_type":"markdown","source":"The boxplots would revel that the target variable is well distributed across the level of each of the three features."},{"metadata":{"trusted":true,"_uuid":"e558189cfa42c3d919dba7379d0d7207a014b745"},"cell_type":"code","source":"plt.figure(figsize = (16,4))\nsns.boxplot(data = train, x='feature_1', y ='target')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"92b0488bcd44fa8d26e196fddfe5afb85a4e2d5f"},"cell_type":"code","source":"plt.figure(figsize = (16,4))\nsns.boxplot(data = train, x='feature_2', y ='target')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"5bc1e41383074355143c866c54b09e43e575e9f5"},"cell_type":"code","source":"plt.figure(figsize = (16,4))\nsns.boxplot(data = train, x='feature_3', y ='target')","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"a4cf43aa736d8f5821260ed8207eda466b54f5bd"},"cell_type":"markdown","source":"If we look at how the target variable correlats to threee other plots it can be seen that the highest degree is with feature_1. All three features are inversely correlated the loyalty score."},{"metadata":{"trusted":true,"_uuid":"c4389792feff154b80290c86ec4c57f98e2db64c"},"cell_type":"code","source":"train.corr()['target'].head(3).plot(kind='bar', color='blue', alpha = 0.5, figsize = (10,4))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"d7e091c7da8ce60d77c8d78d38dc10a2bc905c58"},"cell_type":"code","source":"","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"name":"python","version":"3.6.6","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"}},"nbformat":4,"nbformat_minor":1}