{"cells":[{"metadata":{"_uuid":"31b3c736c94cc0d2e91b1cb3ebdae273e2fdd40c"},"cell_type":"markdown","source":"## Exploration and Visualization for Card Data"},{"metadata":{"_uuid":"9d1b4542bc247009f1eb976bef8679c8d20a5161"},"cell_type":"markdown","source":"### Abstract"},{"metadata":{"_uuid":"3570808c016b9bd1b9986826722db704de835c07"},"cell_type":"markdown","source":"There are some insights for the card data (train.csv and test.csv). The details are in the notebook but mainly the followings are found:\n\n* There is a sharp decrease in the number of cards which start activity at 2018 (Figure: First Active Months Count).\n\n* The customer loyalty is much lower at 2012-04 when you compare it with other months (Figure: Monthly Customer Loyalty).\n\n* Customer loyalty increased continuosly after 2017-01. There is an obvious trend (Figure: Monthly Customer Loyalty).\n\n* Train and Test sets are divided very well according to the features distribution.\n\n* There is a correlation between features and customer loyalty. It is obvious in the last figure. "},{"metadata":{"trusted":false,"_uuid":"49d7da1ed2649eac99181d3433f7e8729351fdc1"},"cell_type":"code","source":"import pandas as pd\nimport matplotlib.pyplot as plt\nimport seaborn as sns","execution_count":null,"outputs":[]},{"metadata":{"trusted":false,"_uuid":"3e2c8736391a155d9c1e1985dad4a9a098e30ef3"},"cell_type":"code","source":"df_train = pd.read_csv(\"../input/train.csv\")\ndf_test = pd.read_csv(\"../input/test.csv\")","execution_count":null,"outputs":[]},{"metadata":{"trusted":false,"_uuid":"3fa53be0020cad9267db3dd18eff50a05e6a39d3"},"cell_type":"code","source":"df_train.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":false,"_uuid":"c4575db789d5c2ec213e4eafd0fb736bb098c1b5"},"cell_type":"code","source":"df_test.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":false,"_uuid":"37ac246559f3e0d7fc383a8b5bfa609c2cbec645"},"cell_type":"code","source":"print(\"length of train dataset:\", len(df_train))\nprint(\"length of test dataset:\", len(df_test))","execution_count":null,"outputs":[]},{"metadata":{"trusted":false,"_uuid":"9da819ef297a605ac70c0199248baa6cf459883a"},"cell_type":"code","source":"df_month_counts_train = pd.DataFrame(df_train['first_active_month'].value_counts())\ndf_month_counts_train = df_month_counts_train.reset_index()\ndf_month_counts_train.columns = ['first_active_month', 'count']\ndf_month_counts_train = df_month_counts_train.sort_values(by=['first_active_month'])\ndf_month_counts_train = df_month_counts_train.reset_index()\ndf_month_counts_train = df_month_counts_train.drop(columns=['index'])","execution_count":null,"outputs":[]},{"metadata":{"trusted":false,"_uuid":"399b60915732c052df802c424a94466f173384a1"},"cell_type":"code","source":"df_month_counts_train.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":false,"_uuid":"297eb5d2637825c993ccda3099acd8a46dc56e3c"},"cell_type":"code","source":"df_month_counts_train.sort_values(by=['first_active_month'], ascending=False).head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":false,"_uuid":"381ea2f36470b9fbb44106c127e01815aaad61a9"},"cell_type":"code","source":"df_month_counts_test = pd.DataFrame(df_test['first_active_month'].value_counts())\ndf_month_counts_test = df_month_counts_test.reset_index()\ndf_month_counts_test.columns = ['first_active_month', 'count']\ndf_month_counts_test = df_month_counts_test.sort_values(by=['first_active_month'])\ndf_month_counts_test = df_month_counts_test.reset_index()\ndf_month_counts_test = df_month_counts_test.drop(columns=['index'])","execution_count":null,"outputs":[]},{"metadata":{"trusted":false,"_uuid":"82e487bd84bc1ea452e14aed99c3ea3fef5f3b36"},"cell_type":"code","source":"df_month_counts_test.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":false,"_uuid":"f933025fa98709f26ee4b3d991997aa4bbd63a7a"},"cell_type":"code","source":"df_month_counts_test.sort_values(by=['first_active_month'], ascending=False).head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":false,"_uuid":"66f8dc17760bc2fd757c8d65212d26beb10caefb"},"cell_type":"code","source":"plt.figure(figsize=(15,10))\nsns.barplot(x=\"first_active_month\", y=\"count\", data=df_month_counts_train)\nplt.xticks(rotation= 90)\nplt.xlabel('Month')\nplt.ylabel('Count')\nplt.title('First Active Month Counts - Train Data')","execution_count":null,"outputs":[]},{"metadata":{"trusted":false,"_uuid":"1c8f4c4d4867c4a97e568bb8fa15e2d8fe31357f"},"cell_type":"code","source":"plt.figure(figsize=(15,10))\nsns.barplot(x=\"first_active_month\", y=\"count\", data=df_month_counts_test)\nplt.xticks(rotation= 90)\nplt.xlabel('Month')\nplt.ylabel('Count')\nplt.title('First Active Month Counts - Test Data')","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"3dcb2be2fc1dabbfdcab31b12b3bfacaa509093d"},"cell_type":"markdown","source":"Note: There is a sharp decrease and very low counts for 2018 data."},{"metadata":{"trusted":false,"_uuid":"a41b116ebe0dd02027c09a0e52b56a552390bb2c"},"cell_type":"code","source":"df_month_means = df_train.groupby(['first_active_month']).mean()\ndf_month_means = df_month_means.reset_index()\ndf_month_means = df_month_means.sort_values(by=['first_active_month'])\ndf_month_means = df_month_means.reset_index()\ndf_month_means = df_month_means.drop(columns=['index'])","execution_count":null,"outputs":[]},{"metadata":{"trusted":false,"_uuid":"f8dd648f0200ea9ddcb40efdeb612ea85146996d"},"cell_type":"code","source":"df_month_means","execution_count":null,"outputs":[]},{"metadata":{"trusted":false,"_uuid":"e4ee9759efb7c4e028de21260ffbdf2b263d5045"},"cell_type":"code","source":"plt.figure(figsize=(15,10))\nsns.barplot(x=\"first_active_month\", y=\"target\", data=df_month_means)\n#sns.relplot(x=\"first_active_month\", y=\"target\", kind=\"line\", data=df_month_means);\nplt.xticks(rotation= 90)\nplt.xlabel('Month')\nplt.ylabel('Target')\nplt.title('Monthly Customer Loyalty')","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"5bf77a599922e588d1170b279a9cf1904ae4d125"},"cell_type":"markdown","source":"Note:\n\nThere are two points to pay attention:\n\n1- What happened to customer loyalty at 2012-04? It is extremely low. It should be investigated.\n\n2- After 2017-01, there is a continous upper trend in customer loyalty."},{"metadata":{"trusted":false,"_uuid":"a1ec0b8950b13fdb7d40ef174cb19bb0871857db"},"cell_type":"code","source":"df_month_means.sort_values(by=['target']).head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":false,"_uuid":"225e30680fe8908e35b195d98772f7e3f5459b19"},"cell_type":"code","source":"plt.figure(figsize=(15,10))\nsns.violinplot(y=\"target\", data=df_train, palette=\"muted\")\nplt.title('Violin Plot for Customer Loyalty')","execution_count":null,"outputs":[]},{"metadata":{"trusted":false,"_uuid":"66b96f781fcdfb5c9cd7f887669702afafa3a5ae"},"cell_type":"code","source":"plt.figure(figsize=(15,10))\nsns.distplot(df_train[\"target\"]);\nplt.title('Histogram for Customer Loyalty')","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"25f55938ed3a08001c63b6befde733db6089437f"},"cell_type":"markdown","source":"Note: It is obvious that customer loyalty values distribution is accumulated around the mean + median but there are some values look like potential outliers or at least should be taken into consideration for prediction."},{"metadata":{"_uuid":"bef9bc377480b7e74e4ef741421e8f6d79d71f6d"},"cell_type":"markdown","source":"Now lets validate it with further analysis by removing some extreme data."},{"metadata":{"trusted":false,"_uuid":"8815c45a2324cda063eb8fbf7b93edf3e710a553"},"cell_type":"code","source":"# Function: print_quantile_info(qu_dataset, qu_field)\n# Print out the following information about the data\n# - interquartile range\n# - upper_inner_fence\n# - lower_inner_fence\n# - upper_outer_fence\n# - lower_outer_fence\n# - percentage of records out of inner fences\n# - percentage of records out of outer fences\n# Input: \n# - pandas dataframe (qu_dataset)\n# - name of the column to analyze (qu_field)\n# Output:\n# None\n\ndef print_quantile_info(qu_dataset, qu_field):\n a = qu_dataset[qu_field].describe()\n \n iqr = a[\"75%\"] - a[\"25%\"]\n print(\"interquartile range:\", iqr)\n \n upper_inner_fence = a[\"75%\"] + 1.5 * iqr\n lower_inner_fence = a[\"25%\"] - 1.5 * iqr\n print(\"upper_inner_fence:\", upper_inner_fence)\n print(\"lower_inner_fence:\", lower_inner_fence)\n \n upper_outer_fence = a[\"75%\"] + 3 * iqr\n lower_outer_fence = a[\"25%\"] - 3 * iqr\n print(\"upper_outer_fence:\", upper_outer_fence)\n print(\"lower_outer_fence:\", lower_outer_fence)\n \n count_over_upper = len(qu_dataset[qu_dataset[qu_field]>upper_inner_fence])\n count_under_lower = len(qu_dataset[qu_dataset[qu_field]upper_outer_fence])\n count_under_lower = len(qu_dataset[qu_dataset[qu_field]upper_inner_fence])\n count_under_lower = len(qu_dataset[qu_dataset[qu_field]upper_outer_fence])\n count_under_lower = len(qu_dataset[qu_dataset[qu_field]=lower_inner_fence]\n elif qu_fence == \"outer\":\n output_dataset = qu_dataset[qu_dataset[qu_field]<=upper_outer_fence]\n output_dataset = output_dataset[output_dataset[qu_field]>=lower_outer_fence]\n else:\n output_dataset = qu_dataset\n \n print(\"length of input dataframe:\", len(qu_dataset))\n print(\"length of new dataframe after outlier removal:\", len(output_dataset))\n \n return output_dataset","execution_count":null,"outputs":[]},{"metadata":{"trusted":false,"_uuid":"9e06b19283974bb8c70d656f992907ce976e0603"},"cell_type":"code","source":"print_quantile_info(df_train, \"target\")","execution_count":null,"outputs":[]},{"metadata":{"trusted":false,"_uuid":"8717dd8e9675e51cf371f7a4518bb4c5c6a0e6bd"},"cell_type":"code","source":"df_train_new = remove_outliers_using_quantiles(df_train, \"target\", \"outer\")","execution_count":null,"outputs":[]},{"metadata":{"trusted":false,"_uuid":"e0adfa2c72ee99b674cd0444957d072241c9701b"},"cell_type":"code","source":"plt.figure(figsize=(15,10))\nsns.violinplot(y=\"target\", data=df_train_new, palette=\"muted\")\nplt.title('Violin Plot for Customer Loyalty After Removing Outer Data')","execution_count":null,"outputs":[]},{"metadata":{"trusted":false,"_uuid":"d6582d285f502b887fc976090783f97b47c35677"},"cell_type":"code","source":"plt.figure(figsize=(15,10))\nsns.distplot(df_train_new[\"target\"]);\nplt.title('Histogram for Customer Loyalty After Removing Outer Data')","execution_count":null,"outputs":[]},{"metadata":{"scrolled":false,"trusted":false,"_uuid":"adda2f28ea159c3ef6fb46c9ff8493786ede9512"},"cell_type":"code","source":"sns.catplot(x=\"feature_1\", kind=\"count\", palette=\"ch:.25\", data=df_train);\nsns.catplot(x=\"feature_1\", kind=\"count\", palette=\"ch:.25\", data=df_test);","execution_count":null,"outputs":[]},{"metadata":{"trusted":false,"_uuid":"d99ff8b549549fbf24953e0f549bb8c6e2629b14"},"cell_type":"code","source":"sns.catplot(x=\"feature_2\", kind=\"count\", palette=\"ch:.25\", data=df_train);\nsns.catplot(x=\"feature_2\", kind=\"count\", palette=\"ch:.25\", data=df_test);","execution_count":null,"outputs":[]},{"metadata":{"trusted":false,"_uuid":"211dfa3f5c4cd0f1e4398fd39f079c342ea3a99a"},"cell_type":"code","source":"sns.catplot(x=\"feature_3\", kind=\"count\", palette=\"ch:.25\", data=df_train);\nsns.catplot(x=\"feature_3\", kind=\"count\", palette=\"ch:.25\", data=df_test);","execution_count":null,"outputs":[]},{"metadata":{"trusted":false,"_uuid":"b3851ec87a3bf3608facec1fdd8ca0c426161535"},"cell_type":"code","source":"sns.catplot(x=\"feature_1\", y=\"target\", kind=\"bar\", palette=\"ch:.25\", data=df_train);","execution_count":null,"outputs":[]},{"metadata":{"trusted":false,"_uuid":"1dcb2638443b122e57a295472e461a762bdc57f2"},"cell_type":"code","source":"sns.catplot(x=\"feature_2\", y=\"target\", kind=\"bar\", palette=\"ch:.25\", data=df_train);","execution_count":null,"outputs":[]},{"metadata":{"trusted":false,"_uuid":"8da184a04820dafed1e9cb24085ed56d4d713457"},"cell_type":"code","source":"sns.catplot(x=\"feature_3\", y=\"target\", kind=\"bar\", palette=\"ch:.25\", data=df_train);","execution_count":null,"outputs":[]},{"metadata":{"trusted":false,"_uuid":"f372e95bf23ca91952f8a42070a7dc6d24be8463"},"cell_type":"code","source":"df_feature_groups = df_train.groupby(['feature_1','feature_2','feature_3']).mean()\ndf_feature_groups = df_feature_groups.reset_index()\ndf_feature_groups.sort_values(\"target\")","execution_count":null,"outputs":[]},{"metadata":{"trusted":false,"_uuid":"7ecd7c67776292e1f457a8cae3534c1424ecabe5"},"cell_type":"code","source":"df_feature_groups[\"features\"] = df_feature_groups[\"feature_1\"].astype('str') + \"+\" + df_feature_groups[\"feature_2\"].astype('str') + \"+\" + df_feature_groups[\"feature_3\"].astype('str')","execution_count":null,"outputs":[]},{"metadata":{"trusted":false,"_uuid":"a41ed92889eb8e1980907925e5d8c757421b015a"},"cell_type":"code","source":"df_feature_groups","execution_count":null,"outputs":[]},{"metadata":{"trusted":false,"_uuid":"47be813872964b8f4f16690c338e0f4e769990d2"},"cell_type":"code","source":"plt.figure(figsize=(15,10))\nsns.barplot(x=\"features\", y=\"target\", palette=\"ch:.25\", data=df_feature_groups)\nplt.xticks(rotation= 90)","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"name":"python","version":"3.6.6","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"}},"nbformat":4,"nbformat_minor":1}