{"cells":[{"metadata":{"_kg_hide-input":true,"_kg_hide-output":false,"trusted":false,"_uuid":"a2613b18f576842fe0c7c9a1fe48ff5f2ef414e3"},"cell_type":"markdown","source":"### Exploration of Merchants Data :"},{"metadata":{"_uuid":"5e5eec116599f384a87c130c7783c8fb21f6b900"},"cell_type":"markdown","source":"#### Exploration of Subsectors, Merchant Category, State, City, Category 2 and 4 :"},{"metadata":{"_uuid":"fec1e5b1f668c30a047a8c334745880100c2bd0f"},"cell_type":"markdown","source":"Let us explore the information about merchants. The pie charts showing the count of merchants by subsector, category, state and city are shown below. For better visualization, the categories for which merchant count is less than a particular value are grouped together. It is to be noted that a significant number of merchants have city as -1 (This may be the case of the missing city values or for the merchants which are operational online)."},{"metadata":{"_kg_hide-input":true,"trusted":true,"_uuid":"4b90cef03c7f3fa86c12d39e21f05933189470c4"},"cell_type":"code","source":"import pandas as pd\nimport plotly\nimport cufflinks as cf\nimport plotly.graph_objs as go\nimport plotly.plotly as py\nimport seaborn as sns\nfrom matplotlib import pyplot as plt\nimport warnings\n\nwarnings.filterwarnings('ignore')\nplotly.offline.init_notebook_mode()\ncf.set_config_file(world_readable=True,offline=True)","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"trusted":true,"_uuid":"879ccd46f3d14233bf9646aa7ac3c6503228e83f"},"cell_type":"code","source":"merchants = pd.read_csv(\"../input/merchants.csv\")\nmerchants.head()","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"trusted":true,"_uuid":"0dd9972f1df4d66d1611693f6bd48cad92051908"},"cell_type":"code","source":"# Merchants by Subsector\ns = merchants.groupby(['subsector_id'])['merchant_id'].count()\ndf = pd.DataFrame({'subsector_id':s.index, 'merchant_count':s.values})\ndf.loc[-1] = ['Others', df.loc[df['merchant_count'] < 1000, 'merchant_count'].sum()]\ndf = df[df['merchant_count'] >= 1000]\ndf.iplot(kind='pie',labels='subsector_id',values='merchant_count', \n         title='Merchants by Subsector (Subsector with count less than 1000 grouped together)')","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"trusted":true,"_uuid":"f0284ec3ddec67cd4482881c224eba2e08af55c6"},"cell_type":"code","source":"# Merchants by Category\ns = merchants.groupby(['merchant_category_id'])['merchant_id'].count()\ndf = pd.DataFrame({'merchant_category_id':s.index, 'merchant_count':s.values})\ndf.loc[-1] = ['Others', df.loc[df['merchant_count'] < 2000, 'merchant_count'].sum()]\ndf = df[df['merchant_count'] >= 2000]\ndf.iplot(kind='pie',labels='merchant_category_id',values='merchant_count', \n         title='Merchants by Category (Category with count less than 2000 grouped together)')","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"trusted":true,"_uuid":"c05f567384b4f30649df128979beb1c1423baddc"},"cell_type":"code","source":"# Merchants by State\ns = merchants.groupby(['state_id'])['merchant_id'].count()\ndf = pd.DataFrame({'state_id':s.index, 'merchant_count':s.values})\ndf.iplot(kind='pie',labels='state_id',values='merchant_count', title='Merchants by State')","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"trusted":true,"_uuid":"956a4ecb54ee17e58d77fe00ccb3be12df8562d4"},"cell_type":"code","source":"# Merchants by City\ns = merchants.groupby(['city_id'])['merchant_id'].count()\ndf = pd.DataFrame({'city_id':s.index, 'merchant_count':s.values})\ndf.loc[-1] = ['Others', df.loc[df['merchant_count'] < 1000, 'merchant_count'].sum()]\ndf = df[df['merchant_count'] >= 1000]\ndf.iplot(kind='pie',labels='city_id',values='merchant_count', \n         title='Merchants by City (City with count less than 1000 grouped together)')","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"e23ba8d666d25016ba2a948aa7faa77bc404b852"},"cell_type":"markdown","source":"Let us explore the merchants based on various anonymized categories as well. The pie charts showing the count of merchants in the various anonymized categories are shown below. The category 1 has two values: 'Y' and 'N'. Almost 98% of merchants belongs to the 'N' category for category 1. The category 4 is also dominated by 'N' with almost 71% merchants belonging to this category."},{"metadata":{"_kg_hide-input":true,"trusted":true,"_uuid":"dbf4830251d9d29ae914d910376f1058ab6a2b22"},"cell_type":"code","source":"# Merchants by category_2\ns = merchants.groupby(['category_2'])['merchant_id'].count()\ndf = pd.DataFrame({'category_2':s.index, 'merchant_count':s.values})\ndf.iplot(kind='pie',labels='category_2',values='merchant_count', \n         title='Merchants by Category 2')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"4d3b54d2f0afd0c5a06f7ba501da6dc74e3806dc","_kg_hide-input":true},"cell_type":"code","source":"# Merchants by category_4\ns = merchants.groupby(['category_4'])['merchant_id'].count()\ndf = pd.DataFrame({'category_4':s.index, 'merchant_count':s.values})\ndf.iplot(kind='pie',labels='category_4',values='merchant_count', \n         title='Merchants by Category 4')","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"31d2ae1d85102a7f6a536d217f62e8e4170b7c0e"},"cell_type":"markdown","source":"#### Exploration of Range of Revenue and Quantity of Transactions :"},{"metadata":{"_uuid":"d03a171b355f2b2fe80669f7527a77e163855ead"},"cell_type":"markdown","source":"The pie charts for the count of merchants based on revenue and quantity of transactions are shown below. The count of merchants belonging to each of the ranges for revenue and transaction quantity are almost same. Let us check for the correlation between these values. Almost 71% of merchants belong to the same range for Revenue and Quantity of Transactions. There exist some amount of correlation between them but either of them can not be ignored."},{"metadata":{"trusted":true,"_uuid":"997f7025d36a41730a5caae39c73b09201b3ab9e","_kg_hide-input":true},"cell_type":"code","source":"# Merchants by most_recent_sales_range\ns = merchants.groupby(['most_recent_sales_range'])['merchant_id'].count()\ndf = pd.DataFrame({'most_recent_sales_range':s.index, 'merchant_count':s.values})\ndf.iplot(kind='pie',labels='most_recent_sales_range',values='merchant_count', \n         title='Merchants by Range of Revenue in the last active month (A > B > C > D > E)')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"9ca867e540926bb323eb39f20da763bd299bdd41","_kg_hide-input":true},"cell_type":"code","source":"# Merchants by most_recent_purchases_range\ns = merchants.groupby(['most_recent_purchases_range'])['merchant_id'].count()\ndf = pd.DataFrame({'most_recent_purchases_range':s.index, 'merchant_count':s.values})\ndf.iplot(kind='pie',labels='most_recent_purchases_range',values='merchant_count', \n         title='Merchants by Range of quantity of Transactions in the last active month (A > B > C > D > E)')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"0641106c993986bc2b79933655daa2b0ed8bbf28","_kg_hide-input":true},"cell_type":"code","source":"print(\"Percentage of merchants who are in the same range of Revenue and Quantity of Transaction is: \"\n     + str(merchants.loc[merchants['most_recent_sales_range'] == merchants['most_recent_purchases_range']]\n           ['merchant_id'].count() / merchants['merchant_id'].count() * 100))","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"8aed2e06bed395a58a85c2dd913acc1730a9046e"},"cell_type":"markdown","source":"Let us further explore the distribution of merchants based on revenue and transaction quantity ranges for subsectors, states and category 2. There does not seem to exist any trend in the data."},{"metadata":{"trusted":true,"_uuid":"548b9edec6334d22662cce8815f5c31ffbeb0338","_kg_hide-input":true},"cell_type":"code","source":"grouped = merchants.groupby([\"subsector_id\", \"most_recent_sales_range\"])['merchant_id'].count().reset_index()\n\ntraceA = go.Bar(\n    x=grouped[grouped['most_recent_sales_range'] == 'A']['subsector_id'],\n    y=grouped[grouped['most_recent_sales_range'] == 'A']['merchant_id'],\n    name='Range A'\n)\n\ntraceB = go.Bar(\n    x=grouped[grouped['most_recent_sales_range'] == 'B']['subsector_id'],\n    y=grouped[grouped['most_recent_sales_range'] == 'B']['merchant_id'],\n    name='Range B'\n)\n\ntraceC = go.Bar(\n    x=grouped[grouped['most_recent_sales_range'] == 'C']['subsector_id'],\n    y=grouped[grouped['most_recent_sales_range'] == 'C']['merchant_id'],\n    name='Range C'\n)\n\ntraceD = go.Bar(\n    x=grouped[grouped['most_recent_sales_range'] == 'D']['subsector_id'],\n    y=grouped[grouped['most_recent_sales_range'] == 'D']['merchant_id'],\n    name='Range D'\n)\n\ntraceE = go.Bar(\n    x=grouped[grouped['most_recent_sales_range'] == 'E']['subsector_id'],\n    y=grouped[grouped['most_recent_sales_range'] == 'E']['merchant_id'],\n    name='Range E'\n)\n\ndata = [traceA, traceB, traceC, traceD, traceE]\nlayout = go.Layout(\n    barmode='stack',\n    title='Merchant Counts (Revenue ranges) for Subsectors',\n    xaxis=dict(\n        title='Subsector ID'\n    ),\n    yaxis=dict(\n        title='Merchant Counts'\n    )\n)\n\nfig = go.Figure(data=data, layout=layout,)\nplotly.offline.iplot(fig)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"6be8e04d59101bc4b45a3e9da8cee7a10f99095d","_kg_hide-input":true},"cell_type":"code","source":"grouped = merchants.groupby([\"subsector_id\", \"most_recent_purchases_range\"])['merchant_id'].count().reset_index()\n\ntraceA = go.Bar(\n    x=grouped[grouped['most_recent_purchases_range'] == 'A']['subsector_id'],\n    y=grouped[grouped['most_recent_purchases_range'] == 'A']['merchant_id'],\n    name='Range A'\n)\n\ntraceB = go.Bar(\n    x=grouped[grouped['most_recent_purchases_range'] == 'B']['subsector_id'],\n    y=grouped[grouped['most_recent_purchases_range'] == 'B']['merchant_id'],\n    name='Range B'\n)\n\ntraceC = go.Bar(\n    x=grouped[grouped['most_recent_purchases_range'] == 'C']['subsector_id'],\n    y=grouped[grouped['most_recent_purchases_range'] == 'C']['merchant_id'],\n    name='Range C'\n)\n\ntraceD = go.Bar(\n    x=grouped[grouped['most_recent_purchases_range'] == 'D']['subsector_id'],\n    y=grouped[grouped['most_recent_purchases_range'] == 'D']['merchant_id'],\n    name='Range D'\n)\n\ntraceE = go.Bar(\n    x=grouped[grouped['most_recent_purchases_range'] == 'E']['subsector_id'],\n    y=grouped[grouped['most_recent_purchases_range'] == 'E']['merchant_id'],\n    name='Range E'\n)\n\ndata = [traceA, traceB, traceC, traceD, traceE]\nlayout = go.Layout(\n    barmode='stack',\n    title='Merchant Counts (Quantity of Transactions ranges) for Subsectors',\n    xaxis=dict(\n        title='Subsector ID'\n    ),\n    yaxis=dict(\n        title='Merchant Counts'\n    )\n)\n\nfig = go.Figure(data=data, layout=layout,)\nplotly.offline.iplot(fig)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"cbc788410b6c413a03271273f2693aa41bb537a0","_kg_hide-input":true},"cell_type":"code","source":"grouped = merchants.groupby([\"state_id\", \"most_recent_sales_range\"])['merchant_id'].count().reset_index()\n\ntraceA = go.Bar(\n    x=grouped[grouped['most_recent_sales_range'] == 'A']['state_id'],\n    y=grouped[grouped['most_recent_sales_range'] == 'A']['merchant_id'],\n    name='Range A'\n)\n\ntraceB = go.Bar(\n    x=grouped[grouped['most_recent_sales_range'] == 'B']['state_id'],\n    y=grouped[grouped['most_recent_sales_range'] == 'B']['merchant_id'],\n    name='Range B'\n)\n\ntraceC = go.Bar(\n    x=grouped[grouped['most_recent_sales_range'] == 'C']['state_id'],\n    y=grouped[grouped['most_recent_sales_range'] == 'C']['merchant_id'],\n    name='Range C'\n)\n\ntraceD = go.Bar(\n    x=grouped[grouped['most_recent_sales_range'] == 'D']['state_id'],\n    y=grouped[grouped['most_recent_sales_range'] == 'D']['merchant_id'],\n    name='Range D'\n)\n\ntraceE = go.Bar(\n    x=grouped[grouped['most_recent_sales_range'] == 'E']['state_id'],\n    y=grouped[grouped['most_recent_sales_range'] == 'E']['merchant_id'],\n    name='Range E'\n)\n\ndata = [traceA, traceB, traceC, traceD, traceE]\nlayout = go.Layout(\n    barmode='stack',\n    title='Merchant Counts (Revenue ranges) for States',\n    xaxis=dict(\n        title='State ID'\n    ),\n    yaxis=dict(\n        title='Merchant Counts'\n    )\n)\n\nfig = go.Figure(data=data, layout=layout,)\nplotly.offline.iplot(fig)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"7a8acc39437a7fa0c26c057aee97fa50adae5ffb","_kg_hide-input":true},"cell_type":"code","source":"grouped = merchants.groupby([\"state_id\", \"most_recent_purchases_range\"])['merchant_id'].count().reset_index()\n\ntraceA = go.Bar(\n    x=grouped[grouped['most_recent_purchases_range'] == 'A']['state_id'],\n    y=grouped[grouped['most_recent_purchases_range'] == 'A']['merchant_id'],\n    name='Range A'\n)\n\ntraceB = go.Bar(\n    x=grouped[grouped['most_recent_purchases_range'] == 'B']['state_id'],\n    y=grouped[grouped['most_recent_purchases_range'] == 'B']['merchant_id'],\n    name='Range B'\n)\n\ntraceC = go.Bar(\n    x=grouped[grouped['most_recent_purchases_range'] == 'C']['state_id'],\n    y=grouped[grouped['most_recent_purchases_range'] == 'C']['merchant_id'],\n    name='Range C'\n)\n\ntraceD = go.Bar(\n    x=grouped[grouped['most_recent_purchases_range'] == 'D']['state_id'],\n    y=grouped[grouped['most_recent_purchases_range'] == 'D']['merchant_id'],\n    name='Range D'\n)\n\ntraceE = go.Bar(\n    x=grouped[grouped['most_recent_purchases_range'] == 'E']['state_id'],\n    y=grouped[grouped['most_recent_purchases_range'] == 'E']['merchant_id'],\n    name='Range E'\n)\n\ndata = [traceA, traceB, traceC, traceD, traceE]\nlayout = go.Layout(\n    barmode='stack',\n    title='Merchant Counts (Quantity of Transactions ranges) for States',\n    xaxis=dict(\n        title='State ID'\n    ),\n    yaxis=dict(\n        title='Merchant Counts'\n    )\n)\n\nfig = go.Figure(data=data, layout=layout,)\nplotly.offline.iplot(fig)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"54bbe81a46de2931be87523ea421298e33f552fe","_kg_hide-input":true},"cell_type":"code","source":"grouped = merchants.groupby([\"category_2\", \"most_recent_sales_range\"])['merchant_id'].count().reset_index()\n\ntraceA = go.Bar(\n    x=grouped[grouped['most_recent_sales_range'] == 'A']['category_2'],\n    y=grouped[grouped['most_recent_sales_range'] == 'A']['merchant_id'],\n    name='Range A'\n)\n\ntraceB = go.Bar(\n    x=grouped[grouped['most_recent_sales_range'] == 'B']['category_2'],\n    y=grouped[grouped['most_recent_sales_range'] == 'B']['merchant_id'],\n    name='Range B'\n)\n\ntraceC = go.Bar(\n    x=grouped[grouped['most_recent_sales_range'] == 'C']['category_2'],\n    y=grouped[grouped['most_recent_sales_range'] == 'C']['merchant_id'],\n    name='Range C'\n)\n\ntraceD = go.Bar(\n    x=grouped[grouped['most_recent_sales_range'] == 'D']['category_2'],\n    y=grouped[grouped['most_recent_sales_range'] == 'D']['merchant_id'],\n    name='Range D'\n)\n\ntraceE = go.Bar(\n    x=grouped[grouped['most_recent_sales_range'] == 'E']['category_2'],\n    y=grouped[grouped['most_recent_sales_range'] == 'E']['merchant_id'],\n    name='Range E'\n)\n\ndata = [traceA, traceB, traceC, traceD, traceE]\nlayout = go.Layout(\n    barmode='stack',\n    title='Merchant Counts (Revenue ranges) for Category 2',\n    xaxis=dict(\n        title='Category 2'\n    ),\n    yaxis=dict(\n        title='Merchant Counts'\n    )\n)\n\nfig = go.Figure(data=data, layout=layout)\nplotly.offline.iplot(fig)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"9f120f07a4ac33d904f0d361d135c8c60ae4e27d","_kg_hide-input":true},"cell_type":"code","source":"grouped = merchants.groupby([\"category_2\", \"most_recent_purchases_range\"])['merchant_id'].count().reset_index()\n\ntraceA = go.Bar(\n    x=grouped[grouped['most_recent_purchases_range'] == 'A']['category_2'],\n    y=grouped[grouped['most_recent_purchases_range'] == 'A']['merchant_id'],\n    name='Range A'\n)\n\ntraceB = go.Bar(\n    x=grouped[grouped['most_recent_purchases_range'] == 'B']['category_2'],\n    y=grouped[grouped['most_recent_purchases_range'] == 'B']['merchant_id'],\n    name='Range B'\n)\n\ntraceC = go.Bar(\n    x=grouped[grouped['most_recent_purchases_range'] == 'C']['category_2'],\n    y=grouped[grouped['most_recent_purchases_range'] == 'C']['merchant_id'],\n    name='Range C'\n)\n\ntraceD = go.Bar(\n    x=grouped[grouped['most_recent_purchases_range'] == 'D']['category_2'],\n    y=grouped[grouped['most_recent_purchases_range'] == 'D']['merchant_id'],\n    name='Range D'\n)\n\ntraceE = go.Bar(\n    x=grouped[grouped['most_recent_purchases_range'] == 'E']['category_2'],\n    y=grouped[grouped['most_recent_purchases_range'] == 'E']['merchant_id'],\n    name='Range E'\n)\n\ndata = [traceA, traceB, traceC, traceD, traceE]\nlayout = go.Layout(\n    barmode='stack',\n    title='Merchant Counts (Quantity of Transactions ranges) for Category 2',\n    xaxis=dict(\n        title='Category 2'\n    ),\n    yaxis=dict(\n        title='Merchant Counts'\n    )\n)\n\nfig = go.Figure(data=data, layout=layout)\nplotly.offline.iplot(fig)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"994ae2fee2238949262eb01e0706d9c853d5bfee"},"cell_type":"markdown","source":"#### Exploration of Anonymized Measure :"},{"metadata":{"_uuid":"baea6be094b62c3d6120443266fafc644d6509d4"},"cell_type":"markdown","source":"Let us explore the two anonymized measures named as: numerical_1, numerical_2. First of all we need to find that whether they are correlated or not. The correlation coefficient is <b>0.99875</b>. They are highly correlated (positive) with each other and hence we need to take care of this fact while developing the model. The distribution of these measures for different revenue and quantity of transactions range is shown below. It should be noted that the merchants who have higher revenue/ quantity of transactions have in general higher value of these numerical measures. The merchants who are in the range E and D have the lowest value of these measures (with low variance as well). \n\nLet us further explore the distribution of these measures for different values of category_2. The value of numerical_1 seems to have the largest spread for category_2=1. But this may happen due to the fact that the category_2 = 1 has almost 50% of merchants and hence the spread. The state-wise distribution of numerical_1 is analyzed as well. The 5 states which have the most wide-spread values for numerical_1 are the ones which have the most number of merchants and hence no conclusion can be drawn from the distribution."},{"metadata":{"trusted":true,"_uuid":"6d0620db6e1a452ec04f1120648e8ce95516ce01","_kg_hide-input":true},"cell_type":"code","source":"print(\"The correlation coefficient between numerical_1 and numerical_2 is: \" + \n     str(merchants['numerical_1'].corr(merchants['numerical_2'])))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"59f34141e970ccfa8d6e4a58429f4c7813d3954a","_kg_hide-input":true},"cell_type":"code","source":"fig = plt.figure(figsize=(15,16))\nax = fig.add_subplot(221)\nsns.scatterplot(x=\"most_recent_sales_range\", y=\"numerical_1\", data=merchants, color='red')\nax.set_xlabel('Revenue Range')\nax.set_ylabel('Numerical 1')\nax.set_title('Numerical 1 by Revenue Range')\nax.grid()\n\nax = fig.add_subplot(222)\nsns.scatterplot(x=\"most_recent_purchases_range\", y=\"numerical_1\", data=merchants, color='red')\nax.set_xlabel('Quantity of Transactions Range')\nax.set_ylabel('Numerical 1')\nax.set_title('Numerical 1 by Quantity of Transactions Range')\nax.grid()\n\nax = fig.add_subplot(223)\nsns.scatterplot(x=\"most_recent_sales_range\", y=\"numerical_2\", data=merchants, color='green')\nax.set_xlabel('Revenue Range')\nax.set_ylabel('Numerical 2')\nax.set_title('Numerical 2 by Revenue Range')\nax.grid()\n\nax = fig.add_subplot(224)\nsns.scatterplot(x=\"most_recent_purchases_range\", y=\"numerical_2\", data=merchants, color='green')\nax.set_xlabel('Quantity of Transactions Range')\nax.set_ylabel('Numerical 2')\nax.set_title('Numerical 2 by Quantity of Transactions Range')\nax.grid()\n\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"3ca28dab44fc165d141a2f58bdea836d280a3975","_kg_hide-input":true},"cell_type":"code","source":"fig = plt.figure(figsize=(15,8))\nax = fig.add_subplot(121)\nsns.scatterplot(x=\"category_2\", y=\"numerical_1\", data=merchants, color='green')\nax.set_xlabel('Category 2')\nax.set_ylabel('Numerical 1')\nax.set_title('Numerical 1 by Category 2')\nax.grid()\n\nax = fig.add_subplot(122)\nsns.scatterplot(x=\"state_id\", y=\"numerical_1\", data=merchants, color='green')\nax.set_xlabel('State ID')\nax.set_ylabel('Numerical 1')\nax.set_title('Numerical 1 by State ID')\nax.grid()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"425c53f6ee598de2a2765a007d847074b1f66aad"},"cell_type":"markdown","source":"#### Exploration of Active Months :"},{"metadata":{"_uuid":"cc91b9c58605b2fd18c33a99b3739ad05ed0bd28"},"cell_type":"markdown","source":"The effect of the period of the activity of the mearchants on the range of revenue is analyzed as well. The plot of the percentage of the merchats who are active for the entire period (of 3, 6 and 12 months) in each of the revenue range is shown below. It can be noted that as the revenue range dips (from A to E), the percentage of active restaurants dips as well. This may depict a phenomenon that as the merchants activity period increases (i.e. they become regular), the revenue tends to improve. This trend needs to be realized further as it may arise due to the fact that there exists more number of merchants in the lower revenue range."},{"metadata":{"trusted":true,"_uuid":"4438f759dccd0d7c5444938d8a4a92a6a240f1aa","_kg_hide-input":true},"cell_type":"code","source":"grouped_12 = merchants.groupby(['most_recent_sales_range','active_months_lag12'])['merchant_id'].count().groupby(\n    level=[0]).apply(lambda x: x *100 / x.sum()).reset_index()\ngrouped_12.columns = ['Sales Range', 'Active Month', 'Percent of Merchants']\ngrouped_12 = grouped_12[grouped_12['Percent of Merchants'] > 80]\n\ngrouped_6 = merchants.groupby(['most_recent_sales_range','active_months_lag6'])['merchant_id'].count().groupby(\n    level=[0]).apply(lambda x: x *100 / x.sum()).reset_index()\ngrouped_6.columns = ['Sales Range', 'Active Month', 'Percent of Merchants']\ngrouped_6 = grouped_6[grouped_6['Percent of Merchants'] > 80]\n\ngrouped_3 = merchants.groupby(['most_recent_sales_range','active_months_lag3'])['merchant_id'].count().groupby(\n    level=[0]).apply(lambda x: x *100 / x.sum()).reset_index()\ngrouped_3.columns = ['Sales Range', 'Active Month', 'Percent of Merchants']\ngrouped_3 = grouped_3[grouped_3['Percent of Merchants'] > 80]\n\ndf = pd.concat([grouped_12, grouped_6, grouped_3],ignore_index=True)\n\nfig = plt.figure(figsize=(15,8))\nax = fig.add_subplot(111)\nsns.stripplot(x=\"Sales Range\", y=\"Percent of Merchants\", hue='Active Month', data=df, size=10.0, jitter=False)\nax.set_xlabel('Revenue Range')\nax.set_ylabel('Percentage of Merchant (Active for the entire period)')\nax.set_title('Percentage of Merchant (Active for the entire period) vs Revenue Range')\nax.grid()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"cc8c4572aeba1fdc714c94835d53986e911a6e70"},"cell_type":"markdown","source":"#### Exploration of various Lag variables (Sales and Purchase Lags) :"},{"metadata":{"_uuid":"d7854da5ee86bbe5e49eb5d544871574edf65fcc"},"cell_type":"markdown","source":"The next step is to analyze the distribution of various lags variables. The distribution of various lags variables are shown below. There does not seem to exist any pattern in the data. If we further analyze the distribution of average purchase lag, it seems that the distribution follows a pattern if we remove the outliers. The third plot shows the distribution of average purchase lags after removing the outliers (values > 200). The data seems to follow a right-skewed normal distribution.\n\nFrom the definition of the lags variables, it is evident that a lag value which is less than 1 (this means that the revenue/transactions in the last active month is greater than the average) is favourable as this indicates that the revenue/transactions of merchants are increasing. The lag variables associated with the revenue can be negative (as the revnue can be negative) and the one associated with transactions will always be positive. The data depicts the same behaviour. Apart from these facts, there are certain NaN and INF values as well. Let us visualize the scatter plot of lag variables segregated by revenue ranges. From the plot it can be observed that for the merchants in the better revenue range (A followed by B and so on), the values of lag variables are lower. The correlation coefficients between the lag variables are shown as well. It can be noted that as the period increases, correlation decreases."},{"metadata":{"trusted":true,"_uuid":"b891d1df9af54c3f0b4753238fd68aa0f38e60c1","_kg_hide-input":true},"cell_type":"code","source":"fig = plt.figure(figsize=(15,12))\nax = fig.add_subplot(311)\n\nsns.distplot(merchants.dropna()['avg_sales_lag3'], hist=False, color='r', label='based on Last 3 months')\nsns.distplot(merchants.dropna()['avg_sales_lag6'], hist=False, color='g', label='based on Last 6 months')\nsns.distplot(merchants.dropna()['avg_sales_lag12'], hist=False, color='b', label='based on Last 12 months')\n\nax.set_xlabel('Average Sales Lag')\nax.set_title('Distribution Plot (Average Sales Lag)')\nax.grid()\nax.legend()\n\nax = fig.add_subplot(312)\n\nsns.distplot(merchants.dropna()['avg_purchases_lag3'], hist=False, color='r', label='based on Last 3 months')\nsns.distplot(merchants.dropna()['avg_purchases_lag6'], hist=False, color='g', label='based on Last 6 months')\nsns.distplot(merchants.dropna()['avg_purchases_lag12'], hist=False, color='b', label='based on Last 12 months')\n\nax.set_xlabel('Average Purchase Lag')\nax.set_title('Distribution Plot (Average Purchase Lag)')\nax.grid()\nax.legend()\n\nax = fig.add_subplot(313)\n\nsns.distplot(merchants.dropna()[merchants['avg_purchases_lag3'] <= 200]['avg_purchases_lag3'], hist=False, color='r', label='based on Last 3 months')\nsns.distplot(merchants.dropna()[merchants['avg_purchases_lag6'] <= 200]['avg_purchases_lag6'], hist=False, color='g', label='based on Last 6 months')\nsns.distplot(merchants.dropna()[merchants['avg_purchases_lag12'] <= 200]['avg_purchases_lag12'], hist=False, color='b', label='based on Last 12 months')\n\nax.set_xlabel('Average Purchase Lag')\nax.set_title('Distribution Plot (Average Purchase Lag after removing outliers)')\nax.grid()\nax.legend()\n\nplt.tight_layout()\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"2a6346098f37ded7424f5cca5d62b62bf3a74eda","_kg_hide-input":true},"cell_type":"code","source":"fig = plt.figure(figsize=(15,12))\n\nq = merchants[\"avg_sales_lag12\"].quantile(0.99)\ndf = merchants[merchants[\"avg_sales_lag12\"] < q]\n\nq = df[\"avg_purchases_lag12\"].quantile(0.99)\ndf = df[df[\"avg_purchases_lag12\"] < q]\n\nax = fig.add_subplot(311)\n\nsns.scatterplot(x=\"avg_sales_lag12\", y=\"avg_purchases_lag12\", hue=\"most_recent_sales_range\", data=df)\n\nax.set_xlabel('Average Sales Lag (12 months)')\nax.set_ylabel('Average Purchase Lag (12 months)')\nax.set_title('Sales Lag vs Purchase Lag (12 months)')\nax.grid()\n\nq = merchants[\"avg_sales_lag6\"].quantile(0.99)\ndf = merchants[merchants[\"avg_sales_lag6\"] < q]\n\nq = df[\"avg_purchases_lag6\"].quantile(0.99)\ndf = df[df[\"avg_purchases_lag6\"] < q]\n\nax = fig.add_subplot(312)\n\nsns.scatterplot(x=\"avg_sales_lag6\", y=\"avg_purchases_lag6\", hue=\"most_recent_sales_range\", data=df)\n\nax.set_xlabel('Average Sales Lag (6 months)')\nax.set_ylabel('Average Purchase Lag (6 months)')\nax.set_title('Sales Lag vs Purchase Lag (6 months)')\nax.grid()\n\nq = merchants[\"avg_sales_lag3\"].quantile(0.99)\ndf = merchants[merchants[\"avg_sales_lag3\"] < q]\n\nq = df[\"avg_purchases_lag3\"].quantile(0.99)\ndf = df[df[\"avg_purchases_lag3\"] < q]\n\nax = fig.add_subplot(313)\n\nsns.scatterplot(x=\"avg_sales_lag3\", y=\"avg_purchases_lag3\", hue=\"most_recent_sales_range\", data=df)\n\nax.set_xlabel('Average Sales Lag (3 months)')\nax.set_ylabel('Average Purchase Lag (3 months)')\nax.set_title('Sales Lag vs Purchase Lag (3 months)')\nax.grid()\n\nplt.tight_layout()\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"db3b8eaa77ce0d86051a9de5797129544bb19fce","_kg_hide-input":true},"cell_type":"code","source":"print(\"The correlation coefficient between avg_sales_lag3 and avg_purchases_lag3 is: \" + \n     str(merchants['avg_sales_lag3'].corr(merchants['avg_purchases_lag3'])))\nprint(\"The correlation coefficient between avg_sales_lag6 and avg_purchases_lag6 is: \" + \n     str(merchants['avg_sales_lag6'].corr(merchants['avg_purchases_lag6'])))\nprint(\"The correlation coefficient between avg_sales_lag12 and avg_purchases_lag12 is: \" + \n     str(merchants['avg_sales_lag12'].corr(merchants['avg_purchases_lag12'])))","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"5bcc8228b0c668751ee9efcecaf7095eca864eef"},"cell_type":"markdown","source":"### Exploration of New Merchant Transactions :"},{"metadata":{"_uuid":"19a8d9021c6d1cd31038665d2acaeb53c2ff09af"},"cell_type":"markdown","source":"The notebook deals with the exploratory data analysis for the new merchant transactions. \n\n<b>Dataset</b>: new_merchant_transactions.csv contains the transactions at new merchants (merchant_ids that this particular card_id has not yet visited) over a period of two months.\n\n<b>Description:</b>"},{"metadata":{"_kg_hide-input":true,"trusted":true,"_uuid":"23c095f85a27d26d48897ef1c8a65837e39e5d1b"},"cell_type":"code","source":"new_merchant_transactions = pd.read_csv(\"../input/new_merchant_transactions.csv\")\nnew_merchant_transactions.head()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"6e2797f6a49a2128ec2a4d41fe92d6b9aa2a09e9"},"cell_type":"markdown","source":"#### Exploration of Anonymized Category 3 :"},{"metadata":{"_uuid":"f07c514dac140d7a7a17fb384cc30e74b57e7109"},"cell_type":"markdown","source":"First of all, there exists a new anonymized category named category_3 in the data set. Let us see the distribution of merchants along this category. The category has some NaN values and it is replaced with UKWN. The count of merchants in each of the category is shown below."},{"metadata":{"_kg_hide-input":true,"trusted":true,"_uuid":"68fff45a3973af12aa95eee04c8080b67b442b1e"},"cell_type":"code","source":"new_merchant_transactions['category_3'] = new_merchant_transactions['category_3'].fillna('UKWN')\ndf = new_merchant_transactions[['category_3', 'merchant_id']].drop_duplicates()\ns = df.groupby(['category_3'])['merchant_id'].count()\ndf1 = pd.DataFrame({'category_3':s.index, 'merchant_count':s.values})\ndf1.iplot(kind='pie',labels='category_3',values='merchant_count', \n         title='Merchants by Category 3 (NaN replaced with UKWN)')","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"592464692e3f16db01bb4602a63ba6b3ae886be2"},"cell_type":"markdown","source":"#### Exploration of Purchase Amount :"},{"metadata":{"_uuid":"f4d7e9c09f0b28521d25ba656aba1815084ebf2e"},"cell_type":"markdown","source":"The distribution of purchase amount is shiown below. The mean of the normalized purchase amount is <b>-0.551</b> with a standard deviation of <b>0.694</b>. The category 3 wise distribution of purchase amount is shown as well. The means of purchase amount for different categories for Category 3 are: <b>A: -0.631, B: -0.606, C: 0.0377, UKWN: 0.034</b> and standard deviations are: <b>A: 0.268, B: 0.444, C: 1.788, UKWN: 1.692</b>. The box plot of purchase amount for various subsector ids shows a lot of variation in it along subsectors as well. Let us furhter explore the variation in purchase amout amongst the states. The box plot of purchase amount across the states shows uniformity amongst it."},{"metadata":{"_kg_hide-input":true,"trusted":true,"_uuid":"042ca708827fe902b740ec238d98f78e53953110"},"cell_type":"code","source":"fig = plt.figure(figsize=(15,8))\nax = fig.add_subplot(211)\n\nax = sns.kdeplot(new_merchant_transactions['purchase_amount'], shade=True, color=\"b\")\nax.set_xlabel('Purchase Amount')\nax.set_title('Distribution Plot (Purchase Amount)')\nax.grid()\n\n\nq = new_merchant_transactions[\"purchase_amount\"].quantile(0.99)\ndf = new_merchant_transactions[new_merchant_transactions[\"purchase_amount\"] < q]\nax = fig.add_subplot(212)\n\nax = sns.kdeplot(df['purchase_amount'], shade=True, color=\"b\")\nax.set_xlabel('Purchase Amount')\nax.set_title('Distribution Plot (Purchase Amount after removing outliers)')\nax.grid()\n\nplt.tight_layout()\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"trusted":true,"_uuid":"881cc4d2ce890b844d3901705a224b5e9dfbc345"},"cell_type":"code","source":"fig = plt.figure(figsize=(15,8))\nax = fig.add_subplot(211)\n\nax = sns.kdeplot(new_merchant_transactions[new_merchant_transactions['category_3'] == 'A']['purchase_amount'],\n                 shade=True, color=\"b\", label='Category 3: A')\nax = sns.kdeplot(new_merchant_transactions[new_merchant_transactions['category_3'] == 'UKWN']['purchase_amount'],\n                 shade=True, color=\"r\", label='Category 3: UKWN')\nax.set_xlabel('Purchase Amount')\nax.set_title('Distribution Plot (Purchase Amount-Category 3: A, UKWN)')\nax.grid()\nax.legend()\n\nax = fig.add_subplot(212)\nax = sns.kdeplot(new_merchant_transactions[new_merchant_transactions['category_3'] == 'B']['purchase_amount'],\n                 shade=True, color=\"b\", label='Category 3: B')\nax = sns.kdeplot(new_merchant_transactions[new_merchant_transactions['category_3'] == 'C']['purchase_amount'],\n                 shade=True, color=\"r\", label='Category 3: C')\nax.set_xlabel('Purchase Amount')\nax.set_title('Distribution Plot (Purchase Amount-Category 3: B, C)')\nax.grid()\nax.legend()\n\nplt.tight_layout()\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"trusted":true,"_uuid":"e618e7a7a4b4de674f1390d62bdb4435d306dd63"},"cell_type":"code","source":"fig = plt.figure(figsize=(15,8))\nax = fig.add_subplot(111)\nsns.boxplot(x=\"subsector_id\", y=\"purchase_amount\", data=new_merchant_transactions,\n                showfliers=False)\n\nax.set_xlabel('Subsector ID')\nax.set_ylabel('Purchase Amount')\nax.set_title('Box Plot (Purchase Amount vs Subsectors)')\n\nplt.tight_layout()\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"trusted":true,"_uuid":"ae0617af4db6c00570b5dfa1f0e18177a3ed2c04"},"cell_type":"code","source":"fig = plt.figure(figsize=(15,8))\nax = fig.add_subplot(111)\nsns.boxplot(x=\"state_id\", y=\"purchase_amount\", data=new_merchant_transactions,\n                showfliers=False)\n\nax.set_xlabel('State ID')\nax.set_ylabel('Purchase Amount')\nax.set_title('Box Plot (Purchase Amount vs States)')\n\nplt.tight_layout()\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"0303e5197a52671692e363e8cc6394191b596a8e"},"cell_type":"markdown","source":"Let us explore the correlation between the purchase amount and number of installments. First of all, we visualize that how the transactions are distributed across the installment counts. The pie chart showing the transaction counts for the number of installments is shown below. Intutively it seems that the count of installments should increase as the purchase amount increases. The box plot of purchase amount against the count of installments somehow verify this trend."},{"metadata":{"_kg_hide-input":true,"trusted":true,"_uuid":"50b06a5efb825bcf7612f5189375627e0a8b381e"},"cell_type":"code","source":"s = df.groupby(['installments'])['merchant_id'].count()\ndf1 = pd.DataFrame({'installments':s.index, 'transaction_count':s.values})\ndf1.iplot(kind='pie',labels='installments',values='transaction_count', \n         title='Transaction Counts by Installments')","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"trusted":true,"_uuid":"dcf0b8882842d269fc683e5a7ed261c050167a11"},"cell_type":"code","source":"fig = plt.figure(figsize=(15,8))\nax = fig.add_subplot(111)\nsns.boxplot(x=\"installments\", y=\"purchase_amount\", data=new_merchant_transactions,\n                showfliers=False)\n\nax.set_xlabel('Installment Counts')\nax.set_ylabel('Purchase Amount')\nax.set_title('Box Plot (Purchase Amount vs Installment Counts)')\n\nplt.tight_layout()\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"0f47c74b3974fc8de8b7328127c8a04df830ac0c"},"cell_type":"markdown","source":"#### Exploration of Transactions based on date :"},{"metadata":{"_uuid":"d350e412312a9b0ed00140127782ed893703fedd"},"cell_type":"markdown","source":"We can also explore the relation of purchase amount purchase date. Let us first convert the column purchase_date into proper date-time format. Purchase hour, date of the month and day of the week is extracted from it as well. There does not seem to exist any pattern in the purchase amount on the basis of date of the month. If we look at the plot for the purchase amount against the hour of the day, there exists a slight spike during the early hour of the day and a little dip during the later part of the night. With respect to the day of the week, apart from Sunday, the distribution seems uniform. Purchase amount dips slightly on Sunday."},{"metadata":{"_kg_hide-input":true,"trusted":true,"_uuid":"ecbc2079327617fe42de499cef24e67cdbf91192"},"cell_type":"code","source":"# Convert Purchase time to DateTime\nnew_merchant_transactions['purchase_date'] = pd.to_datetime(new_merchant_transactions['purchase_date'],\n                                                         format='%Y%m%d %H:%M:%S')\nnew_merchant_transactions['purchase_hour'] = new_merchant_transactions['purchase_date'].dt.hour\nnew_merchant_transactions['purchase_day'] = new_merchant_transactions['purchase_date'].dt.day_name()\nnew_merchant_transactions['purchase_date_only'] = new_merchant_transactions['purchase_date'].dt.day","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"trusted":true,"_uuid":"ad620a4a912eedc135cb4a745c91a98598526836"},"cell_type":"code","source":"fig = plt.figure(figsize=(15,8))\nax = fig.add_subplot(111)\nsns.boxplot(x=\"purchase_date_only\", y=\"purchase_amount\", data=new_merchant_transactions,\n                showfliers=False)\n\nax.set_xlabel('Date of the Month')\nax.set_ylabel('Purchase Amount')\nax.set_title('Box Plot (Purchase Amount vs Date of the month)')\n\nplt.tight_layout()\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"trusted":true,"_uuid":"327bdbc1fdc69f2f502aac0f9350bfa4fee36fc6"},"cell_type":"code","source":"fig = plt.figure(figsize=(15,8))\nax = fig.add_subplot(111)\nsns.boxplot(x=\"purchase_hour\", y=\"purchase_amount\", data=new_merchant_transactions,\n                showfliers=False)\n\nax.set_xlabel('Hour of the Day')\nax.set_ylabel('Purchase Amount')\nax.set_title('Box Plot (Purchase Amount vs Hour of the Day)')\n\nplt.tight_layout()\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"trusted":true,"_uuid":"53c6b3479191f5e5e1ec14bfaace75c888637948"},"cell_type":"code","source":"fig = plt.figure(figsize=(15,8))\nax = fig.add_subplot(111)\nsns.boxplot(x=\"purchase_day\", y=\"purchase_amount\", data=new_merchant_transactions,\n                showfliers=False, \n            order=['Monday', 'Tuesday', 'Wednesday', 'Thursday', 'Friday', 'Saturday', 'Sunday'])\n\nax.set_xlabel('Day of the Week')\nax.set_ylabel('Purchase Amount')\nax.set_title('Box Plot (Purchase Amount vs Day of the Week)')\n\nplt.tight_layout()\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"96eedfaad0dba0bf83e4d027894bc620b0798bd5"},"cell_type":"markdown","source":"### Combining Merchants and New Merchant Transactions :"},{"metadata":{"_uuid":"b368154c300699032e99f43ed5cd050782a6de67"},"cell_type":"markdown","source":"Let us merge the merchant information with the merchant new transcations to further examine the effects of features that are associated with differnt mnerchants. First of all, the plot of purchase amount vs revenue range and quantity of transaction range is analyzed. Purchase amount shows a slight uptick for revenue range B and C and a dip for D and E. This may result due to the fact that the merchants in the revenue range B and C are selling low margin frequent selling items (and hence a decent revenue with high amount of purchase amount).\n\nThe second box-plot shows the plot of purchase amount against the quantity of transaction. With E being the least quantity of transaction, the purchase amount associated with it has the highest mean and variance. This may happen due to the fact that the merchants in the range E for the quantity of transaction would have been selling higher value items and hence higher purchase amount.\n\nThe scatter plot of purchase amount vs the anonymized numerical measure 1 (distinguised based on revenue range and quantity of transaction) is shown below. The plot confirms our claim about the variability of numerical measure for different revenue and quantity of transaction range and it also confirms the variability of purchase amount amongst them."},{"metadata":{"_kg_hide-input":true,"trusted":true,"_uuid":"9d0422d121be4529e1a177df0414776074fc7ea8"},"cell_type":"code","source":"df1 = new_merchant_transactions[['merchant_id', 'card_id', 'category_3', 'purchase_amount', 'purchase_day', \n                                 'purchase_hour', 'purchase_date_only']]\ndf2 = merchants[['merchant_id', 'numerical_1', 'numerical_2', 'most_recent_sales_range', 'most_recent_purchases_range'\n                 ,'category_4']]\ndf = df1.merge(df2, on='merchant_id', how='left')\ndf.head()","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"trusted":true,"_uuid":"5e173543837d8e1e4dbf0d7bc1eec0d4dd1692cb"},"cell_type":"code","source":"fig = plt.figure(figsize=(15,8))\nax = fig.add_subplot(111)\nsns.boxplot(x=\"most_recent_sales_range\", y=\"purchase_amount\", data=df,\n                showfliers=False, \n            order=['A', 'B', 'C', 'D', 'E'])\n\nax.set_xlabel('Revenue Range (A > B > C > D > E)')\nax.set_ylabel('Purchase Amount')\nax.set_title('Box Plot (Purchase Amount vs Revenue Range)')\n\nplt.tight_layout()\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"trusted":true,"_uuid":"b3cd2138bc704aac6241630ebcac20f796a01f43"},"cell_type":"code","source":"fig = plt.figure(figsize=(15,8))\nax = fig.add_subplot(111)\nsns.boxplot(x=\"most_recent_purchases_range\", y=\"purchase_amount\", data=df,\n                showfliers=False, \n            order=['A', 'B', 'C', 'D', 'E'])\n\nax.set_xlabel('Quantity of Transaction (A > B > C > D > E)')\nax.set_ylabel('Purchase Amount')\nax.set_title('Box Plot (Purchase Amount vs Quantity of Transaction)')\n\nplt.tight_layout()\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"cce8dcccd2f653445065ea13f7e33e7633fd0e93"},"cell_type":"markdown","source":"#### Exploration of training data :"},{"metadata":{"_kg_hide-input":true,"trusted":true,"_uuid":"27b5299bb8203eae65afd1b0ab814d8a7bd490cc"},"cell_type":"code","source":"train = pd.read_csv(\"../input/train.csv\")\ntrain.describe()","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"trusted":true,"_uuid":"c9c349a481340437e0af61969042ff3ef77e6256"},"cell_type":"code","source":"train.head()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"7484ca9577a6a4e27ea7845523fd9b9fb17cb194"},"cell_type":"markdown","source":"Let us examine the distribution of target variable first. It almost follows a normal distribution with certain outliers."},{"metadata":{"_kg_hide-input":true,"trusted":true,"_uuid":"37389111616f9707113de47308c44867cebbbb1c"},"cell_type":"code","source":"fig = plt.figure(figsize=(15,8))\nax = fig.add_subplot(211)\n\nax = sns.kdeplot(train['target'], shade=True, color=\"b\")\nax.set_xlabel('Target')\nax.set_title('Distribution Plot (Target)')\nax.grid()\n\n\nq = train[\"target\"].quantile(0.99)\ndf = train[train[\"target\"] < q]\nax = fig.add_subplot(212)\n\nax = sns.kdeplot(df['target'], shade=True, color=\"b\")\nax.set_xlabel('Target')\nax.set_title('Distribution Plot (Target after removing outliers)')\nax.grid()\n\nplt.tight_layout()\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"e71486fc983b59eed64586c420a186a0f0fcce83"},"cell_type":"markdown","source":"The individual counts of samples for different features is shown in the below pie charts. The distribution of target variable for different features is shown in the following figure. There does not seem to be any discernable pattern amongst them as well. While trying to further breakdown the distribution of target based on various combinations of Feature 1 and 2 (color-coded by Feature 3), no pattern is found. One interesting insight that is extricated while analysing this is: <b>the training data has single value of Feature 3 (either 0 or 1) for each combination of Feature 1 and 2.</b> Let us look at the test data to see that the whether the same trend exist in it or not. The count plot for test data is analyzed and it is found that it follows the same trend."},{"metadata":{"_kg_hide-input":true,"trusted":true,"_uuid":"72a1993b2e9aa4dcf5907c6846e199af9abe26e2"},"cell_type":"code","source":"s = train.groupby(['feature_1'])['card_id'].count()\ndf = pd.DataFrame({'feature_1':s.index, 'count':s.values})\ndf.iplot(kind='pie',labels='feature_1',values='count', \n         title='Count for Feature 1')","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"trusted":true,"_uuid":"ea2c5c789a77b405b9e4beb64e07a7dfdcf02f16"},"cell_type":"code","source":"s = train.groupby(['feature_2'])['card_id'].count()\ndf = pd.DataFrame({'feature_2':s.index, 'count':s.values})\ndf.iplot(kind='pie',labels='feature_2',values='count', \n         title='Count for Feature 2')","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"trusted":true,"_uuid":"44ab2edcda288ddf7029d9039213fa283bf9a8ab"},"cell_type":"code","source":"s = train.groupby(['feature_3'])['card_id'].count()\ndf = pd.DataFrame({'feature_3':s.index, 'count':s.values})\ndf.iplot(kind='pie',labels='feature_3',values='count', \n         title='Count for Feature 3')","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"trusted":true,"_uuid":"290fd7d9c14beeae5f48a8fa6b1b7170ad5dbe3c"},"cell_type":"code","source":"fig = plt.figure(figsize=(15,24))\n\nax = fig.add_subplot(311)\nax = sns.violinplot(x=\"feature_1\", y=\"target\", data=train, palette=\"muted\")\nax.set_xlabel('Feature 1')\nax.set_title('Distribution (Target vs Feature 1)')\nax.grid()\n\nax = fig.add_subplot(312)\nax = sns.violinplot(x=\"feature_2\", y=\"target\", data=train, palette=\"muted\")\nax.set_xlabel('Feature 2')\nax.set_title('Distribution (Target vs Feature 2)')\nax.grid()\n\nax = fig.add_subplot(313)\nax = sns.violinplot(x=\"feature_3\", y=\"target\", data=train, palette=\"muted\")\nax.set_xlabel('Feature 3')\nax.set_title('Distribution (Target vs Feature 3)')\nax.grid()\n\nplt.tight_layout()\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"trusted":true,"_uuid":"0d39a1b07cde5b36da565c1724e9902ddbe71065"},"cell_type":"code","source":"feature_1 = [1,2,3,4,5]\nfeature_2 = [1,2,3]\n\nrows = ['Feature 1: {}'.format(row) for row in feature_1]\ncols = ['Feature 2: {}'.format(col) for col in feature_2]\n\n\nfig = plt.figure(figsize=(15,24))\n\nfor f1 in feature_1:\n    for f2 in feature_2:\n        df = train[(train['feature_1'] == f1) & (train['feature_2'] == f2)]\n        if(len(df) > 0):\n            ax = fig.add_subplot(5,3,3*(f1-1)+f2)\n            ax = sns.violinplot(x=\"feature_3\", y=\"target\", data=df, palette=\"muted\")\n            ax.set_xlabel('Feature 3')\n            ax.set_title(cols[f2-1])\n            ax.set_ylabel(rows[f1-1])\n            ax.grid()\n\nplt.tight_layout()\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"trusted":true,"_uuid":"2c83df9a8386623112873dde58c06850c17741d7"},"cell_type":"code","source":"test = pd.read_csv(\"../input/test.csv\")\n\nfeature_1 = [1,2,3,4,5]\nfeature_2 = [1,2,3]\n\nrows = ['Feature 1: {}'.format(row) for row in feature_1]\ncols = ['Feature 2: {}'.format(col) for col in feature_2]\n\n\nfig = plt.figure(figsize=(15,24))\n\nfor f1 in feature_1:\n    for f2 in feature_2:\n        df = test[(test['feature_1'] == f1) & (test['feature_2'] == f2)]\n        if(len(df) > 0):\n            ax = fig.add_subplot(5,3,3*(f1-1)+f2)\n            ax = sns.countplot(x=\"feature_3\", data=df, palette=\"muted\")\n            ax.set_xlabel('Feature 3')\n            ax.set_title(cols[f2-1])\n            ax.set_ylabel(rows[f1-1])\n            ax.grid()\n\nplt.tight_layout()\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"4581c029dcd54baf746589d35a42aeeea61c5862"},"cell_type":"markdown","source":"For further exploration, distribution of target for first active month and year is analyzed. The distribution shows no discerning trend on the basis of month. But for the year, there exist some pattern in the distribution."},{"metadata":{"_kg_hide-input":true,"trusted":true,"_uuid":"13cc4b4897c345f93b8c09dc3ed61accd68ade6f"},"cell_type":"code","source":"# Convert first active month into datetime and extract year and month\ntrain['first_active_month'] = pd.to_datetime(train['first_active_month'], format='%Y-%m')\ntrain['first_active_year'] = train['first_active_month'].dt.year\ntrain['first_active_month'] = train['first_active_month'].dt.month\ntrain.head()","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"trusted":true,"_uuid":"1e01d8739ea401adcca1ccaa06d83f4ed270b2a4"},"cell_type":"code","source":"fig = plt.figure(figsize=(15,24))\n\nax = fig.add_subplot(311)\nax = sns.violinplot(x=\"first_active_year\", y=\"target\", data=train, palette=\"muted\")\nax.set_xlabel('First Active Year')\nax.set_title('Distribution (Target vs First active year)')\nax.grid()","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"trusted":true,"_uuid":"f6119c2c054e0a5991c2f6d74221804368e36d97"},"cell_type":"code","source":"fig = plt.figure(figsize=(15,24))\n\nax = fig.add_subplot(311)\nax = sns.violinplot(x=\"first_active_month\", y=\"target\", data=train, palette=\"muted\")\nax.set_xlabel('First Active Month')\nax.set_title('Distribution (Target vs First active month)')\nax.grid()","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"_uuid":"dedf574664fd3d499099c1f84596955e2efa39c6"},"cell_type":"markdown","source":"#### Decision Tree Regressor :\n\nA simple decision tree regressor based on training data is implemented (no historical transactions and merchants data is used). \n\n<b>Score: 3.921</b>\n\nParameters: {'max_depth': 5, 'max_features': 0.9}\n\nWe need to try different models after doing some sort of feature engineering to improve the score."},{"metadata":{"_kg_hide-input":true,"trusted":true,"_uuid":"6cbf3628727c5bf6408bc7a8c8b5b80f0ee36bfe"},"cell_type":"code","source":"train = train.join(pd.get_dummies(train['feature_1'], prefix='Feature_1'))\ntrain = train.drop('feature_1', axis=1)\ntrain = train.join(pd.get_dummies(train['feature_2'], prefix='Feature_2'))\ntrain = train.drop('feature_2', axis=1)\ntrain = train.join(pd.get_dummies(train['feature_3'], prefix='Feature_3'))\ntrain = train.drop('feature_3', axis=1)\ntrain = train.join(pd.get_dummies(train['first_active_month'], prefix='first_active_month'))\ntrain = train.drop('first_active_month', axis=1)\ntrain = train.join(pd.get_dummies(train['first_active_year'], prefix='first_active_year'))\ntrain = train.drop('first_active_year', axis=1)\nX_train = train.drop(['card_id', 'target'], axis=1)\ny_train = train['target']","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"trusted":true,"_uuid":"34b25f32b6a2618a69f46650cf76fb3ee6854577"},"cell_type":"code","source":"# Decision Tree Regressor\nfrom sklearn import tree\nfrom sklearn.model_selection import GridSearchCV\nfrom sklearn.metrics import mean_squared_error, make_scorer\nfrom sklearn.model_selection import cross_val_score\nfrom sklearn.tree import DecisionTreeRegressor\n\nparameters = {'max_depth':range(1,30),\n             'max_features': [0.5, 0.6, 0.7, 0.8, 0.9]}\n\n# Best Model\n# {'max_depth': 5, 'max_features': 0.9}\nparameters_best_model = {'max_depth':[5],\n             'max_features': [0.9]}\nmse_scorer = make_scorer(mean_squared_error, greater_is_better=False) \n\nclf = GridSearchCV(tree.DecisionTreeRegressor(random_state=1), parameters_best_model, n_jobs=4, cv=10, \n                   scoring=mse_scorer)\nclf.fit(X=X_train, y=y_train)\ntree_model = clf.best_estimator_\nprint (clf.best_score_, clf.best_params_) ","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"trusted":true,"_uuid":"32162b188b97b0a2b1c839f63dd12405b985717d"},"cell_type":"code","source":"# Preprocessing of test data\n# Convert first active month into datetime and extract year and month\ntest['first_active_month'] = test['first_active_month'].fillna('2018-01')\ntest['first_active_month'] = pd.to_datetime(test['first_active_month'], format='%Y-%m')\ntest['first_active_year'] = test['first_active_month'].dt.year\ntest['first_active_month'] = test['first_active_month'].dt.month.astype(int)\ntest.head()","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"trusted":true,"_uuid":"38c1a964104699f15da20d67424ac12afc5ce3ad"},"cell_type":"code","source":"# Preprocessing of test data\ntest = test.join(pd.get_dummies(test['feature_1'], prefix='Feature_1'))\ntest = test.drop('feature_1', axis=1)\ntest = test.join(pd.get_dummies(test['feature_2'], prefix='Feature_2'))\ntest = test.drop('feature_2', axis=1)\ntest = test.join(pd.get_dummies(test['feature_3'], prefix='Feature_3'))\ntest = test.drop('feature_3', axis=1)\ntest = test.join(pd.get_dummies(test['first_active_month'], prefix='first_active_month'))\ntest = test.drop('first_active_month', axis=1)\ntest = test.join(pd.get_dummies(test['first_active_year'], prefix='first_active_year'))\ntest = test.drop('first_active_year', axis=1)\ntest.head()","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"trusted":true,"_uuid":"2dc682d645634c7b5c5dc8b1037231df411cf5d7"},"cell_type":"code","source":"X_test = test.drop(['card_id'], axis=1)\np = tree_model.predict(X_test)\n# Submission File\nsub = pd.read_csv('../input/sample_submission.csv')\nsub[\"target\"] = p\nsub.to_csv(\"submission_decisiontree.csv\", index=False)","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"_uuid":"463442869a966d51ac3210bb1066ee742048b120"},"cell_type":"markdown","source":"#### LGBM :\n\n<b>Score: 3.922</b>\n\nParameters: {'bagging_fraction': 0.8, 'bagging_freq': 2, 'feature_fraction': 0.9, 'lambda_l1': 0.2, 'learning_rate': 0.01}[](http://)"},{"metadata":{"_kg_hide-input":true,"trusted":true,"_uuid":"694a07bcd3bc9edeaf72d891397295b27f8ed8c2"},"cell_type":"code","source":"import lightgbm as lgb\nfrom lightgbm.sklearn import LGBMRegressor\n\nlgbm_parameters = {'learning_rate': [0.1, 0.01, 0.005],\n                   'bagging_freq': [1, 2],\n                   'bagging_fraction': [0.8, 0.9],\n                   'feature_fraction': [0.8, 0.9],\n                   'lambda_l1' : [0.1, 0.2]}\n\n# Best Model\n# {'bagging_fraction': 0.8, 'bagging_freq': 2, 'feature_fraction': 0.9, 'lambda_l1': 0.2, 'learning_rate': 0.01}\nlgbm_parameters_best_model = {'bagging_fraction': [0.8], 'bagging_freq': [2], 'feature_fraction': [0.9],\n                              'lambda_l1': [0.2], 'learning_rate': [0.01]}\nreg = GridSearchCV(LGBMRegressor(objective='regression', boosting_type='gbdt', metric='rmse', random_state=1),\n                   lgbm_parameters_best_model, n_jobs=1, cv=10)\nreg.fit(X=X_train, y=y_train)\nmodel = reg.best_estimator_\nprint (reg.best_score_, reg.best_params_) ","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"trusted":true,"_uuid":"952eb42500fea10285566602b9f423de6b3fa7bb"},"cell_type":"code","source":"# Submission File\np = model.predict(X_test)\nsub = pd.read_csv('../input/sample_submission.csv')\nsub[\"target\"] = p\nsub.to_csv(\"submission_lgbm.csv\", index=False)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"b627b591b34eabafab1bea20125f71353fbab0ac"},"cell_type":"markdown","source":"#### XGBoost :\n\n<b>Score: 3.923</b>\n\nParameters: {'colsample_bytree': 0.7, 'eta': 0.1, 'max_depth': 6, 'subsample': 1.0}****"},{"metadata":{"_kg_hide-input":true,"trusted":true,"_uuid":"da9d9b107ee21e4aa718d32f0f843e496f7b8028"},"cell_type":"code","source":"import xgboost as xgb\nfrom xgboost.sklearn import XGBRegressor\n\nxgboost_parameters = {'eta': [0.1, 0.01, 0.05], \n                      'max_depth': [6, 7, 8], \n                      'subsample': [0.8, 1.0],\n                      'colsample_bytree': [0.7, 0.8]}\n\n# Best model parameters\n# {'colsample_bytree': 0.7, 'eta': 0.1, 'max_depth': 6, 'subsample': 1.0}\nxgboost_parameters_best_model = {'colsample_bytree': [0.7], 'eta': [0.1], 'max_depth': [6], 'subsample': [1.0]}\nxgboost_reg = GridSearchCV(XGBRegressor(objective='reg:linear', booster='gbtree', eval_metric='rmse', random_state=1),\n                   xgboost_parameters_best_model, n_jobs=10, cv=5)\nxgboost_reg.fit(X=X_train, y=y_train)\nmodel = xgboost_reg.best_estimator_\nprint (xgboost_reg.best_score_, xgboost_reg.best_params_)","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"trusted":true,"_uuid":"61ccc47accec10a7f9fc58aef9c366c1c9cb73a5"},"cell_type":"code","source":"# Submission File\np = model.predict(X_test)\nsub = pd.read_csv('../input/sample_submission.csv')\nsub[\"target\"] = p\nsub.to_csv(\"submission_xgboost.csv\", index=False)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"f0fe5b3b1f83142f9e9e545c22c84416da359e35"},"cell_type":"markdown","source":"**Any suggestion is much appreciated!!!**"},{"metadata":{"_uuid":"041f762196b486204477e9315a8e85080b9e647a"},"cell_type":"markdown","source":"Reference:\n\nhttps://www.kaggle.com/samaxtech/eda-clean-feng-lgbm-xgboost-stacked-model"},{"metadata":{"_uuid":"b69b4d5e05ad16ba101aa5f84c7a476d6d78e8d0"},"cell_type":"markdown","source":""},{"metadata":{"_uuid":"e116ac3855ca3e85886e37b4a360fcea074035f5"},"cell_type":"markdown","source":""}],"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"name":"python","version":"3.6.6","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"}},"nbformat":4,"nbformat_minor":1}