{"cells":[{"metadata":{"_uuid":"e8e112aa5b5db7303e39d3a9cf1702848962c6c2"},"cell_type":"markdown","source":"## Introduction\n\n#### Analysis feature on this competitions \n* Analysis all feature that used and do data visualize\n* Hep to find the key features on this competitions"},{"metadata":{"_uuid":"34dd77a7bf27124a6ec7b6bb05e19967771629ef"},"cell_type":"markdown","source":"## Training  kernel ==> https://www.kaggle.com/super13579/revisit-one-hot-ranking"},{"metadata":{"_uuid":"bf938f8069d8698670d3633c4bb28d3fabce0bb3"},"cell_type":"markdown","source":"### Version 1 ( LB = 1.5057)\n* Process Date feature (add day, week, hour, \"revisit time\")\n* Totals feature don't do anything, only fillna(0)\n* Do one hot encoding for categorical feature that have unique value <15\n* Do ranking encoding for categorical feature that have unique value >15"},{"metadata":{"_uuid":"53f67e514822a666c5b6a5829bc95857296d84c5"},"cell_type":"markdown","source":"**Data Fields**\n* fullVisitorId- A unique identifier for each user of the Google Merchandise Store.\n* channelGrouping - The channel via which the user came to the Store.\n* date - The date on which the user visited the Store.\n* device - The specifications for the device used to access the Store.\n* geoNetwork - This section contains information about the geography of the user.\n* sessionId - A unique identifier for this visit to the store.\n* socialEngagementType - Engagement type, either \"Socially Engaged\" or \"Not Socially Engaged\".\n* totals - This section contains aggregate values across the session.\n* trafficSource - This section contains information about the Traffic Source from which the session originated.\n* visitId - An identifier for this session. This is part of the value usually stored as the _utmb cookie. This is only unique to the user. For a completely unique ID, we should use a combination of fullVisitorId and visitId.\n* visitNumber - The session number for this user. If this is the first session, then this is set to 1.\n* visitStartTime - The timestamp (expressed as POSIX time)."},{"metadata":{"_uuid":"bf88f05a8719a3913b8c282dcfecfeaa27c7390a"},"cell_type":"markdown","source":"## Load data\n* import module that data analysis need to used\n* load original data to see feature and data type\n\n★There are some feature have json object"},{"metadata":{"_uuid":"1e6d855c53be752ddaf290854843cd33b294e1e1","trusted":true,"_kg_hide-output":true},"cell_type":"code","source":"import os\nprint(os.listdir(\"../input\"))\nimport numpy as np \nimport pandas as pd \nimport json\nimport bq_helper\nfrom pandas.io.json import json_normalize\nimport warnings\nimport gc\ngc.enable()\nwarnings.filterwarnings(\"ignore\")\n","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"0426a539e4235a6459bddb0567863cfc294b0215"},"cell_type":"markdown","source":"## Flatten feature with json object\n* There are json objects in some features.\n* We used this function written by julian in his kernel to flat these features.\n\nhttps://www.kaggle.com/julian3833/1-quick-start-read-csv-and-flatten-json-fields/notebook."},{"metadata":{"trusted":true,"_uuid":"75b4763d19e53ae2e16ca73676ddfbcbbfd43bb1"},"cell_type":"code","source":"json_columns = ['device', 'geoNetwork','totals', 'trafficSource']\ndef load_dataframe(filename):\n    path = \"../input/\" + filename\n    df = pd.read_csv(path, converters={column: json.loads for column in json_columns}, \n                     dtype={'fullVisitorId': 'str'})\n    \n    for column in json_columns:\n        column_as_df = json_normalize(df[column])\n        column_as_df.columns = [f\"{column}_{subcolumn}\" for subcolumn in column_as_df.columns]\n        df = df.drop(column, axis=1).merge(column_as_df, right_index=True, left_index=True)\n    return df","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"10013c07db2eb5066cece44c1e72fba7ba379ce2"},"cell_type":"markdown","source":"### Show the flatten data"},{"metadata":{"trusted":true,"_uuid":"7c3d7025140c8590f60841e6077a1ecc792ab547","_kg_hide-output":false,"_kg_hide-input":true},"cell_type":"code","source":"train = load_dataframe(\"ga-customer-revenue-prediction/train.csv\")\nprint(train.shape)\ntrain.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"587d5782ace07133bd05e57ad08433ff1d327213","_kg_hide-output":false,"_kg_hide-input":true},"cell_type":"code","source":"test = load_dataframe(\"ga-customer-revenue-prediction/test.csv\")\nprint(test.shape)\ntest.head()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"4991f38a69b6f37d4b4d8590fa0022abbbccfc7e"},"cell_type":"markdown","source":"## Align test and train features\n* We found train set have 2 more feature than test set\n* 1 is target featrue, find another feature that not include in test set and remove it on train set"},{"metadata":{"trusted":true,"_uuid":"e016306c6823674a1b9f12396f11e0e32850b992","_kg_hide-input":true},"cell_type":"code","source":"not_in_test_feature = [i for i in train.columns if i not in test.columns]\nprint (not_in_test_feature)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"bee2b1e154bd5d5f2e394a3a23eec8a14bcbf588","_kg_hide-input":true},"cell_type":"code","source":"train = train.drop(['trafficSource_campaignCode'], axis = 1)\ntrain.shape,test.shape","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"fe8b06c96d7c8364f3933466e5910767bd0de51d"},"cell_type":"markdown","source":"## Analysis unique count of each feature and remove only one unique feature\n* There are some features that contains only one value \"not available in demo dataset\" as constant values\n* constant value dont have any effective on model prdictions, let's remove it."},{"metadata":{"_uuid":"4f60d65fdf84dfdcae264cb9b3d990f9c5347877"},"cell_type":"markdown","source":"* Train Set"},{"metadata":{"trusted":true,"_uuid":"929d4641a49f4837a8eb9a3925fbf33f53c32a27","_kg_hide-input":false},"cell_type":"code","source":"#import module of matplot\nimport matplotlib.pyplot as plt\nimport seaborn as sns \nfrom plotly.offline import init_notebook_mode, iplot\nimport plotly.graph_objs as go\nfrom plotly import tools\nimport matplotlib as mpl\nimport matplotlib.pyplot as plt\nimport scipy.stats as st\n%matplotlib inline","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"770d8ece057a1f551eab95ae7f51ed5ace972fa8","_kg_hide-output":true,"_kg_hide-input":false},"cell_type":"code","source":"#analysis count of each features\nunique_count_train = []\nfor i in train.columns:\n    unique_count_train.append(len(train[str(i)].value_counts()))\nunique_datafram_train = {\n    \"Feature\":train.columns,\n    \"unique_count\":unique_count_train\n}\nunique_train = pd.DataFrame(unique_datafram_train)\nunique_train\n#=========================or you can use unique to do this =================================#\n#train.nunique()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"8b47454a8002884befff9f2e069a52d1f8b8fb0a"},"cell_type":"markdown","source":"### Remove constant features"},{"metadata":{"trusted":true,"_uuid":"947fe3649e06d585e11b113271ad12c1f323b5c3"},"cell_type":"code","source":"const_columns_train = unique_train.loc[unique_train['unique_count'] == 1,\"Feature\"]\nconst_columns_train = list(const_columns_train)\n#=========================or you can use unique to do this =================================#\n#const_columns_train = [col for col in train.columns if train[col].nunique() == 1]\nprint (\"Shape before remove constant feature on train set = \" + str(train.shape))\ntrain = train.drop(const_columns_train,axis = 1)\nprint (\"Shape after remove constant feature on train set = \" + str(train.shape))","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"f9a8cbaccc3b1946319a05c746bd173125c9047c"},"cell_type":"markdown","source":"* Test Set"},{"metadata":{"trusted":true,"_uuid":"5ce0e3321f0f16ab89c1fcd9a4fd6868c415ac30","_kg_hide-input":false,"_kg_hide-output":true},"cell_type":"code","source":"#analysis count of each features\nunique_count_test = []\nfor i in test.columns:\n    unique_count_test.append(len(test[str(i)].value_counts()))\nunique_datafram_test = {\n    \"Feature\":test.columns,\n    \"unique_count\":unique_count_test\n}\nunique_test = pd.DataFrame(unique_datafram_test)\nunique_test\n#=========================or you can use unique to do this =================================#\n#test.nunique()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"de9d0f0c5a6e8204c14d4aaf1072af2f55dbb5a4"},"cell_type":"code","source":"const_columns_test = unique_test.loc[unique_test['unique_count'] == 1,\"Feature\"]\nconst_columns_test = list(const_columns_test)\n#=========================or you can use unique to do this =================================#\n#const_columns_test = [col for col in test.columns if test[col].nunique() == 1]\nprint (\"Shape before remove constant feature on train set = \" + str(test.shape))\ntest = test.drop(const_columns_test,axis = 1)\nprint (\"Shape after remove constant feature on train set = \" + str(test.shape))\n","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"fb6d69cc03bbf2a7426b8ede2cc70df30f519ce6"},"cell_type":"markdown","source":"## Analysis Date feature\n* Extracting new feature from date\n* Analysis new feature "},{"metadata":{"trusted":true,"_uuid":"8ae1d1accdbf5451e4ed874d4dd78b215e5bad27"},"cell_type":"code","source":"def extract_new_feature(df): \n    print(\"Start extract date...\")\n    df['date'] = pd.to_datetime(df['visitStartTime'], unit='s')\n    df['day_of_week'] = df['date'].dt.dayofweek\n    df['hour'] = df['date'].dt.hour\n    df['day'] = df['date'].dt.day\n    df['month'] = df['date'].dt.month\n    print(\"Finished extract date...\")\nextract_new_feature(train)\nextract_new_feature(test)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"46bd2b8f838b8529ae3aefd19b393aaf0d237833"},"cell_type":"markdown","source":"### Plot Visit of extracted date feature"},{"metadata":{"_kg_hide-input":true,"_kg_hide-output":false,"trusted":true,"_uuid":"1a43793dc83032bb6ff75de14f5e07253e6d7508"},"cell_type":"code","source":"t1 = train['day_of_week'].value_counts()\nt2 = train['hour'].value_counts()\nt3 = train['day'].value_counts()\nt4 = train['month'].value_counts()\n\nplt.subplots(figsize=(15,10))\nplt.subplot(2,2,1)\nplt.title('Visits by day of week',color='b',fontsize=12)\nplt.bar(t1.index,t1.values,color='orange')\nplt.xticks(t1.index,rotation='horizontal',fontsize=12)\nplt.yticks(fontsize=12)\n\nplt.subplot(2,2,2)\nplt.title('Visits by hour',color='b',fontsize=12)\nplt.bar(t2.index,t2.values,color='pink')\nplt.xticks(t2.index,rotation='horizontal',fontsize=10)\nplt.yticks(fontsize=12)\n\nplt.subplot(2,2,3)\nplt.title('Visits by day',color='b',fontsize=12)\nplt.bar(t3.index,t3.values,color='blue')\nplt.xticks(t3.index,rotation='horizontal',fontsize=10)\nplt.yticks(fontsize=12)\n\nplt.subplot(2,2,4)\nplt.title('Visits by month',color='b',fontsize=12)\nplt.bar(t4.index,t4.values,color='purple')\nplt.xticks(t4.index,rotation='horizontal',fontsize=12)\nplt.yticks(fontsize=12)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"245b01fc172147fd4aae7c8348cf98867b015ff1"},"cell_type":"markdown","source":"### Plot total trascations revenue of extracted date feature"},{"metadata":{"trusted":true,"_uuid":"49985833c0d052b58c6de143b7b091b731fc8141","scrolled":true,"_kg_hide-input":true},"cell_type":"code","source":"train['totals_transactionRevenue'] = train['totals_transactionRevenue'].astype('float64')\na = train[['day_of_week','totals_transactionRevenue']].groupby('day_of_week').mean().reset_index()\na1 = train[['hour','totals_transactionRevenue']].groupby('hour').mean().reset_index()\na2 = train[['day','totals_transactionRevenue']].groupby('day').mean().reset_index()\na3 = train[['month','totals_transactionRevenue']].groupby('month').mean().reset_index()\n\nplt.subplots(figsize=(15,10))\nplt.subplot(2,2,1)\nplt.title('Total transactionRevenue mean by day of week',color='b',fontsize=12)\nplt.bar(a['day_of_week'],a['totals_transactionRevenue'],color='orange')\nplt.xticks(a['day_of_week'],rotation='horizontal',fontsize=12)\nplt.yticks(fontsize=12)\n\nplt.subplot(2,2,2)\nplt.title('Total transactionRevenue mean by hour',color='b',fontsize=12)\nplt.bar(a1['hour'],a1['totals_transactionRevenue'],color='pink')\nplt.xticks(a1['hour'],rotation='horizontal',fontsize=10)\nplt.yticks(fontsize=12)\n\nplt.subplot(2,2,3)\nplt.title('Total transactionRevenue mean by day',color='b',fontsize=12)\nplt.bar(a2['day'],a2['totals_transactionRevenue'],color='blue')\nplt.xticks(a2['day'],rotation='horizontal',fontsize=10)\nplt.yticks(fontsize=12)\n\nplt.subplot(2,2,4)\nplt.title('Total transactionRevenue mean by month',color='b',fontsize=12)\nplt.bar(a3['month'],a3['totals_transactionRevenue'],color='purple')\nplt.xticks(a3['month'],rotation='horizontal',fontsize=12)\nplt.yticks(fontsize=12)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"1728a08b33263512b79e55405304db0df6718552"},"cell_type":"markdown","source":"### Find the time period of same ID revisit \nidea comes from Aahish kernel :  https://www.kaggle.com/ashishpatel26/future-is-here"},{"metadata":{"trusted":true,"_uuid":"3a8f2da50b9a939285050b53f878696213475626"},"cell_type":"code","source":"def add_time_period_of_same_ID(df): \n    print(\"Start add time period feature...\")\n    df.sort_values(['fullVisitorId', 'date'], ascending=True, inplace=True)\n    df['next_revisit_time'] = (\n        df['date'] - df[['fullVisitorId', 'date']].groupby('fullVisitorId')['date'].shift(1)\n    ).astype(np.int64) // 1e9 // 60 // 60\n    df['prev_revisit_time'] = (\n        df['date'] - df[['fullVisitorId', 'date']].groupby('fullVisitorId')['date'].shift(-1)\n    ).astype(np.int64) // 1e9 // 60 // 60\n    print(\"Finished time periodfeature...\")\nadd_time_period_of_same_ID(train)\nadd_time_period_of_same_ID(test)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"f9127618115ffa8915c95de46d6c590efe223ba2"},"cell_type":"markdown","source":"* Plot the epriod time that revisit cause totals_transactionRevenue\n* We found that the totals_transactionRevenue are locate at small revisit time portion\n### We keep the feature of next and previous revisit time"},{"metadata":{"trusted":true,"_uuid":"071ef65a43847e1773b663b5fed4f796bd7589cf","_kg_hide-input":true},"cell_type":"code","source":"b = train[['next_revisit_time','totals_transactionRevenue']].groupby('next_revisit_time').mean().reset_index()\nb[b['next_revisit_time']== -2562048] = 0 #Process no revisit ID to 0\nb.sort_values(by = \"next_revisit_time\",ascending=False).fillna(0)\nb2 = train[['prev_revisit_time','totals_transactionRevenue']].groupby('prev_revisit_time').mean().reset_index()\nb2[b2['prev_revisit_time']== -2562048] = 0\nb2.sort_values(by = \"prev_revisit_time\",ascending=False).fillna(0)\n\nplt.subplots(figsize=(15,10))\nplt.subplot(2,1,1)\nplt.title('Total transactionRevenue mean by sort of next revisit time',color='b',fontsize=12)\nplt.scatter(b['next_revisit_time'],b['totals_transactionRevenue'],color='orange')\n#plt.xticks(b.index,rotation='horizontal',fontsize=12)\nplt.yticks(fontsize=12)\n\nplt.subplot(2,1,2)\nplt.title('Total transactionRevenue mean by sort of previous revisit time',color='b',fontsize=12)\nplt.scatter(b2['prev_revisit_time'],b2['totals_transactionRevenue'],color='orange')\n#plt.xticks(b.index,rotation='horizontal',fontsize=12)\nplt.yticks(fontsize=12)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"f136cf03fb42ad1e54cac8049decadf97045f1e5"},"cell_type":"markdown","source":"## Analysis Visit count and Revenue relation\n* Result show the ditribution have no any relation with Revenue, No need to add this feature"},{"metadata":{"trusted":true,"_uuid":"25fc00d7e2cbcf2abf8199e024709cdb09f28c7a","_kg_hide-input":true},"cell_type":"code","source":"c = train[['fullVisitorId','totals_transactionRevenue']]\nc['counts'] = c['fullVisitorId']\nc = c.groupby(['fullVisitorId']).agg({'totals_transactionRevenue':'mean','counts':'count'}).reset_index()\nc = c.sort_values(by='counts').fillna(0).groupby('counts').mean().reset_index()\nlines = c.plot.scatter(x='counts', y='totals_transactionRevenue',figsize=(15,5))","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"97da33d0a0295db5ef1e7a53868dee3371777e64"},"cell_type":"markdown","source":"## Find the categorical feature and process data \n* We can see there are missing value  \"NaN\" on the data\n* Find the categorical feature and visulize the data to analysis\n* Analysis Unique count < 15, and unique count>15"},{"metadata":{"_uuid":"e3e0f0e21893598d11e583edbb0bb32cd54015d8"},"cell_type":"markdown","source":"### Find the categorical feature"},{"metadata":{"trusted":true,"_uuid":"058d959d837e49e6a7fedee4dfd298d1b2502320"},"cell_type":"code","source":"categorical_features_train = train.select_dtypes(include=[np.object])\ncategorical_features_test = test.select_dtypes(include=[np.object])\ncategorical_features_train.columns","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"68c73fd40909018d2fd92302f072892d9da0bad6"},"cell_type":"markdown","source":"### Analysis missing values for all categorical features "},{"metadata":{"_uuid":"60c05416d5a6c9e7993371a62abdb6881055719e"},"cell_type":"markdown","source":"* Training Set"},{"metadata":{"trusted":true,"_uuid":"e6d0e5002a014f46ba41c896af0edb4ca8f61b87"},"cell_type":"code","source":"Nan_value_train = categorical_features_train.isnull().sum().sort_values(ascending=False)\nPercent_Nan_value_train = Nan_value_train/categorical_features_train.shape[0]\nNan_train_data = pd.concat([Nan_value_train,Percent_Nan_value_train], axis= 1,keys = ['Missing value count','Percent'])\nNan_train_data.head(10)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"bb0f5df494d21b2016d89e25611c76178c539a8a"},"cell_type":"markdown","source":"* Test Set"},{"metadata":{"trusted":true,"_uuid":"2635cfb594449858e5476bf2bfafc68883690844"},"cell_type":"code","source":"Nan_value_test = categorical_features_test.isnull().sum().sort_values(ascending=False)\nPercent_Nan_value_test = Nan_value_test/categorical_features_test.shape[0]\nNan_test_data = pd.concat([Nan_value_test,Percent_Nan_value_test], axis= 1,keys = ['Missing value count','Percent'])\nNan_test_data.head(10)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"0ca88b630db7d0365b8e48c05d8a2df604c57638"},"cell_type":"markdown","source":"## Analysis Total feature\n* Analysis total feature by bropuby full ID"},{"metadata":{"trusted":true,"_uuid":"0624a0d3641e056eaeecd8a518fa171f255b8fa0"},"cell_type":"code","source":"Totals_features = [i for i in train.columns if i.find(\"totals\")>=0 ]\nTotals_features","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"b35fbe747462769b9298889155fb08457c55bc34","_kg_hide-input":false},"cell_type":"code","source":"agg_dict = {}\nfor col in Totals_features:\n    train[col] = train[col].astype('float')\n    if col == 'totals_transactionRevenue':\n        agg_dict[col] = 'mean'\n    agg_dict[col] = \"sum\"\ntmp = train.groupby(\"fullVisitorId\").agg(agg_dict).reset_index()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"24df351eb43c1275c037099c4933fac8e43d6a82"},"cell_type":"markdown","source":"### Plot scatter to see the distribution\n* Looks the totals feature don't need to do any process, fillna(0)"},{"metadata":{"trusted":true,"_uuid":"6a55ed60b5ed1eecd7422f43a1a99ac2ec8222b1","_kg_hide-input":true},"cell_type":"code","source":"plt.subplots(figsize=(12,6))\nplt.subplot(1,2,1)\nplt.title('Revenue mean by totals hits',color='b',fontsize=12)\nplt.scatter(tmp['totals_hits'],tmp['totals_transactionRevenue'],color='b')\n#plt.xticks(b.index,rotation='horizontal',fontsize=12)\nplt.yticks(fontsize=12)\n\nplt.subplot(1,2,2)\nplt.title('Revenue mean by totals pageviews',color='b',fontsize=12)\nplt.scatter(tmp['totals_pageviews'],tmp['totals_transactionRevenue'],color='b')\n#plt.xticks(b.index,rotation='horizontal',fontsize=12)\nplt.yticks(fontsize=12)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"ffba6fb490d7dce6d3b0ef108cc443abf2be4c67"},"cell_type":"code","source":"train['totals_pageviews']=train['totals_pageviews'].astype('float64')\ntrain['totals_hits']=train['totals_hits'].astype('float64')\ntest['totals_pageviews']=test['totals_pageviews'].astype('float64')\ntest['totals_hits']=test['totals_hits'].astype('float64')","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"54a6096cfb4165bb5c3d3a589094613a6bb14c9f"},"cell_type":"markdown","source":"## Analysis unique count <15 on categorical features\n* Cause we have different unique count of test set and train set, lets concate it to do one hot encoding"},{"metadata":{"trusted":true,"_uuid":"bb98d070f700d8f65cff85288046870efa9f6441"},"cell_type":"code","source":"categorical_unique = train[list(categorical_features_test)].nunique().reset_index()\ncategorical_unique.columns = ['features','unique_count']\ncategorical_unique_less_15 = categorical_unique.loc[categorical_unique['unique_count'] < 15,\"features\"]\ncategorical_unique_less_15 = list(categorical_unique_less_15)\ncategorical_unique_less_15","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"d46a6ccfba5f0db7afd9e65f0c092052570719ee"},"cell_type":"markdown","source":"## Analysis the feature that have unique count >= 15 \n* Anaysis the revenue of these feature\n* Process different type feature to be training feature"},{"metadata":{"trusted":true,"_uuid":"3114604fa1ea92ae9d6c6dcafe89df4b0e7b71a6","_kg_hide-input":true},"cell_type":"code","source":"excluded_features = [\n    'date', 'fullVisitorId', 'sessionId', 'totals.transactionRevenue', \n    'visitId', 'visitStartTime']\ncategorical_larger_15_feature = [i for i in categorical_features_train if i not in categorical_unique_less_15 and i not in excluded_features]\nprint(categorical_larger_15_feature)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"3783b0f5009e6956e4df4681250700d1b3039646"},"cell_type":"markdown","source":"### Analysis Revene correlation on geoNetwork features\n* Plot the Count / Revenue correlation for all geoNetwork features"},{"metadata":{"trusted":true,"scrolled":true,"_uuid":"efb58dcfc13b32171797abe93b29569502b1391f","_kg_hide-input":true},"cell_type":"code","source":"geoNetwork_feature = [i for i in categorical_larger_15_feature if i.find('geoNetwork')>=0]\nprint(geoNetwork_feature)","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"trusted":true,"_uuid":"77ceeb85fef046da8f94874bd28b9cc25acde844"},"cell_type":"code","source":"plt.subplots(figsize=(20,60))\nfor k,i in enumerate(geoNetwork_feature): \n    train_geoNetwork = train[[i]+[\"totals_transactionRevenue\"]]\n    train_geoNetwork['totals_transactionRevenue']=train_geoNetwork['totals_transactionRevenue'].fillna(0).astype('float64')\n    train_geoNetwork[i+'_counts'] = train_geoNetwork[i].fillna('others')\n    train_geoNetwork = train_geoNetwork.groupby(i).agg({'totals_transactionRevenue':'mean',i+'_counts':'count'}).reset_index()\n    train_geoNetwork = train_geoNetwork.sort_values(by='totals_transactionRevenue',ascending=False)\n    \n\n    plt.subplot(len(geoNetwork_feature),2,(2*k)+1)\n    plt.title('mean revenue by '+i,color='b',fontsize=12)\n    plt.xlabel(str(i),color='b',fontsize=12)\n    plt.barh(range(10),train_geoNetwork.totals_transactionRevenue[:10],color='grey')\n    plt.yticks(range(10),train_geoNetwork[i][:10],rotation='horizontal',fontsize=10)\n    plt.xticks(fontsize=12)\n    \n    plt.subplot(len(geoNetwork_feature),2,(2*k)+2)\n    plt.title('Count by '+i,color='b',fontsize=12)\n    plt.xlabel(str(i),color='b',fontsize=12)\n    plt.barh(range(10),train_geoNetwork[i+'_counts'][:10],color='orange')\n    plt.yticks(range(10),train_geoNetwork[i][:10],rotation='horizontal',fontsize=10)\n    plt.xticks(fontsize=12)\n    \nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"d33d0dceb3c106618f361eed1a5448af0912527d"},"cell_type":"markdown","source":"### Analysis Revene correlation on device and traffic features\n* Plot the Count / Revenue correlation for device and traffic features"},{"metadata":{"_uuid":"a2b88b4768dcbc3fa906dfd3fac53e03189bf6a3"},"cell_type":"markdown","source":"* Device features"},{"metadata":{"trusted":true,"_uuid":"be7142b35e19df8118c583533850327bb9c0bd72"},"cell_type":"code","source":"device_feature = [i for i in categorical_larger_15_feature if i.find('device')>=0]\nprint(device_feature)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"25c3289af1de1be378bde1d1efe860a515d2d9eb","_kg_hide-input":true},"cell_type":"code","source":"plt.subplots(figsize=(20,40))\nfor k,i in enumerate(device_feature): \n    train_device = train[[i]+[\"totals_transactionRevenue\"]]\n    train_device['totals_transactionRevenue']=train_device['totals_transactionRevenue'].fillna(0).astype('float64')\n    train_device[i+'_counts'] = train_device[i].fillna('others')\n    train_device = train_device.groupby(i).agg({'totals_transactionRevenue':'mean',i+'_counts':'count'}).reset_index()\n    train_device = train_device.sort_values(by='totals_transactionRevenue',ascending=False)\n    \n\n    plt.subplot(len(geoNetwork_feature),2,(2*k)+1)\n    plt.title('mean revenue by '+i,color='b',fontsize=12)\n    plt.xlabel(str(i),color='b',fontsize=12)\n    plt.barh(range(10),train_device.totals_transactionRevenue[:10],color='grey')\n    plt.yticks(range(10),train_device[i][:10],rotation='horizontal',fontsize=10)\n    plt.xticks(fontsize=12)\n    \n    plt.subplot(len(geoNetwork_feature),2,(2*k)+2)\n    plt.title('Count by '+i,color='b',fontsize=12)\n    plt.xlabel(str(i),color='b',fontsize=12)\n    plt.barh(range(10),train_device[i+'_counts'][:10],color='orange')\n    plt.yticks(range(10),train_device[i][:10],rotation='horizontal',fontsize=10)\n    plt.xticks(fontsize=12)\n    \nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"862c6af1b1aebc9e9fbaa874fab465811ea9161c"},"cell_type":"markdown","source":"* Traffic features\nRemove path and gclId feature"},{"metadata":{"trusted":true,"_uuid":"c745bac590182c09d9c0005e05979a1dbd149d34"},"cell_type":"code","source":"Traffic_feature = [i for i in categorical_larger_15_feature if i.find('traffic')>=0 and i !='trafficSource_adwordsClickInfo.gclId' and i!='trafficSource_referralPath']\nprint(Traffic_feature)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"31a7c755e51c53228378fd72570d2f1b5c33117e","_kg_hide-input":true},"cell_type":"code","source":"plt.subplots(figsize=(20,60))\nfor k,i in enumerate(Traffic_feature): \n    if i !='trafficSource_adwordsClickInfo.gclId' and i!='trafficSource_referralPath':\n        train_traffic = train[[i]+[\"totals_transactionRevenue\"]]\n        train_traffic['totals_transactionRevenue']=train_traffic['totals_transactionRevenue'].fillna(0).astype('float64')\n        train_traffic[i+'_counts'] = train_traffic[i].fillna('others')\n        train_traffic = train_traffic.groupby(i).agg({'totals_transactionRevenue':'mean',i+'_counts':'count'}).reset_index()\n        train_traffic = train_traffic.sort_values(by='totals_transactionRevenue',ascending=False)\n\n\n        plt.subplot(len(geoNetwork_feature),2,(2*k)+1)\n        plt.title('mean revenue by '+i,color='b',fontsize=12)\n        plt.xlabel(str(i),color='b',fontsize=12)\n        plt.barh(range(10),train_traffic.totals_transactionRevenue[:10],color='grey')\n        plt.yticks(range(10),train_traffic[i][:10],rotation='horizontal',fontsize=10)\n        plt.xticks(fontsize=12)\n\n        plt.subplot(len(geoNetwork_feature),2,(2*k)+2)\n        plt.title('Count by '+i,color='b',fontsize=12)\n        plt.xlabel(str(i),color='b',fontsize=12)\n        plt.barh(range(10),train_traffic[i+'_counts'][:10],color='orange')\n        plt.yticks(range(10),train_traffic[i][:10],rotation='horizontal',fontsize=10)\n        plt.xticks(fontsize=12)\n    \nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"f1948fb400cf6e299133fa81c2f6e9cd83c611a8"},"cell_type":"markdown","source":"## Analysis Revenue \n* First we analysis the assist value  of Transations Revenue\n* Plot Distribution of total Revenue\n* Add visiter level Revenue"},{"metadata":{"trusted":true,"_uuid":"808f40a3c47ddf1b28aa6093d89cfe6f81d199f0"},"cell_type":"code","source":"Revenue_assist_count = train[\"totals_transactionRevenue\"].count()\nRevenue_nonassist_count = train.shape[0]-Revenue_assist_count\nlabel_name = [\"Revenue_assist_count\",\"Revenue_nonassist_count\"]\n\nRevenue_assist_list ={\n    \"Revenue\":label_name,\n    \"Revenue_count\":[Revenue_assist_count,Revenue_nonassist_count]\n}\n\nRevenue_assist_dataframe = pd.DataFrame(Revenue_assist_list)\n\n#Revenue_non_zero = train[train[\"totals_transactionRevenue\"]>0][\"totals_transactionRevenue\"]\nRevenue_non_zero = train[\"totals_transactionRevenue\"].fillna(0).reset_index().astype('float64')\nRevenue_non_zero = Revenue_non_zero[Revenue_non_zero[\"totals_transactionRevenue\"]>0][\"totals_transactionRevenue\"]\n\n\nplt.figure(figsize=(18,18))\nax1=plt.subplot(221)\nRevenue_assist_dataframe.plot(kind='pie', y = 'Revenue_count', ax=ax1,autopct='%1.1f%%',\n startangle=90, shadow=False, labels=Revenue_assist_dataframe['Revenue'], legend = False, fontsize=14)\nplt.subplot(222)\nsns.distplot(Revenue_non_zero)\nplt.title(\"Distribution of Non-Zero Total Revenue\")\nplt.xlabel('Total TransactionRevenue')\nplt.subplot(212)\nsns.distplot(np.log1p(Revenue_non_zero))\nplt.title(\"Natural Log Distribution of Non Zero Total Transactions\");\nplt.xlabel(\"Natural Log - Total Transactions\");\n","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"261a31e8aeee958e6cc440e1c7c7803886be6ad7"},"cell_type":"markdown","source":"# Future Plan\n* findout the key feature on training (you can refer the upper record of different feature)\n* doing esemble learning"},{"metadata":{"trusted":true,"_uuid":"6b71545579a68255d07bb6c77c201cf948a33e4f"},"cell_type":"code","source":"train.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"bb1d3b578647d15ef7bc8f3c416e8ea14d52bb29"},"cell_type":"code","source":"test.head()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"792d1ff520a3fac3308000df73e272bf9ecf1164"},"cell_type":"markdown","source":""}],"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"name":"python","version":"3.6.6","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"}},"nbformat":4,"nbformat_minor":1}