{"cells":[{"metadata":{"_uuid":"e53103c6b97ab64a10f7df9516992cb487639fbb"},"cell_type":"markdown","source":"Please upvote if you find notebook useful
\n\n

Key Machine Learning Techniques used in this Notebook

\n\nNote: This is not a tutorial to above mentioned techniques. But their implementation on a complex dataset"},{"metadata":{"_uuid":"489f511eadce516ec2e72d7f22ac15b2ce501b45"},"cell_type":"markdown","source":"\n

Problem Statement

\n\nIn this competition, you’re challenged to analyze a Google Merchandise Store (also known as GStore, where Google swag is sold) customer dataset to predict revenue per customer. \n\n

\nSubmissions are scored on the root mean squared error. RMSE or RMSD is defined as:\n\n![RSME or RSMD](https://wikimedia.org/api/rest_v1/media/math/render/svg/eeb88fa0f90448e9d1a67cd7a70164f674aeb300)\n\n

\n Final submission deadline: November 15, 2018\n

GStore EDA

\n
    \n
  1. Data Preparation\n \n
  2. Further Data Preparation\n \n
  3. Exploratory Analysis Summary(work in progress)\n\n
  4. Developing baseline model\n \n
  5. While preparing submission we have to keep few points in mind\n\n\n
\n"},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true,"scrolled":true},"cell_type":"code","source":"#IMPORTING REQUIRED LIBRARIES\nimport numpy as np # linear algebra\nimport pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\nimport os\nimport matplotlib.pyplot as plt\nimport seaborn as sns\nimport math\n\nfrom lightgbm.sklearn import LGBMRegressor\nfrom sklearn.model_selection import train_test_split\nfrom sklearn.metrics import mean_squared_error\nfrom sklearn.model_selection import KFold,StratifiedKFold\nfrom sklearn.feature_extraction.text import TfidfVectorizer\nfrom sklearn.decomposition import TruncatedSVD\n\nimport gc\ngc.enable()\n\n\nimport warnings\nwarnings.filterwarnings(\"ignore\")\n%matplotlib inline\n\n","execution_count":null,"outputs":[]},{"metadata":{"_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","trusted":true,"scrolled":false},"cell_type":"code","source":"#DATASET VIEW\npath1=\"../input/\"\ndata_files=list(os.listdir(path1))\ndf_files=pd.DataFrame(data_files,columns=['File_Name'])\ndf_files['Size_in_MB']=df_files.File_Name.apply(lambda x:round(os.stat(path1+x).st_size/(1024*1024),2))\ndf_files","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"206f185dde0f0d7bb4aa98031d65c9a58cb3c6ed","scrolled":false},"cell_type":"code","source":"#DATASET VIEW\npath1=\"../input/ga-customer-revenue-prediction/\"\npath2='../input/gstore-prepared-dataset/'\ndata_files=['../input/gstore-prepared-dataset/prepared_train/prepared_train',\n '../input/gstore-prepared-dataset/prepared_test/prepared_test']\ndf_files=pd.DataFrame(data_files,columns=['File_Name'])\ndf_files['Size_in_MB']=df_files.File_Name.apply(lambda x:round(os.stat(x).st_size/(1024*1024),2))\ndf_files","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"5cba841622718acac545f2949d73e4f04faa3a01","scrolled":false},"cell_type":"code","source":"#All functions\n\n#FUNCTION FOR PROVIDING FEATURE SUMMARY\ndef feature_summary(df_fa):\n print('DataFrame shape')\n print('rows:',df_fa.shape[0])\n print('cols:',df_fa.shape[1])\n col_list=['Null','Unique_Count','Data_type','Max/Min','Mean','Std','Skewness','Sample_values']\n df=pd.DataFrame(index=df_fa.columns,columns=col_list)\n df['Null']=list([len(df_fa[col][df_fa[col].isnull()]) for i,col in enumerate(df_fa.columns)])\n #df['%_Null']=list([len(df_fa[col][df_fa[col].isnull()])/df_fa.shape[0]*100 for i,col in enumerate(df_fa.columns)])\n df['Unique_Count']=list([len(df_fa[col].unique()) for i,col in enumerate(df_fa.columns)])\n df['Data_type']=list([df_fa[col].dtype for i,col in enumerate(df_fa.columns)])\n for i,col in enumerate(df_fa.columns):\n if 'float' in str(df_fa[col].dtype) or 'int' in str(df_fa[col].dtype):\n df.at[col,'Max/Min']=str(round(df_fa[col].max(),2))+'/'+str(round(df_fa[col].min(),2))\n df.at[col,'Mean']=df_fa[col].mean()\n df.at[col,'Std']=df_fa[col].std()\n df.at[col,'Skewness']=df_fa[col].skew()\n df.at[col,'Sample_values']=list(df_fa[col].unique())\n \n return(df.fillna('-'))\n\n#FUNCTION FOR READING DICTIONARY ITEMS AND HANDLING KEYERROR\ndef get_val(x,col):\n try:\n y=x[col]\n except:\n y=np.nan\n return(y)\n\n#FUNCTION FOR CALCULATING RSME\ndef rsme(y,pred):\n return(mean_squared_error(y,pred)**0.5)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"86a4600deb10f1dac4b21039b172f72470a503b5","scrolled":false},"cell_type":"code","source":"%%time\n#READING TRAINING AND TEST DATASET\nprint('reading train dataset...')\ndf_train=pd.read_csv(data_files[0],dtype={'fullVisitorId':str})\nprint('reading test dataset...')\ndf_test=pd.read_csv(data_files[1],dtype={'fullVisitorId':str})\nprint('data reading complete')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"944e90ca010087fd75c9e5bd99e4aa7d8065a2ba","scrolled":false},"cell_type":"code","source":"#CHECKING TOP FIVE TRAIN OBSERVATIONS OR ROWS\ndf_train.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"4c279ad830babe55c2a93246976eb8bcac571414","scrolled":false},"cell_type":"code","source":"#FEATURE SUMMARY FOR TRAIN DATASET\nfeature_summary(df_train)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"a09edeb18f935507c07f5b3e7f05a67aa93b71cd","scrolled":false},"cell_type":"code","source":"#CHECKING TOP 5 TEST OBSERVATIONS OR ROWS\ndf_test.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"2adb6124e9bd85a751d325c0fd72d01b571d53fe","scrolled":false},"cell_type":"code","source":"#FEATURE SUMMARY FOR TEST DATASET\nfeature_summary(df_test)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"5408637ababf47250cd04ee0473083cf8bf2f979"},"cell_type":"code","source":"#CREATING COPY OF TRAIN DATA SET\ntrain_cpy=df_train.copy()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"9e379907215f55d1a10530f8e66b81423f1d3187"},"cell_type":"code","source":"#ADDING ANOTHER FEATURE revenue_status TO INDICATE PRESENCE/ABSENCE OF REVENUE FOR EACH OBSERVATION\ndf_train['revenue_status']=df_train.totals_transactionRevenue.apply(lambda x: 0 if x==0 else 1)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"01057562b61b71cf78412ba31004c6ebd1147abf"},"cell_type":"markdown","source":"

Analyzing Each Feature one by one

\nWill try to understand following points\n\nwork in progress..."},{"metadata":{"_kg_hide-input":true,"trusted":true,"_uuid":"78cea5feb52f3a7c06c27c6682c7990d431e57fe"},"cell_type":"code","source":"#VISUALIZATION FUNCTIONS\ndef revenue_transaction_visualization(df_t,col,col_name):\n df_con=df_t[[col,'totals_transactionRevenue','revenue_status']].groupby(col).aggregate(\n {'totals_transactionRevenue':['mean'],'revenue_status':['count']}).reset_index()\n df_con.columns=[col,'totals_transactionRevenue_mean','revenue_status_count']\n df=df_con.sort_values(by='totals_transactionRevenue_mean',ascending=False)[:20]\n df1=df_con.sort_values(by='revenue_status_count',ascending=False)[:20]\n display('SORTED BY Mean Revenue')\n display(df.style.format(formatter))\n\n\n plt.subplots(figsize=(20,5))\n plt.subplot(1,2,1)\n plt.title('REVENUE MEAN BY '+col_name,color='b',fontsize=12)\n plt.xlabel(col_name,color='b',fontsize=12)\n plt.ylabel('Mean Revenue',color='b',fontsize=12)\n plt.bar(range(len(df)),df.totals_transactionRevenue_mean,color='grey')\n plt.xticks(range(len(df)),df[col],rotation=90,fontsize=12)\n plt.yticks(fontsize=12)\n\n\n plt.subplot(1,2,2)\n plt.title('NUMBER OF TRANSACTIONS WITH REVENUE BY '+col_name,color='b',fontsize=12)\n plt.xlabel(col_name,color='b',fontsize=12)\n plt.ylabel('Count of Transactions with Revenue',color='b',fontsize=12)\n plt.bar(range(len(df)),df.revenue_status_count,color='orange')\n plt.yticks(fontsize=12)\n plt.xticks(range(len(df)),df[col],rotation=90,fontsize=12)\n plt.show()\n\n display('SORTED BY Count of Transactions with Revenue')\n display(df1.style.format(formatter))\n\n plt.subplots(figsize=(20,5))\n plt.subplot(1,2,1)\n plt.title('REVENUE MEAN BY '+col_name,color='b',fontsize=12)\n plt.xlabel(col_name,color='b',fontsize=12)\n plt.ylabel('Mean Revenue',color='b',fontsize=12)\n plt.bar(range(len(df1)),df1.totals_transactionRevenue_mean,color='grey')\n plt.xticks(range(len(df1)),df1[col],rotation=90,fontsize=12)\n plt.yticks(fontsize=12)\n\n\n plt.subplot(1,2,2)\n plt.title('NUMBER OF TRANSACTIONS WITH REVENUE BY '+col_name,color='b',fontsize=12)\n plt.xlabel(col_name,color='b',fontsize=12)\n plt.ylabel('Count of Transactions with Revenue',color='b',fontsize=12)\n plt.bar(range(len(df1)),df1.revenue_status_count,color='orange')\n plt.yticks(fontsize=12)\n plt.xticks(range(len(df1)),df1[col],rotation=90,fontsize=12)\n plt.show()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"07103d895f291b6d9dbcbe5c98eca1e4000bd09e"},"cell_type":"markdown","source":"

Understanding Number Of Transations Generating Revenue

\n"},{"metadata":{"trusted":true,"_uuid":"6d639c10df7e4580b894f7ca8a7c96a40afb73e0","_kg_hide-input":true,"scrolled":false},"cell_type":"code","source":"#UNDERSTANDING NUMBER OF TRANSACTIONS GENERATING REVENUE\npie_labels=['Revenue Generated -'+str(df_train['revenue_status'][df_train.revenue_status==1].count()),'No Revenue Generated-'+\n str(df_train['revenue_status'][df_train.revenue_status==0].count())]\npie_share=[df_train['revenue_status'][df_train.revenue_status==1].count()/df_train['revenue_status'].count(),\n df_train['revenue_status'][df_train.revenue_status==0].count()/df_train['revenue_status'].count()]\nfigureObject, axesObject = plt.subplots(figsize=(6,6))\npie_colors=('green','orange')\npie_explode=(.30,.15)\naxesObject.pie(pie_share,labels=pie_labels,explode=pie_explode,autopct='%.2f%%',colors=pie_colors,startangle=45,shadow=True)\naxesObject.axis('equal')\nplt.title('Percentage of Transactions Generating Revenue and Not Generating Revenue',color='blue',fontsize=12)\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"83e0df0d10b210a6a4c7c143a658acbe4640cf83"},"cell_type":"markdown","source":"

Understanding Number Of Transations Generating Revenue by Browser

\n"},{"metadata":{"trusted":true,"_uuid":"607c79540614ddf8ef2454f108889939ae6866ff","_kg_hide-input":true},"cell_type":"code","source":"#REVENUE GENERATED BY BROWSERS\ndf_browser=df_train[['device_browser','totals_transactionRevenue','revenue_status']].groupby(df_train.device_browser).aggregate({'totals_transactionRevenue':['mean'],\n 'revenue_status':['count']}).reset_index()\ndf_browser.columns=['device_browser','totals_transactionRevenue_mean','revenue_status_count']\ndf=df_browser.sort_values(by='totals_transactionRevenue_mean',ascending=False)[df_browser.totals_transactionRevenue_mean>0]\nformatter = {'totals_transactionRevenue_mean':'{:4.2f}'}\ndisplay(df.style.format(formatter))\n\nplt.subplots(figsize=(20,5))\nplt.subplot(1,2,1)\nplt.title('REVENUE MEAN BY BROWSER',color='b',fontsize=12)\nplt.xlabel('Browsers',color='b',fontsize=12)\nplt.ylabel('Mean Revenue',color='b',fontsize=12)\nplt.bar(range(len(df)),df.totals_transactionRevenue_mean,color='grey')\nplt.xticks(range(len(df)),df.device_browser,rotation=90,fontsize=12)\nplt.yticks(fontsize=12)\n\n\nplt.subplot(1,2,2)\nplt.title('NUMBER OF TRANSACTIONS WITH REVENUE BY BROWSER',color='b',fontsize=12)\nplt.xlabel('Browsers',color='b',fontsize=12)\nplt.ylabel('Count of Transactions with Revenue',color='b',fontsize=12)\nplt.bar(range(len(df)),df.revenue_status_count,color='orange')\nplt.xticks(range(len(df)),df.device_browser,rotation=90,fontsize=12)\nplt.yticks(fontsize=12)\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"1f1563a521bd046ae76b3e4d0b630044c9b675b7"},"cell_type":"markdown","source":"

Understanding Number Of Transations Generating Revenue by Operating System

\n"},{"metadata":{"trusted":true,"_uuid":"fa75f6c5263fa8141a80324d7bd4f6497d491332","_kg_hide-input":true},"cell_type":"code","source":"#REVENUE GENERATED BY OPERATING SYSTEM\ndf_OS=df_train[['device_operatingSystem','totals_transactionRevenue','revenue_status']].groupby(df_train.device_operatingSystem).aggregate({'totals_transactionRevenue':['mean'],\n 'revenue_status':['count']}).reset_index()\ndf_OS.columns=['device_operatingSystem','totals_transactionRevenue_mean','revenue_status_count']\ndf=df_OS.sort_values(by='totals_transactionRevenue_mean',ascending=False)[df_OS.totals_transactionRevenue_mean>0]\ndisplay(df.style.format(formatter))\n\nplt.subplots(figsize=(20,5))\nplt.subplot(1,2,1)\nplt.title('REVENUE MEAN BY OPERATING SYSTEM',color='b',fontsize=12)\nplt.xlabel('Operating Systems',color='b',fontsize=12)\nplt.ylabel('Mean Revenue',color='b',fontsize=12)\nplt.bar(range(len(df)),df.totals_transactionRevenue_mean,color='grey')\nplt.xticks(range(len(df)),df.device_operatingSystem,rotation=90,fontsize=12)\nplt.yticks(fontsize=12)\n\n\nplt.subplot(1,2,2)\nplt.title('NUMBER OF TRANSACTIONS WITH REVENUE BY OPERATING SYSTEM',color='b',fontsize=12)\nplt.xlabel('Operating Systems',color='b',fontsize=12)\nplt.ylabel('Count of Transactions with Revenue',color='b',fontsize=12)\nplt.bar(range(len(df)),df.revenue_status_count,color='orange')\nplt.yticks(fontsize=12)\nplt.xticks(range(len(df)),df.device_operatingSystem,rotation=90,fontsize=12)\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"edd17ea7852c30c0dd4dd58060aa058d24788875"},"cell_type":"markdown","source":"

Understanding Revenue, Transactions with Revenue and Total Transactions by Date

\n"},{"metadata":{"trusted":true,"_uuid":"fa603cdf220e770fee019b543b7d165124fa74b7","_kg_hide-input":true,"scrolled":false},"cell_type":"code","source":"#UNDERSTANDING REVENUE, TRANSACTIONS WITH REVENUE AND TOTAL TRANSACTIONS BY DATE\ndf=df_train[['date','revenue_status','totals_transactionRevenue']]\ndf['date']=pd.to_datetime(df['date'],format=\"%Y%m%d\")\ndf=df.groupby('date').aggregate({'revenue_status':['sum','count'],'totals_transactionRevenue':['sum']}).reset_index()\ndf.columns=['date','transactions_withRevenue','total_transactions','total_revenue']\n\n\n#PLOT FOR REVENUE BY DATES\nplt.figure(figsize=(40,10))\nplt.title(\"REVENUE BY DATES\",fontsize=30,color='b')\nplt.xlabel(\"Dates\",fontsize=30,color='b')\nplt.ylabel(\"Total Revenue\",fontsize=30,color='b')\nplt.xticks(fontsize=30)\nplt.yticks(fontsize=30)\nplt.plot(df['date'],df['total_revenue'],color='orange',linewidth=3)\nplt.hlines(xmin='2016-07-20',xmax='2017-08-10',y=np.mean(df['total_revenue']),color='r',linestyle='dashed')\nplt.text('2017-01-10',np.mean(df['total_revenue'])+2,str('Mean: '+str(round(np.mean(df['total_revenue'])/(10**10),4))),\n color='r',fontsize=25)\nplt.hlines(xmin='2016-07-20',xmax='2017-08-10',y=np.max(df['total_revenue']),color='r',linestyle='dashed')\nplt.text('2017-01-10',np.max(df['total_revenue'])+2,str('Max: '+str(round(np.max(df['total_revenue'])/(10**10),4))),\n color='r',fontsize=25)\nplt.show()\n\n#PLOT FOR TRANSACTIONS WITH REVENUE BY DATES\nplt.figure(figsize=(40,10))\nplt.title(\"TRANSACTIONS WITH REVENUE BY DATES\",fontsize=30,color='b')\nplt.xlabel(\"Dates\",fontsize=30,color='b')\nplt.ylabel(\"Transactions with Revenue\",fontsize=30,color='b')\nplt.xticks(fontsize=30)\nplt.yticks(fontsize=30)\nplt.plot(df['date'],df['transactions_withRevenue'],color='g',linewidth=3)\nplt.hlines(xmin='2016-07-20',xmax='2017-08-10',y=np.mean(df['transactions_withRevenue']),color='r',linestyle='dashed')\nplt.text('2017-01-10',np.mean(df['transactions_withRevenue'])+2,str('Mean: '+str(round(np.mean(df['transactions_withRevenue']),4))),\n color='r',fontsize=25)\nplt.hlines(xmin='2016-07-20',xmax='2017-08-10',y=np.max(df['transactions_withRevenue']),color='r',linestyle='dashed')\nplt.text('2017-01-10',np.max(df['transactions_withRevenue'])+1,str('Max: '+str(round(np.max(df['transactions_withRevenue']),4))),\n color='r',fontsize=25)\nplt.show()\n\n\n#PLOT FOR TOTAL TRANSACTIONS BY DATES\nplt.figure(figsize=(40,10))\nplt.title(\"TOTAL TRANSACTIONS BY DATES\",fontsize=30,color='b')\nplt.xlabel(\"Dates\",fontsize=30,color='b')\nplt.ylabel(\"Total Transactions\",fontsize=30,color='b')\nplt.xticks(fontsize=30)\nplt.yticks(fontsize=30)\nplt.plot(df['date'],df['total_transactions'],color='r',linewidth=3)\nplt.hlines(xmin='2016-07-20',xmax='2017-08-10',y=np.mean(df['total_transactions']),color='g',linestyle='dashed')\nplt.text('2017-01-10',np.mean(df['total_transactions'])+2,str('Mean: '+str(round(np.mean(df['total_transactions']),0))),\n color='g',fontsize=25)\nplt.hlines(xmin='2016-07-20',xmax='2017-08-10',y=np.max(df['total_transactions']),color='g',linestyle='dashed')\nplt.text('2017-01-10',np.max(df['total_transactions'])+2,str('Max: '+str(round(np.max(df['total_transactions']),0))),\n color='g',fontsize=25)\nplt.show()\n\n\n\n","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"8bd1ff46801774c44a3b99a6b6d5e2bd295de4bb"},"cell_type":"markdown","source":"

Understanding Revenue generated by Device (mobile/non mobile)

\n"},{"metadata":{"trusted":true,"_uuid":"968c394ebec12d9a850fb8e8bea895973848ab1c","_kg_hide-input":true,"scrolled":false},"cell_type":"code","source":"#REVENUE GENERATED BY OPERATING SYSTEM\ndf_isM=df_train[['device_isMobile','totals_transactionRevenue','revenue_status']].groupby(df_train.device_isMobile).aggregate({'totals_transactionRevenue':['mean'],\n 'revenue_status':['count']}).reset_index()\ndf_isM.columns=['device_isMobile','totals_transactionRevenue_mean','revenue_status_count']\ndf=df_isM.sort_values(by='totals_transactionRevenue_mean',ascending=False)[df_isM.totals_transactionRevenue_mean>0]\ndisplay(df.style.format(formatter))\n\nplt.subplots(figsize=(20,5))\nplt.subplot(1,2,1)\nplt.title('REVENUE MEAN BY MOBILE',color='b',fontsize=12)\nplt.xlabel('Mobile',color='b',fontsize=12)\nplt.ylabel('Mean Revenue',color='b',fontsize=12)\nplt.bar(range(len(df)),df.totals_transactionRevenue_mean,color='grey')\nplt.xticks(range(len(df)),df.device_isMobile,rotation=90,fontsize=12)\nplt.yticks(fontsize=12)\n\n\nplt.subplot(1,2,2)\nplt.title('NUMBER OF TRANSACTIONS WITH REVENUE BY MOBILE',color='b',fontsize=12)\nplt.xlabel('Mobile',color='b',fontsize=12)\nplt.ylabel('Count of Transactions with Revenue',color='b',fontsize=12)\nplt.bar(range(len(df)),df.revenue_status_count,color='orange')\nplt.yticks(fontsize=12)\nplt.xticks(range(len(df)),df.device_isMobile,rotation=90,fontsize=12)\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"b71402b4dea033c079c84353601fc6816947993a"},"cell_type":"markdown","source":"

Understanding Revenue generated by Country

\n"},{"metadata":{"_kg_hide-input":true,"trusted":true,"scrolled":false,"_uuid":"7ea884caa7d56c0c8fd1c5188219c98f89d20689"},"cell_type":"code","source":"#REVENUE GENERATED BY COUNTRY\nrevenue_transaction_visualization(df_train,'geoNetwork_country','COUNTRY')","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"a1b7801c60d2e3d68d3903b9cc35381729fb8818"},"cell_type":"markdown","source":"

Understanding Revenue generated by City

\n"},{"metadata":{"_kg_hide-input":true,"trusted":true,"scrolled":false,"_uuid":"215de81508f9223970cce7927fe69f959b3ecf57"},"cell_type":"code","source":"#REVENUE GENERATED BY CITY\nrevenue_transaction_visualization(df_train,'geoNetwork_city','CITY')","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"470f98bfee29870ae871074cbf87fa17bc011df3"},"cell_type":"markdown","source":"

Understanding Revenue generated by networkDomain

\n"},{"metadata":{"_kg_hide-input":true,"trusted":true,"scrolled":false,"_uuid":"1d220ea405a4172cbfb63a7940bfe7b1d7a21a3d"},"cell_type":"code","source":"#REVENUE GENERATED BY NETWORKDOMAIN\nrevenue_transaction_visualization(df_train,'geoNetwork_networkDomain','NETWORKDOMAIN')","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"d1cd8d53066d2d87fc61e255ec9d8a4a628389c8"},"cell_type":"markdown","source":"

Understanding Revenue generated by subContinent

\n"},{"metadata":{"trusted":true,"scrolled":false,"_uuid":"db90333ba4ba5c1f2421070cfe6afea101b3d74a","_kg_hide-input":true},"cell_type":"code","source":"#REVENUE GENERATED BY SUBCONTINENT\nrevenue_transaction_visualization(df_train,'geoNetwork_subContinent','SUBCONTINENT')","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"d4f199280b9de82cb149eeb658797e2e910deb77"},"cell_type":"markdown","source":"

Understanding Revenue generated by Region

\n"},{"metadata":{"_kg_hide-input":true,"trusted":true,"scrolled":false,"_uuid":"0fe0b26f2f1c90ce263ae8706f49df64e58d432c"},"cell_type":"code","source":"#REVENUE GENERATED BY REGION\nrevenue_transaction_visualization(df_train,'geoNetwork_region','REGION')","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"5a1d07e078dc76dc6d183b9f5ec0354ef7da9734"},"cell_type":"markdown","source":"

Understanding Revenue generated by Continent

\n"},{"metadata":{"_kg_hide-input":true,"trusted":true,"_uuid":"0c057051fcd2de312ca3ded52a3a4060ef617b1d"},"cell_type":"code","source":"#REVENUE GENERATED BY CONTINENT\nrevenue_transaction_visualization(df_train,'geoNetwork_continent','CONTINENT')","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"f34299f9708a0fb7e87d6e1442bdd09f6d7c46d3"},"cell_type":"markdown","source":"

Understanding Revenue generated by Metro

\n"},{"metadata":{"_kg_hide-input":true,"trusted":true,"scrolled":false,"_uuid":"dd45bad0fde5eb2b566df8adf5f448c78e5a0055"},"cell_type":"code","source":"#REVENUE GENERATED BY METRO\nrevenue_transaction_visualization(df_train,'geoNetwork_metro','METRO')","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"5d045581654650c572d672df6758487bf0e1251f"},"cell_type":"markdown","source":"

Understanding Revenue generated by adContent

\n"},{"metadata":{"_kg_hide-input":true,"trusted":true,"scrolled":true,"_uuid":"17d7146def7b3d45752f58ca8add910b0f803471"},"cell_type":"code","source":"#REVENUE GENERATED BY AdCONTENT with Train dataset\nrevenue_transaction_visualization(df_train,'trafficSource_adContent','AdCONTENT')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"scrolled":false,"_uuid":"66b08d06f0fb49fefed6f96cb28dcde6291f1f01"},"cell_type":"code","source":"#ANALYZING AdContent in train and test dataset\nprint('Train data: Unique AdContent value count')\ndisplay(df_train['trafficSource_adContent'].groupby(df_train.trafficSource_adContent).count())\nprint('Test data: Unique AdContent value count')\ndisplay(df_test['trafficSource_adContent'].groupby(df_test.trafficSource_adContent).count())\n","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"fe41cf370bf8bf76614d1a459f144e73f1192648"},"cell_type":"markdown","source":"

Understanding Revenue generated by isTrueDirect

\n"},{"metadata":{"_kg_hide-input":true,"trusted":true,"_uuid":"cd00f06a09b895ae33dbeded6ea05ea9003d7123"},"cell_type":"code","source":"#REVENUE GENERATED BY isTrueDirect\ndf_train['trafficSource_isTrueDirect'].replace({np.nan:'False'},inplace=True)\ndf_isTD=df_train[['trafficSource_isTrueDirect','totals_transactionRevenue','revenue_status']].groupby(df_train.trafficSource_isTrueDirect).aggregate({'totals_transactionRevenue':['mean'],\n 'revenue_status':['count']}).reset_index()\ndf_isTD.columns=['trafficSource_isTrueDirect','totals_transactionRevenue_mean','revenue_status_count']\ndf=df_isTD.sort_values(by='totals_transactionRevenue_mean',ascending=False)[df_isTD.totals_transactionRevenue_mean>0]\ndisplay(df.style.format(formatter))\n\nplt.subplots(figsize=(20,5))\nplt.subplot(1,2,1)\nplt.title('REVENUE MEAN BY isTrueDirect',color='b',fontsize=12)\nplt.xlabel('isTrueDirect',color='b',fontsize=12)\nplt.ylabel('Mean Revenue',color='b',fontsize=12)\nplt.bar(range(len(df)),df.totals_transactionRevenue_mean,color='grey')\nplt.xticks(range(len(df)),df.trafficSource_isTrueDirect,rotation=90,fontsize=12)\nplt.yticks(fontsize=12)\n\n\nplt.subplot(1,2,2)\nplt.title('NUMBER OF TRANSACTIONS WITH REVENUE BY isTrueDirect',color='b',fontsize=12)\nplt.xlabel('isTrueDirect',color='b',fontsize=12)\nplt.ylabel('Count of Transactions with Revenue',color='b',fontsize=12)\nplt.bar(range(len(df)),df.revenue_status_count,color='orange')\nplt.yticks(fontsize=12)\nplt.xticks(range(len(df)),df.trafficSource_isTrueDirect,rotation=90,fontsize=12)\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"c0608bb559cd4ead1b31d260764e2305adf9d183"},"cell_type":"markdown","source":"

Understanding Revenue generated by Medium

\n"},{"metadata":{"_kg_hide-input":true,"trusted":true,"_uuid":"4406db897f26a6374aef9f0081ee2881c213552b"},"cell_type":"code","source":"#REVENUE GENERATED BY MEDIUM\nrevenue_transaction_visualization(df_train,'trafficSource_medium','MEDIUM')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"855d034fb83f2fc701375b2502720bc8b9561b90"},"cell_type":"code","source":"","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"0fa4fa0e98275ba4a66c33091b58ce61d3ee0a84"},"cell_type":"code","source":"#REVENUE GENERATED BY MEDIUM\nrevenue_transaction_visualization(df_train,'trafficSource_referralPath','referralPath')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"be510c6e7cb96948787123cba176b1e792b292be"},"cell_type":"code","source":"#AS IDENTIFIED IN EDA COUNTRY Anguilla HAS A VERY HIGH VALUE SINGLE VISIT TRANSACTION\nprint('Dropping following observation:')\ndisplay(train_cpy[train_cpy.geoNetwork_country=='Anguilla'])\n\n#DROPING THIS OUTLIER\ntrain_cpy.drop(train_cpy[train_cpy.geoNetwork_country=='Anguilla'].index,axis=0,inplace=True)\n\n#RESETING INDEX\ntrain_cpy.reset_index(drop=True,inplace=True)\n\n#COMBINING TRAIN AND TEST DATASET\ndf_combi=pd.concat([train_cpy,df_test],ignore_index=True)\n\n#FEATURE SUMMARY FOR COMBINED DATASET\ndf_combi_fs=feature_summary(df_combi)\ndisplay(df_combi_fs)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"scrolled":true,"_uuid":"4c9648a37ab3d37e555e5993259897ac2d599c4d"},"cell_type":"code","source":"#EXTRACTING DAY_OF_WEEK, HOUR, DAY, MONTH FROM DATE \ndf_combi['date'] = pd.to_datetime(df_combi['visitStartTime'], unit='s')\ndf_combi['day_of_week'] = df_combi['date'].dt.dayofweek\ndf_combi['hour'] = df_combi['date'].dt.hour\ndf_combi['day'] = df_combi['date'].dt.day\ndf_combi['month'] = df_combi['date'].dt.month\n\n#ADDING ANOTHER FEATURE revenue_status TO INDICATE PRESENCE/ABSENCE OF REVENUE FOR EACH OBSERVATION\ndf_combi['revenue_status']=df_combi.totals_transactionRevenue.apply(lambda x: 0 if x==0 else 1)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"443756ea7c7a573b58bf4ab7eb6d598946b324d7"},"cell_type":"code","source":"%%time\n#CONVERTING ALL THE STRINGS IN CATEGORICAL FEATURES TO LOWER CASE\nfor col in df_combi.columns:\n if ((df_combi[col].dtype=='object') & (col!='fullVisitorId')):\n df_combi[col]=df_combi[col].apply(lambda x:str(x).lower())\n \n#REPLACING STRING 'nan' WITH np.nan\ndf_combi.replace('nan',np.nan,inplace=True)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"3cd972b3f3d106c69d20d84b0f3bfa32110d51ab"},"cell_type":"code","source":"%%time\n#CONVERTING CATEGORICAL FEATURES (LESS THAN 10 UNIQUE VALUES) TO DUMMIES\ndf_combi.drop(['device_isMobile'],axis=1,inplace=True)\n\ncat_col=['channelGrouping','device_deviceCategory','tsadclick_slot','tsadclick_adNetworkType','tsadclick_isVideoAd','trafficSource_medium',\n 'geoNetwork_continent']\n\nfor col in cat_col:\n df_combi[col]=df_combi[col].apply(lambda x: str(x).replace(\" \",\"_\"))\n \ndummy=pd.DataFrame()\n\nfor col in cat_col:\n if col.find('_')!=-1:\n col_name=col.split('_')[1]\n else:\n col_name=col\n dummy=pd.concat([dummy,pd.get_dummies(df_combi[col],prefix=col_name)],axis=1)\n \nprint('Newly created dummy cols:',len(dummy.columns))\ndf_combi=pd.concat([df_combi,dummy],axis=1)\n\ndf_combi.drop(cat_col,axis=1,inplace=True)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"5a1f49bce3d72d0fb7a45d301386442abca18fb9"},"cell_type":"code","source":"df_combi.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"23ea867b07beae33e3a6f58e0f749392173a93df"},"cell_type":"code","source":"#SOME BASIC DATA CLEANUP\ndf_combi['totals_newVisits'].fillna(0,inplace=True) \ndf_combi['totals_bounces'].fillna(0,inplace=True)\ndf_combi['tsadclick_page'].fillna(0,inplace=True)\ndf_combi['trafficSource_isTrueDirect'].replace({np.nan:0,'true':1},inplace=True)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"94d3b214826b6e9979ae4250a0d6220d6374b000"},"cell_type":"code","source":"%%time\n#GENERATING RANKS FOR CATEGORICAL FEATURES WITH UNIQUE VALUES GREATER THAN 10\n#RANKS ARE GENERATED USING REVENUE PERCENTAGE\ncols=[x for x in df_combi.columns if x not in ['fullVisitorId','sessionId','geoNetwork_networkDomain','tsadclick_gclId']]\n\nfor col in cols:\n if df_combi[col].dtype=='object':\n df_combi[col].fillna('others',inplace=True)\n col_list=['revenue_status','totals_transactionRevenue']\n col_list.append(col)\n print(col_list)\n df=df_combi[col_list].groupby(col).aggregate({col:['count'],'revenue_status':['sum'],'totals_transactionRevenue':['sum']}).reset_index()\n df.columns=[col,col+\"_count\",'revenue_status_sum','totals_transactionRevenue_sum']\n df['revenue_perc']=df['totals_transactionRevenue_sum']/df[col+\"_count\"]\n df['rank']=df['revenue_perc'].rank(ascending=1)\n \n replace_dict={}\n final_dict={}\n for k,col_val in enumerate(df[col].values):\n replace_dict[col_val]=df.iloc[k,5]\n final_dict[col]=replace_dict\n df_combi.replace(final_dict,inplace=True)\n del df,replace_dict,final_dict\n gc.collect()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"458ae861e04e5f76a858eb7e8a52c4d7d44d11b8"},"cell_type":"code","source":"#SPLITING COMBINED DATASET BACK TO TRAIN AND TEST SETS\ntrain=df_combi[:len(train_cpy)]\ntest=df_combi[len(train_cpy):]","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"9dc717a86b38860fa0b3c50a07fe7fd13ebb5c9f"},"cell_type":"code","source":"train.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"e718bf400bf6414c4e4f76830b0d17706b16a660"},"cell_type":"code","source":"test.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"68a2af82f4c1c1aa32ccbf265ba08f7bce3a27e1"},"cell_type":"code","source":"%%time\n#REPLACING DOT WITH SPACE IN FEATURE geoNetwork_networkDomain\n#THIS IS DONE TO TREAT IT AS TEXT AND WE WILL USE TfidfVectorizer TO EXTRACT FEATURES\ntrain['geoNetwork_networkDomain'].fillna('unknown.unknown',inplace=True)\ntest['geoNetwork_networkDomain'].fillna('unknown.unknown',inplace=True)\n\ntrain['geoNetwork_networkDomain']=train.geoNetwork_networkDomain.apply(lambda x: x.replace('.',' '))\ntest['geoNetwork_networkDomain']=test.geoNetwork_networkDomain.apply(lambda x: x.replace('.',' '))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"99297786d8e7765ec411fb72beb2f64004fd3c34"},"cell_type":"code","source":"%%time\n#USING TfidfVectorizer TO EXTRACT FEATURES FROM geoNetwork_networkDomain\nTvect=TfidfVectorizer(ngram_range=(1,2),max_features=20000)\nvect=Tvect.fit(train['geoNetwork_networkDomain'])\ntrain_vect=vect.transform(train['geoNetwork_networkDomain'])\ntest_vect=vect.transform(test['geoNetwork_networkDomain'])\n\n#DIMENSIONALITY REDUCTION ON EXTRACTED FEATURES\nsvd=TruncatedSVD(n_components=10)\n\n#CREATING DATAFRAMES AFTER FEATURE EXTRACTION AND REDUCTION\nvect_cols=['vect'+str(x) for x in range(1,11)]\ndf_train_vect=pd.DataFrame(svd.fit_transform(train_vect),columns=vect_cols)\ndf_test_vect=pd.DataFrame(svd.fit_transform(test_vect),columns=vect_cols)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"9c828807902796ff416abdc5ce75767d43b082d5"},"cell_type":"code","source":"#VIEW OF EXTRACTED AND REDUCED FEATURES\nprint(train_vect.shape,test_vect.shape)\ndisplay(df_train_vect.head())\ndisplay(df_test_vect.head())\nprint('Shape of vector dataframes:',df_train_vect.shape,df_test_vect.shape)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"a245e44665de16bc5c777605f78b710930c3c323"},"cell_type":"code","source":"X=train.drop(['sessionId','visitId','date','geoNetwork_networkDomain','tsadclick_gclId'],axis=1)\nX_test=test.drop(['sessionId','visitId','date','geoNetwork_networkDomain','tsadclick_gclId'],axis=1) ","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"eec5b9bcc114fd059d79d1d47cb17d9c8faa7cfb"},"cell_type":"code","source":"#REORDERING INDEX FOR TEST DATASET\n#THIS IS REQUIRED BEFORE CONCATENATING DATAFRAMES\nX_test.reset_index(drop=True,inplace=True)\nX_test.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"91a702b8b74f91cec86ba3e0af1b92dfce9681ca"},"cell_type":"code","source":"#CONCATENATING WITH TEXT FEATURES \nX=pd.concat([X,df_train_vect],axis=1)\nX_test=pd.concat([X_test,df_test_vect],axis=1)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"07c43157c852df36c7e6018ace7ac401aa2a9d37"},"cell_type":"code","source":"#VIEW OF TRAIN AND TEST DATASET SHAPE\nprint('Before creating aggregated features')\nprint('Train shape:',X.shape,' Test shape:',X_test.shape)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"75d2a558f131e49419c3c984a8d3b1b95289e99f"},"cell_type":"code","source":"%%time\nagg_func={}\nagg_col=['fullVisitorId']\nfor col in [x for x in X.columns if x not in ['fullVisitorId']]:\n if col=='totals_transactionRevenue':\n agg_func[col]=['sum']\n agg_col.append(str(col)+'_sum')\n elif col=='revenue_status':\n agg_func[col]=['sum']\n agg_col.append(str(col)+'_sum')\n else:\n agg_func[col]=['sum','max','min','mean','var','std']\n agg_col.append(str(col)+'_sum')\n agg_col.append(str(col)+'_max')\n agg_col.append(str(col)+'_min')\n agg_col.append(str(col)+'_mean')\n agg_col.append(str(col)+'_var')\n agg_col.append(str(col)+'_std')\n \nX=X.groupby(X.fullVisitorId).aggregate(agg_func).reset_index()\nX.columns=agg_col\n\nX_test=X_test.groupby(X_test.fullVisitorId).aggregate(agg_func).reset_index()\nX_test.columns=agg_col","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"68c058f96d22b6f89958e2bc1ff09f01e40f5a3f"},"cell_type":"code","source":"%%time\n\n#CREATING y_dummy FOR USING STRATIFIED KFOLD\ny_dummy=X.revenue_status_sum.apply(lambda x: 0 if x==0 else 1)\n\n#TARGET FEATURE CONVERTED TO NATURAL LOG\n# y=pd.Series(X['totals_transactionRevenue_sum'])\ny=X.totals_transactionRevenue_sum.apply(lambda x: np.log1p(x))\n\n#PEPARING DATA FOR TRAINING LGBM MODEL\nX=X.drop(['totals_transactionRevenue_sum','fullVisitorId','revenue_status_sum'],axis=1)\n\n#FINAL DATAFRAME FOR SUBMISSION\ncol=['fullVisitorId','totals_transactionRevenue_sum']\nfinal=X_test[col] \nfinal.columns=['fullVisitorId','PredictedLogRevenue']\n\n#FINAL TEST FEATURES USED FOR PREDICTING SUBMISSION\nX_test=X_test.drop(['fullVisitorId','totals_transactionRevenue_sum','revenue_status_sum'],axis=1)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"be598ba6ee117475189ea03f89491baae02a0958"},"cell_type":"code","source":"print('After creating aggregated features')\nprint('Train shape:',X.shape,' Test shape:',X_test.shape)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"e83492334dc869c1ea22b72bbccb02184b803993","scrolled":false},"cell_type":"code","source":"%%time\n#LGBMRegressor. THIS REQUIRES FURTHER PARAMETER TUNINIG\nmodel=LGBMRegressor(boosting_type='gbdt',num_leaves=31,max_depth=-1,learning_rate=0.01,n_estimators=1000,max_bin=255,subsample_for_bin=50000,\n objective=None,min_split_gain=0,min_child_weight=3,min_child_samples=10,subsample=1,subsample_freq=1,colsample_bytree=1,\n reg_alpha=0.1,reg_lambda=0,seed=17,silent=False,nthread=-1,n_jobs=-1)\n\n\nk=1\nsplits=5\navg_score=0\n\n\nskf = StratifiedKFold(n_splits=splits, shuffle=True, random_state=200)\nprint('\\nStarting KFold iterations...')\nfor train_index,test_index in skf.split(X,y_dummy):\n df_X=X.iloc[train_index,:]\n df_y=y.iloc[train_index]\n val_X=X.iloc[test_index,:]\n val_y=y.iloc[test_index]\n\n model.fit(df_X,df_y)\n\n preds_x=pd.Series(model.predict(val_X))\n acc=rsme(val_y,preds_x)\n print('Iteration:',k,' rmse:',acc)\n \n if k==1:\n score=acc\n model1=model\n preds=pd.Series(model.predict(X_test))\n \n else:\n preds1=pd.Series(model.predict(X_test))\n preds=preds+preds1\n if score>acc:\n score=acc\n model1=model\n avg_score=avg_score+acc \n k=k+1\nprint('\\n Best score:',score,' Avg Score:',avg_score/splits)\npreds=preds/splits","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"f3cbc4c2333cc294bd2381f3c00248b925490f61"},"cell_type":"markdown","source":"

Preparing Submission

\nWhile preparing submission we have to keep few points in mind\n\n

Score log

\n
    \n
  1. Score improved from 2.005 to 1.4431 by preparing predicted data\n
  2. By further data preparation score improved from 1.4431 to 1.4389\n
  3. Grouped data by fullVisitorId and applied aggregates score improved from 1.4389 to 1.4281 \n
  4. Score further improved from 1.4281 to 1.470 by converting feature geoNetwork_networkDomain into text and using TfidfVectorizer for text feature extraction\n
  5. Score is improved from 1.4270 to 1.4260 by dropping observation for country Anguilla. This improvement is direct outcome of EDA\n
"},{"metadata":{"trusted":true,"_uuid":"e151490a2176a0ca10fc70591cb1d34f155bcedc"},"cell_type":"code","source":"#PREPARING PREDICTED DATA\nfinal['PredictedLogRevenue']=pd.Series(preds)\n#GROUPING PREDICTED DATA ON fullVisitorId\nfinal = final.groupby(\"fullVisitorId\")[\"PredictedLogRevenue\"].sum().reset_index()\nfinal.columns = [\"fullVisitorId\", \"PredictedLogRevenue\"]\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"ac23467bcde2ad5fbaa2f5a4af12913e322381b4"},"cell_type":"code","source":"final","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"6db0dd952d8c6f8e7d2821266e9d2b6abebb33a2"},"cell_type":"code","source":"#READING SUMISSION FILE\nsubmission=pd.read_csv(path1+'sample_submission.csv')\n\n#CREATING JOIN BETWEEN PREDICTED DATA WITH SUBMISSION FILE\nsubmission=submission.join(final.set_index('fullVisitorId'),on='fullVisitorId',lsuffix='_sub')\nsubmission.drop('PredictedLogRevenue_sub',axis=1,inplace=True)\n\n#HANDLING NaN IN CASE OF MISSING fullVisitorId\nsubmission.fillna(0,inplace=True)\n\n#SUBMITING FILE\nsubmission.to_csv('LGBM_submission.csv',index=False)","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"name":"python","version":"3.6.6","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"}},"nbformat":4,"nbformat_minor":1}