{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"code","source":"If you want see another interesting Kernels please check here https://www.kaggle.com/kabure/kernels\n\n### *Please, don't forget to upvote this Kernel to keep me motivated ! *","metadata":{"_uuid":"e8cc38f081fa109d3b8477d3beb06c809d000563"},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"English is not my native language, so sorry for any error.","metadata":{"_uuid":"8d2102167ab6bfebd6143032ef5e074cb94e487e"}},{"cell_type":"markdown","source":"# Google Analytics Customer Revenue Prediction","metadata":{"_uuid":"47748c0985b935b4e46d2a6c83af0eecf1516f65"}},{"cell_type":"markdown","source":"## Presenting the initial data: \n\n<b>Data Fields: </b>\n<b>fullVisitorIdv</b> - A unique identifier for each user of the Google Merchandise Store. <br>\n<b>channelGrouping</b> - The channel via which the user came to the Store.<br>\n<b>date</b> - The date on which the user visited the Store.<br>\n<b>device </b>- The specifications for the device used to access the Store.<br>\n<b>geoNetwork</b> - This section contains information about the geography of the user.<br>\n<b>sessionId</b> - A unique identifier for this visit to the store.<br>\n<b>socialEngagementType</b> - Engagement type, either \"Socially Engaged\" or \"Not Socially Engaged\".<br>\n<b>totals</b> - This section contains aggregate values across the session.<br>\n<b>trafficSource</b> - This section contains information about the Traffic Source from which the session originated.<br>\n<b>visitId</b> - An identifier for this session. This is part of the value usually stored as the _utmb cookie. This is only unique to the user. For a completely unique ID, you should use a combination of fullVisitorId and visitId.<br>\n<b>visitNumber</b> - The session number for this user. If this is the first session, then this is set to 1.<br>\n<b>visitStartTime</b> - The timestamp (expressed as POSIX time).<br>","metadata":{"_uuid":"7fbc08fd2ec0ec81825c6d794576fbb7a06ff9c1"}},{"cell_type":"markdown","source":"First of all, the data are becoming in Json format, so we will need to handle with it and I will use a chunk that I saw in almost all kernel of this competition.","metadata":{"_uuid":"f8f57d502c6739ff78f74aa89519987d05641cae"}},{"cell_type":"markdown","source":"# Objectives: \n\n- I will explore if we have some difference between the browser and if browser is significant to predict sells. <br>\n- Which countrys and continents have more acesses and sales ? How it's distributed?!  <br>\n- Which type of device are most normal in our dataset?<br>\n- What's the mobile % of accesses? <br>\n- Which is the most frequent Operational System? <br>\n- What's the most frequent channelGrouping ?<br> \n- Whats the most frequent Weekdays, months, days, year with highest accesses and revenue? \n\nAnd another bunch of ideas that I will have when start exploring.\n\n\n","metadata":{"_uuid":"d5db3cb7d849d9fad6163f3f8b1e671cd6f17b93"}},{"cell_type":"markdown","source":"## Importing necessary librarys","metadata":{"_uuid":"6ea6f1fa6570a74176e88faf94c3da6782824043"}},{"cell_type":"code","source":"# Necessary librarys\nimport os # it's a operational system library, to set some informations\nimport random # random is to generate random values\n\nimport pandas as pd # to manipulate data frames \nimport numpy as np # to work with matrix\nfrom scipy.stats import kurtosis, skew # it's to explore some statistics of numerical values\n\nimport matplotlib.pyplot as plt # to graphics plot\nimport seaborn as sns # a good library to graphic plots\nimport squarify # to better understand proportion of categorys - it's a treemap layout algorithm\n\n# Importing librarys to use on interactive graphs\nfrom plotly.offline import init_notebook_mode, iplot, plot \nimport plotly.graph_objs as go \n\nimport json # to convert json in df\nfrom pandas.io.json import json_normalize # to normalize the json file\n\n# to set a style to all graphs\nplt.style.use('fivethirtyeight')\ninit_notebook_mode(connected=True)","metadata":{"_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Some columns are in Json format so it will be necessary to handle with this problem.\n\nI will use a chunk code inspiration that almost all kernels are using I dont know who did first, but I got on SRK kernel and I did some modifications","metadata":{"_uuid":"59dd8b9cf783a9a649fd59eddf9108f6341551f7"}},{"cell_type":"code","source":"columns = ['device', 'geoNetwork', 'totals', 'trafficSource'] # Columns that have json format\n\ndir_path = \"../input/\" # you can change to your local \n\n# p is a fractional number to skiprows and read just a random sample of the our dataset. \np = 0.07 # *** In this case we will use 50% of data set *** #\n\n#Code to transform the json format columns in table\ndef json_read(df):\n    #joining the [ path + df received]\n    data_frame = dir_path + df\n    \n    #Importing the dataset\n    df = pd.read_csv(data_frame, \n                     converters={column: json.loads for column in columns}, # loading the json columns properly\n                     dtype={'fullVisitorId': 'str'}, # transforming this column to string\n                     skiprows=lambda i: i>0 and random.random() > p)# Number of rows that will be imported randomly\n    \n    for column in columns: #loop to finally transform the columns in data frame\n        #It will normalize and set the json to a table\n        column_as_df = json_normalize(df[column]) \n        # here will be set the name using the category and subcategory of json columns\n        column_as_df.columns = [f\"{column}.{subcolumn}\" for subcolumn in column_as_df.columns] \n        # after extracting the values, let drop the original columns\n        df = df.drop(column, axis=1).merge(column_as_df, right_index=True, left_index=True)\n        \n    # Printing the shape of dataframes that was imported     \n    print(f\"Loaded {os.path.basename(data_frame)}. Shape: {df.shape}\")\n    return df # returning the df after importing and transforming","metadata":{"_uuid":"17c123bd94775fa116a13113e729dc5113f01aad","trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Importing the datasets","metadata":{"_uuid":"86ee07b708307e5bfb907e41672393d374216bfe"}},{"cell_type":"code","source":"%%time \n# %%time is used to calculate the timing of code chunk execution #\n\n# We will import the data using the name and extension that will be concatenated with dir_path\ndf_train = json_read(\"train.csv\") \n# The same to test dataset\n#df_test = json_read(\"test.csv\") ","metadata":{"_uuid":"5ea0bc7871409e13de182189ebc0f30ddf7f3573","trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Nice. <br>\nAfter the import and transformation, we have 54 columns. <br>\nNow, let's see our data and handle with problemns that we will find","metadata":{"_uuid":"4b5cd86c5d9a59ed86a7be4f3fa19b079ab5dfbb"}},{"cell_type":"code","source":"# This command shows the first 5 rows of our dataset\ndf_train.head()","metadata":{"_uuid":"fb4b16cc25fe51887b9b4ebb3a161a0a6d683866","trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"It's interesting because we can see that <b>SessionId</b> has the <b>fullVisitorId</b> and <b>VisitStartTime</b> and <b>visitId</b>\n\nAlso, the date column we need to transform in datetime format and extract another datetime informations contained in the columns that I quoted above","metadata":{"_uuid":"8feee60f1daee4adba7db4836a25348977a65ba9"}},{"cell_type":"markdown","source":"## Knowing the missing values","metadata":{"_uuid":"473c414cf04168f496cd66a053029b39d5d19973"}},{"cell_type":"code","source":"# code chunk that I saw in Gabriel Preda kernel\ndef missing_values(data):\n    total = data.isnull().sum().sort_values(ascending = False) # getting the sum of null values and ordering\n    percent = (data.isnull().sum() / data.isnull().count() * 100 ).sort_values(ascending = False) #getting the percent and order of null\n    df = pd.concat([total, percent], axis=1, keys=['Total', 'Percent']) # Concatenating the total and percent\n    print(\"Total columns at least one Values: \")\n    print (df[~(df['Total'] == 0)]) # Returning values of nulls different of 0\n    \n    print(\"\\n Total of Sales % of Total: \", round((df_train[df_train['totals.transactionRevenue'] != np.nan]['totals.transactionRevenue'].count() / len(df_train['totals.transactionRevenue']) * 100),4))\n    \n    return ","metadata":{"_uuid":"fec2d2803b7bd55c5867242cb88af572b600290d","trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# calling the missing values function\nmissing_values(df_train) ","metadata":{"_uuid":"8e82a6e0f0f5e949c8142c7f7ecdc464441a0256","execution":{"iopub.status.busy":"2021-09-08T18:37:01.478869Z","iopub.execute_input":"2021-09-08T18:37:01.479213Z","iopub.status.idle":"2021-09-08T18:37:01.735116Z","shell.execute_reply.started":"2021-09-08T18:37:01.479144Z","shell.execute_reply":"2021-09-08T18:37:01.73357Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Nice. We can see that we have: <br>\nOur target have just 1.3% of non-null values  <br> \n6 columns with 97%+ of missing values  <br>\n4 columns with 50%+ of missing values <br>\n1 column with 22.22%  <br>\n1 column with 0.004% <br> \n\nWe will explore to understand what it looks like\n                                          ","metadata":{"_uuid":"51ec5f59a8ae95afffd418e95f7f878f269e7c36"}},{"cell_type":"markdown","source":"## Let's take a look on datatypes of all columns","metadata":{"_uuid":"e8d964f6e05fca00667ec2a3c2b0e62e7e7266d0"}},{"cell_type":"markdown","source":"- If you want see the code click in \"code\"\n- If you want see the ouput click in \"output\"","metadata":{"_uuid":"13f65c013634d3793ec70c2143c8e9cee926eb2b"}},{"cell_type":"code","source":"print(df_train.info())","metadata":{"_kg_hide-output":true,"_uuid":"fbe23d09172475042da33a2185600b7ffc5a7653","trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Nice! \n\nData Types contained in our dataframe: <br>\n- bool(1) <br>\n- int64(4) <br>\n- object(49) <br>","metadata":{"_uuid":"17e0aa2c0b6f99008990785e8f360ee49be48a85"}},{"cell_type":"markdown","source":"## Creating the function to handle with date ","metadata":{"_uuid":"dc1ed41a1c9a92a3f552b42207c116adf50e21bf"}},{"cell_type":"code","source":"# library of datetime\nfrom datetime import datetime\n\n# This function is to extract date features\ndef date_process(df):\n    df[\"date\"] = pd.to_datetime(df[\"date\"], format=\"%Y%m%d\") # seting the column as pandas datetime\n    df[\"_weekday\"] = df['date'].dt.weekday #extracting week day\n    df[\"_day\"] = df['date'].dt.day # extracting day\n    df[\"_month\"] = df['date'].dt.month # extracting day\n    df[\"_year\"] = df['date'].dt.year # extracting day\n    df['_visitHour'] = (df['visitStartTime'].apply(lambda x: str(datetime.fromtimestamp(x).hour))).astype(int)\n    \n    return df #returning the df after the transformations","metadata":{"_uuid":"4284e6cb9d9930d26460c4653c5935a2deeed422","trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"### Calling the function ","metadata":{"_uuid":"80bc44e6661561bdb69f06eab88215cb31311abb","trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df_train = date_process(df_train) #calling the function that we created above\n\ndf_train.head(n=2) #printing the first 2 rows of our dataset","metadata":{"_uuid":"7619fa329dcb6c0b5bea70b032145fe59a6ec70d","trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Before look the unique values in each column, I will drop the constant values that is not useful and will make the df lighter\nfor it, I will need to give some attention to numerical values","metadata":{"_uuid":"6f3bb45b16bf649d463276cb50eb6e49da5b8a43"}},{"cell_type":"markdown","source":"Defining some functions that I will use to call clean the data\n- If you want see, click in \"code\"","metadata":{"_uuid":"82d578ffa3e569ff729f973868a51ad99812bc74"}},{"cell_type":"code","source":"def FillingNaValues(df):    # fillna numeric feature\n    df['totals.pageviews'].fillna(1, inplace=True).astype(int) #filling NA's with 1\n    df['totals.newVisits'].fillna(0, inplace=True).astype(int) #filling NA's with 0\n    df['totals.bounces'].fillna(0, inplace=True).astype(int)   #filling NA's with 0\n    df[\"totals.transactionRevenue\"] = df[\"totals.transactionRevenue\"].fillna(0.0).astype(float) #filling NA with zero\n    df['trafficSource.isTrueDirect'].fillna(False, inplace=True) # filling boolean with False\n    df['trafficSource.adwordsClickInfo.isVideoAd'].fillna(True, inplace=True) # filling boolean with True\n    df_train.loc[df_train['geoNetwork.city'] == \"(not set)\", 'geoNetwork.city'] = np.nan\n    df_train['geoNetwork.city'].fillna(\"NaN\", inplace=True)\n\n    return df #return the transformed dataframe","metadata":{"_uuid":"ad6d4dd89351fda4bcf87f45a0236ec6c1168b29","trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def NumericalColumns(df):    # fillna numeric feature\n    df['totals.pageviews'].fillna(1, inplace=True) #filling NA's with 1\n    df['totals.newVisits'].fillna(0, inplace=True) #filling NA's with 0\n    df['totals.bounces'].fillna(0, inplace=True)   #filling NA's with 0\n    df['trafficSource.isTrueDirect'].fillna(False, inplace=True) # filling boolean with False\n    df['trafficSource.adwordsClickInfo.isVideoAd'].fillna(True, inplace=True) # filling boolean with True\n    df[\"totals.transactionRevenue\"] = df[\"totals.transactionRevenue\"].fillna(0.0).astype(float) #filling NA with zero\n    df['totals.pageviews'] = df['totals.pageviews'].astype(int) # setting numerical column as integer\n    df['totals.newVisits'] = df['totals.newVisits'].astype(int) # setting numerical column as integer\n    df['totals.bounces'] = df['totals.bounces'].astype(int)  # setting numerical column as integer\n    df[\"totals.hits\"] = df[\"totals.hits\"].astype(float) # setting numerical to float\n    df['totals.visits'] = df['totals.visits'].astype(int) # seting as int\n\n    return df #return the transformed dataframe","metadata":{"_uuid":"18f9f89cbe2074b38e687d33ad06c63499df7d69","_kg_hide-input":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Normalize","metadata":{"_uuid":"f94022c26f4a61c2d3c3693254b51e7a1e57cb98"}},{"cell_type":"code","source":"from sklearn import preprocessing\n\ndef Normalizing(df):\n    # Use MinMaxScaler to normalize the column\n    df[\"totals.hits\"] =  (df['totals.hits'] - min(df['totals.hits'])) / (max(df['totals.hits'])  - min(df['totals.hits']))\n    # normalizing the transaction Revenue\n    df['totals.transactionRevenue'] = df_train['totals.transactionRevenue'].apply(lambda x: np.log1p(x))\n    # return the modified df\n    return df ","metadata":{"_uuid":"9ae19cdafba7f9a37c64b4d398a9f8be342a56fc","_kg_hide-input":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Let's investigate some constant columns","metadata":{"_uuid":"751aca8f7276d7ba4968664b9349d063f0b4009b","trusted":true}},{"cell_type":"code","source":"# We will takeoff all columns where we have a unique value (constants)\n# It is useful because this columns don't give us none information\ndiscovering_consts = [col for col in df_train.columns if df_train[col].nunique() == 1]\n\n# printing the total of columns dropped and the name of columns \nprint(\"Columns with just one value: \", len(discovering_consts), \"columns\")\nprint(\"Name of constant columns: \\n\", discovering_consts)","metadata":{"_uuid":"42321bff6c5bdf02b01bdc6f4fec332e0696cc13","trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"","metadata":{"_uuid":"f58c88c804aeeca68d597f6cf0ebf61709467d84"}},{"cell_type":"code","source":"#Here are all columns that the unique value is 'not available in demo dataset'\n\nnot_aval_cols = ['socialEngagementType','device.browserSize','device.browserVersion', 'device.flashVersion', \n                 'device.language' ,'device.mobileDeviceBranding', 'device.mobileDeviceInfo','device.mobileDeviceMarketingName',\n                 'device.mobileDeviceModel', 'device.mobileInputSelector' , 'device.operatingSystemVersion','device.screenColors',\n                 'device.screenResolution', 'geoNetwork.cityId', 'geoNetwork.latitude' ,'geoNetwork.longitude',\n                 'geoNetwork.networkLocation','trafficSource.adwordsClickInfo.criteriaParameters']","metadata":{"_uuid":"622fda5f6a0b8ffb3a008e755f19b1bee77c4597","trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# It's useul to we have notion that might we have 23 constant columns\n\n- Below I will set a function to better investigate our data and correctly categorize them\n","metadata":{"_uuid":"c8171bdb0370f576f7962687f81aef8cae73d386","trusted":true}},{"cell_type":"code","source":"# seting the function to show \ndef knowningData(df, data_type=object, limit=3): #seting the function with df, \n    n = df.select_dtypes(include=data_type) #selecting the desired data type\n    for column in n.columns: #initializing the loop\n        print(\"##############################################\")\n        print(\"Name of column \", column, ': \\n', \"Uniques: \", df[column].unique()[:limit], \"\\n\",\n              \" | ## Total nulls: \", (round(df[column].isnull().sum() / len(df[column]) * 100,2)),\n              \" | ## Total unique values: \", df_train.nunique()[column]) #print the data and % of nulls)\n        # print(\"Percentual of top 3 of: \", column)\n        # print(round(df[column].value_counts()[:3] / df[column].value_counts().sum() * 100,2))\n        print(\"#############################################\")","metadata":{"_uuid":"356bfb3c7fae18c60e524d069aea3fd7ff3a974d","trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"\n### I will by object data Type. <br>\nClick on \"Output\" to see the result ","metadata":{"_uuid":"08e06b3957406abc12ec9d781297de6c5d175852"}},{"cell_type":"code","source":"# calling our function: object is default\nknowningData(df_train)","metadata":{"_kg_hide-output":true,"_uuid":"070121f44dbf08c698aae911696f612c94aa07f8","trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Printing Integers","metadata":{"_uuid":"721eb4c604e2e7f8c6811d9af128a13ee0d1db9d"}},{"cell_type":"code","source":"knowningData(df_train, data_type=int)","metadata":{"_kg_hide-output":true,"_uuid":"212795fc1d2d738a8365d1139d5a0205a8e790a1","trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Printing Float","metadata":{"_uuid":"82938d22dcc4c7b69b58c1ac847e039a811990c7"}},{"cell_type":"code","source":"knowningData(df_train, data_type=float)","metadata":{"_kg_hide-output":true,"_uuid":"bbebcb86dafb074cc93cace4f2e19777617add30","trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"We haven't float datatype yet. <br>","metadata":{"_uuid":"c6dc9005bec4736a6f2a08d6c47020e5f4a7a2d4"}},{"cell_type":"markdown","source":"## I will drop some of this features and fillna or missing in some of them","metadata":{"_uuid":"5799654350f222aacbe552535cda00b21d583a84"}},{"cell_type":"code","source":"to_drop = [\"socialEngagementType\",'device.browserVersion', 'device.browserSize', 'device.flashVersion', 'device.language', \n           'device.mobileDeviceBranding', 'device.mobileDeviceInfo', 'device.mobileDeviceMarketingName', 'device.mobileDeviceModel',\n           'device.mobileInputSelector', 'device.operatingSystemVersion', 'device.screenColors', 'device.screenResolution', \n           'geoNetwork.cityId', 'geoNetwork.latitude', 'geoNetwork.longitude','geoNetwork.networkLocation', \n           'trafficSource.adwordsClickInfo.criteriaParameters', 'trafficSource.adwordsClickInfo.gclId', 'trafficSource.campaign',\n           'trafficSource.adwordsClickInfo.page', 'trafficSource.referralPath', 'trafficSource.adwordsClickInfo.slot',\n           'trafficSource.adContent', 'trafficSource.keyword']\n\n\n","metadata":{"_uuid":"b8561c320415f62f4cd245091a47de9c62dd7659","trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df_train.drop(to_drop, axis=1, inplace=True)","metadata":{"_uuid":"1a4b221505ff10c5e4c51e842365b6fe70c87d59","trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"print(\"Total features dropped: \", len(to_drop))\nprint(\"Shape after dropping: \", df_train.shape)","metadata":{"_uuid":"5ec423223d17d45ab7c5cf8e2678a852abc1c863","trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# call the function to transform the numerical columns\ndf_train = NumericalColumns(df_train)\n\n# Call the function that will normalize some features\ndf_train = Normalizing(df_train)","metadata":{"_uuid":"ccf9c8f37da21f67b244b80506654ecd51d58fed","trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# looking if we have any mistake on \n\nfor c in dummy_feaures:\n    if c in to_drop: print(c)","metadata":{"_uuid":"de06e72aa764f9a9ac4575e124358bfc209ddd60","trusted":true}},{"cell_type":"code","source":"","metadata":{"_uuid":"4c933bff1466c8ba84d45e5aa691f4ca12ee307c","trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Let's see the unique values in our dataset. <br>\n\n- if you want see click in \"output\"","metadata":{"_uuid":"83fdb2b3c0572195dc601187643a5622a2ce2587","trusted":true}},{"cell_type":"code","source":"# We will takeoff all columns where we have a unique value\n# It is useful because this columns don't give us none information\nclean_consts = [col for col in df_train.columns if df_train[col].nunique() == 1]\n\n\n# this function drop all constant columns, inplacing the data \ndf_train.drop('trafficSource.adwordsClickInfo.adNetworkType', axis=1, inplace=True) \n\n# printing the total of columns dropped and the name of columns \nprint(\"This useful action will drop: \", len(clean_consts), \"columns\")\nprint(\"All dropped columns: \\n\", clean_consts)","metadata":{"_uuid":"df45d83eba81c1688403c0219685818e1df43a8e","trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"The output show us totals.visits and trafficSource.adwordsClickInfo.adNetworkType, but totals,visits can be useful, so I will drop just trafficSource feature","metadata":{"_uuid":"3965cebcc69089630b042f7a1df40040b7eb0309"}},{"cell_type":"code","source":"df_train.nunique()","metadata":{"_uuid":"cf78dc9e490c5ca03ec89893de56d6b2636e95ea","_kg_hide-output":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Excellent. Now we don't have more constant values","metadata":{"_uuid":"f3e67ac43a41f8976098caddc7b2c5f215aae8cd"}},{"cell_type":"markdown","source":"### Based on this output I will select and set a variable with all features by category","metadata":{"_uuid":"609ecb02fbeb7b45e0b434592d2e2d4c837f5062"}},{"cell_type":"code","source":"'trafficSource.adwordsClickInfo.adNetworkType'","metadata":{"_uuid":"463656bcaa564dc8530e96c3b645209927d989b8","trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"dummy_feaures =['channelGrouping', 'device.browser', 'device.deviceCategory', 'geoNetwork.city', 'device.operatingSystem', \n                'trafficSource.medium', 'trafficSource.source',\n                'geoNetwork.continent', 'geoNetwork.country', 'geoNetwork.metro', 'geoNetwork.networkDomain', 'geoNetwork.region', \n                'geoNetwork.subContinent']\n\n\nnumericals = ['totals.visits', '_visitHour', '_day', '_month', '_weekday']","metadata":{"_kg_hide-output":true,"_kg_hide-input":true,"_uuid":"7a67f1a41573aada51a45e0bef3055210c96ff9e","trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# First, let see the distribuition of transactions Revenues\n\nI will start exploring the quantile ","metadata":{"_uuid":"646b37c2ea75c5da1fe2bc88b92b22424ab96b3f"}},{"cell_type":"code","source":"# Printing some statistics of our data\nprint(\"Transaction Revenue Min Value: \", \n      df_train[df_train['totals.transactionRevenue'] > 0][\"totals.transactionRevenue\"].min()) # printing the min value\nprint(\"Transaction Revenue Mean Value: \", \n      df_train[df_train['totals.transactionRevenue'] > 0][\"totals.transactionRevenue\"].mean()) # mean value\nprint(\"Transaction Revenue Median Value: \", \n      df_train[df_train['totals.transactionRevenue'] > 0][\"totals.transactionRevenue\"].median()) # median value\nprint(\"Transaction Revenue Max Value: \", \n      df_train[df_train['totals.transactionRevenue'] > 0][\"totals.transactionRevenue\"].max()) # the max value\n\n# It I did to plot the quantiles but are not working\n#print(round(df_train['totals.transactionRevenue'].quantile([.025,.25,.5,.75,.975]),2))\n\n# seting the figure size of our plots\nplt.figure(figsize=(14,5))\n\n# Subplot allow us to plot more than one \n# in this case, will be create a subplot grid of 2 x 1\nplt.subplot(1,2,1)\n# seting the distribuition of our data and normalizing using np.log on values highest than 0 and + \n# also, we will set the number of bins and if we want or not kde on our histogram\nax = sns.distplot(np.log(df_train[df_train['totals.transactionRevenue'] > 0][\"totals.transactionRevenue\"] + 0.01), bins=40, kde=True)\nax.set_xlabel('Transaction RevenueLog', fontsize=15) #seting the xlabel and size of font\nax.set_ylabel('Distribuition', fontsize=15) #seting the ylabel and size of font\nax.set_title(\"Distribuition of Revenue Log\", fontsize=20) #seting the title and size of font\n\n# setting the second plot of our grid of graphs\nplt.subplot(1,2,2)\n# ordering the total of users and seting the values of transactions to understanding \nplt.scatter(range(df_train.shape[0]), np.sort(df_train['totals.transactionRevenue'].values))\nplt.xlabel('Index', fontsize=15) # xlabel and size of words\nplt.ylabel('Revenue value', fontsize=15) # ylabel and size of words\nplt.title(\"Revenue Value Distribution\", fontsize=20) # Setting Title and fontsize\n\nplt.show()","metadata":{"_uuid":"8fddcf823310f3ac3b47a280817ac10361be34f9","trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Nice distribuition... We have very high values on the Transactions Revenue. \n","metadata":{"_uuid":"c8527ffb1923d433cd68c7cdcb06cc98687bf34f","trusted":true}},{"cell_type":"markdown","source":"## I will see the kurtosis and Skewness of Transaction Revenue\nSkew and Kurtosis: 2 Important Statistics terms you need to know \n\n\n## Skewness\nIt is the degree of distortion from the symmetrical bell curve or the normal distribution. It measures the lack of symmetry in data distribution.<br>\nIt differentiates extreme values in one versus the other tail. A symmetrical distribution will have a skewness of 0.\n\n<b>Positive Skewness </b>means when the tail on the right side of the distribution is longer or fatter. The mean and median will be greater than the mode.<br>\n<b>Negative Skewness </b> is when the tail of the left side of the distribution is longer or fatter than the tail on the right side. The mean and median will be less than the mode.\n\n#### So, when is the skewness too much?\nThe rule of thumb seems to be:<br>\nIf the skewness is between -0.5 and 0.5, the data are fairly symmetrical. <br>\nIf the skewness is between -1 and -0.5(negatively skewed) or between 0.5 and 1(positively skewed), the data are moderately skewed.<br>\nIf the skewness is less than -1(negatively skewed) or greater than 1(positively skewed), the data are highly skewed.<br>\n\n\n## Kurtosis\nKurtosis is all about the tails of the distribution — not the peakedness or flatness. It is used to describe the extreme values in one versus the other tail. <br>It is actually the measure of outliers present in the distribution.\n\n<b>High kurtosis</b> in a data set is an indicator that data has heavy tails or outliers. If there is a high kurtosis, then, we need to investigate why do we have so many outliers. It indicates a lot of things, maybe wrong data entry or other things. Investigate! <br>\n<b>Low kurtosis</b> in a data set is an indicator that data has light tails or lack of outliers. If we get low kurtosis(too good to be true), then also we need to investigate and trim the dataset of unwanted results","metadata":{"_uuid":"c71ed04299a99b634d251443a76cc8b2b960285a"}},{"cell_type":"code","source":"print('Excess kurtosis of normal distribution (should be 0): {}'.format(\n    kurtosis(df_train[df_train['totals.transactionRevenue'] > 0][\"totals.transactionRevenue\"])))\nprint( 'Skewness of normal distribution (should be 0): {}'.format(\n    skew((df_train[df_train['totals.transactionRevenue'] > 0][\"totals.transactionRevenue\"]))))","metadata":{"_uuid":"1a471f060bdea1f011dbdd9dead1838f56a75ef8","trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Our data are fairly symmetrical skewed and have a High Kurtosis.\n\nI will see how many outliers we have on this dataset.\n","metadata":{"_uuid":"ec787d718c72215b52417e7bf1f6d2b3272e5065"}},{"cell_type":"markdown","source":"### Function that I created to find the map outlier values\n- Click on \"code\" to see the function","metadata":{"_uuid":"711c63585ac0f1fecad8ef51f0fb3ed2da1c9fda"}},{"cell_type":"code","source":"def CalcOutliers(df_num): \n    '''\n    \n    Leonardo Ferreira 20/10/2018\n    Set a numerical value and it will calculate the upper, lower and total number of outliers\n    It will print a lot of statistics of the numerical feature that you set on input\n    \n    '''\n    # calculating mean and std of the array\n    data_mean, data_std = np.mean(df_num), np.std(df_num)\n\n    # seting the cut line to both higher and lower values\n    # You can change this value\n    cut = data_std * 3\n\n    #Calculating the higher and lower cut values\n    lower, upper = data_mean - cut, data_mean + cut\n\n    # creating an array of lower, higher and total outlier values \n    outliers_lower = [x for x in df_num if x < lower]\n    outliers_higher = [x for x in df_num if x > upper]\n    outliers_total = [x for x in df_num if x < lower or x > upper]\n\n    # array without outlier values\n    outliers_removed = [x for x in df_num if x > lower and x < upper]\n    \n    print('Identified lowest outliers: %d' % len(outliers_lower)) # printing total number of values in lower cut of outliers\n    print('Identified upper outliers: %d' % len(outliers_higher)) # printing total number of values in higher cut of outliers\n    print('Identified outliers: %d' % len(outliers_total)) # printing total number of values outliers of both sides\n    print('Non-outlier observations: %d' % len(outliers_removed)) # printing total number of non outlier values\n    print(\"Total percentual of Outliers: \", round((len(outliers_total) / len(outliers_removed) )*100, 4)) # Percentual of outliers in points\n    \n    return\n","metadata":{"_uuid":"d12790b819bd9bce414203233c2c1578fe59f2e7","trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"CalcOutliers(df_train['totals.transactionRevenue']) # Call ","metadata":{"_uuid":"31a1e992143d98e30aabd5569d97f70bd92fcc72","trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"CalcOutliers(df_train['totals.pageviews']) # Call ","metadata":{"_uuid":"5d132e66fde158c6a1d8daede257b93282a2fa5e","trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Device Browsers","metadata":{"_uuid":"ed7eb5dda27ed49dc8208f258f1d5377750da132"}},{"cell_type":"code","source":"# the top 10 of browsers represent % of total\nprint(\"Percentual of Browser usage: \")\nprint(df_train['device.browser'].value_counts()[:7] ) # printing the top 7 percentage of browsers\n\n# seting the graph size\nplt.figure(figsize=(14,6))\n\n# Let explore the browser used by users\nsns.countplot(df_train[df_train['device.browser']\\\n                       .isin(df_train['device.browser']\\\n                             .value_counts()[:10].index.values)]['device.browser'], palette=\"hls\") # It's a module to count the category's\nplt.title(\"TOP 10 Most Frequent Browsers\", fontsize=20) # Adding Title and seting the size\nplt.xlabel(\"Browser Names\", fontsize=16) # Adding x label and seting the size\nplt.ylabel(\"Count\", fontsize=16) # Adding y label and seting the size\nplt.xticks(rotation=45) # Adjust the xticks, rotating the labels\n\nplt.show() #use plt.show to render the graph that we did above","metadata":{"_uuid":"fd2613fb0a19aa87a0ec016378a9bca180fb3887","trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### In our top 5 browsers we have more than 94% of total\n- TOP 1 - CHROME - 69,08%\n- TOP 2 - SAFARI - 20,04%\n- TOP 3 - FIREFOX - 3,77%\n\nNothing new under the sun... Chrome is the most used followed by Safari and firefox.\n","metadata":{"_uuid":"a9993b94cbee5276e05776fd137a196e32b847ee","trusted":true}},{"cell_type":"markdown","source":"## What if we cross the Revenue and Browser?\n","metadata":{"_uuid":"ffb50b603e096223e98bc11d90711156998e8489"}},{"cell_type":"code","source":"plt.figure(figsize=(13,6)) #figure size\n\n#It's another way to plot our data. using a variable that contains the plot parameters\ng1 = sns.boxenplot(x='device.browser', y='totals.transactionRevenue', \n                   data=df_train[(df_train['device.browser'].isin((df_train['device.browser'].value_counts()[:10].index.values))) &\n                                  df_train['totals.transactionRevenue'] > 0])\ng1.set_title('Browsers Name by Transactions Revenue', fontsize=20) # title and fontsize\ng1.set_xticklabels(g1.get_xticklabels(),rotation=45) # It's the way to rotate the xticks when we use variable to our graphs\ng1.set_xlabel('Device Names', fontsize=18) # Xlabel\ng1.set_ylabel('Trans Revenue(log) Dist', fontsize=18) #Ylabel\n\nplt.show()","metadata":{"_uuid":"c3ab87bfb285973429b671b966a60dc2c28881d6","trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"I think that it's very insightful information.\n\nChrome have highest values in general but the highest value of transactions was did on Firefox.<br>\nWe can see a \"small\" but consistent sells in Safari. Also IE and Edge give some results to Google;","metadata":{"_uuid":"7a1c76caa6384c3d353c87e051990dccab49534c"}},{"cell_type":"markdown","source":"## Let's see the Channel Grouping\n - The channel via which the user came to the Store.","metadata":{"_uuid":"7e8b0a9780d18ad0db49a53d7c996a85d16c632f"}},{"cell_type":"code","source":"# the top 10 of browsers represent % of total\nprint(\"Percentual of Channel Grouping used: \")\nprint((df_train['channelGrouping'].value_counts()[:5])) # printing the top 7 percentage of browsers\n\n# seting the graph size\nplt.figure(figsize=(14,7))\n\n# let explore the browser used by users\nsns.countplot(df_train[\"channelGrouping\"], palette=\"hls\") # It's a module to count the category's\nplt.title(\"Channel Grouping Count\", fontsize=20) # seting the title size\nplt.xlabel(\"Channel Grouping Name\", fontsize=18) # seting the x label size\nplt.ylabel(\"Count\", fontsize=18) # seting the y label size\n\nplt.show() #use plt.show to render the graph that we did above","metadata":{"_uuid":"1019d4bf94a102d9ca256cd222fbc716afc94a58","trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"The TOP 5 Grouping Channels represents 97% of total values. \nRespectivelly: \n- TOP 1 => Organic Search - 42.99%\n- TOP 2 => Social - 24.39%\n- TOP 3 => Direct - 15.42%\n- TOP 4 => Referral - 11.89%\n- TOP 5 => Paid Search - 2.55%\n\nI have a new insight that I will explore furthuer. How wich channel are distributed by browsers? \n","metadata":{"_uuid":"56670cf50c09a55ee50200fec29a6fac3e1c4d7a"}},{"cell_type":"markdown","source":"## Crossing Channel Grouping x Browsers","metadata":{"_uuid":"c9672347d2dc14c59050af7e86499cc509b822ec"}},{"cell_type":"code","source":"## I will use the crosstab to explore two categorical values\n\n# At index I will use set my variable that I want analyse and cross by another\ncrosstab_eda = pd.crosstab(index=df_train['channelGrouping'], normalize=True,\n                           # at this line, I am using the isin to select just the top 5 of browsers\n                           columns=df_train[df_train['device.browser'].isin(df_train['device.browser']\\\n                                                                            .value_counts()[:5].index.values)]['device.browser'])\n# Ploting the crosstab that we did above\ncrosstab_eda.plot(kind=\"bar\",    # select the bar to plot the count of categoricals\n                 figsize=(14,7), # adjusting the size of graphs\n                 stacked=True)   # code to unstack \nplt.title(\"Channel Grouping % for which Browser\", fontsize=20) # seting the title size\nplt.xlabel(\"The Channel Grouping Name\", fontsize=18) # seting the x label size\nplt.ylabel(\"Count\", fontsize=18) # seting the y label size\nplt.xticks(rotation=0)\nplt.show() # rendering","metadata":{"_uuid":"3f461736ae52ed6a36c892e77a265399ad3b0484","trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Very cool! Interesting patterns","metadata":{"_uuid":"270e9319019ef6f1fd0d25260d33e5455886eef2"}},{"cell_type":"markdown","source":"# Operational System ","metadata":{"_uuid":"9dca0278af87c758626cefe003e9fb428b0542cc"}},{"cell_type":"code","source":"# the top 5 of browsers represent % of total\nprint(\"Percentual of Operational System: \")\nprint(df_train['device.operatingSystem'].value_counts()[:5]) # printing the top 7 percentage of browsers\n\n# seting the graph size\nplt.figure(figsize=(14,7))\n\n# let explore the browser used by users\nsns.countplot(df_train[\"device.operatingSystem\"], palette=\"hls\") # It's a module to count the category's\nplt.title(\"Operational System used Count\", fontsize=20) # seting the title size\nplt.xlabel(\"Operational System Name\", fontsize=16) # seting the x label size\nplt.ylabel(\"OS Count\", fontsize=16) # seting the y label size\nplt.xticks(rotation=45) # Adjust the xticks, rotating the labels\n\nplt.show() #use plt.show to render the graph that we did above","metadata":{"_uuid":"74d59c258d5102197b8c4e0ec15097bfed8aaafd","trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"The TOP 5 of Operational System corresponds to 96%.\n\nTOP 1 => Windows - 38.75% <br>\nTOP 2 => Macintosh - 28.04% <br>\nTOP 3 => Android - 14.15% <br>\nTOP 4 => iOS - 11.75% <br>\nTOP 5 => Linux - 3.91% <br>\n\nIt's very interestign to me. In my country macbook isn't the most common SO. I will investigate further the SO by Country's","metadata":{"_uuid":"b55e6e5f6d48e7fa519294f466ff287d0b77cfa7"}},{"cell_type":"markdown","source":"## Now let's investigate the most used brower by Operational System","metadata":{"_uuid":"93e7fe418401f38b0dd67cf211194f9f089ee797"}},{"cell_type":"code","source":"\n# At index I will use isin to substitute the loop and get just the values with more than 1%\ncrosstab_eda = pd.crosstab(index=df_train[df_train['device.operatingSystem']\\\n                                          .isin(df_train['device.operatingSystem']\\\n                                                .value_counts()[:6].index.values)]['device.operatingSystem'], \n                           \n                           # at this line, I am using the isin to select just the top 5 of browsers\n                           columns=df_train[df_train['device.browser'].isin(df_train['device.browser']\\\n                                                                            .value_counts()[:5].index.values)]['device.browser'])\n# Ploting the crosstab that we did above\ncrosstab_eda.plot(kind=\"bar\",    # select the bar to plot the count of categoricals\n                 figsize=(14,7), # adjusting the size of graphs\n                 stacked=True)   # code to unstack \nplt.title(\"Most frequent OS's by Browsers of users\", fontsize=22) # adjusting title and fontsize\nplt.xlabel(\"Operational System Name\", fontsize=19) # adjusting x label and fontsize\nplt.ylabel(\"Count OS\", fontsize=19) # adjusting y label and fontsize\nplt.xticks(rotation=0) # Adjust the xticks, rotating the labels\n\nplt.show() # rendering","metadata":{"_uuid":"13d40e390372c97905e30d0ca10ebad16719db6e","trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Cool! <br>\n\nIt's visually clear to see that chrome is the most used in all OS, less in iOS, that is a mobile OS. \n\nI will see if we can see a diference between the Revenues of transactions are different ","metadata":{"_uuid":"f94c946788533ecedd3d397fc773c2f77317155a","trusted":true}},{"cell_type":"markdown","source":"## I will explore the distribuition of transaction Revenue by each OS","metadata":{"_uuid":"5dabdf87282a0408a54388c08540f1a575edef7b"}},{"cell_type":"code","source":"(sns.FacetGrid(df_train[(df_train['device.operatingSystem']\\\n                        .isin(df_train['device.operatingSystem']\\\n                              .value_counts()[:6].index.values)) & df_train['totals.transactionRevenue'] > 0],\n               hue='device.operatingSystem', height=5, aspect=2)\n  .map(sns.kdeplot, 'totals.transactionRevenue', shade=True)\n .add_legend()\n)\nplt.show()","metadata":{"_uuid":"d34317b1ba6d6e9f1f66c58a23c487a65e32ab11","trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Cool, we can have a better understanding of the distribution of Revenue by OS","metadata":{"_uuid":"2fa6a06048c63a484b21d6295e9d5e56beea7bfd"}},{"cell_type":"markdown","source":"## Let's investigate the Device Category","metadata":{"_uuid":"47aec47461b0fe1b8dce656e1f1e5ce678c98a9a","trusted":true}},{"cell_type":"code","source":"# the top 5 of browsers represent % of total\nprint(\"Percentual of Operational System: \")\nprint(round(df_train['device.deviceCategory'].value_counts() / len(df_train['device.deviceCategory']) * 100, 2)) # printing the top 7 percentage of browsers\n\n# seting the graph size\nplt.figure(figsize=(14,5))\n\nplt.subplot(1,2,1)\n# let explore the browser used by users\nsns.countplot(df_train[\"device.deviceCategory\"], palette=\"hls\") # It's a module to count the category's\nplt.title(\"Device Category Count\", fontsize=20) # seting the title size\nplt.xlabel(\"Device Category\", fontsize=18) # seting the x label size\nplt.ylabel(\"Count\", fontsize=16) # seting the y label size\nplt.xticks(fontsize=18) # Adjust the xticks, rotating the labels\n\nplt.subplot(1,2,2)\nsns.boxenplot(x=\"device.deviceCategory\", y = 'totals.transactionRevenue', \n              data=df_train[df_train['totals.transactionRevenue'] > 0], palette=\"hls\") # It's a module to count the category's\nplt.title(\"Device Category Revenue Distribuition\", fontsize=20) # seting the title size\nplt.xlabel(\"Device Category\", fontsize=18) # seting the x label size\nplt.ylabel(\"Revenue(Log)\", fontsize=16) # seting the y label size\nplt.xticks(fontsize=18) # Adjust the xticks, rotating the labels\n\nplt.subplots_adjust(hspace = 0.9, wspace = 0.5)\n\nplt.show() #use plt.show to render the graph that we did above","metadata":{"_uuid":"c27df69fb49cb019460d6d611e28970922dbfcd0","trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"In percentual, we can see that :\n- desktop represents 73.5%\n- mobile represents 23.12%\n- tablet represents 3.38%\n\nI thought that the Revenue is almost all did by desktops. Let's explore it further.","metadata":{"_uuid":"61c436b3b60cfa1e02720aae8c8a26dc2c2267fe"}},{"cell_type":"markdown","source":"## Let's see the difference distribution between Devices","metadata":{"_uuid":"8607632f5c289d212d4192bc9ba5ee3d910d8bec"}},{"cell_type":"code","source":"(sns.FacetGrid(df_train[df_train['totals.transactionRevenue'] > 0],\n               hue='device.deviceCategory', height=5, aspect=2)\n  .map(sns.kdeplot, 'totals.transactionRevenue', shade=True)\n .add_legend()\n)\nplt.show()","metadata":{"_uuid":"e6fe29c132df19a74cc391c13d74b778a70c8d6f","trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"We have We can see the distribuition of ","metadata":{"_uuid":"af8ec25d7455e41340fe78fa43be71c4aa12db14"}},{"cell_type":"markdown","source":"## Now, lets investigate the Device Category by Browsers","metadata":{"_uuid":"c1aa33b590e5f129f6b737e6cfe8888938aed000"}},{"cell_type":"code","source":"# At index I will use isin to substitute the loop and get just the values with more than 1%\ncrosstab_eda = pd.crosstab(index=df_train['device.deviceCategory'], # at this line, I am using the isin to select just the top 5 of browsers\n                           columns=df_train[df_train['device.operatingSystem']\\\n                                            .isin(df_train['device.operatingSystem']\\\n                                                  .value_counts()[:6].index.values)]['device.operatingSystem'])\n# Ploting the crosstab that we did above\ncrosstab_eda.plot(kind=\"bar\",    # select the bar to plot the count of categoricals\n                 figsize=(14,7), # adjusting the size of graphs\n                 stacked=True)   # code to unstack \nplt.title(\"Most frequent OS's by Device Categorys of users\", fontsize=22) # adjusting title and fontsize\nplt.xlabel(\"Device Name\", fontsize=19)                # adjusting x label and fontsize\nplt.ylabel(\"Count Device x OS\", fontsize=19)                               # adjusting y label and fontsize\nplt.xticks(rotation=0)                                            # Adjust the xticks, rotating the labels\n\n\nplt.show() # rendering","metadata":{"_uuid":"815651b3895bf595d523d82c3b0d619edcbdf295","trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Very interesting values.","metadata":{"_uuid":"fb3fb2e737a1aaf4bf2ea56d6ac906c66ce93432","trusted":true}},{"cell_type":"markdown","source":"# SubContinent ","metadata":{"_uuid":"0854b097c081212058908ef9c6ae156f1ff6ec44"}},{"cell_type":"code","source":"# the top 8 of browsers represent % of total\nprint(\"Description of SubContinent count: \")\nprint(df_train['geoNetwork.subContinent'].value_counts()[:8]) # printing the top 7 percentage of browsers\n\n# seting the graph size\nplt.figure(figsize=(16,7))\n\n# let explore the browser used by users\nsns.countplot(df_train[df_train['geoNetwork.subContinent']\\\n                       .isin(df_train['geoNetwork.subContinent']\\\n                             .value_counts()[:15].index.values)]['geoNetwork.subContinent'], palette=\"hls\") # It's a module to count the category's\nplt.title(\"TOP 15 most frequent SubContinents\", fontsize=20) # seting the title size\nplt.xlabel(\"subContinent Names\", fontsize=18) # seting the x label size\nplt.ylabel(\"SubContinent Count\", fontsize=18) # seting the y label size\nplt.xticks(rotation=45) # Adjust the xticks, rotating the labels\n\nplt.show() #use plt.show to render the graph that we did above","metadata":{"_uuid":"2affc90a0388834255a5bb4114ee6ea773cff3d1","trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"WoW, We have a very high number of users from North America. \n\nTOP 5 regions are equivalent of almost 70% +-  of total \n\nTOP 1 => Northern America - 44.18% <br>\nTOP 2 => Southeast Asia - 8.29% <br>\nTOP 3 => Northern Europe - 6.73% <br>\nTOP 4 => Southern Asia - 6.33% <br>\nTOP 5 => Western Europe - 6.23% <br>","metadata":{"_uuid":"636413869fa5b17720448f9824bccebf1530d030"}},{"cell_type":"markdown","source":"## Let's cross the SubContinent by Browser","metadata":{"_uuid":"5bd94c1e9c0408a7faa215775f6df1d2915d8354"}},{"cell_type":"code","source":"## I will use the crosstab to explore two categorical values\n\n# At index I will use isin to substitute the loop and get just the values with more than 1%\ncrosstab_eda = pd.crosstab(index=df_train[df_train['geoNetwork.subContinent']\\\n                                          .isin(df_train['geoNetwork.subContinent']\\\n                                                .value_counts()[:10].index.values)]['geoNetwork.subContinent'], \n                           \n                           # at this line, I am using the isin to select just the top 5 of browsers\n                           columns=df_train[df_train['device.browser'].isin(df_train['device.browser']\\\n                                                                            .value_counts()[:5].index.values)]['device.browser'])\n# Ploting the crosstab that we did above\ncrosstab_eda.plot(kind=\"bar\",    # select the bar to plot the count of categoricals\n                 figsize=(16,7), # adjusting the size of graphs\n                 stacked=True)   # code to unstack \nplt.title(\"TOP 10 Most frequent Subcontinents by Browsers used\", fontsize=22) # adjusting title and fontsize\nplt.xlabel(\"Subcontinent Name\", fontsize=19) # adjusting x label and fontsize\nplt.ylabel(\"Count Subcontinent\", fontsize=19) # adjusting y label and fontsize\nplt.xticks(rotation=45) # Adjust the xticks, rotating the labels\nplt.legend(loc=1, prop={'size': 12}) # to \n\nplt.show() # rendering","metadata":{"_uuid":"27b2fd721d60cd80f9a5e3dc26e40fb91a8463a0","trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Nice, this graph is very insightful.\nThe North America have a low ratio of Safari x Chrome... I thought that it was the contrary\n\nFirefox have a relative high presence in North America too.","metadata":{"_uuid":"87d3c4aed0e28994d300405382507983aff5afe5"}},{"cell_type":"code","source":"print('train date:', min(df_train['date']), 'to', max(df_train['date']))","metadata":{"_uuid":"02e16f6e9bb5fa0d3195bb079012cabb08cbccea","trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"year = df_train['_year'].value_counts()         # counting the Year with value counts\nmonth = df_train['_month'].value_counts()      # coutning months\nweeday = df_train['_weekday'].value_counts()    # Couting weekday\nday = df_train['_day'].value_counts()              # counting Day\ndate = df_train['date'].value_counts()           # Counting date","metadata":{"_uuid":"bc77dd4a9a1aed2aafe58484a4e8e51db06ed276","trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## INTERACTIVE DATE FEATURES","metadata":{"_uuid":"4e851acd6ae4f98f5e63c8a7befdbc3ec0e64016"}},{"cell_type":"markdown","source":"## First I will explore revenue and number of visits by day","metadata":{"_uuid":"45e293c2c406fdb42409cd56c76597bbdbfdc010"}},{"cell_type":"code","source":"# I saw and take a lot of inspiration to this interactive plots in kernel: \n# https://www.kaggle.com/jsaguiar/complete-exploratory-analysis-all-columns\n# I learned a lot in this kernel and I will implement and adapted some ideas\n\n#seting some static color options\ncolor_op = ['#5527A0', '#BB93D7', '#834CF7', '#6C941E', '#93EAEA', '#7425FF', '#F2098A', '#7E87AC', \n            '#EBE36F', '#7FD394', '#49C35D', '#3058EE', '#44FDCF', '#A38F85', '#C4CEE0', '#B63A05', \n            '#4856BF', '#F0DB1B', '#9FDBD9', '#B123AC']\n\n# Visits by time train\n\n# couting all entries by date to get number of visits by each date\ndates_temp = df_train['date'].value_counts().to_frame().reset_index().sort_values('index') \n# renaming the columns to apropriate names\ndates_temp = dates_temp.rename(columns = {\"date\" : \"visits\"}).rename(columns = {\"index\" : \"date\"})\n\n# creating the first trace with the necessary parameters\ntrace = go.Scatter(x=dates_temp.date.astype(str), y=dates_temp.visits,\n                    opacity = 0.8, line = dict(color = color_op[3]), name= 'Visits by day')\n\n# Below we will get the total values by Transaction Revenue Log by date\ndates_temp_sum = df_train.groupby('date')['totals.transactionRevenue'].sum().to_frame().reset_index()\n\n# using the new dates_temp_sum we will create the second trace\ntrace1 = go.Scatter(x=dates_temp_sum.date.astype(str), line = dict(color = color_op[1]), name=\"RevenueLog by day\",\n                        y=dates_temp_sum['totals.transactionRevenue'], opacity = 0.8)\n\n# Getting the total values by Transactions by each date\ndates_temp_count = df_train[df_train['totals.transactionRevenue'] > 0].groupby('date')['totals.transactionRevenue'].count().to_frame().reset_index()\n\n# using the new dates_temp_count we will create the third trace\ntrace2 = go.Scatter(x=dates_temp_count.date.astype(str), line = dict(color = color_op[5]), name=\"Sellings by day\",\n                        y=dates_temp_count['totals.transactionRevenue'], opacity = 0.8)\n\n#creating the layout the will allow us to give an title and \n# give us some interesting options to handle with the outputs of graphs\nlayout = dict(\n    title= \"Informations by Date\",\n    xaxis=dict(\n        rangeselector=dict(\n            buttons=list([\n                dict(count=1, label='1m', step='month', stepmode='backward'),\n                dict(count=3, label='3m', step='month', stepmode='backward'),\n                dict(count=6, label='6m', step='month', stepmode='backward'),\n                dict(step='all')\n            ])\n        ),\n        rangeslider=dict(visible = True),\n        type='date'\n    )\n)\n\n# creating figure with the both traces and layout\nfig = dict(data= [trace, trace1, trace2], layout=layout)\n\n#rendering the graphs\niplot(fig) #it's an equivalent to plt.show()","metadata":{"_uuid":"2cb8cdf11837f745da37f1faf9eb7d7215115d4b","_kg_hide-input":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Creating an Sofistcated interactive graphics to better understanding of date features\n\nTo see the code click in \"code\". \n## SELECT THE OPTION: ","metadata":{"_uuid":"58dbc1d65e92ba700c741839c9a4222bf35c11c7"}},{"cell_type":"code","source":"# Setting the first trace\ntrace1 = go.Histogram(x=df_train[\"_year\"],\n                      name='Year Count')\n\n# Setting the second trace\ntrace2 = go.Histogram(x=df_train[\"_month\"],\n                name='Month Count')\n\n# Setting the third trace\ntrace3 = go.Bar(y=day.values,\n                x=day.index.values, \n                name='Day Count')\n\n# Setting the fourth trace\ntrace4 = go.Bar(y=weeday.values,\n                x=weeday.index.values,\n                name='Weekday Count')\n\n# puting all traces in the same \"array of graphics\" to we render it below\ndata = [trace1, trace2, trace4, trace3]\n\n#Creating the options to be posible we use in our \nupdatemenus = list([\n    dict(active=-1,\n         x=-0.15,\n         buttons=list([  \n             dict(\n                 label = 'Years Count',\n                 method = 'update',\n                 args = [{'visible': [True, False, False, False,False]}, \n                         {'title': 'Count of Year'}]),\n             dict(\n                 label = 'Months Count',\n                 method = 'update',\n                 args = [{'visible': [False, True, False, False,False]},\n                         {'title': 'Count of Months'}]),\n             dict(\n                 label = 'WeekDays Count',\n                 method = 'update',\n                 args = [{'visible': [False, False, True, False, False]},\n                         {'title': 'Count of WeekDays'}]),\n            dict(\n                label = 'Days Count ',\n                method = 'update',\n                args = [{'visible': [False, False, False, True,False]},\n                        {'title': 'Count of Day'}]) ])\n    )\n])\n\n\nlayout = dict(title='The percentual Distribuitions of Date Features (Select from Dropdown)',\n              showlegend=False,\n              updatemenus=updatemenus,\n#              xaxis = dict(\n#                  type=\"category\"\n#                      ),\n              barmode=\"group\"\n             )\nfig = dict(data=data, layout=layout)\nprint(\"SELECT BELOW: \")\niplot(fig)","metadata":{"_uuid":"864f5577ef8b0ab3ac4f3a14537196b8b40eb2e3","_kg_hide-output":false,"_kg_hide-input":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"******* *How can I set order to my year, months and days?* *******","metadata":{"_uuid":"0a9bcb4e07e0416ac16ab3a7d31ae21e717270b8"}},{"cell_type":"markdown","source":"### Very Cool graphs.\n\nWE can see that the number of access are clearly downing by the through the time. \n\n- The months with highest accesses are October and November.\n- On the Weekend the trafic is lower than other days.\n- The 5 days with highest number of accesses is 1 and 5\n- Considering the full count of dates, we can see that the days with highest accesses are almost all in november/2016\n","metadata":{"_uuid":"b144feec1b14923c1237d7eaa8666906ebf1c101"}},{"cell_type":"markdown","source":"#### Let's investigate the VisitHour and weekday to see if we can find some interesting patterns","metadata":{"_uuid":"27d53dc8b53550a8a937f7b30b96f6da9b9ef08c"}},{"cell_type":"code","source":"date_sales = ['_visitHour', '_weekday'] #seting the desired \n\ncm = sns.light_palette(\"green\", as_cmap=True)\npd.crosstab(df_train[date_sales[0]], df_train[date_sales[1]], \n            values=df_train[\"totals.transactionRevenue\"], aggfunc=[np.sum]).style.background_gradient(cmap = cm)\n\n# tab.columns.levels[1] = [\"Sun\", \"Mon\", \"Thu\", \"wed\", \"Thi\",\"Fri\",\"Sat\"]","metadata":{"_uuid":"e1975529c7fbc0866acd700cba2fd6b6e234070e","trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Very interesting, we can see that from 17  to 20 hour we have the highest numbers of ","metadata":{"_uuid":"b22a4329690db833383cde3d3aedb351ddb1666b"}},{"cell_type":"markdown","source":"## I will use a interesting graphic called Squarify \n- I will apply it in feature Country to discovery where the user access the store","metadata":{"_uuid":"32dc0d6b64c8d5bb05139b472abc47d17176f00e"}},{"cell_type":"code","source":"number_of_colors = 20 # total number of different collors that we will use\n\n# Here I will generate a bunch of hexadecimal colors \ncolor = [\"#\"+''.join([random.choice('0123456789ABCDEF') for j in range(6)])\n             for i in range(number_of_colors)]","metadata":{"_uuid":"963ce4b784c466f9aeb7490df57935f440c61096","trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Exploring Countrys","metadata":{"_uuid":"66ede41d597a59aa2c44955b29858855b8f37123"}},{"cell_type":"code","source":"country_tree = df_train[\"geoNetwork.country\"].value_counts() #counting the values of Country\n\nprint(\"Description most frequent countrys: \")\nprint(country_tree[:15]) #printing the 15 top most \n\ncountry_tree = round((df_train[\"geoNetwork.country\"].value_counts()[:30] \\\n                       / len(df_train['geoNetwork.country']) * 100),2)\n\nplt.figure(figsize=(14,5))\ng = squarify.plot(sizes=country_tree.values, label=country_tree.index, \n                  value=country_tree.values,\n                  alpha=.4, color=color)\ng.set_title(\"'TOP 30 Countrys - % size of total\",fontsize=20)\ng.set_axis_off()\nplt.show()","metadata":{"_uuid":"3b4f42be8780ba5abb86491b95b6c05384d43309","trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"USA have a very highest value than another countrys. \n\nBelow I will take a look on cities and find for the highest revenues from them\n","metadata":{"_uuid":"a4b96e6e0e11be37f004dd6e1a1d49a3e416292d"}},{"cell_type":"markdown","source":"## Now, I will look on City feature and see the principal cities in the dataset","metadata":{"_uuid":"a39f2bd479c36e09ad97fc5cb7233b625b700b81"}},{"cell_type":"code","source":"df_train.loc[df_train[\"geoNetwork.city\"] == \"not available in demo dataset\", 'geoNetwork.city'] = np.nan","metadata":{"_uuid":"5142be3a58f77f0de9b8d8b03cbfb8230155179f","trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"city_tree = df_train[\"geoNetwork.city\"].value_counts() #counting \n\nprint(\"Description most frequent Citys: \" )\nprint(city_tree[:15])\n\ncity_tree = round((city_tree[:30] / len(df_train['geoNetwork.city']) * 100),2)\n\nplt.figure(figsize=(14,5))\ng = squarify.plot(sizes=city_tree.values, label=city_tree.index, \n                  value=city_tree.values,\n                  alpha=.4, color=color)\ng.set_title(\"'TOP 30 Citys - % size of total\",fontsize=20)\ng.set_axis_off()\nplt.show()","metadata":{"_uuid":"744e3f288212cfcd87fb077569b04314bbeb4fef","trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Nicelly distributed clients that accessed the store. \n(non set) have 3.81% of total, so I dont will consider in top five, but it was the top 2 most frequent. \n\nThe top 5 are:\n- Montain View\n- New York \n- San Francisco\n- Sunnyvale \n- London\n\nAnd in terms of money, how the Countrys and Cities are ? \n","metadata":{"_uuid":"8cb4ca4f8fca49baeab424aa9fd9140f0567effe"}},{"cell_type":"markdown","source":"____________________","metadata":{"_uuid":"9344c85fbc4351d8e35912e1bce2c388e421a486"}},{"cell_type":"markdown","source":"### Creating a function with plotly to better investigate the dataset\n\n- Click in \"code\" to see the commented code","metadata":{"_uuid":"a4b4b66d07c41db917702540e6758d959a6a6c19"}},{"cell_type":"code","source":"def PieChart(df_colum, title, limit=15):\n    \"\"\"\n    This function helps to investigate the proportion of visits and total of transction revenue \n    by each category\n    \"\"\"\n\n    count_trace = df_train[df_colum].value_counts()[:limit].to_frame().reset_index()\n    rev_trace = df_train.groupby(df_colum)[\"totals.transactionRevenue\"].sum().nlargest(10).to_frame().reset_index()\n\n    trace1 = go.Pie(labels=count_trace['index'], values=count_trace[df_colum], name= \"% Acesses\", hole= .5, \n                    hoverinfo=\"label+percent+name\", showlegend=True,domain= {'x': [0, .48]}, \n                    marker=dict(colors=color))\n\n    trace2 = go.Pie(labels=rev_trace[df_colum], \n                    values=rev_trace['totals.transactionRevenue'], name=\"% Revenue\", hole= .5, \n                    hoverinfo=\"label+percent+name\", showlegend=False, domain= {'x': [.52, 1]})\n\n    layout = dict(title= title, height=450, font=dict(size=15),\n                  annotations = [\n                      dict(\n                          x=.25, y=.5,\n                          text='Visits', \n                          showarrow=False,\n                          font=dict(size=20)\n                      ),\n                      dict(\n                          x=.80, y=.5,\n                          text='Revenue', \n                          showarrow=False,\n                          font=dict(size=20)\n                      )\n        ])\n\n    fig = dict(data=[trace1, trace2], layout=layout)\n    iplot(fig)","metadata":{"_kg_hide-input":true,"_uuid":"ba3cacf3d0cc127d8b35b41133604e6e26cbd571","trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Device Category feature","metadata":{"_uuid":"23e1f787c9d000554066aea293f8808a8de424af"}},{"cell_type":"code","source":"PieChart(\"device.deviceCategory\", \"Device Category\")","metadata":{"_uuid":"104d01069fe11a66ff3738f206b791d18150af3b","trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## I will apply the Prie Chart in Country's again","metadata":{"_uuid":"38b54ca57e57bf4ddade4cd7889485c7c88911e7"}},{"cell_type":"code","source":"# call the function\nPieChart(\"geoNetwork.city\", \"Top Cities by Accesses and Revenue\", limit=12)","metadata":{"_uuid":"1648ff7cdb2548efe8d357f0a9e88b6549451ce9","trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"- New York is responsible by 14% of visits and 31% of revenues.\n- Montain view have 19% in visists but just 16% of revenues\n- Chicago have just 3.5% of visits but have a high significance in revenues","metadata":{"_uuid":"6e77cfdac5dea3046ae40f35713b61d1272a2232","trusted":true}},{"cell_type":"markdown","source":"## Seeing again Channel Grouping more specified","metadata":{"_uuid":"3b95a7ba1b2337466a8e708019a668b8a4ed49e2"}},{"cell_type":"code","source":"PieChart(\"channelGrouping\", \"Channel Grouping Visits and Revenues\")","metadata":{"_uuid":"b3296caa545f703647e2e8e46acb0fb422370715","trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"It's interesting to note that Referral have a less number of Visits but is responsible for almost 40% of revenues****","metadata":{"_uuid":"73c48a5cbeb35949101fc3f909246563b457482a"}},{"cell_type":"markdown","source":"### Months in pizza graph","metadata":{"_uuid":"3706ce71368c3957579e38bdb40e032d5e22d18d"}},{"cell_type":"markdown","source":"## Let's see the NetWork Domain\n- I will plot visits and revenues by each category, including the non-set and unknown accesses and revenues\n","metadata":{"_uuid":"036612794be2040bac93d4918d73bb6d9f9b8515"}},{"cell_type":"code","source":"PieChart('geoNetwork.networkDomain', \"Network Domain\")","metadata":{"_uuid":"202155e1496d32e68b2e611bbfa9674b4e0ebc7f","trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Wow, another very cool information. \n\n- (not set) domain have almost 50% of total visits and 62% of Revenues.\n- Unknown is responsible by 28% of visits but just 2.70% of Revenues\n- comcast.net have 5.5% of visits and 7.4% Revenues.","metadata":{"_uuid":"b3dd496495bed44e84d481d2a8c2241ca138a212"}},{"cell_type":"markdown","source":"Let's take a look on Mobile and Browser proportions","metadata":{"_uuid":"81305283f850873c66a777edb7315611fd257680"}},{"cell_type":"code","source":"PieChart(\"device.deviceCategory\", \"Device Category\")","metadata":{"_uuid":"e1081175860b562099b45c115c751d3487f9c24c","trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"The absolutelly high part of revenues are from Desktop Devices","metadata":{"_uuid":"f11af04271ed24a89c23fde30403cff01020d129"}},{"cell_type":"markdown","source":"## Trafic Source Medium","metadata":{"_uuid":"620cc879e9f23ebf410b2f37ccf2bdedcca9fcbf"}},{"cell_type":"code","source":"PieChart(\"trafficSource.medium\", \"Trafic Source - Medium\")","metadata":{"_uuid":"04fb4fe637312415b073ecd3528bc1198881844a","trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"- Organic have highest number of visits but is the third in revenues\n- Referral have almost 40% in both Visits and Revenues\n- The none category have almost 16% of visits but almost 40% of revenues","metadata":{"_uuid":"e8755bf3ed97eca7561b5d27f1e3cd1311223c22","trusted":true}},{"cell_type":"markdown","source":"Now I will take a look on trafficSource section, the Source to access the store","metadata":{"_uuid":"dc86a9a774f05532d6c45ec817cfb7d2451db84f"}},{"cell_type":"code","source":"PieChart('trafficSource.source', \"Visits and Revenue by TOP Sources\", limit=8)","metadata":{"_uuid":"d86ecf58c561858347391fe8e16bf518dfcb4bd2","trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"We have a high number of visits from youtube but 0 sales. <br>\nthe mall.googleplex is have a low number of access but have the highest value in revenues","metadata":{"_uuid":"4cabc13925d8cc56bc28591806ac9975ce4cd11d"}},{"cell_type":"markdown","source":"# I will continue this notebook! Votes up the kernel and stay tuned to next updates","metadata":{"_uuid":"6be8ccdae823f3ccaad68ec01ac87a7c88862407"}},{"cell_type":"code","source":"df_train.corr()['totals.transactionRevenue']","metadata":{"_uuid":"6b63bc0f0ba6b6c9f7350878d11759576b370a08","trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Seeing the crosstab with heatmap","metadata":{"_uuid":"3259db00fe1f717acb96ae78a6f068b518c81cee"}},{"cell_type":"code","source":"country_repayment = ['channelGrouping', '_weekday'] #seting the desired \n\ncm = sns.light_palette(\"green\", as_cmap=True)\npd.crosstab(df_train[country_repayment[0]], df_train[country_repayment[1]], \n            values=df_train[\"totals.transactionRevenue\"], aggfunc=[np.sum]).style.background_gradient(cmap = cm)\n\n# tab.columns.levels[1] = [\"Sun\", \"Mon\", \"Thu\", \"wed\", \"Thi\",\"Fri\",\"Sat\"]","metadata":{"_uuid":"fe02aae49c1f4962bebdd02013d8a89f63e66380","trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"##  Geolocation plot to visually understand the data","metadata":{"_uuid":"4fd8dfe3a2014e8eadfdb3b8190ea866de926f23","trusted":true}},{"cell_type":"code","source":"# Counting total visits by countrys\ncountMaps = pd.DataFrame(df_train['geoNetwork.country'].value_counts()).reset_index()\ncountMaps.columns=['country', 'counts'] #renaming columns\ncountMaps = countMaps.reset_index().drop('index', axis=1) #reseting index and droping the column\n\ndata = [ dict(\n        type = 'choropleth',\n        locations = countMaps['country'],\n        locationmode = 'country names',\n        z = countMaps['counts'],\n        text = countMaps['country'],\n        autocolorscale = False,\n        marker = dict(\n            line = dict (\n                color = 'rgb(180,180,180)',\n                width = 0.5\n            ) ),\n        colorbar = dict(\n            autotick = False,\n            tickprefix = '',\n            title = 'Number of Visits'),\n      ) ]\n\nlayout = dict(\n    title = 'Couting Visits Per Country',\n    geo = dict(\n        showframe = False,\n        showcoastlines = True,\n        projection = dict(\n            type = 'Mercator'\n        )\n    )\n)\n\nfigure = dict( data=data, layout=layout )\niplot(figure, validate=False, filename='map-countrys-count')","metadata":{"_uuid":"f35db04e9e4dbceeb77029df6af763b1700d55cc","_kg_hide-input":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Total Revenues by Country ","metadata":{"_uuid":"932679723c6b45dd99e3d10709650571545b8361"}},{"cell_type":"code","source":"# I will crete a variable of Revenues by country sum\nsumRevMaps = df_train[df_train['totals.transactionRevenue'] > 0].groupby(\"geoNetwork.country\")[\"totals.transactionRevenue\"].count().to_frame().reset_index()\nsumRevMaps.columns = [\"country\", \"count_sales\"] # renaming columns\nsumRevMaps = sumRevMaps.reset_index().drop('index', axis=1) #reseting index and drop index column\n\ndata = [ dict(\n        type = 'choropleth',\n        locations = sumRevMaps['country'],\n        locationmode = 'country names',\n        z = sumRevMaps['count_sales'],\n        text = sumRevMaps['country'],\n        autocolorscale = False,\n        marker = dict(\n            line = dict (\n                color = 'rgb(180,180,180)',\n                width = 0.5\n            ) ),\n        colorbar = dict(\n            autotick = False,\n            tickprefix = '',\n            title = 'Count of Sales'),\n      ) ]\n\nlayout = dict(\n    title = 'Total Sales by Country',\n    geo = dict(\n        showframe = False,\n        showcoastlines = True,\n        projection = dict(\n            type = 'Mercator'\n        )\n    )\n)\n\nfigure = dict( data=data, layout=layout )\n\niplot(figure, validate=False, filename='map-countrys-total')","metadata":{"_uuid":"349766424b33710198240cc3f068b0a0926a250d","_kg_hide-input":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Some tests that I am doing to try find interesting feature engineering approaches","metadata":{"_uuid":"d0bce740990e19317f2495dbe741eda30a2b1aea"}},{"cell_type":"code","source":"df_train['month_unique_user_count'] = df_train.groupby('_month')['fullVisitorId'].transform('nunique')\ndf_train['day_unique_user_count'] = df_train.groupby('_day')['fullVisitorId'].transform('nunique')\ndf_train['weekday_unique_user_count'] = df_train.groupby('_weekday')['fullVisitorId'].transform('nunique')\n\n\ndf_train['traf_sourc_browser_count'] = df_train.groupby(['trafficSource.medium', 'device.browser'])['totals.pageviews'].transform('nunique')\ndf_train['Id_browser_pageviews_sumprod'] = df_train.groupby(['fullVisitorId', 'device.browser'])['totals.pageviews'].transform('cumprod')\ndf_train['Id_browser_hits_sumprod'] = df_train.groupby(['fullVisitorId', 'device.browser'])['totals.hits'].transform('cumprod')\ndf_train['Id_browser_hits_sumprod'] = df_train.groupby(['fullVisitorId', 'device.browser'])['totals.hits'].transform('cumprod')\ndf_train['Id_browser_hits_sumprod_mob'] = df_train.groupby(['fullVisitorId', 'device.browser', 'device.isMobile'])['totals.hits'].transform('sum')\n\ndf_train['Id_networkDomain_hits'] = df_train.groupby(['fullVisitorId', 'geoNetwork.networkDomain'])['totals.hits'].transform('var')\ndf_train['Id_networkDomain_country_hits'] = df_train.groupby(['fullVisitorId', 'geoNetwork.networkDomain', 'geoNetwork.country'])['totals.hits'].transform('unique')\n\n","metadata":{"_uuid":"0cb4f9106082d8d96af3c1236550dc33d0cd4175","_kg_hide-output":true,"_kg_hide-input":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df_train[[\"totals.transactionRevenue\", 'Id_browser_hits_sumprod', 'Id_networkDomain_hits','Id_networkDomain_country_hits', 'Id_browser_hits_sumprod_mob']].corr()","metadata":{"_uuid":"6437a9d5ff5bbb8b0eb6791e832cc2401926fc2c","_kg_hide-output":true,"_kg_hide-input":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Preprocessing the fulldataset and creating new features","metadata":{"_uuid":"da64b44e601cf75d0d75673f90f0641a4c0950d6","trusted":true}},{"cell_type":"code","source":"aggs = {\n    'date': ['min', 'max'],\n    'totals.hits': ['sum', 'min', 'max', 'mean', 'median'],\n    'totals.pageviews': ['sum', 'min', 'max', 'mean', 'median'],\n    'totals.bounces': ['sum', 'mean', 'median'],\n    'totals.newVisits': ['sum', 'mean', 'median']\n}\n\n# Previous applications categorical features\ncat_aggregations = {}\n\nfor cat in dummy_feaures:\n    cat_aggregations[cat] = ['min', 'max', 'mean']\n\nprev_agg = df_train.groupby('fullVisitorId').agg({**aggs})\n\nprev_agg.columns = pd.Index(['Agg_' + e[0] + \"_\" + e[1].upper() for e in prev_agg.columns.tolist()])","metadata":{"_uuid":"133ea13b036829110319da4f4e69eee0b6334185","trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"prev_agg","metadata":{"_uuid":"52dc6b6e78b98fb547f4a6012095c3fbb6b7b870","trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"new_columns = [\n        k + '_' + agg for k in aggs.keys() for agg in aggs[k]\n    ]\nnew_columns","metadata":{"_uuid":"a7ef38820e9fb49ad5f1d71f253b2fa803e80029","trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"dummy_feaures","metadata":{"_uuid":"d4b67b55347c758ce33b2e725c399cbb666e7e84","trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"### Testing some grouping approaches","metadata":{"_uuid":"b5bfd985138a4abbe288f12a8838a97e763b1bcf","trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df_train['cumcount'] = df_train.groupby('fullVisitorId').cumcount() + 1","metadata":{"_uuid":"55e6693014ca3cb1bc2938d64b6b3371b8a42ddd","trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Some tests to feature engineering","metadata":{"_uuid":"3139a6d399c80ee47c84ee0d05d5f9d9066bdbd6"}},{"cell_type":"code","source":"aggs = {\n    'date': ['min', 'max'],\n    'totals.transactionRevenue': ['sum', 'size'],\n    'totals.hits': ['sum', 'min', 'max', 'count', 'median'],\n    'totals.pageviews': ['sum', 'min', 'max', 'mean', 'median'],\n    'totals.bounces': ['sum', 'mean', 'median'],\n    'totals.newVisits': ['sum', 'mean', 'median']\n}\n\n# Previous applications categorical features\ncat_aggregations = {}\n\nfor cat in dummy_feaures:\n    cat_aggregations[cat] = ['min', 'max', 'mean']\n\nprev_agg = df_train.groupby('fullVisitorId').agg({**aggs})\n\nprev_agg.head()\n\n","metadata":{"_uuid":"29532ab5a4ec9c636d6d9937f678d708dc2b25e6","trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# I will continue working on this kernel, stay tuned\n\n******** Please, if you liked this kernel don't forget to votes up and give your feedback ******** ","metadata":{"_uuid":"23b75275bdf104dc889476245e3e9bbc43723b27","trusted":true}},{"cell_type":"code","source":"prev_agg.columns = [\"_\".join(x) for x in prev_agg.columns.ravel()]","metadata":{"_uuid":"8c8ee2fd6974dacf00d5590a9b701f7c10d7639c","trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"prev_agg.head()","metadata":{"_uuid":"6fb8904cdf0549f7011328aba6a18f5baac97f11","trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"","metadata":{"_uuid":"8f46e405dd6662edc927dd42aa30c97d57ab1e2b","trusted":true},"execution_count":null,"outputs":[]}]}