{"cells":[{"metadata":{"_uuid":"bf416e0b8eab0c94525b3bd32b7625436cc8ee27"},"cell_type":"markdown","source":"# Recalculating the Vote Number\nAccording to a [discussion section](https://www.kaggle.com/c/ga-customer-revenue-prediction/discussion/65989), Abhimanyu Dikshit\nsays that \"a  'bounced session' is usually one where less than a certain threshold amount of time was spent/actions taken(usually attributed to users who were looking for something else and landed on your site by mistake)\". We have a feature called 'totals.bounces'  which equals 1 when the session bound. 100% of the times that 'totals.bounces' equals 1 then totals.transactionRevenue is NaN (AKA 0). Therefore, I want to ignore these sessions in a new feature that gives the \"real\" Vote number. This code is very slow and I would love feedback on how to make it faster"},{"metadata":{"_uuid":"3196b360ea771f5adc1f6aeefcdefca8246333c7"},"cell_type":"markdown","source":"## Load Packages"},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"code","source":"# This Python 3 environment comes with many helpful analytics libraries installed\n# It is defined by the kaggle/python docker image: https://github.com/kaggle/docker-python\n# For example, here's several helpful packages to load in \n\nimport numpy as np # linear algebra\nimport pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\nimport matplotlib.pyplot as plt # to graphics plot\n\n# Input data files are available in the \"../input/\" directory.\n# For example, running this (by clicking run or pressing Shift+Enter) will list the files in the input directory\n\nimport os\nprint(os.listdir(\"../input\"))\n\n# Any results you write to the current directory are saved as output.","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"ca7a9e31e9a18336d2a547a776fb2293f226cc87"},"cell_type":"code","source":"import json # to convert json in df\nfrom pandas.io.json import json_normalize # to normalize the json file","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"683a77a4aba1de5228364dda025600f7d87f7e7d"},"cell_type":"code","source":"# keep track of how long the code takes...\nfrom datetime import datetime, date, time","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"03b69d92a45afbd61238a69caa640d23cb6b60e7"},"cell_type":"markdown","source":"## Import Datasets\nThis is the code from the [current kernel with the most votes](https://www.kaggle.com/sudalairajkumar/simple-exploration-baseline-ga-customer-revenue)"},{"metadata":{"trusted":true,"_uuid":"3d56c3e1f347aba4a0bae1c09ac06faf21f0b7b7"},"cell_type":"code","source":"def load_df(csv_path='../input/train.csv', nrows=None):\n    JSON_COLUMNS = ['device', 'geoNetwork', 'totals', 'trafficSource']\n    \n    df = pd.read_csv(csv_path, \n                     converters={column: json.loads for column in JSON_COLUMNS}, \n                     dtype={'fullVisitorId': 'str'}, # Important!!\n                     nrows=nrows)\n    \n    for column in JSON_COLUMNS:\n        column_as_df = json_normalize(df[column])\n        column_as_df.columns = [f\"{column}.{subcolumn}\" for subcolumn in column_as_df.columns]\n        df = df.drop(column, axis=1).merge(column_as_df, right_index=True, left_index=True)\n    print(f\"Loaded {os.path.basename(csv_path)}. Shape: {df.shape}\")\n    return df","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"dee970e64bc667c8b9ddfd8cf559d0c36b4b4e11"},"cell_type":"code","source":"%%time\ntrain_df = load_df()\ntest_df = load_df(\"../input/test.csv\")","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"832d671571634fb8b1eba86911199112723b408f"},"cell_type":"markdown","source":"### Add Column with \"True\" Vote Number"},{"metadata":{"_uuid":"3749969cec5e7a2592db899cf4b2f0b89930b4e8"},"cell_type":"markdown","source":"first I just filled some of the columns I will be using..."},{"metadata":{"trusted":true,"_uuid":"567ba158ea84f5a660c4645a4755eb6356197565"},"cell_type":"code","source":"train_df[\"totals.transactionRevenue\"].fillna(0, inplace=True)\ntrain_df[\"totals.bounces\"].fillna(0, inplace=True)\ntest_df[\"totals.bounces\"].fillna(0, inplace=True)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"41984e5f4081c86373a033b7866609d415a6032a"},"cell_type":"code","source":"def addVisitNumMinusBounces(df, newcol=\"visitNumber.Unbounced\"):\n    sorted_df = df.sort_values(by=[\"fullVisitorId\",\"visitNumber\"])\n    reindex_df = sorted_df.reset_index(drop=True)\n    reindex_df[newcol]=0\n    prev_val=0\n    prev_id=\"\"\n    sizeOfdf=len(reindex_df)\n    for index, row in reindex_df.iterrows():\n        if row[\"fullVisitorId\"] != prev_id:\n            if row[\"totals.bounces\"] == 1:\n                prev_val=0\n            else:\n                reindex_df.loc[index,newcol] = 1\n                prev_val=1\n        else:\n            if row[\"totals.bounces\"] != 1:\n                reindex_df.loc[index,newcol] = prev_val\n                prev_val = prev_val + 1\n        if index%10000==0:\n            print(\"%i rows complete out of %i\" % \n                  (index, sizeOfdf))\n            t =datetime.now()\n            print(t.strftime(\"%d/%m/%y %H:%M\"))\n        prev_id = row[\"fullVisitorId\"]\n    print(\"%i rows complete out of %i\\nDONE!\" %\n          (sizeOfdf, sizeOfdf))\n    return(reindex_df)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"9dcab0017c2e5a2a3d0abf64a385f094dd46e279"},"cell_type":"code","source":"%%time\ntrain_df = addVisitNumMinusBounces(train_df)\ntest_df = addVisitNumMinusBounces(test_df)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"8a3d8c64d6576d1b76930bd3ea97252f8a4fec05","trusted":true},"cell_type":"code","source":"train_df.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"76b2dcf43bd56a0ae100bd0ad63e79b015c14318"},"cell_type":"code","source":"test_df.head()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"1c58eb488a0508e90d9b52284af9b760a6dea9f5"},"cell_type":"markdown","source":""}],"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"name":"python","version":"3.6.6","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"}},"nbformat":4,"nbformat_minor":1}