{"cells":[{"metadata":{"_uuid":"380a02f3c84e16e558a802196dba5ba145c720a8"},"cell_type":"markdown","source":"This is a profile of the training data - i.e., an overview of variables with counts, distributions and so on. It's a good start to a more thorough EDA. It includes unpacking the JSON columns and looking at the contents."},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true,"scrolled":true,"_kg_hide-output":true},"cell_type":"code","source":"import json\nimport pandas as pd\nimport hvplot.pandas","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"b6a2ca31fc7d68e54a757a0316517f3f08c46245"},"cell_type":"markdown","source":"Here's our train data - notice the lovely JSON!"},{"metadata":{"_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"train = pd.read_csv('../input/train.csv', dtype={'fullVisitorId': str, 'date': str}, parse_dates=['date'],\n        index_col=('fullVisitorId', 'sessionId'))\ntrain.head()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"1227f2a1b80ce675ddd371398363d5605208ea77"},"cell_type":"markdown","source":"We'll start with the non-json columns first. Note that the bar plots only include the top 50 categories."},{"metadata":{"trusted":true,"_uuid":"3e1b45534baaa381dacbae4be73ecb5f0082867a","scrolled":false},"cell_type":"code","source":"json_cols = ['device', 'geoNetwork', 'totals', 'trafficSource']\ntrain_jsons = train[json_cols]\ntrain_nonjsons = train.drop(json_cols, axis=1)\n\ndef profile(df):\n    numerics = df.select_dtypes(exclude=['bool', 'object']).columns.tolist()\n    try:\n        display(df.hvplot.hist(numerics, height=300, width=350, subplots=True, shared_axes=False).cols(2))\n    except: print(\"no numerics\")\n    for n in df.select_dtypes(['bool', 'object']).columns:\n        dcounts = df[n].value_counts(normalize=True)\n        dcounts_df = pd.DataFrame({'label': dcounts.index.tolist(), 'percent of total': dcounts})\n        dcounts_df.reset_index(drop=True, inplace=True)\n        display(dcounts_df[0:50].hvplot.bar(x='label', y='percent of total', invert=True, flip_yaxis=True, \n               height=450, width=450, ylim=(0,1), title=n))\nprofile(train_nonjsons)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"00b0752e14872767b48bdef186245b38e861f0ba"},"cell_type":"markdown","source":"Now we can do the json columns. You'll see the target to predict, transactionRevenue, as part of the 'totals' display."},{"metadata":{"trusted":true,"_uuid":"f7e89fdf63bef256d197a8a56a0886137b886459","scrolled":true},"cell_type":"code","source":"for jc in json_cols: \n    print(jc)\n    flat_df = pd.io.json.json_normalize(train_jsons[jc].apply(json.loads)) \n    flat_df = flat_df.apply(pd.to_numeric, errors='ignore')\n    profile(flat_df)\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"6afb28d315a8eefa6af18457d2f928f12866b777"},"cell_type":"markdown","source":"All for now, good luck!"}],"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"name":"python","version":"3.6.6","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"}},"nbformat":4,"nbformat_minor":1}