{"cells":[{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"markdown","source":"## Overview\n\nThe purpose of this kernel is to take a look at the data, come up with some insights, and attempt to create a predictive model or two. This notebook is still **very** raw. I will work on it as my very limited time permits, and hope to expend it in the upcoming days and weeks.\n\n## Packages\n\nFirst, let's load a few useful Python packages. This section will keep growing in subsequent versions of this EDA."},{"metadata":{"_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","trusted":true},"cell_type":"code","source":"import os\nimport json\nimport numpy as np\nimport pandas as pd\nfrom pandas.io.json import json_normalize\nimport time\nimport lightgbm as lgb\nfrom sklearn.metrics import mean_squared_error\nfrom sklearn.model_selection import KFold\nimport matplotlib.pyplot as plt\nimport seaborn as sns\n\n%matplotlib inline","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"f9221ca3aca037d9275ef115903243f981d36191"},"cell_type":"markdown","source":"Now let us look at the input folder. Here we find all the relevant files for this competition."},{"metadata":{"trusted":true,"_uuid":"fee5b94da477ee40ec3dad4a000e570c68b114b2"},"cell_type":"code","source":"print(os.listdir(\"../input\"))","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"e2b09bd1c39d03b0a204ed0d9897608f0fb11c93"},"cell_type":"markdown","source":"We see that the input folder only contains three files ```train.csv```, ```test.csv```, and ```sample_submission.csv```. It seems that for this competition we don't have to do any complicated combination and mergers of files.\n\nNow let's import and take a glimpse at these files."},{"metadata":{"trusted":true,"_uuid":"2344fe25d5c601b813bdc7ca0419627dfc75f057"},"cell_type":"code","source":"train_df = pd.read_csv('../input/train.csv')\ntrain_df.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"f2f1aff76b2db5129cc7675cbb8ba6a823222a62"},"cell_type":"code","source":"train_df.shape","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"c115e6bf62a075c9f20686f0b2db28e82339d037"},"cell_type":"code","source":"train_df.info()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"f7797944d311cbf574b4a54315179c11715cf5e1"},"cell_type":"code","source":"train_df.isnull().values.any()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"919fea800f6acf326f1bdb8160a0889fe7792fb8"},"cell_type":"markdown","source":"A few things immediately stand out:\n\n1. There is almost a million datapoints, with 12 raw \"features\"\n2. The most interesting features seem to be in JSON format. We will need to do something about them to get them into a format that is suitable for modeling.\n3. There don't seem to be any missing values in the dataset, at least as far as the core raw features are concerned. Once we process JSON, it is possible that we'll find some missing values. Stay tuned.\n\nNow let's take a look at the test dataset."},{"metadata":{"trusted":true,"_uuid":"f3aeb3bf1e998b487babd034596f400e10389528"},"cell_type":"code","source":"test_df = pd.read_csv('../input/test.csv')\ntest_df.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"99ca5ad67baaf30f74f783db70b7e590a61f72d4"},"cell_type":"code","source":"test_df.shape","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"0df3f84193e746ca1c3026691769080f133450b1"},"cell_type":"code","source":"test_df.info()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"af7879f59c73b675c68363a2e0c3759f47df6ad5"},"cell_type":"code","source":"test_df.isnull().values.any()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"4d42851b751629ccac8b484cb8c49b659e46302c"},"cell_type":"markdown","source":"A few things about the test set that we notice:\n\n1. There is a close match in terms of the size with the train set (800,000 vs 900,000)\n2. It also contains the JSON values features.\n3. There don't seem to be any missing values in the dataset either.\n\nNow let's reload the data, dealing with the JSON fields. For this we'll use [Julian's kernel function](https://www.kaggle.com/julian3833/1-quick-start-read-csv-and-flatten-json-fields/notebook):\n"},{"metadata":{"trusted":true,"_uuid":"baed2990e9ec4d218a104dc319aa87ca2d2c7129"},"cell_type":"code","source":"def load_df(csv_path='../input/train.csv', nrows=None):\n    JSON_COLUMNS = ['device', 'geoNetwork', 'totals', 'trafficSource']\n    \n    df = pd.read_csv(csv_path, \n                     converters={column: json.loads for column in JSON_COLUMNS}, \n                     dtype={'fullVisitorId': 'str'}, # Important!!\n                     nrows=nrows)\n    \n    for column in JSON_COLUMNS:\n        column_as_df = json_normalize(df[column])\n        column_as_df.columns = [f\"{column}.{subcolumn}\" for subcolumn in column_as_df.columns]\n        df = df.drop(column, axis=1).merge(column_as_df, right_index=True, left_index=True)\n    print(f\"Loaded {os.path.basename(csv_path)}. Shape: {df.shape}\")\n    return df","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"60abf3708d98df571bc80e05d44f93e7a735cdaf"},"cell_type":"code","source":"%%time\ntrain_df = load_df()\ntest_df = load_df(\"../input/test.csv\")","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"fea47f40f0e19b4f8ab16719895c0585b4c42f22"},"cell_type":"markdown","source":"Let's now look at the structure of the flattened files:"},{"metadata":{"trusted":true,"_uuid":"3c43712b32c5a0440eaccd043b1d712d6255de17"},"cell_type":"code","source":"train_df.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"2e0c9b6488281579c1d0d4279d5f8bf3d76ea844"},"cell_type":"code","source":"train_df.info()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"9b18b7feeb72866e94ba87679dbe40097e0bd782"},"cell_type":"code","source":"train_df.isnull().values.any()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"6837f9c9a00f2887663afeaf2622e15792548645"},"cell_type":"code","source":"train_df_describe = train_df.describe()\ntrain_df_describe","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"7fc7a98893c764683d65aedcc12dc3d6c38b177f"},"cell_type":"markdown","source":"And now we'll look at the structure of the flattened test files:"},{"metadata":{"trusted":true,"_uuid":"ec60b40e68fbe969158eec973a44d72d36984a18"},"cell_type":"code","source":"test_df.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"a208fb43cc8913bb3e8183a3377bae5518fba2b8"},"cell_type":"code","source":"test_df.shape","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"e4eb9bb410e5f6935c6b30624cd910ebece6b962"},"cell_type":"code","source":"test_df.info()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"36f90c6f44aa263cbaf4bddfa1f1713a8fce9c94"},"cell_type":"code","source":"test_df_describe = test_df.describe()\ntest_df_describe","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"6e5823cce4dcede5aecd80740e2402a80a81fc2e"},"cell_type":"markdown","source":"The simplest \"model\" that we could build is to predict the mean value fro all test entries:"},{"metadata":{"trusted":true,"_uuid":"16d688ab0218a0f22f242a9adcba9a437276e131"},"cell_type":"code","source":"sample_submission = pd.read_csv('../input/sample_submission.csv')\nsample_submission.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"85e24279b3e8dd20a8ba4280f4f17d3a5d78cf09"},"cell_type":"code","source":"sample_submission['PredictedLogRevenue'] = np.log(1.26)\nsample_submission.to_csv('simple_mean.csv', index=False)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"eead169e430404c1d8c5a137c3f7d7ea290663b7"},"cell_type":"code","source":"sample_submission.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"db4c6fefab51ce0cf8c222788ed211da61fdb3bf"},"cell_type":"code","source":"","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"name":"python","version":"3.6.6","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"}},"nbformat":4,"nbformat_minor":1}