{"cells":[{"metadata":{},"cell_type":"markdown","source":"# Beginner Friendly Guide: Flatten JSON Fields\nThanks to this [wonderful kernel](https://www.kaggle.com/julian3833/1-quick-start-read-csv-and-flatten-json-fields/notebook) by [Julian](https://www.kaggle.com/julian3833), we can flatten the json fields in the input file. I created this kernel based on his in order to add some basic explanations that weren't so clear to me when I first read his kernel (this is my very first kaggle competition without any tutorials :D)"},{"metadata":{},"cell_type":"markdown","source":"# Necessary Imports:"},{"metadata":{"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","trusted":true},"cell_type":"code","source":"import numpy as np\nimport pandas as pd\nimport os\nimport json\nfrom pandas.io.json import json_normalize","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Finding the csv Path in Kaggle:"},{"metadata":{"trusted":true},"cell_type":"code","source":"import os\nfor dirname, _, filenames in os.walk('/kaggle/input'):\n    for filename in filenames:\n        print(os.path.join(dirname, filename))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Loading the Data:\n\n   * ``pd.read_csv`` takes has an optional parameter called `converters` which is a dict of functions for converting values in certain columns. Keys can either be integers or column labels.\n   * ``json.loads(s)``: is a function that deserializes ``s`` (a ``str``, ``bytes`` or ``bytearray`` instance containing a JSON document) to a Python object.\n   * The dataset contains many json objects in which every field must be converted to a seperate column, for instance the `totals` column looks like this:\n   ```\n   {\"visits\": \"1\", \"hits\": \"1\", \"pageviews\": \"1\", \"bounces\": \"1\", \"newVisits\": \"1\"}\n   ```\nWhat we need is to have each field as a seperate column: totals_visits,totals_hits...etc.\n   * We need to deserialize each one of these json objects to a Python object. This can be done by assigning a dict of json_column names each one of them mapped to ``json.loads``. Then, each one of these python object, that resulted from the deserialization of a json object, needs to be normalized to a flat table, i.e. every field must be converted to a seperate column. Thsi is achieved using pandas utility ``json_normalize``  "},{"metadata":{"trusted":true},"cell_type":"code","source":"CSV_PATH='/kaggle/input/ga-customer-revenue-prediction/train.csv'\nJSON_COLUMNS = ['device','geoNetwork', 'totals', 'trafficSource']\ndef load_data(csv_path=CSV_PATH, nrows=None, json_cols=JSON_COLUMNS):\n    df = pd.read_csv(csv_path, # engine='python', \n                     converters={col: json.loads for col in json_cols}, \n                     dtype={'fullVisitorId': 'str'},\n                     nrows=nrows)\n    for col in json_cols:\n        # normalizing (flattening) each json column\n        col_as_df = json_normalize(df[col])\n        # renaming each column of the new dataframe that resulted form\n        # normalization by concatenating its name to the name of the json column \n        # from which it was extracted so that we can keep track of the \n        # significance of the columns\n        col_as_df.columns = [f\"{col}.{subcol}\" for subcol in col_as_df.columns]\n        # replacing the original json column by the new dataframe we obtained above\n        df = df.drop(col, axis=1).merge(col_as_df, right_index=True, left_index=True)\n    print(f\"Loaded {os.path.basename(csv_path)}. Shape: {df.shape}\")\n    return df\n        ","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"%%time\ncustomer_data_flattened = load_data()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Saving to Feather Format:\n## What is Feather?\n\nFeather is a fast, lightweight, and easy-to-use binary file format for storing data frames. It has a few specific design goals:\n\n   * **Lightweight, minimal API**: make pushing data frames in and out of memory as simple as possible\n\n   * **Language agnostic**: Feather files are the same whether written by Python or R code. Other languages can read and write Feather files, too.\n\n   * **High read and write performance**: When possible, Feather operations should be bound by local disk performance.\n\n## Limitations:\nSome features of pandas are not supported in Feather:\n\n   * Non-string column names\n   * Row indexes\n   * Object-type columns with non-homogeneous data\n"},{"metadata":{"trusted":true},"cell_type":"code","source":"os.makedirs('tmp', exist_ok=True)\ncustomer_data_flattened.to_feather('tmp/ga-customer-data-flattened.feather')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# to read your data in the futur uncomment the following line of code:\ncustomer_data_flattened = pd.read_feather('tmp/ga-customer-data-flattened.feather')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"***I hope this kernel helped at least one beginner like myself. If you have any questions or feedback I'd love to hear them. Enjoy :D***"}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":1}