{"cells":[{"metadata":{"_uuid":"48c2a59b308779c1b04df11a853ddd2742326c1a"},"cell_type":"markdown","source":"This kernel aims to give a way of importing fairly big datasets when there is not much RAM available. Or, more simply, when you want to use your RAM for something more than reading a csv file.\n\nThe idea is simply to import column by column and drop it if the column is not useful. Thus I will drop a column if it contains only 1 unique value or, but in this case it is just an example, if more than 70% of the entries are missing.\n\nIn this way, I was able to load both datasets in pandas DataFrames on a laptop with only 8 GB of RAM.\n\nA word of caution before going into the code: dropping missing data is not necessarily a good strategy because data might be missing for a reason.\n\nI thank Julian Peller for his [nice kernel](https://www.kaggle.com/julian3833/1-quick-start-read-csv-and-flatten-json-fields) that showed me the core procedure."},{"metadata":{"trusted":false,"_uuid":"83f83a4bed138a3cf373575c1af9dae4a4040a62"},"cell_type":"code","source":"import json\nimport numpy as np\nimport pandas as pd\nfrom pandas.io.json import json_normalize\nimport gc","execution_count":null,"outputs":[]},{"metadata":{"trusted":false,"_uuid":"e80a4b9e4705f7ee231a717b47df071d11ecadcd"},"cell_type":"code","source":"def zero_entropy(data):\n    const_cols = [c for c in data.columns if data[c].nunique(dropna=False) == 1]\n    if len(const_cols) > 0:\n        print(\"The following columns will be dropped since they have only one value: \\n\")\n        print(const_cols)\n        for col in const_cols:\n            del data[col]\n    return data\n\n\ndef na_dropper(data):\n    tot = data.shape[0]\n    for col in data.columns:\n        mis = data[col].isna().sum()\n        if ((mis/tot) > 0.7) and ('transactionRevenue' not in col): # quick escape from making a mistake\n            print(\"The column {} will be dropped because more than 70% of the entries are missing\".format(col))\n            del data[col]\n    return data\n\n            \ndef light_import(data_path):\n    # first, simple columns\n    simple_cols = ['channelGrouping', 'fullVisitorId', 'sessionId', \n              'visitId', 'visitNumber', 'visitStartTime']\n    result = pd.read_csv(data_path, usecols=simple_cols, dtype={'fullVisitorId': 'str'})\n    # cleaning useless columns\n    result = zero_entropy(result)\n    result = na_dropper(result)\n    # then focus on the complex column\n    complex_cols = ['geoNetwork', 'device', 'totals', 'trafficSource']\n    for col in complex_cols:\n        print(\"Importing {}...\".format(col)) # to watch something happening\n        tmp = pd.read_csv(data_path, usecols=[col])\n        tmp = json_normalize(tmp[col].apply(json.loads))\n        tmp.columns = [f\"{col}_{subcolumn}\" for subcolumn in tmp.columns]\n        # cleaning columns\n        tmp = zero_entropy(tmp)\n        tmp = na_dropper(tmp)\n        # mergin what is left\n        result = result.merge(tmp, left_index=True, right_index=True)\n        # remove the garbage\n        del tmp\n        gc.collect()\n    return result","execution_count":null,"outputs":[]},{"metadata":{"trusted":false,"_uuid":"fdc46d7ce096d32a3f15f63b78f5a650582fa34b"},"cell_type":"code","source":"%%time\ndf_train = light_import('../input/train.csv')","execution_count":null,"outputs":[]},{"metadata":{"trusted":false,"_uuid":"8e2e63f9811f19a9909f61932365fabaf1bce1a2"},"cell_type":"code","source":"%%time\ndf_test = light_import('../input/test.csv')","execution_count":null,"outputs":[]},{"metadata":{"trusted":false,"_uuid":"b73e5924b18e1cc40ac0b2007e8894a373536d63"},"cell_type":"code","source":"df_train.columns","execution_count":null,"outputs":[]},{"metadata":{"trusted":false,"_uuid":"39fb9882b9b9ac1e99786d3e857c4ffcf4a6f76e"},"cell_type":"code","source":"","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"codemirror_mode":{"name":"ipython","version":3},"file_extension":".py","mimetype":"text/x-python","name":"python","nbconvert_exporter":"python","pygments_lexer":"ipython3","version":"3.6.6"}},"nbformat":4,"nbformat_minor":1}