{"cells":[{"metadata":{},"cell_type":"markdown","source":"# Goal of the kernel\nI just discovered (thanks to the NFL BigDataBowl Competition) an awesome package called `pandas_profiling`. It allows to generate a comprehensive report on a dataset, way more detailed than a `.describe()` function would do. \nIn detail, it prints interesting visualisations for the variables (distribution, missing values, types, etc.), the correlations, the missing values, and then it shows some samples of the dataframe. \nThe report is generated in HTML and allows you to dive into it by examining extreme values, histograms, plots, etc.\nHope you'll enjoy it!"},{"metadata":{},"cell_type":"markdown","source":"- **v_1 update**: I added the analysis of the test set. This way, you can compare both distributions to see if they match (and if not, you can correct the data).\n- **v_2 update**: increased the size of the samples to make them more representative (but cannot increase them much more due to memory limitations) (+ typo fixes)."},{"metadata":{},"cell_type":"markdown","source":"From the documentation of the Github page (https://github.com/pandas-profiling/pandas-profiling): for each column the following statistics – if relevant for the column type – are presented in an HTML report :\n- Essentials : type, unique values, missing values\n- Quantile statistics like minimum value, Q1, median, Q3, maximum, range, interquartile range and other descriptive statistics\n- Descriptive statistics like mean, mode, standard deviation, sum, median absolute deviation, coefficient of variation, high-order moments, mode\n- Histograms\n- Correlations highlighting of highly correlated variables, Spearman, Pearson and Kendall matrices\n- Missing values matrix, count, heatmap and dendrogram of missing values"},{"metadata":{},"cell_type":"markdown","source":"### Imports"},{"metadata":{"trusted":true},"cell_type":"code","source":"# Uncomment behind to install the package\n# !pip install pandas-profiling","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"import gc\nimport pandas_profiling\n\nimport pandas as pd\nimport numpy as np\nimport matplotlib.pyplot as plt\n\nfrom time import time\nfrom pandas.api.types import is_datetime64_any_dtype as is_datetime\nfrom pandas.api.types import is_categorical_dtype","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Loading and quick preprocessing"},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"def reduce_mem_usage(df, use_float16=False, log=False):\n    \"\"\" Iterate through all the columns of a dataframe and modify the data type to reduce memory usage \"\"\"\n\n    start_mem = df.memory_usage().sum() / 1024**2\n    if log: print(\"Memory usage of dataframe is {:.2f} MB\".format(start_mem))\n    \n    for col in df.columns:\n        if is_datetime(df[col]) or is_categorical_dtype(df[col]):      # manage categorical columns\n            continue\n        col_type = df[col].dtype\n        \n        if col_type != object:\n            c_min, c_max = df[col].min(), df[col].max()\n            if str(col_type)[:3] == \"int\":                             # manage columns of type int\n                if c_min > np.iinfo(np.int8).min and c_max < np.iinfo(np.int8).max:\n                    df[col] = df[col].astype(np.int8)\n                elif c_min > np.iinfo(np.int16).min and c_max < np.iinfo(np.int16).max:\n                    df[col] = df[col].astype(np.int16)\n                elif c_min > np.iinfo(np.int32).min and c_max < np.iinfo(np.int32).max:\n                    df[col] = df[col].astype(np.int32)\n                elif c_min > np.iinfo(np.int64).min and c_max < np.iinfo(np.int64).max:\n                    df[col] = df[col].astype(np.int64)  \n            else:                                                     # manage columns of type float\n                if use_float16 and c_min > np.finfo(np.float16).min and c_max < np.finfo(np.float16).max:\n                    df[col] = df[col].astype(np.float16)\n                elif c_min > np.finfo(np.float32).min and c_max < np.finfo(np.float32).max:\n                    df[col] = df[col].astype(np.float32)\n                else:\n                    df[col] = df[col].astype(np.float64)\n        else:\n            df[col] = df[col].astype(\"category\")\n\n    end_mem = df.memory_usage().sum() / 1024**2\n    if log: print(\"Memory usage after optimization is: {:.2f} MB\".format(end_mem))\n    if log: print(\"Decreased by {:.1f}%\".format(100 * (start_mem - end_mem)/start_mem))\n    \n    return df","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Credits to https://www.kaggle.com/rohanrao/ashrae-half-and-half for adapting the `reduce_mem_usage` function"},{"metadata":{"trusted":true},"cell_type":"code","source":"start = time()\n\nprint('Loading csvs \\n')\nbuilding      = pd.read_csv('../input/ashrae-energy-prediction/building_metadata.csv')\nweather_train = pd.read_csv('../input/ashrae-energy-prediction/weather_train.csv')\nweather_test  = pd.read_csv('../input/ashrae-energy-prediction/weather_test.csv')\ntrain         = pd.read_csv('../input/ashrae-energy-prediction/train.csv')\ntest          = pd.read_csv('../input/ashrae-energy-prediction/test.csv')\n\nprint('Reducing memory usage for train and test \\n')\ntrain = reduce_mem_usage(train, use_float16=True)\ntest = reduce_mem_usage(test, use_float16=True)\n\nprint('Merging train and test datasets with building dataset \\n')\ntrain = train.merge(building, on='building_id', how='left')\ntest = test.merge(building,   on='building_id', how='left')\n\nprint('Merging train and test datasets with weather dataset \\n')\ntrain = train.merge(weather_train, on=['site_id', 'timestamp'], how='left')\ntest = test.merge(weather_test,    on=['site_id', 'timestamp'], how='left')\n\nprint('gc \\n')\ndel weather_train, weather_test, building\ngc.collect()\n\nprint(round(time()-start, 1), ' seconds elapsed')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"As the dataset is very big, I have to sample just a few columns to make it runnable on Kaggle kernels but don't hesitate to take more columns if you can!"},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"train_small = train.sample(int(1e5))\ntest_small = test.sample(int(1e5))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Train Visualisation Report\n"},{"metadata":{"trusted":true},"cell_type":"code","source":"pandas_profiling.ProfileReport(train_small)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Test Visualisation Report"},{"metadata":{"trusted":true},"cell_type":"code","source":"pandas_profiling.ProfileReport(test_small)","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":1}