{"cells":[{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"code","source":"# This Python 3 environment comes with many helpful analytics libraries installed\n# It is defined by the kaggle/python docker image: https://github.com/kaggle/docker-python\n# For example, here's several helpful packages to load in \n\nimport numpy as np # linear algebra\nimport pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\npd.options.display.max_rows = 999\npd.options.display.max_columns = 999\nimport missingno as msno \n\n# Input data files are available in the \"../input/\" directory.\n# For example, running this (by clicking run or pressing Shift+Enter) will list all files under the input directory\n\nimport os\nfor dirname, _, filenames in os.walk('/kaggle/input'):\n    for filename in filenames:\n        print(os.path.join(dirname, filename))\n\n# Any results you write to the current directory are saved as output.","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"First off... this is intended to by my first notebook to share. It is basic, and simple, and probably useless for this particular competition (spoiler: it kind of is). But I think the process taught me a lot, and should be useful in real word datasets!\n\nIn this notebook I plan to analyze the missing data in our dataset. In the real world we typically have very messy data, and this data can be lost when aquiring, or deleted accidentally afterwards. The workflow will be:\n\n1. Check for missing values (are they NaN? are they some other representation for NULL, if so, convert these to null values).\n2. Analyze the amount of missing values, and the type.\n3. Either delete, or impute these values.\n4. Evaluate and compare the performance of each imputed option.\n\nThis notebook will cover the first 2 steps in this process using the missingno python library.  Intuitively, I expect this data to be completely random with regards to missing values, but that is an assumption!\n\nWe will concatenate our training and test sets for now (simply for analysis). And we will drop the target column, assuming our training set is complete.\n"},{"metadata":{"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","trusted":true},"cell_type":"code","source":"X_train = pd.read_csv('/kaggle/input/cat-in-the-dat-ii/train.csv', index_col='id')\nX_test = pd.read_csv('/kaggle/input/cat-in-the-dat-ii/test.csv', index_col='id')\n\ncat_df = X_train.append(X_test, sort=False)\ncat_df.drop('target', axis=1, inplace=True)\n\ncat_df.head()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Now let's do some basic EDA."},{"metadata":{"trusted":true},"cell_type":"code","source":"cat_df.info()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Some unique values on our features."},{"metadata":{"trusted":true},"cell_type":"code","source":"cols = list(cat_df.columns)\ncols\n\nfor col in cols:\n    temp = cat_df[col].unique()\n    print('Column Name: ', col)\n    print('Column Unique Values: ', temp)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"It certainly appears that all our missing values are correctly stored as NaN's. Lets get some basic statistics of our numeric data. (We will have to deal with the non-numeric data in the future)"},{"metadata":{"trusted":true},"cell_type":"code","source":"cat_df.describe()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Now lets see how much data is missing from each column."},{"metadata":{"trusted":true},"cell_type":"code","source":"for col in cols:\n    print('Column Name: ', col, ' ', sum(cat_df[col].isnull()), ' is missing.')\n    ","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"This looks like each column has a similar amount of data missing. Let's see if there are any patterns to our missing data. First using a matrix plotted through the missingno library."},{"metadata":{"trusted":true},"cell_type":"code","source":"msno.matrix(cat_df)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Now let's see a heatmap. "},{"metadata":{"trusted":true},"cell_type":"code","source":"msno.heatmap(cat_df)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"The heatmap shows no correlations, and a dendrogram can be done as well (but... I think we can see where this is heading). Any closely correlated features will be seen \"close to each other\" on leaves.  Long, \"deep\" branches show no correlations."},{"metadata":{"trusted":true},"cell_type":"code","source":"msno.dendrogram(cat_df)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"We can classify this dataset as having missing values completely at Random (MCAR).  From here on out, we will have to properly encode our data before imputing the missing values.  But we can safely say, that our data has no unique patterns within the missingness, that we will be concerned about!"}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":1}