{"cells":[{"metadata":{"_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","trusted":false},"cell_type":"code","source":"# This Python 3 environment comes with many helpful analytics libraries installed\n# It is defined by the kaggle/python Docker image: https://github.com/kaggle/docker-python\n# For example, here's several helpful packages to load\n\nimport numpy as np # linear algebra\nimport pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\n\n# Input data files are available in the read-only \"../input/\" directory\n# For example, running this (by clicking run or pressing Shift+Enter) will list all files under the input directory\n\nimport os\nfor dirname, _, filenames in os.walk('/kaggle/input'):\n    for filename in filenames:\n        print(os.path.join(dirname, filename))\n\n# You can write up to 5GB to the current directory (/kaggle/working/) that gets preserved as output when you create a version using \"Save & Run All\" \n# You can also write temporary files to /kaggle/temp/, but they won't be saved outside of the current sessionj","execution_count":null,"outputs":[]},{"metadata":{"_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","trusted":false},"cell_type":"code","source":"train = pd.read_csv(\"/kaggle/input/cat-in-the-dat-ii/train.csv\")\ntrain.head()\n                    ","execution_count":null,"outputs":[]},{"metadata":{"trusted":false},"cell_type":"code","source":"test = pd.read_csv(\"/kaggle/input/cat-in-the-dat-ii/test.csv\")\ntest.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":false},"cell_type":"code","source":"train.info()\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":false},"cell_type":"code","source":"train_target = train.target\ntest_id = test.id\ntrain.drop([\"target\",\"id\"],1,inplace=True)\ntest.drop(\"id\",1,inplace = True)\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Few important points about the data we have in the dataset\nGiven dataset contains Numeric as well as Categorical Variables:-\n\n1. Following kind of categorical variabels are present in the dataset:\n\n    1.1 Binary Variables - values like Male & Female.\n\n    1.2 Ordinal Variables (low and high cardinality) - ordered values like \"Good\" \"V Good\" \"Excellent\".\n\n    1.3 Nominal Variables (low and high cardinality) - do not have any intrinsic order, values like blood groups \"A\" \"B\" etc","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"We will convert all the non-categorical variables to categorical variables, lets first get a list of numeric variables","execution_count":null},{"metadata":{"trusted":false},"cell_type":"code","source":"# get list of numeric variables\nnum_vars = [] \nfor col in train:\n    if train[col].dtypes != 'O':\n        num_vars.append(col)\nnum_vars ","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Lets convert all numeric variables to categorical variables","execution_count":null},{"metadata":{"trusted":false},"cell_type":"code","source":"# converting numeric variables to categorical variables\nfor cols in num_vars:\n    train[cols] = train[cols].astype('category')\n    test[cols] = test[cols].astype('category')\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Lets cross verify whether we have successfully converted the numeric fields to categorical fields or not","execution_count":null},{"metadata":{"trusted":false},"cell_type":"code","source":"# verify the dataset \ntrain.info()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"We have successfully converted all the viables to categorical type.","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"Lets check the number of unique values for each variable ","execution_count":null},{"metadata":{"trusted":false},"cell_type":"code","source":"# number of unique variables for each of the column present in the dataset\nfor col in train.columns:\n    print(col,\":\\n\",train[col].nunique(),\"\\n\")\n    ","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"There are lots of high cardinality features in the dataset.","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"# Model Based on DeepTables\n\nDeepTables(DT) is a easy-to-use toolkit that enables deep learning to unleash great power on tabular data.\n\nMLP (also known as Fully-connected neural networks) have been shown inefficient in learning distribution representation. The “add” operations of the perceptron layer have been proven poor performance to exploring multiplicative feature interactions. In most cases, manual feature engineering is necessary and this work requires extensive domain knowledge and very cumbersome.","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"DT follow these steps to build a neural network:\n\n1. Category features -> Embedding Layer.\n\n2. Continuous feature -> Dense Layer or to Embedding Layer after discretization/categorization.\n\n3. Embedding/Dense layers -> Feature Interactions/Extractions nets.\n\n4. Stacking(add/concat) outputs of nets as the output of the model.","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"# ModelConfig\n\nModelConfig is the most important parameter in DT. It is used to set how to clean and preprocess the data automatically, and how to assemble various network components to building a neural nets for prediction tasks, as well as the setting of hyper-parameters of nets, etc. If you do not change any settings in ModelConfig, DT will work in most cases as well. However, you can get a better performance by tuning the parameters in ModelConfig.","execution_count":null},{"metadata":{"trusted":false},"cell_type":"code","source":"!pip install deeptables\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":false},"cell_type":"code","source":"from deeptables.models.deeptable import DeepTable, ModelConfig\nfrom tensorflow.keras.utils import plot_model\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":false},"cell_type":"code","source":"n_folds=5\nepochs=1\nbatch_size=128","execution_count":null,"outputs":[]},{"metadata":{"trusted":false},"cell_type":"code","source":"%%time\n\nconf = ModelConfig(\n    dnn_params={\n        'hidden_units':((300, 0.3, True),(300, 0.3, True),), #hidden_units\n        'dnn_activation':'relu',\n    },\n    fixed_embedding_dim=True,\n    embeddings_output_dim=20,\n    nets =['linear','cin_nets','dnn_nets'],\n    stacking_op = 'add',\n    output_use_bias = False,\n    cin_params={\n       'cross_layer_size': (200, 200),\n       'activation': 'relu',\n       'use_residual': False,\n       'use_bias': True,\n       'direct': True, \n       'reduce_D': False,\n    },\n)\n\ndt = DeepTable(config = conf)\noof_proba, eval_proba, test_prob = dt.fit_cross_validation(\n    train, train_target, X_eval=None, X_test=test, \n    num_folds=n_folds, stratified=False, iterators=None, \n    batch_size=batch_size, epochs=epochs, verbose=1, callbacks=[], n_jobs=1)","execution_count":null,"outputs":[]},{"metadata":{"trusted":false},"cell_type":"code","source":"# lets prepare for the prediction submission \nsub = pd.DataFrame() \nsub['Id'] = test_id \nsub['target'] = test_prob\nsub.to_csv('submission_dt.csv',index=False)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Using the above model, I got Private Score of 0.78676 and Public Score of 0.78527.\nI used epoch = 1 and n_folds = 5 just to make this submission possible quickly.\n\nModel will produce much better results if you use n_folds = 50 and epoch = 100.\n\n","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"# Kindly upvote if you liked my kernel!","execution_count":null}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}