{"cells":[{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"markdown","source":"## Categorical Feature Encoding Challenge II\n\nIn this kernel you can find EDA for Categorical Feature Encoding Challenge. Basic feature engineering. Creating of dataset for PyTorch. PyTorch fully connected NN with Embeddings for categorical features. CatBoost usage. \n***\nContents:\n* [Imports](#Imports)\n\n* [EDA](#EDA)\n\n* [Feature engineering](#Feature-engineering)\n\n* [Features encoding](#Features-encoding)\n\n* [EDA p.2](#EDA-p.2)\n\n* [Features encoding p.2](#Features-encoding-p.2)\n\n* [PyTorch Dataset](#PyTorch-Dataset)\n\n* [PyTorch Model](#PyTorch-Model)\n\n* [PyTorch Train](#PyTorch-Train)\n\n* [NN Predictions](#NN-Predictions)\n\n* [CatBoost](#CatBoost)\n\n* [CatBoost predictions](#CatBoost-predictions)\n\n* [Average predictions](#Average-predictions)\n\n***\n\n#### Acknowledgments\n\n1. [A Neural Network in PyTorch for Tabular Data with Categorical Embeddings](https://yashuseth.blog/2018/07/22/pytorch-neural-network-for-tabular-data-with-categorical-embeddings/) - for great explanation of PyTorch magic.\n2. [3 Ways to Encode Categorical Variables for Deep Learning](https://machinelearningmastery.com/how-to-prepare-categorical-data-for-deep-learning-in-python/) - for the best explanation of encodings, as always. \n3. [Categorical Data EDA & Visualization](https://www.kaggle.com/subinium/categorical-data-eda-visualization) - for awesome EDA.\n4. [Cat in dat 2: OHE,Target + Logit](https://www.kaggle.com/pavelvpster/cat-in-dat-2-ohe-target-logit) - for new features."},{"metadata":{"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","collapsed":true,"_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","trusted":false},"cell_type":"markdown","source":"## Imports"},{"metadata":{"trusted":true},"cell_type":"code","source":"import warnings\nwarnings.filterwarnings('ignore')\n\nimport os\n\nimport pandas as pd\nimport numpy as np\nfrom tqdm import tqdm_notebook as tqdm\n\nimport matplotlib.pyplot as plt\nplt.style.use('ggplot')\nimport seaborn as sns\n\nfrom sklearn.model_selection import train_test_split\nfrom sklearn import metrics, preprocessing","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"for dirname, _, filenames in os.walk('/kaggle/input'):\n    for filename in filenames:\n        print(os.path.join(dirname, filename))\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train = pd.read_csv('/kaggle/input/cat-in-the-dat-ii/train.csv')\ntest = pd.read_csv('/kaggle/input/cat-in-the-dat-ii/test.csv')\nsample_submission = pd.read_csv('/kaggle/input/cat-in-the-dat-ii/sample_submission.csv')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"print(f'Train shape: {train.shape}',\n      f'Test shape: {test.shape}',\n      f'Submission shape: {sample_submission.shape}', sep=' | ')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train.head()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## EDA"},{"metadata":{"trusted":true},"cell_type":"code","source":"sns.countplot(train.target)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"There is a class disbalance. "},{"metadata":{},"cell_type":"markdown","source":"It is easier to do feature engineering or data cleaning, if we will connect <i>train</i> and <i>test</i> data."},{"metadata":{"trusted":true},"cell_type":"code","source":"all_df = pd.concat([train, test], axis=0, ignore_index=True)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"all_df.index.nunique(), len(all_df.index)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"all_df = all_df.drop('id', axis=1)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"all_df.head()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Let's check number of unique values in every column of our dataset. It will help us in finding out, which type of encoding we can use."},{"metadata":{"trusted":true},"cell_type":"code","source":"nunique_vals = list()\n\nfor column in all_df:\n    nunique_vals.append(all_df[column].nunique())\n    \npd.DataFrame({'columns': all_df.columns,\n              'num_of_unique': nunique_vals})","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"> In statistics, nominal data (also known as nominal scale) is a type of data that is used to label variables without providing any quantitative value. It is the simplest form of a scale of measure. Unlike ordinal data, nominal data cannot be ordered and cannot be measured.\n\n> In statistics, ordinal data are the type of data in which the data values follow a natural order. One of the most notable features of ordinal data is that the differences between the data values cannot be determined or are meaningless. Generally, the data categories lack the width representing the equal increments of the underlying attribute.\n\nSo, for <i>bin_{0-4}</i> there is no need in some special type of encodings. We can just change strings to numbers. Next: to encode day, month and <i>nom_{0-4}</i> we cant use [OHE](https://hackernoon.com/what-is-one-hot-encoding-why-and-when-do-you-have-to-use-it-e3c6186d008f). It will be great to encode <i>num_{5-9}</i> using OHE, too. But my hardware is not ready for such experiments... Features <i>ord_{0-5}</i> we will encode using [LabelEncoding](https://scikit-learn.org/stable/modules/generated/sklearn.preprocessing.LabelEncoder.html)."},{"metadata":{},"cell_type":"markdown","source":"Let's print all features with the unique values and amount of NAN values."},{"metadata":{"trusted":true},"cell_type":"code","source":"for column in all_df.columns:\n\n    unique_values = all_df[column].unique()\n    \n    print(f'Statistics fot column: {column}')\n    print(f'Column unique values:\\n {unique_values}')\n    print(f'Number of unique values: {len(unique_values)}')\n    print(f'Number of NAN values: {all_df[column].isna().sum()}')\n    print('_' * 50)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"So, it is really hard to understand something from this data. Maybe there is a connection between <i>ord_2</i> (it can be temperature) and <i>month</i>. Let's try to check."},{"metadata":{"trusted":true},"cell_type":"code","source":"month_temperature = all_df.groupby(['month', 'ord_2'])['ord_2'].count().to_frame()\nmonth_temperature = month_temperature.rename(columns={'ord_2': 'num_of_days'})\nmonth_temperature = month_temperature.reset_index()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"month_temperature.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"plt.rcParams.update({'font.size': 25})\n\nmonth = 1\n\nfor i in range(6):\n    fig, ax = plt.subplots(1, 2, figsize=(60, 20))\n    \n    for j in range(2):\n        \n        mt_part = month_temperature.loc[month_temperature.month == month]\n\n        ax[j].set_title(month)\n        sns.barplot(x=mt_part['ord_2'], y=mt_part['num_of_days'], ax=ax[j])\n        \n        month += 1\n    \n    plt.show()\n    plt.pause(0.1)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"There is almost no difference between months. Let's try some feature engineering, and after it again, EDA. Maybe we will find something."},{"metadata":{},"cell_type":"markdown","source":"## Feature engineering"},{"metadata":{},"cell_type":"markdown","source":"Feature <i>ord_5</i> consist of two letters, so we can divide it on two features."},{"metadata":{"trusted":true},"cell_type":"code","source":"all_df['ord_5_1'] = all_df['ord_5'].str[0]\nall_df['ord_5_2'] = all_df['ord_5'].str[1]\n\nall_df = all_df.drop('ord_5', axis=1)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"all_df['nan_features'] = all_df.isna().sum(axis=1)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"It is great to use [sin and cos transformations](https://www.kaggle.com/avanwyk/encoding-cyclical-features-for-deep-learning) to encode cyclical features."},{"metadata":{"trusted":true},"cell_type":"code","source":"all_df['month_sin'] = np.sin((all_df['month'] - 1) * (2.0 * np.pi / 12))\nall_df['month_cos'] = np.cos((all_df['month'] - 1) * (2.0 * np.pi / 12))\n\nall_df['day_sin'] = np.sin((all_df['day'] - 1) * (2.0 * np.pi / 7))\nall_df['day_cos'] = np.cos((all_df['day'] - 1) * (2.0 * np.pi / 7))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"So, now we have continuous features, there is no need to encode them, of course."},{"metadata":{},"cell_type":"markdown","source":"## Features encoding"},{"metadata":{"trusted":true},"cell_type":"code","source":"categorical = ['bin_0', 'bin_1', 'bin_2', 'bin_3',\n               'bin_4', 'day', 'month', 'nom_0',\n               'nom_1', 'nom_2', 'nom_3', 'nom_4',\n               'nom_5', 'nom_6', 'nom_7', 'nom_8',\n               'nom_9', 'ord_0', 'ord_1', 'ord_2', \n               'ord_3', 'ord_4', 'ord_5_1', 'ord_5_2']\n\nnom_5_9 = ['nom_5', 'nom_6', 'nom_7', 'nom_8', 'nom_9']\n\ncontinuous = ['month_sin', 'month_cos',\n              'day_sin', 'day_cos']","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"features = [x for x in all_df.columns \n            if x not in ['id', 'target'] + continuous]","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Using LabelEncoding we just change string values to numbers."},{"metadata":{"trusted":true},"cell_type":"code","source":"for feat in tqdm(features):\n    lbl_enc = preprocessing.LabelEncoder()\n    \n    all_df[feat] = lbl_enc.fit_transform(all_df[feat]. \\\n                                         fillna('-1'). \\\n                                         astype(str).values)\n    \nall_df['target'] = all_df['target'].fillna(-1)\nall_df[continuous] = all_df[continuous].fillna(-2)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## EDA p.2"},{"metadata":{"trusted":true},"cell_type":"markdown","source":"It is great to find if there is some correlation between categorical features. So, what can we do?\n\n> What we need is something that will look like correlation, but will work with categorical values — or more formally, we’re looking for a measure of association between two categorical features. Introducing: Cramér’s V. It is based on a nominal variation of Pearson’s Chi-Square Test, and comes built-in with some great benefits\n\nAnd another method:\n\n> Theil’s U, also referred to as the Uncertainty Coefficient, is based on the conditional entropy between x and y — or in human language, given the value of x, how many possible states does y have, and how often do they occur.\n\nHuge thanks to Shaked Zychlinski for his great [article](https://towardsdatascience.com/the-search-for-categorical-correlation-a1cf7f1888c9) and [code](https://github.com/shakedzy/dython/blob/master/dython/nominal.py)."},{"metadata":{"trusted":true},"cell_type":"code","source":"from scipy.stats import chi2_contingency, entropy\nfrom collections import Counter\n\n\ndef cramers_v(x, y):\n    \"\"\"\n        Calculates Cramer's V statistic for categorical-categorical association.\n        \n        :param x: pd.Series or np.array\n        :param y: pd.Series or np.array \n        \n        :return: Cramer's V statistic, float in range of [0, 1]\n    \"\"\"\n    \n    confusion_matrix = pd.crosstab(x, y)\n    \n    chi2 = chi2_contingency(confusion_matrix)[0]\n    \n    n = confusion_matrix.sum().sum()\n    \n    r, k = confusion_matrix.shape\n    phi_2 = chi2 / n\n    \n    phi2corr = max(0, phi_2 - ((k - 1) * (r - 1)) / (n - 1))\n    \n    rcorr = r - ((r - 1) ** 2) / (n - 1)\n    kcorr = k - ((k - 1) ** 2) / (n - 1)\n    \n    res = np.sqrt(phi2corr / min(kcorr - 1, rcorr - 1))\n    \n    return res\n\ndef conditional_entropy(x, y):\n    \"\"\"\n        Calculates the conditional entropy of x given y: S(x|y)\n    \n        :param x: pd.Series or np.array\n        :param y: pd.Series or np.array \n        \n        :return: float\n    \"\"\"\n    \n    y_counter = Counter(y)\n    xy_counter = Counter((list(zip(x, y))))\n    \n    total_occurrences = sum(y_counter.values())\n    entropy = 0.0\n    \n    for xy in xy_counter.keys():\n        p_xy = xy_counter[xy] / total_occurrences\n        p_y = y_counter[xy[1]] / total_occurrences\n        \n        entropy += p_xy * np.log(p_y / p_xy)\n        \n    return entropy\n\ndef theils_u(x, y):\n    \"\"\"\n        Calculates Theil's U statistic (Uncertainty coefficient) for categorical-categorical association.\n    \n        :param x: pd.Series or np.array\n        :param y: pd.Series or np.array \n        \n        :return: Theil's U statistic, float in range of [0, 1]\n    \"\"\"\n    \n    s_xy = conditional_entropy(x, y)\n    x_counter = Counter(x)\n    \n    total_occurrences = sum(x_counter.values())\n    \n    p_x = list(map(lambda n: n / total_occurrences, x_counter.values()))\n    s_x = entropy(p_x)\n    \n    if s_x == 0:\n        return 1\n    \n    else:\n        return (s_x - s_xy) / s_x","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"plt.rcParams.update({'font.size': 10})","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"%%time\n\nplt.subplots(figsize=(18, 18))\nplt.title('Cramers V')\n\ncorr_res = round(all_df.corr(method=cramers_v), 2)\nsns.heatmap(corr_res, annot=True)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"%%time\n\nplt.subplots(figsize=(18, 18))\nplt.title('Pearson')\n\n\ncorr_simple_res = round(all_df.corr(), 2)\nsns.heatmap(corr_simple_res, annot=True)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"%%time\n\nplt.subplots(figsize=(18, 18))\nplt.title('Uncertainty coefficient')\n\ncorr_res = round(all_df.corr(method=theils_u), 2)\nsns.heatmap(corr_res, annot=True)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Using this heatmaps we can understand that features nom_{5-9} are somehow connected between each other. If somebody knew what we can do with it, please explain in comments."},{"metadata":{},"cell_type":"markdown","source":"## Features encoding p.2"},{"metadata":{},"cell_type":"markdown","source":"We can encode some of the featrues using OneHotEncoding."},{"metadata":{"trusted":true},"cell_type":"code","source":"to_dummies = ['day', 'month', 'nom_0',\n              'nom_1', 'nom_2', 'nom_3', 'nom_4']","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"%%time\n\nall_df = pd.get_dummies(all_df,\n                        columns=to_dummies,\n                        sparse=True,\n                        dtype=np.int8)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":" Dimensionality of our data is pretty huge. And data is sparse, so, [pandas](https://pandas.pydata.org/pandas-docs/stable/user_guide/sparse.html) help us with it:\n \n> Pandas provides data structures for efficiently storing sparse data. These are not necessarily sparse in the typical “mostly 0”. Rather, you can view these objects as being “compressed” where any data matching a specific value (NaN / missing value, though any value can be chosen, including 0) is omitted. The compressed values are not actually stored in the array."},{"metadata":{"trusted":true},"cell_type":"code","source":"all_df.shape","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"all_df.isna().sum().sum()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Let's get train and test again."},{"metadata":{"trusted":true},"cell_type":"code","source":"%%time\n\ntrain = all_df[:train.shape[0]]\ntest = all_df[train.shape[0]:]","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"print(f'Train shape: {train.shape}',\n      f'Test shape: {test.shape}', sep=' | ')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train.isna().sum().sum(), test.isna().sum().sum() ","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"%%time\n\ntrain_data = train.drop('target', axis=1).to_numpy()\ntrain_target = train['target'].to_numpy()\n\ntest_data = test.drop('target', axis=1).to_numpy()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"To create dataset we use numpy arrays, and our model needs to understand, which features are categorical, which are continuous. So we need to find their indexes."},{"metadata":{"trusted":true},"cell_type":"code","source":"%%time\n\ncategorical = all_df.drop(['target'] + continuous,\n                          axis=1).columns\n\ncat_cols_idx, cont_cols_idx = list(), list()\n\nfor idx, column in enumerate(all_df.drop('target',\n                                         axis=1).columns):\n    if column in categorical:\n        cat_cols_idx.append(idx)\n    elif column in continuous:\n        cont_cols_idx.append(idx)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train_data, train_target, test_data","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## PyTorch"},{"metadata":{"trusted":true},"cell_type":"code","source":"import torch\nfrom torch.utils.data import Dataset, DataLoader\nfrom torch.utils.data.sampler import SubsetRandomSampler","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"If we want [reproducible](https://pytorch.org/docs/stable/notes/randomness.html) results, we should fix seeds."},{"metadata":{"trusted":true},"cell_type":"code","source":"import random\n\ndef set_seed(seed):\n    random.seed(seed)\n    np.random.seed(seed)\n    \n    torch.manual_seed(seed)\n    torch.backends.cudnn.deterministick = True\n    torch.backends.cudnn.benchmark = False \n    \nset_seed(27)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"We can create PyTorch dataset for using it with NN. The point is, that we have continuous and categorical features. It is good to divide them, so NN can use Embeddings with categorical data."},{"metadata":{},"cell_type":"markdown","source":"## PyTorch Dataset"},{"metadata":{"trusted":true},"cell_type":"code","source":"class ClassificationDataset(Dataset):\n    def __init__(self, data, targets=None,\n                 is_train=True, cat_cols_idx=None,\n                 cont_cols_idx=None):\n        self.data = data\n        self.targets = targets\n        self.is_train = is_train\n        self.cat_cols_idx = cat_cols_idx\n        self.cont_cols_idx = cont_cols_idx\n    \n    def __getitem__(self, idx):\n        row = self.data[idx].astype('float32')\n        \n        data_cat = []\n        data_cont = []\n        \n        result = None\n        \n        if self.cat_cols_idx:\n            data_cat = torch.tensor(row[self.cat_cols_idx])\n            \n        if self.cont_cols_idx:\n            data_cont = torch.tensor(row[self.cont_cols_idx])\n                \n        data = [data_cat, data_cont]\n                \n        if self.is_train:\n            result = {'data': data,\n                      'target': torch.tensor(self.targets[idx])}\n        else:\n            result = {'data': data}\n            \n        return result\n            \n    \n    def __len__(self):\n        return(len(self.data))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train_dataset = ClassificationDataset(train_data, \n                                      targets=train_target,\n                                      cat_cols_idx=cat_cols_idx,\n                                      cont_cols_idx=cont_cols_idx)\ntest_dataset = ClassificationDataset(test_data,\n                                     cat_cols_idx=cat_cols_idx,\n                                     cont_cols_idx=cont_cols_idx,\n                                     is_train=False)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"len(test_dataset)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"print(f'First element of train_dataset: {train_dataset[1]}',\n      f'First element of test_dataset: {test_dataset[1]}', sep='\\n')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"We should split our dataset to create validation and train parts."},{"metadata":{"trusted":true},"cell_type":"code","source":"def split_dataset(trainset, valid_size=0.2, batch_size=64):\n    num_train = len(trainset)\n    \n    indices = list(range(num_train))\n    np.random.shuffle(indices)\n    \n    split = int(np.floor(valid_size * num_train))\n    \n    valid_idx, train_idx = indices[:split], indices[split:]\n    \n    valid_sampler = SubsetRandomSampler(valid_idx)\n    train_sampler = SubsetRandomSampler(train_idx)\n    \n    valid_loader = DataLoader(trainset, \n                              batch_size=batch_size, \n                              sampler=valid_sampler)\n    train_loader = DataLoader(trainset, \n                              batch_size=batch_size, \n                              sampler=train_sampler)\n    \n    return train_loader, valid_loader","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train_loader, valid_loader = split_dataset(train_dataset, \n                                           valid_size=0.2, \n                                           batch_size=2000)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"next(iter(train_loader))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"len(train_loader)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## PyTorch Model"},{"metadata":{"trusted":true},"cell_type":"code","source":"class ClassificationEmbdNN(torch.nn.Module):\n    \n    def __init__(self, emb_dims, no_of_cont=None):\n        super(ClassificationEmbdNN, self).__init__()\n        \n        self.emb_layers = torch.nn.ModuleList([torch.nn.Embedding(x, y)\n                                               for x, y in emb_dims])\n        \n        no_of_embs = sum([y for x, y in emb_dims])\n        self.no_of_embs = no_of_embs\n        self.emb_dropout = torch.nn.Dropout(0.2)\n        \n        self.no_of_cont = 0\n        if no_of_cont:\n            self.no_of_cont = no_of_cont\n            self.bn_cont = torch.nn.BatchNorm1d(no_of_cont)\n        \n        self.fc1 = torch.nn.Linear(in_features=self.no_of_embs + self.no_of_cont, \n                                   out_features=256)\n        self.dropout1 = torch.nn.Dropout(0.2)\n        self.bn1 = torch.nn.BatchNorm1d(256)\n        self.act1 = torch.nn.ReLU()\n        \n        self.fc2 = torch.nn.Linear(in_features=256, \n                                   out_features=256)\n        self.dropout2 = torch.nn.Dropout(0.2)\n        self.bn2 = torch.nn.BatchNorm1d(256)\n        self.act2 = torch.nn.ReLU()\n        \n        self.fc3 = torch.nn.Linear(in_features=256, \n                                   out_features=64)\n        self.dropout3 = torch.nn.Dropout(0.2)\n        self.bn3 = torch.nn.BatchNorm1d(64)\n        self.act3 = torch.nn.ReLU()\n        \n        self.fc4 = torch.nn.Linear(in_features=64, \n                                   out_features=1)\n        self.act4 = torch.nn.Sigmoid()\n        \n    def forward(self, x_cat, x_cont=None):\n        if self.no_of_embs != 0:\n            x = [emb_layer(x_cat[:, i])\n                 for i, emb_layer in enumerate(self.emb_layers)]\n        \n            x = torch.cat(x, 1)\n            x = self.emb_dropout(x)\n            \n        if self.no_of_cont != 0:\n            x_cont = self.bn_cont(x_cont)\n            \n            if self.no_of_embs != 0:\n                x = torch.cat([x, x_cont], 1)\n            else:\n                x = x_cont\n        \n        x = self.fc1(x)\n        x = self.dropout1(x)\n        x = self.bn1(x)\n        x = self.act1(x)\n        \n        x = self.fc2(x)\n        x = self.dropout2(x)\n        x = self.bn2(x)\n        x = self.act2(x)\n        \n        x = self.fc3(x)\n        x = self.dropout3(x)\n        x = self.bn3(x)\n        x = self.act3(x)\n        \n        x = self.fc4(x)\n        x = self.act4(x)\n        \n        return x","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## PyTorch Train"},{"metadata":{},"cell_type":"markdown","source":"The coolest thing about PyTorch is reusability of code. You can use almost the same method for training a huge amount of architectures."},{"metadata":{"trusted":true},"cell_type":"code","source":"def train_network(model, train_loader, valid_loader,\n                  loss_func, optimizer, n_epochs=20,\n                  saved_model='model.pt'):\n    \n    device = torch.device('cuda:0' if torch.cuda.is_available() else 'cpu')\n    model.to(device)\n    \n    train_losses = list()\n    valid_losses = list()\n    \n    valid_loss_min = np.Inf\n    \n    for epoch in range(n_epochs):\n        train_loss = 0.0\n        valid_loss = 0.0\n        \n        train_auc = 0.0\n        valid_auc = 0.0\n        \n        model.train()\n        for batch in tqdm(train_loader):\n            optimizer.zero_grad()\n            \n            output = model(batch['data'][0].to(device, \n                                               dtype=torch.long),\n                           batch['data'][1].to(device, \n                                               dtype=torch.float))\n            \n            \n            loss = loss_func(output, batch['target'].to(device, \n                                                        dtype=torch.float))\n            \n            loss.backward()\n            optimizer.step()\n            \n            train_auc += metrics.roc_auc_score(batch['target'].cpu().numpy(),\n                                               output.detach().cpu().numpy())\n\n            train_loss += loss.item() * batch['data'][0].size(0)  #!!!\n    \n\n        model.eval()\n        for batch in tqdm(valid_loader):\n            output = model(batch['data'][0].to(device, \n                                               dtype=torch.long),\n                           batch['data'][1].to(device, \n                                               dtype=torch.float))\n            \n            \n            loss = loss_func(output, batch['target'].to(device, \n                                                        dtype=torch.float))\n            \n            valid_auc += metrics.roc_auc_score(batch['target'].cpu().numpy(),\n                                               output.detach().cpu().numpy())\n            valid_loss += loss.item() * batch['data'][0].size(0)  #!!!\n           \n        \n        train_loss = np.sqrt(train_loss / len(train_loader.sampler.indices))\n        valid_loss = np.sqrt(valid_loss / len(valid_loader.sampler.indices))\n\n        train_auc = train_auc / len(train_loader)\n        valid_auc = valid_auc / len(valid_loader)\n        \n        train_losses.append(train_loss)\n        valid_losses.append(valid_loss)\n\n        print('Epoch: {}. Training loss: {:.6f}. Validation loss: {:.6f}'\n              .format(epoch, train_loss, valid_loss))\n        print('Training AUC: {:.6f}. Validation AUC: {:.6f}'\n              .format(train_auc, valid_auc))\n        \n        if valid_loss < valid_loss_min:  # let's save the best weights to use them in prediction\n            print('Validation loss decreased ({:.6f} --> {:.6f}). Saving model...'\n                  .format(valid_loss_min, valid_loss))\n            \n            torch.save(model.state_dict(), saved_model)\n            valid_loss_min = valid_loss\n            \n    \n    return train_losses, valid_losses\n        ","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Using embedding in NN we can change dimensionality of categorical features. So, we'll choose new dimensionality for every categorical feature."},{"metadata":{"trusted":true},"cell_type":"code","source":"cat_dim = [int(all_df[col].nunique()) for col in categorical]\ncat_dim = [[x, min(200, (x + 1) // 2)] for x in cat_dim]\n\nfor el in cat_dim:\n    if el[0] < 10:\n        el[1] = el[0]\n\ncat_dim","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"model = ClassificationEmbdNN(emb_dims=cat_dim, \n                             no_of_cont=len(continuous))\n\nloss_func = torch.nn.BCELoss()\noptimizer = torch.optim.Adam(model.parameters(), lr=0.1)\n\ntrain_losses, valid_losses = train_network(model=model, \n                                           train_loader=train_loader, \n                                           valid_loader=valid_loader, \n                                           loss_func=loss_func, \n                                           optimizer=optimizer,\n                                           n_epochs=3, \n                                           saved_model='simple_nn.pt')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## NN Predictions"},{"metadata":{"trusted":true},"cell_type":"code","source":"def predict(data_loader, model):\n    model.eval()\n    device = torch.device('cuda:0' if torch.cuda.is_available() else 'cpu')\n    \n    model.to(device)\n    \n    with torch.no_grad():\n        predictions = None\n        \n        for i, batch in enumerate(tqdm(data_loader)):   \n            \n            output = model(batch['data'][0].to(device, \n                                               dtype=torch.long), \n                           batch['data'][1].to(device, \n                                               dtype=torch.float)).cpu().numpy()\n            \n            if i == 0:\n                predictions = output\n                \n            else: \n                \n                predictions = np.vstack((predictions, output))\n                \n    return predictions","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"model.load_state_dict(torch.load('simple_nn.pt'))\n\ntest_loader = DataLoader(test_dataset, \n                         batch_size=1000)\n\nnn_predictions = predict(test_loader, model)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"nn_predictions","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"nn_predictions_df = pd.DataFrame({'id': sample_submission['id'], 'target': nn_predictions.squeeze()})","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"nn_predictions_df.head()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## CatBoost"},{"metadata":{},"cell_type":"markdown","source":"I will use [CatBoost](https://catboost.ai) as a second model, because it is damn great with data which consists of huge amount of categorical data."},{"metadata":{"trusted":true},"cell_type":"code","source":"from catboost import CatBoostClassifier","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"X = train.drop('target', axis=1)\ny = train['target']\nX_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=27)\n\ntest_data = test.drop('target', axis=1)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"We should give CatBoost names of categorical features. So, it will process them not like continuous values. "},{"metadata":{"trusted":true},"cell_type":"code","source":"cat_features = ['bin_0', 'bin_1', 'bin_2', 'bin_3', 'bin_4', 'nom_5', 'nom_6', 'nom_7',\n                'nom_8', 'nom_9', 'ord_0', 'ord_1', 'ord_2', 'ord_3', 'ord_4',\n                'ord_5_1', 'ord_5_2', 'nan_features', 'day_0', 'day_1', 'day_2',\n                'day_3', 'day_4', 'day_5', 'day_6', 'day_7', 'month_0', 'month_1',\n                'month_2', 'month_3', 'month_4', 'month_5', 'month_6', 'month_7',\n                'month_8', 'month_9', 'month_10', 'month_11', 'month_12', 'nom_0_0',\n                'nom_0_1', 'nom_0_2', 'nom_0_3', 'nom_1_0', 'nom_1_1', 'nom_1_2',\n                'nom_1_3', 'nom_1_4', 'nom_1_5', 'nom_1_6', 'nom_2_0', 'nom_2_1', \n                'nom_2_2', 'nom_2_3','nom_2_4', 'nom_2_5', 'nom_2_6', 'nom_3_0',\n                'nom_3_1', 'nom_3_2','nom_3_3', 'nom_3_4', 'nom_3_5', 'nom_3_6',\n                'nom_4_0', 'nom_4_1', 'nom_4_2', 'nom_4_3', 'nom_4_4']","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"best_params = {\n    'bagging_temperature': 0.8, \n    'depth': 5, \n    'iterations': 1000,\n    'l2_leaf_reg': 30,\n    'learning_rate': 0.05,\n    'random_strength': 0.8\n}\n\nmodel_cat = CatBoostClassifier(**best_params,\n                               loss_function='Logloss',\n                               eval_metric='AUC', \n                               nan_mode='Min',\n                               thread_count=4,\n                               task_type='GPU', \n                               verbose=True)\n\nmodel_cat.fit(X_train, y_train,\n              eval_set=(X_test, y_test), \n              cat_features=cat_features,\n              verbose_eval=300, \n              early_stopping_rounds=500, \n              use_best_model=True,\n              plot=False)\n\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## CatBoost predictions"},{"metadata":{"trusted":true},"cell_type":"code","source":"cat_predictions = model_cat.predict_proba(test_data)[:, 1]","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"cat_predictions_df = pd.DataFrame({'id': sample_submission['id'], \n                                   'target': cat_predictions})","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"cat_predictions_df.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"nn_predictions_df.head()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Average predictions"},{"metadata":{"trusted":true},"cell_type":"code","source":"res_sub = pd.DataFrame({'id': sample_submission['id']})\nres_sub.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"res_sub['target'] = round((cat_predictions_df['target'] + nn_predictions_df['target']) / 2, 2)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"res_sub.head(5)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"res_sub.to_csv('res_sub.csv', index=False)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Thanks for your attention. If you'd like this notebook, upvote!"}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":1}