{"cells":[{"metadata":{},"cell_type":"markdown","source":"<h1>Introduction</h1>\n\nHere is my work, pre-processing and different tests for the competition.\n\nI took some ideas, changing the code a little, from [this notebook](https://www.kaggle.com/siavrez/comparing-imputation-for-ordinal-data).\n\nI was also insipred by [this discussion](https://www.kaggle.com/c/cat-in-the-dat-ii/discussion/126199). \n\nSeveral remarks : \n\n1. After testing several kernels, CatBoostClassifier provided the best result.\n2. For ordinal features (ord_0 to ord_5 + day and month, which I considered as ordinal), imputing is better than OneHotEncoding\n3. OneHotEncoding works for bin_0 -> bin_4, nom_0-> nom_9\n4. Imputing with TargerEncoding provides the best result.\n"},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"code","source":"import csv\nimport numpy as np \nimport pandas as pd \n\nimport category_encoders as ce\n\nfrom catboost import CatBoostClassifier, Pool\n\nfrom sklearn.preprocessing import OrdinalEncoder, OneHotEncoder, MinMaxScaler\nfrom sklearn.experimental import enable_iterative_imputer\nfrom sklearn.impute import SimpleImputer, IterativeImputer, MissingIndicator\n\nfrom sklearn.tree import DecisionTreeClassifier, DecisionTreeRegressor\nfrom sklearn.linear_model import LogisticRegression, BayesianRidge, RidgeClassifier, RidgeClassifierCV\nfrom sklearn.experimental import enable_hist_gradient_boosting\nfrom sklearn.ensemble import RandomForestClassifier, AdaBoostClassifier, BaggingClassifier, ExtraTreesClassifier, GradientBoostingClassifier, HistGradientBoostingClassifier, StackingClassifier \nfrom sklearn.naive_bayes import GaussianNB\nfrom sklearn.discriminant_analysis import QuadraticDiscriminantAnalysis, LinearDiscriminantAnalysis\nfrom sklearn.neural_network import MLPClassifier\n\nfrom sklearn.model_selection import GridSearchCV, train_test_split, KFold, StratifiedKFold\n\nfrom sklearn.metrics import roc_auc_score, auc\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Constants"},{"metadata":{"trusted":true},"cell_type":"code","source":"FILE_TRAIN = \"/kaggle/input/cat-in-the-dat-ii/train.csv\"\nFILE_TEST = \"/kaggle/input/cat-in-the-dat-ii/test.csv\"\n\nNAN_STRING_TO_REPLACE = 'zz'\nNAN_VALUE_FLOAT = 8888.0\nNAN_VALUE_INT = 8888\nNAN_VALUE_STRING = '8888'\n\nPARAMS_CATBOOST = dict()\nPARAMS_CATBOOST['logging_level'] = 'Silent'\nPARAMS_CATBOOST['eval_metric'] = 'AUC'\nPARAMS_CATBOOST['loss_function'] = 'Logloss'\nPARAMS_CATBOOST['iterations'] = 1000\nPARAMS_CATBOOST['od_type'] = 'Iter' # IncToDec, Iter\nPARAMS_CATBOOST['l2_leaf_reg'] = 300 # lambda, default 3\nPARAMS_CATBOOST['learning_rate'] = 0.1 # alpha, default 0.3 if no l2_leaf_reg\nPARAMS_CATBOOST['task_type'] = 'CPU'\nPARAMS_CATBOOST['use_best_model']: True","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"<h1>Functions</h1> "},{"metadata":{"trusted":true},"cell_type":"code","source":"\n#Description: Read Data from CSV file into Pandas DataFrame\ndef readData(inFile, sep=','):\n    df_op = pd.read_csv(filepath_or_buffer=inFile, low_memory=False, encoding='utf-8', sep=sep)\n    return df_op\n\n# Description: Write Pandas DataFrame into CSV file\ndef writeData(df, outFile):\n    f = open(outFile+'.csv', 'w')\n    r = df.to_csv(index=False, path_or_buf=f)\n    f.close()\n\n# Write submission into file    \ndef print_submission_into_file(y_pred, df_test_id):\n    l = []\n    for myindex in range(y_pred.shape[0]):\n        Y0 = y_pred[myindex]\n        l.insert(myindex, Y0)\n    \n    df_pred = pd.DataFrame(pd.Series(l), columns=[\"target\"])\n    df_result = pd.concat([df_test_id, df_pred], axis=1, sort=False)\n     \n    f = open('submission.csv', 'w')\n    r = df_result.to_csv(index=False, path_or_buf=f)\n    f.close()\n    return df_result\n    \n# Create a column with the number of missing values in each row:    \ndef set_missing_values_column(df):\n    df['nb_missing_values'] = df.shape[1] - df.count(axis=1)\n    return df\n\n\n# Description; Ballance data for a pd.DataFrame, from scratch\n# Input: df : pd.DataFrame without the Id column, containing the label column\n# Output: X_res: pd.Dataframe for training ballanced containing the label column\n\ndef ballance_data_with_y(df):\n    df_1 =  df[df[\"target\"]==1]\n    df_0 =  df[df[\"target\"]==0]\n    len1 = df_1.shape[0]\n    len0 = df_0.shape[0]\n    \n    vmax = 0\n    vmin = 1\n    if len1 > len0:\n        vmax = 1\n        vmin = 0\n        df_max = df_1\n        df_min = df_0\n    elif len1 < len0:\n        vmax = 0\n        vmin = 1\n        df_max = df_0\n        df_min = df_1\n    else:\n        return (df, Y)\n    \n    len_max = df_max.shape[0]\n    len_min = df_min.shape[0]\n    \n    to_multiply = int(round(len_max/len_min))\n    df_to_append = pd.concat([df_min] * to_multiply, ignore_index=True)\n    \n    len_append = df_to_append.shape[0]\n    \n    X_res = pd.concat([df_max, df_to_append], ignore_index=True)\n    \n    to_add = len_max - len_append\n    if to_add > 0:\n        df_to_add = df_min.sample(n=to_add, random_state=1)\n        X_res = pd.concat([X_res, df_to_add], ignore_index=True)\n    \n    X_res = X_res.reset_index(drop=True)\n    return X_res\n\n# Convert categories to numeric\ndef convert_data_to_numeric_2(df, df_all, ar_train_transformed=None, enc=None):\n    columns_for_ordinal_encoder = [cn for cn in df.columns if cn not in ['id', 'target', 'bin_0', 'bin_1', 'day', 'month', 'ord_0', 'ord_1', 'ord_2', 'nb_missing_values']]\n    #print(columns_for_ordinal_encoder)\n    enc_new = enc\n    \n    # Fillna with ZZ which will be the last value in alphabetical order. \n    df_ordinal = df[columns_for_ordinal_encoder].fillna(NAN_STRING_TO_REPLACE).applymap(lambda x: str(x))\n    df_ordinal_all = df_all[columns_for_ordinal_encoder].fillna(NAN_STRING_TO_REPLACE).applymap(lambda x: str(x))\n    ar_train_transformed_new = ar_train_transformed\n    if enc == None and ar_train_transformed == None:\n        enc_new = OrdinalEncoder(dtype=np.int16)\n        enc_new.fit(df_ordinal_all)\n        #print(enc_new.categories_)\n        \n        ar_train_transformed_new = enc_new.transform(df_ordinal_all)\n    \n    ar_ordinal_transformed = enc_new.transform(df_ordinal)\n    count_columns = 0\n    for cn in columns_for_ordinal_encoder:\n        this_col_all = ar_train_transformed_new[:,count_columns]\n        mx = this_col_all.max() # Find the index of ZZ, last one always, in our case, as OrdinalEncoder encodes by alphabetic order\n        \n        this_col_train = ar_ordinal_transformed[:,count_columns]\n        this_col_nan_train = np.where(this_col_train==mx, NAN_VALUE_INT, this_col_train)\n    \n        ar_ordinal_transformed[:,count_columns] = this_col_nan_train\n        count_columns = count_columns+1\n\n    df_ordinal_transformed = pd.DataFrame(ar_ordinal_transformed, columns=columns_for_ordinal_encoder)\n    \n    df.update(df_ordinal_transformed)      \n\n    ord_1_mapping = {'Novice' : 0, 'Contributor' : 1, 'Expert' : 2, 'Master': 3, 'Grandmaster': 4}\n    ord_2_mapping = { 'Freezing': 0, 'Cold': 1, 'Warm' : 2, 'Hot': 3, 'Boiling Hot' : 4, 'Lava Hot' : 5}\n    \n    df['ord_1'] = df.loc[df.ord_1.notnull(), 'ord_1'].map(ord_1_mapping)\n    df['ord_2'] = df.loc[df.ord_2.notnull(), 'ord_2'].map(ord_2_mapping)\n    \n    df['ord_0'] = df.loc[df.ord_0.notnull(), 'ord_0'].apply(lambda x: x-1)\n    df['day'] = df.loc[df.day.notnull(), 'day'].apply(lambda x: x-1)\n    df['month'] = df.loc[df.month.notnull(), 'month'].apply(lambda x: x-1)\n\n    # Fill NaN with -1 first : \n    df = df.fillna(-1)\n     \n    # Optimize : \n    columns_only_16 = ['nom_5', 'nom_6', 'nom_7', 'nom_8', 'nom_9', 'ord_5']\n    \n    # columns with more than 127 and less than 3000 unique values:\n    columns_int16 =  [cn for cn in df.columns if cn not in ['id', 'target'] and cn in columns_only_16]\n    \n    # columns with less than 127 unique values:\n    columns_int8 = [cn for cn in df.columns if cn not in ['id', 'target', 'nom_5', 'nom_6', 'nom_7', 'nom_8', 'nom_9', 'ord_5']]\n\n    df[columns_int8] = df[columns_int8].astype('int8')\n    df[columns_int16] = df[columns_int16].astype('int16')\n    \n    df = df.applymap(lambda x: NAN_VALUE_INT if x == -1 else x)\n    \n    return (df, ar_train_transformed_new, enc_new)\n\n# Parse OneHotEncoding for train and test, and for the features in cols : \ndef get_ohe(df_train, df_test, cols):\n    new_cols = [\"ohe_\"+col for col in cols]\n    \n    size_train = df_train.shape[0]\n    size_test = df_test.shape[0]    \n        \n    df = df_train.append(df_test, ignore_index=True, sort=False)\n    df_ohe = df.loc[:, cols].astype('category')\n    \n    df_ohe = df_ohe.applymap(lambda x: np.nan if x==NAN_VALUE_INT else x) # replace NAN_VALUE_INT with nan\n    \n    df_ohe = pd.get_dummies(df_ohe, prefix=new_cols, sparse=True, columns=cols)\n     \n    index_test_start = size_train\n    index_test_end = size_train+size_test-1\n    \n    df_train = df_ohe.loc[0:size_train-1,:].astype('int16')\n    df_test = df_ohe.loc[index_test_start:index_test_end,:].astype('int16')\n    return (df_train, df_test)\n\n# Encodes with simple imputer for DataFrame, cols, strategy, and returns the encoder\ndef encode_missing_simple_imputer(df, cols, strategy, si=None):\n    df_to_transform = df[cols]\n    \n    if si==None:\n        si = SimpleImputer(strategy=strategy, missing_values=NAN_VALUE_INT)\n        ar = si.fit_transform(df_to_transform)\n    else:\n        ar = si.transform(df_to_transform)\n    \n    df_transformed = pd.DataFrame(ar, columns=cols)\n    df.update(df_transformed)\n    return (df, si)\n\n\n# Encode with SimpleImputer\ndef encode_missing_simple_imputer(df, cols, strategy, si=None):\n    df_to_transform = df[cols]\n    \n    if si==None:\n        si = SimpleImputer(strategy=strategy, missing_values=NAN_VALUE_INT)\n        ar = si.fit_transform(df_to_transform)\n    else:\n        ar = si.transform(df_to_transform)\n    \n    df_transformed = pd.DataFrame(ar, columns=cols)\n    if strategy == 'most_frequent':\n        df_transformed = df_transformed.applymap(lambda x: str(x)).astype('category')\n       \n    df.update(df_transformed)\n    \n    return (df, si)\n\ndef parse_all_kernels(X, Y, cat_features):\n    names = [\n        \"RidgeClassifier\",\n        \"RidgeClassifierCV\",\n        \"CatBoostClassifier\",        \"GradientBoostingClassifier\",\n        \"HistGradientBoostingClassifier\",\n        \"ExtraTreesClassifier\",\n        \"LinearDiscriminantAnalysis\",\n        \"QuadraticDiscriminantAnalysis\",\n        \"DecisionTreeClassifier\",\n        \"RandomForestClassifier\",\n        \"AdaBoostClassifier\",\n        \"GaussianNB\",\n        \"LogisticRegression\",\n        \"MLPClassifier\"\n    ]\n\n    classifiers = [\n        RidgeClassifier(),\n        RidgeClassifierCV(),\n        CatBoostClassifier(**PARAMS_CATBOOST),\n        GradientBoostingClassifier(),\n        HistGradientBoostingClassifier(),\n        ExtraTreesClassifier(),\n        LinearDiscriminantAnalysis(),\n        QuadraticDiscriminantAnalysis(),\n        DecisionTreeClassifier(max_depth=5),\n        RandomForestClassifier(max_depth=5, n_estimators=10),\n        AdaBoostClassifier(),\n        GaussianNB(), # Naive Bayes :-( \n        LogisticRegression(max_iter=10000),\n        MLPClassifier(alpha=0.1, max_iter=1000, batch_size=1000, tol=0.0001, verbose=False)\n    ]\n\n    SPLITS = 3\n    for name, clf in zip(names, classifiers):\n        kf = StratifiedKFold(n_splits=SPLITS, shuffle=True)\n        current_roc = 0.0\n\n        count = 0\n        for train_index, test_index in kf.split(X, Y):\n            count = count+1\n            #print(\"Split \"+str(count)+\" ... \")\n\n            X_train, X_test = X.iloc[train_index], X.iloc[test_index]\n            y_train, y_test = Y.iloc[train_index], Y.iloc[test_index]\n\n            if name == \"CatBoostClassifier\":\n                clf = CatBoostClassifier(**PARAMS_CATBOOST)\n\n                train_dataset = Pool(data=X_train,\n                     label=y_train,\n                     cat_features=cat_features)\n\n                eval_dataset = Pool(data=X_test,\n                     label=y_test,\n                     cat_features=cat_features)\n\n                clf.fit(train_dataset,\n                    use_best_model=True,\n                    eval_set=eval_dataset)\n\n                print(\"Count of trees in model = {}\".format(clf.tree_count_))\n            else:\n                clf.fit(X_train, y_train)\n\n            if name in [\"RidgeClassifier\", \"RidgeClassifierCV\"]:\n                y_test_predict = clf.decision_function(X_test)\n            else:\n                y_test_predict = clf.predict_proba(X_test)[:,1]\n\n            oof_auc_score_test = roc_auc_score(y_test, y_test_predict)\n            current_roc += oof_auc_score_test\n        final_roc = current_roc/float(SPLITS)\n\n        print(name+\": \"+str(final_roc))\n        print(\"-------------------\")\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"<h1>Preprocessing</h1>"},{"metadata":{"trusted":true},"cell_type":"code","source":"# Read data :\ndf_train = readData(FILE_TRAIN)\ndf_test = readData(FILE_TEST)\n\n# Add nb_missing_values\ndf_train = set_missing_values_column(df_train)\ndf_test = set_missing_values_column(df_test)\n\n# Convert data to numeric. \n# Be careful tu use the same encoder for both train and test! The encoder is fit on both train + test.\ndf_all = df_train.append(df_test, sort=True)\n\n(df_train, ar_train, enc_ord) = convert_data_to_numeric_2(df=df_train, df_all=df_all, ar_train_transformed=None, enc=None)\n(df_test, ar_train, enc_ord) = convert_data_to_numeric_2(df=df_test, df_all=df_all, ar_train_transformed=ar_train, enc=enc_ord)\n\n# Ballance data:\ndf_train = ballance_data_with_y(df_train)\nY = df_train[\"target\"]\n\n# I prefer to split the datasets, rather than making several copies, so as not to charge the RAM:\ndf_train = df_train.drop(\"id\", axis=1) # we do not longer need this one\ndf_train = df_train.drop(\"target\", axis=1) # saved already\ndf_train = df_train.reset_index(drop=True)\nY = Y.reset_index(drop=True)\n\ndf_test_id = df_test[\"id\"] # we need this for the submission, so save ... \ndf_test = df_test.drop(\"id\", axis=1) # ... and drop afterwards\ndf_test = df_test.reset_index(drop=True)\n\ncols = df_train.columns\n\nX = df_train\nX_testset = df_test\n\nprint(X.shape)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"<h1>Imputations</h1> \n\nImputation by groups, I used five groups : \n\n* bin_0 -> bin_4\n* ord_0 -> ord_5\n* nom_0 -> nom_4\n* nom_5 -> nom_9\n* day, month\n\nFor each one I could use either OheHotEncoding, or encoding by most frequent value.\nI combined these encodings for each group and compared the results.\n\nI obtained a better result for OHE than most frequent for ord and nom_0 -> nom_5. \n\nIn the following example I used most_frequent value for ord, day, month and OHE for nom_0 -> nom_9.\n\n"},{"metadata":{"trusted":true},"cell_type":"code","source":"cols_bin = [\"bin_0\",  \"bin_1\", \"bin_2\", \"bin_3\", \"bin_4\"]\ncols_ord_dm = ['day', 'month', 'ord_0', 'ord_1', 'ord_2', 'ord_3', 'ord_4', 'ord_5']\ncols_ord_nom_04 = ['nom_0', 'nom_1', 'nom_2', 'nom_3', 'nom_4']\ncols_ord_nom_59 = ['nom_5', 'nom_6', 'nom_7', 'nom_8', 'nom_9']\n\n# Encode F for ord, dm, nom_5 -> nom_9:\nstrategy_F = 'most_frequent'\ncols_F = cols_ord_dm + cols_ord_nom_59\n(df_train, si) = encode_missing_simple_imputer(df_train, cols_F, strategy_F)\n(df_test, si) = encode_missing_simple_imputer(df_test, cols_F, strategy_F, si)\n    \n# Encode OHE bin + nom_0 -> nom_4\ncols_ohe = cols_bin + cols_ord_nom_04\n(X_ohe_train, X_ohe_test) = get_ohe(df_train, df_test, cols_ohe)\n    \ndf_train.drop(cols_ohe, axis=1, inplace=True)\ndf_test.drop(cols_ohe, axis=1, inplace=True)\n\ndf_train = df_train.reset_index(drop=True)\ndf_test = df_test.reset_index(drop=True)\nX_ohe_train = X_ohe_train.reset_index(drop=True)\nX_ohe_test = X_ohe_test.reset_index(drop=True)\nY = Y.reset_index(drop=True)\n\nX = pd.concat([df_train, X_ohe_train], axis=1)\nX_testset = pd.concat([df_test, X_ohe_test], axis=1)\ncat_features = cols_F\ncolumns = X.columns\n\nprint(X.shape)\nprint(X_testset.shape)\nprint(\"End\")","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"<h1>Kernels</h1>\n\nI tested the following kernels, on different imputations and encoding strategies : \n\nSeveral remarks : \n\n* For most kernels I used the default settings, at this stage\n* *Logistic regression* : I increased the number of iterations to 10000 as it did not converge at default (1000)\n* *SVC* : could not test as too long (I stopped after 2 hours of run)\n* *KNeighborsClassifier* : could not test, OOM (needed about 600Gb RAM)\n\nBy far, the best result was provided by ***CatBoostClassifier***.\n\nSo I made different tests using only this kernel, with option *use_best_model*.\n"},{"metadata":{"trusted":true},"cell_type":"code","source":"# parse_all_kernels(X, Y, cat_features)\n        \n    ","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}