{"cells":[{"metadata":{},"cell_type":"markdown","source":"# Cat in the Dat II - Grid Search and Pipeline\nThis kernel is an exploration of using grid search and pipeline to optimize a binary classifier on the Cat in the Dat II data set.  Only the ordinal features are considered for this example, but the code easily supports the remaining feature classes, as shown in my other kernels.\n\nThe structure follows my pattern for building a pipeline and examining the transformed features at the end of the pipe.  While this adds complexity to an otherwise simple flow, the traceability of transformed features is priceless to understand the interactions in more sophisticated problems.\n\n### 🔴 CTL\nThe CTL class is a singleton with static attributes used to control the behavior throughout the kernel.  Most notably, each of the input features can be turned on/off independently without digging through the code before invoking a long running commit.\n\n### 🔴 No data visualization\nThis kernel does not do any visualization of the data, as we are only doing gridsearch/pipeline development here. My other, real, kernels contain input and label exploration.\n\n### 🔴 Scikit-learn get_feature_names()\nThere is an ongoing effort in Scikit-learn's GitHub repo to implement `get_feature_names()` throughout the pipeline infrastructure (transformers, imputers, etc.).  The version of sklearn available by default in Kaggle contains only a partial implementation, and is most notably missing from Pipeline which is what glues it all together.\n\nThis kernel contains a limited sub-set of the code from sklearn's PR, with minor tweaks to get it to run.  Since this kernel only uses Pandas DataFrame, the implementation is much simpler than what is required in the full Scikit-learn solution.  The classes in this kernel that extend sklearn append *WithNames* to the classname and are contained in a single code cell.\n\n### 🔴 Custom Transformers\nThis kernel contains several custom transformers useful in the Cat in the Dat II competition.  The transformations that they impelemnt are typically simple one-liners.  The surrounding code is there to support `get_feature_names()`, and to support analysis of the behaviors and datatypes passed through the Pipeline stack.\n\n### 🔴 Grid Search\n`RandomizedSearchCV` is used to evaluate different hyperparameter settings for optimal performance.  Scoring of `roc_auc` is used for Cat in the Dat II, as specified in the competition documents.\n\nMost examples you see in other kernels run a grid search on the model only, not on a whole pipeline that includes input feature preprocessing.  Doing it on the full pipeline is a little slower since it will run preprocessing for each cross validation, but you gain so much in reduced bias that can be introduced by mixing validation data into training, either by design or accident.\n\n### 🔴 Transformed Feature Traceability\nData about the transformed features in the best model is visualized at the end.  This is where all the effort to maintain `get_feature_names()` pays off, with transparent mappings of raw inputs to transformed output."},{"metadata":{"trusted":true},"cell_type":"code","source":"%%javascript\n/* Disable autoscrolling in kernel notebook */\nIPython.OutputArea.auto_scroll_threshold = -1","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"code","source":"# This Python 3 environment comes with many helpful analytics libraries installed\n# It is defined by the kaggle/python docker image: https://github.com/kaggle/docker-python\n# For example, here's several helpful packages to load in \n\nimport numpy as np # linear algebra\nimport pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\n\n# Input data files are available in the \"../input/\" directory.\n# For example, running this (by clicking run or pressing Shift+Enter) will list all files under the input directory\n\nimport os\nfor dirname, _, filenames in os.walk('/kaggle/input'):\n    for filename in filenames:\n        print(os.path.join(dirname, filename))\n\n# Any results you write to the current directory are saved as output.","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Import libraries"},{"metadata":{"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","trusted":true},"cell_type":"code","source":"from datetime import datetime, timedelta\nimport numpy as np\nimport pandas as pd\nfrom scipy import stats\nimport matplotlib.pyplot as plt\nimport seaborn as sns\n%matplotlib inline\npd.set_option(\"display.max_columns\", 40)\n\nfrom sklearn.compose import ColumnTransformer\nfrom sklearn.pipeline import Pipeline, FeatureUnion\nfrom sklearn.impute import SimpleImputer, MissingIndicator\nfrom sklearn.preprocessing import OneHotEncoder, FunctionTransformer, OrdinalEncoder, StandardScaler\nfrom sklearn.ensemble import RandomForestRegressor\nfrom sklearn.metrics import roc_auc_score\nfrom sklearn.model_selection import train_test_split, RandomizedSearchCV\nfrom sklearn.base import BaseEstimator, TransformerMixin\n\nimport xgboost as xgb","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# set seaborn to work better with jupyter dark theme\nsns.set_style(\"whitegrid\")","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Setup control constants"},{"metadata":{"trusted":true},"cell_type":"code","source":"class CTL:\n    \"\"\"\n    Constants and control switches to let me turn on/off certain code blocks from one spot.\n    \"\"\"\n    random_state = 17\n    \n    input_dir = \"/kaggle/input/cat-in-the-dat-ii\"\n    # set to None to write to current directory.\n    output_dir = None\n    \n    train_file_name = \"train.csv\"\n    test_file_name = \"test.csv\"\n    \n    # kaggle settings:\n    sub_filename_pattern = \"submission.csv\"\n    sub_file_date_pattern = None\n    # local PC settings:\n    #sub_filename_pattern = \"sub-{0:s}-{1:0.5f}.csv\"\n    #sub_file_date_pattern = \"%Y%m%d%H%M%S\"\n    \n    enable_bin_features = False\n    \n    # enable_nom_features is global control for all the nom's\n    enable_nom_features = False\n    enable_nom_oh = True\n    # The nominal high cardinality features are the hex strings, let's try mean encoding\n    enable_nom_mean = True\n    \n    # enable_ord_features is global control for all the ord's\n    enable_ord_features = True\n    enable_ord0 = True\n    enable_ord1 = True\n    enable_ord2 = True\n    enable_ord345 = True\n\n   # enable_cyclical_features is global control for all the cyc's\n    enable_cyclical_features = False\n    enable_cyclical_month = True\n    enable_cyclical_day = True\n    \n    # set active_model to one of the configured models\n    model_xgboost = \"xgboost\"\n    model_randomforest = \"randomforest\"\n    active_model = \"\"\n    \n    gridsearch_n_iter = 10\n    gridsearch_cv = 5\n    gridsearch_scoring = \"roc_auc\"\n    \n    # for nested params, prefix with the name of the proper step in the pipeline\n    gridsearch_param_grid = {\n        'model__max_depth': [3, 4, 5, 6, 7],\n        'model__learning_rate': [0.01, 0.05, 0.10, 0.20],\n        'model__subsample': [0.5, 0.6, 0.7, 0.8, 0.9],\n        'model__colsample_bytree': [0.5, 0.6, 0.7, 0.8, 0.9],\n        'model__reg_alpha': [0.5, 1.0, 2.0, 5.0, 10.0],\n        'model__reg_lambda': [0.5, 1.0, 2.0, 5.0, 10.0],\n        'model__num_leaves': [7, 15, 31, 63, 127],\n        'model__n_estimators': [50, 100, 150, 200, 250, 300, 350, 400, 450],\n        'model__min_data_in_leaf': [1, 3, 5, 10, 15, 25],\n    }\n\n    \n\n# specify the active model outside the class def\nCTL.active_model = CTL.model_xgboost","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Load the Train and Test data"},{"metadata":{"trusted":true},"cell_type":"code","source":"train_df = pd.read_csv(os.path.join(CTL.input_dir, CTL.train_file_name), index_col='id')\ntest_df = pd.read_csv(os.path.join(CTL.input_dir, CTL.test_file_name), index_col='id')\n\nbinary_features = [x for x in train_df.columns if x.startswith(\"bin\")]\nnominal_features = [x for x in train_df.columns if x.startswith(\"nom\")]\nordinal_features = [x for x in train_df.columns if x.startswith(\"ord\")]\ncyclical_features = [\"day\", \"month\"]\nlabel_column = \"target\"\n\nprint(\"Shape of raw train data:\", train_df.shape)\nprint(\"Shape of raw test data :\", test_df.shape)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Implementing Feature Names\nThere is an unfinished PR in sklearn's GitHub repository, \"RFC Implement Pipeline get feature names #12627\".\n\nThe following code block is a minimalist way to implement get_feature_names in the transformers that I'm using in this notebook."},{"metadata":{"trusted":true},"cell_type":"code","source":"# This is minor modification of GitHub PR \"RFC Implement Pipeline get feature names #12627\"\nclass PipelineWithNames(Pipeline):\n    def __init__(self, steps, memory=None, verbose=0):\n        super().__init__(steps, memory, verbose)\n        \n    def get_feature_names(self, input_features=None):\n        \"\"\"Get feature names for transformation.\n        Transform input features using the pipeline.\n        If the last step is a transformer, it's included\n        in the transformation, otherwise it's not.\n        Parameters\n        ----------\n        input_features : array-like of string\n            Input feature names.\n        Returns\n        -------\n        feature_names : array-like of string\n            Transformed feature names\n        \"\"\"\n        feature_names = input_features\n        with_final = hasattr(self._final_estimator, \"transform\")\n        \n        for i, name, transform in self._iter(with_final=with_final):\n            if not hasattr(transform, \"get_feature_names\"):\n                raise TypeError(\"Transformer {} does provide\"\n                                \" get_feature_names\".format(name))\n            try:\n                feature_names = transform.get_feature_names(\n                    input_features=feature_names)\n            except TypeError:\n                    feature_names = transform.get_feature_names()\n        return feature_names\n\n\nclass SimpleImputerWithNames(SimpleImputer):\n    def __init__(self, missing_values=np.NaN, strategy='mean', fill_value=None, verbose=0,\n                 copy=True, add_indicator=False):\n        super().__init__(missing_values, strategy, fill_value, verbose, copy, add_indicator)\n        self.feature_names = None\n        \n    def fit(self, X, y=None):\n        if hasattr(X, \"columns\"):\n            self.feature_names = X.columns\n            if self.verbose:\n                print(\"SimpleImputerWithNames.fit(\", type(X), \") saving feature names:\",\n                      self.feature_names)\n        else:\n            if self.verbose:\n                print(\"SimpleImputerWithNames.fit() input X (type=\", type(X),\n                      \") has no 'columns' attr, cannot save feature names\")\n\n        return super().fit(X, y)\n\n    def transform(self, X):\n        X_transform = super().transform(X)\n        if self.verbose:\n            print(\"SimpleImputerWithNames.transform(type=\", type(X),\n                  \") super return type is \", type(X_transform))\n        # this next line ties this impl to pandas...\n        # ...but this is only used for one notebook that's all pandas, so i'm ok with it\n        X_df = pd.DataFrame(data=X_transform, columns=self.feature_names)\n        return X_df\n    \n    def get_feature_names(self, input_features=None):\n        return self.feature_names\n\n    \nclass MissingIndicatorWithNames(MissingIndicator):\n    def __init__(self, verbose=0, feature_suffix=\"_missing\", **kwargs):\n        super().__init__(**kwargs)\n        self.verbose = verbose\n        self.feature_suffix = feature_suffix\n        self.feature_names = None\n        \n    def fit(self, X, y=None):\n        if hasattr(X, \"columns\"):\n            self.feature_names = [str(col)+self.feature_suffix for col in X.columns]\n            if self.verbose:\n                print(\"MissingIndicatorWithNames.fit(\", type(X), \") saving feature names:\",\n                      self.feature_names)\n        else:\n            if self.verbose:\n                print(\"MissingIndicatorWithNames.fit() input X (type=\" + type(X) +\n                      \") has no 'columns' attr, cannot save feature names\")\n\n        return super().fit(X, y)\n\n    def get_feature_names(self, input_features=None):\n        return self.feature_names\n    \n    \nclass StandardScalerWithNames(StandardScaler):\n    def __init__(self, verbose=0, feature_suffix=\"_scaler\", **kwargs):\n        super().__init__(**kwargs)\n        self.verbose = verbose\n        self.feature_suffix = feature_suffix\n        self.feature_names = None\n        \n    def fit(self, X, y=None):\n        if hasattr(X, \"columns\"):\n            self.feature_names = [str(col)+self.feature_suffix for col in X.columns]\n            if self.verbose:\n                print(\"StandardScalerWithNames.fit(\", type(X), \") saving feature names:\",\n                      self.feature_names)\n        else:\n            if self.verbose:\n                print(\"StandardScalerWithNames.fit() input X (type=\", type(X),\n                      \") has no 'columns' attr, cannot save feature names\")\n\n        return super().fit(X, y)\n\n    def get_feature_names(self, input_features=None):\n        return self.feature_names\n\n\nclass OneToOneMixin(object):\n    \"\"\"Provides get_feature_names for simple transformers\n    Assumes there's a 1-to-1 correspondence between input features\n    and output features.\n    \"\"\"\n\n    def get_feature_names(self, input_features=None):\n        \"\"\"Get feature names for transformation.\n        Returns input_features as this transformation\n        doesn't add or drop features.\n        Parameters\n        ----------\n        input_features : array-like of string\n            Input feature names.\n        Returns\n        -------\n        feature_names : array-like of string\n            Transformed feature names\n        \"\"\"\n        if input_features is not None:\n            return input_features\n        else:\n            raise ValueError(\"Don't know how to get\"\n                             \" input feature names for {}\".format(self))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Custom Transformers\nThis next section has custom transformers, partly to implement new function, partly to include get_feature_names in the pipeline.\n\nThis is a contrast to the section above, which is copy-paste of minimal changes to sklearn library classes to make get_feature_names working prior to installation of the pull request."},{"metadata":{"trusted":true},"cell_type":"code","source":"def int2str(x, width):\n    return str(x).rjust(width, \" \")\n\n\ndef print_stamp(text: str = \"\", earlier: datetime = None):\n    \"\"\"\n    Prints a timestamp with an optional text message, and an optional duration from an earlier\n    datetime\n    \n    :return: the current timestamp, so you can pass it back in later to show a duration\n    \"\"\"\n    d = datetime.now()\n    if earlier is not None:\n        print(text, str(d), \"elapsed:\", str(d - earlier), flush=True)\n    else:\n        print(text, str(d), flush=True)\n    return d\n\n\nclass SinCosTransformer(BaseEstimator, TransformerMixin, OneToOneMixin):\n    def __init__(self, max_vals: int, feature_suffix_sin: str = \"_sin\",\n                 feature_suffix_cos: str = \"_cos\", label_feature: str = \"target\",\n                 verbose: int = 0):\n        super().__init__()\n        self.max_vals = max_vals\n        self.feature_suffix_sin = feature_suffix_sin\n        self.feature_suffix_cos = feature_suffix_cos\n        self.label_feature = label_feature\n        self.verbose = verbose\n        self.feature_names = []\n    \n    def debug(self, *args, **kwargs):\n        if self.verbose:\n            print(*args, **kwargs)\n    \n    def fit(self, X, y=None):\n        \"\"\"Does nothing at fit time\n        \n        GitHub PR \"RFC Implement Pipeline get feature names #12627\" will introduce\n        proper argument passing in the get_feature_names chain.\n        Until that is available, we'll store the feature names at fit time.\n        \n        :return: self, so we can daisy-chain .transform()\n        \"\"\"\n        self.debug(\"SinCosTransformer.fit(X=\", type(X), \"y=\", type(y), \")\")\n        if hasattr(X, \"columns\"):\n            for col in X.columns:\n                self.feature_names.append(col + self.feature_suffix_sin)\n                self.feature_names.append(col + self.feature_suffix_cos)\n            self.debug(\"SinCosTransformer.fit() saving feature names:\", self.feature_names)       \n        else:\n            self.debug(\"SinCosTransformer.fit() input X has no 'columns' attr,\",\n                       \"cannot save feature names\")\n        return self\n\n    def get_feature_names(self, input_features=None):\n        \"\"\"Overriding method in OneToOneMixin, because ColumnTransformer does not pass any value\n        for input_features until GitHub PR \"RFC Implement Pipeline get feature names #12627\" is\n        available\n        \"\"\"\n        return self.feature_names\n    \n    def transform(self, X):\n        X_new = pd.DataFrame()\n        self.debug(\"SinCosTransformer.transform(\", type(X), \") running\")\n        \n        if hasattr(X, \"columns\"):\n            self.debug(\"SinCosTransformer.transform() generating sin/cos values, X cols:\", X.columns)\n            \n            for col_name in X.columns:\n                if col_name == self.label_feature:\n                    self.debug(\"SinCosTransformer.transform() skipping label_feature '\",\n                               col_name, \"', not sure why sklearn passes it\",\n                               \"for this transformer and not the others\")\n                else:\n                    X_new[col_name + self.feature_suffix_sin] = np.sin(\n                        2 * np.pi * X[col_name] / self.max_vals)\n                    X_new[col_name + self.feature_suffix_cos] = np.cos(\n                        2 * np.pi * X[col_name] / self.max_vals)\n        else:\n            raise AttributeError(\"SinTransformer input X (\", type(X),\n                                 \") has no attribute 'columns'\")\n        return X_new\n\n    \nclass MeanTransformer(BaseEstimator, TransformerMixin, OneToOneMixin):\n    def __init__(self, feature_suffix: str = \"_mean\", label_feature: str = \"target\",\n                 verbose: int = 0):\n        super().__init__()\n        self.feature_suffix = feature_suffix\n        self.label_feature = label_feature\n        # mean_maps will be populated in fit().  It is a dict, k=col_name, value is another dict:\n        #    k=value seen in traning, v=mean target label seen in training\n        self.mean_maps = {}\n        self.feature_names = []\n        self.verbose = verbose\n    \n    def debug(self, *args, **kwargs):\n        if self.verbose:\n            print(*args, **kwargs)\n    \n    def fit(self, X, y=None):\n        \"\"\"Calculates the mean target label value for each input value in X and retains\n        a map to apply at transform() time.\n        \n        GitHub PR \"RFC Implement Pipeline get feature names #12627\" will introduce\n        proper argument passing in the get_feature_names chain.\n        Until that is available, we'll store the feature names at fit time.\n        \n        :return: self, so we can daisy-chain .transform()\n        \"\"\"\n        self.debug(\"MeanTransformer.fit(X=\", type(X), \"y=\", type(y), \")\")\n        \n        if hasattr(X, \"columns\"):\n            df = pd.DataFrame(X)\n            \n            df[self.label_feature] = y\n            self.debug(\"MeanTransformer.fit(), df cols after adding labels:\", str(df.columns))\n            \n            for col in df:\n                self.debug(\"MeanTransformer.fit(), iterating on col:\", col,\n                           \"comparing it to:\", self.label_feature)\n                \n                if str(col) == self.label_feature:\n                    self.debug(\"MeanTransformer.fit(), skipping fit on label column\")\n                else:\n                    self.debug(\"MeanTransformer.fit(), adding column '\", col,\n                               \"' to feature_names\")\n                    self.feature_names.append(col + self.feature_suffix)\n                    self.mean_maps[col] = np.round(\n                        df.groupby(col)[self.label_feature].mean(), decimals=2).to_dict()\n            # drop the labels, lest they be given back to us in transform()\n            df.drop(self.label_feature, axis=1, inplace=True)\n        else:\n            self.debug(\"MeanTransformer.fit() input X (\", type(X),\n                       \") has no 'columns' attr, cannot save feature names\")\n        return self\n\n    def get_feature_names(self, input_features=None):\n        \"\"\"Overriding method in OneToOneMixin, because ColumnTransformer does not pass any value\n        for input_features until GitHub PR \"RFC Implement Pipeline get feature names #12627\" is\n        available\n        \"\"\"\n        return self.feature_names\n    \n    def transform(self, X):\n        X_new = pd.DataFrame()\n        self.debug(\"MeanTransformer.transform(\", type(X), \") running\")\n        \n        if hasattr(X, \"columns\"):\n            self.debug(\"MeanTransformer.transform() mapping mean values, X cols:\", X.columns)\n            \n            for col_name in X.columns:\n                if col_name == self.label_feature:\n                    self.debug(\"MeanTransformer.transform() skipping label_feature '\",\n                                col_name, \"', not sure why sklearn passes it\",\n                               \"for this transformer and not the others\")\n                else:\n                    if hasattr(X[col_name], \"map\"):\n                        X_new[col_name+self.feature_suffix] = X[col_name].map(\n                            self.mean_maps[col_name])\n                    else:\n                        raise AttributeError(\"MeanTransformer column (name=\", col_name,\n                                             \", type=\", type(X[col_name]), \") has no attribute 'map'\")\n        else:\n            raise AttributeError(\"MeanTransformer input X (\", type(X),\n                                 \") has no attribute 'columns'\")\n        return X_new\n   \n    \nclass MapTransformer(BaseEstimator, TransformerMixin, OneToOneMixin):\n    def __init__(self, value_map: dict = {}, verbose: int = 0):\n        super().__init__()\n        self.value_map = value_map\n        self.verbose = verbose\n        self.feature_names = None\n    \n    def fit(self, X, y=None):\n        \"\"\"This transformer does nothing at fit time.\n        WHY does TransformerMixin have no fit() method for us to invoke/override?\n        \n        GitHub PR \"RFC Implement Pipeline get feature names #12627\" will introduce\n        proper argument passing in the get_feature_names chain.\n        Until that is available, we'll store the feature names at fit time.\n        \n        :return: self, so we can daisy-chain .transform()\n        \"\"\"\n        self.debug(\"MapTransformer.fit(\", type(X), \") running\")\n        if hasattr(X, \"columns\"):\n            self.feature_names = X.columns\n            self.debug(\"MapTransformer.fit() saving feature names:\", self.feature_names)\n                   \n        else:\n            self.debug(\"MapTransformer.fit() input X has no 'columns' attr,\",\n                       \"cannot save feature names\")\n        return self\n\n    def get_feature_names(self, input_features=None):\n        \"\"\"Overriding method in OneToOneMixin, because ColumnTransformer does not pass any value\n        for input_features until GitHub PR \"RFC Implement Pipeline get feature names #12627\" is\n        available\n        \"\"\"\n        return self.feature_names\n    \n    def transform(self, X):\n        self.debug(\"MapTransformer.transform(\", type(X), \") running, cols:\", X.columns)\n                   \n        if hasattr(X, \"replace\"):\n            #for k, v in self.value_map.items():\n            #    X.replace(k, v, inplace=True)\n            X.replace(self.value_map, inplace=True)\n            self.debug(\"MapTransformer.transform() replacing with value_map\",\n                       str(self.value_map))\n        else:\n            raise AttributeError(\"MapTransformer input X (\", type(X),\n                                 \") has no attribute 'replace'\")\n        return X\n\n    def debug(self, *args, **kwargs):\n        if self.verbose:\n            print(*args, **kwargs)\n\n\nclass AsciiTransformer(BaseEstimator, TransformerMixin, OneToOneMixin):\n    def __init__(self, feature_suffix: str = '_ascii', verbose: int = 0):\n        super().__init__()\n        self.verbose = verbose\n        self.feature_names = None\n        self.feature_suffix = feature_suffix\n    \n    def fit(self, X, y=None):\n        \"\"\"This transformer does nothing at fit time.\n        WHY does TransformerMixin have no fit() method for us to invoke/override?\n        \n        GitHub PR \"RFC Implement Pipeline get feature names #12627\" will introduce\n        proper argument passing in the get_feature_names chain.\n        Until that is available, we'll store the feature names at fit time.\n        \n        :return: self, so we can daisy-chain .transform()\n        \"\"\"\n        if hasattr(X, \"columns\"):\n            # pd.DataFrame\n            self.feature_names = [col + self.feature_suffix for col in X.columns]\n            self.debug(\"AsciiTransformer.fit(\", type(X), \") saving feature names:\",\n                       self.feature_names)\n            \n        elif hasattr(X, \"dtype\") and hasattr(X.dtype, \"names\") and X.dtype.names is not None:\n            # np.ndarray\n            self.feature_names = [col + self.feature_suffix for col in X.dtype.names]\n            self.debug(\"AsciiTransformer.fit(\", type(X), \") saving feature names:\",\n                       self.feature_names)\n            \n        else:\n            raise AttributeError(\"AsciiTransformer.fit(), input X (\", type(X),\n                                 \") has no 'columns' or 'dtype.names' attr,\",\n                                 \"cannot save feature names\")\n        return self\n\n    def get_feature_names(self, input_features=None):\n        \"\"\"Overriding method in OneToOneMixin, because ColumnTransformer does not pass any value\n        for input_features until GitHub PR \"RFC Implement Pipeline get feature names #12627\" is\n        available\n        \"\"\"\n        return self.feature_names\n    \n    def transform(self, X):\n        X_new = pd.DataFrame()\n        self.debug(\"AsciiTransformer.transform(\", type(X), \") running\")\n        \n        if hasattr(X, \"columns\"):\n            self.debug(\"AsciiTransformer.transform() replacing with ascii values\")\n\n            for col_name in X.columns:\n                if hasattr(X[col_name], \"map\"):\n                    X_new[col_name+self.feature_suffix] = X[col_name].map(\n                        ascii_ord, na_action='ignore')\n                else:\n                    raise AttributeError(\"AsciiTransformer column (name=\", col_name,\n                                         \", type=\", type(X[col_name]), \") has no attribute 'map'\")\n        else:\n            raise AttributeError(\"AsciiTransformer input X (\", type(X),\n                                 \") has no attribute 'columns'\")\n        return X_new\n\n    def debug(self, *args, **kwargs):\n        if self.verbose:\n            print(*args, **kwargs)\n\n                   \ndef ascii_ord(s):\n    \"\"\"\n    Walks through each char in s backwards, and accumulates the ascii ord values\n    \n    :param s: str - the string to convert\n    :return: int, the accumulated value\n    \"\"\"\n    acc = 0\n    for index, c in enumerate(reversed(s)):\n        acc += ord(c) * (128 ** index)\n    return acc\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Create a simple model\nThis is an exploration of grid search, so just build a preprocessor and model pipeline that is simple and quick to train."},{"metadata":{"trusted":true},"cell_type":"code","source":"###############################################################################################\n# Initialize Preprocessor Steps\n###############################################################################################\n\npreprocessor_steps = []\n\n\n###############################################################################################\n# Ordinal Transformation Pipeline\n###############################################################################################\n\n# ord_0 is already ordinal numbers, so only need to replace missing and scale\nord0_transformer = PipelineWithNames(\n    steps=[\n        ('feature', SimpleImputerWithNames(strategy=\"most_frequent\")),\n        ('scaler', StandardScalerWithNames()),\n    ])\n\nif CTL.enable_ord_features and CTL.enable_ord0:\n    preprocessor_steps.append((\"ord0\", ord0_transformer, [\"ord_0\"]))\n    print(\"Enabling ordinal feature [ord_0]\")\nelse:\n    print(\"Disabling ord_0 feature in pipeline\")\n\n# Novice is the most common, so map NaN to Novice:1\nord1_value_map = {\n    \"Novice\": 1,\n    \"Contributor\": 2,\n    \"Expert\": 3,\n    \"Master\": 4,\n    \"Grandmaster\": 5,\n    np.nan: 1\n}\n\nord1_transformer = PipelineWithNames(\n    steps=[\n        ('value_map', MapTransformer(value_map=ord1_value_map, verbose=0)),\n        ('scaler', StandardScalerWithNames()),\n    ])\n\nif CTL.enable_ord_features and CTL.enable_ord1:\n    preprocessor_steps.append(('ord1', ord1_transformer, ['ord_1']))\n    print(\"Enabling ordinal feature [ord_1]\")\nelse:\n    print(\"Disabling ord_1 feature in pipeline\")\n\n# Freezing is the most common, so map NaN to Freezing:1\nord2_value_map = {\n    \"Freezing\": 1,\n    \"Cold\": 2,\n    \"Warm\": 3,\n    \"Hot\": 4,\n    \"Boiling Hot\": 5,\n    \"Lava Hot\": 6,\n    np.nan: 1\n}\n\nord2_transformer = PipelineWithNames(\n    steps=[\n        ('value_map', MapTransformer(value_map=ord2_value_map, verbose=0)),\n        ('scaler', StandardScalerWithNames()),\n    ])\n\nif CTL.enable_ord_features and CTL.enable_ord2:\n    preprocessor_steps.append(('ord2', ord2_transformer, ['ord_2']))\n    print(\"Enabling ordinal feature [ord_2]\")\nelse:\n    print(\"Disabling ord_2 feature in pipeline\")\n\n# ord 3 & 4 get the ascii treatment\nascii_transformer = PipelineWithNames(\n    steps=[\n        ('missing', SimpleImputerWithNames(strategy=\"most_frequent\", verbose=0)),\n        ('ascii', AsciiTransformer()),\n        ('scaler', StandardScalerWithNames()),\n    ]\n)\nif CTL.enable_ord_features and CTL.enable_ord345:\n    ordinal_ascii_features = ['ord_3', 'ord_4', 'ord_5']\n    preprocessor_steps.append(\n        ('ord345', ascii_transformer, ordinal_ascii_features)\n    )\n    print(\"Enabling ordinal ascii features\", ordinal_ascii_features)\nelse:\n    print(\"Disabling ordinal ascii features in pipeline\")\n\n    \n###############################################################################################\n# Assemble the All-Feature Preprocessor\n###############################################################################################\nfeature_preprocessor = ColumnTransformer(transformers=preprocessor_steps)\n\n\nprint(\"\")\nprint(\"Feature Preprocessor:\")\nprint(str(feature_preprocessor))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Split training and validation data"},{"metadata":{"trusted":true},"cell_type":"code","source":"t = print_stamp(\"Splitting train and validation data...\")\nX_train, X_valid, y_train, y_valid = train_test_split(\n    train_df.drop(label_column, axis=1),\n    train_df[label_column],\n    random_state=CTL.random_state,\n    stratify=train_df[label_column]\n)\nprint_stamp(\"Splitting complete\", t)\nprint(\"\")\n\nprint(\"X_train shape:\", X_train.shape)\nprint(\"X_valid shape:\", X_valid.shape)\nprint(\"y_train shape:\", y_train.shape)\nprint(\"y_valid shape:\", y_valid.shape)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Train and Evaluate the Model"},{"metadata":{"trusted":true},"cell_type":"code","source":"t = print_stamp(\"Setting up model...\")\n\nmodel = None\n\nif CTL.active_model == CTL.model_randomforest:\n    print(\"Using randomforest model\")\n    model = RandomForestRegressor(\n        n_estimators=100,\n        random_state=CTL.random_state\n    )\n    \nelif CTL.active_model == CTL.model_xgboost:\n    print(\"Using xgboost model\")\n    params = {\n        'objective': 'binary:logistic',\n    }\n    model = xgb.XGBRegressor(random_state=CTL.random_state)\n    model.set_params(**params)\n\nprint_stamp(\"Model setup complete.\", t)\nprint(\"\")\n    \nt = print_stamp(\"Assembling final pipeline...\")\n\neval_pipeline = Pipeline(\n    steps=[\n        (\"preprocessor\", feature_preprocessor),\n        (\"model\", model)\n    ])\nprint_stamp(\"Assembly complete\", t)\n\n\nprint(\"\")\nt = print_stamp(\"Setting up grid search...\")\n\ngrid_search = RandomizedSearchCV(\n    estimator=eval_pipeline, param_distributions=CTL.gridsearch_param_grid,\n    scoring=CTL.gridsearch_scoring, n_iter=CTL.gridsearch_n_iter, cv=CTL.gridsearch_cv,\n    random_state=CTL.random_state,\n    )\nprint_stamp(\"Setup complete\", t)\n\nprint(\"\")\ntrain_start = print_stamp(\"Training grid search...\")\ngrid_search.fit(X_train, y_train)\ntrain_end = print_stamp(\"Training complete\", train_start)\nround_count = CTL.gridsearch_n_iter * CTL.gridsearch_cv\nprint(\"Grid search / CV rounds:\", round_count)\nprint(\"Duration per round:\", str((train_end - train_start) / round_count))\nprint(\"\")\n\nbest_model = grid_search.best_estimator_\nbest_params = grid_search.best_params_\n\nprint(\"Transformed feature names:\")\nprint(best_model.named_steps[\"preprocessor\"].get_feature_names())\nprint(\"\")\n\nt = print_stamp(\"Predicting on validation set...\")\ny_preds = best_model.predict(X_valid)\nprint_stamp(\"Predicting complete\", t)\nprint(\"\")\n\nauc = roc_auc_score(y_valid, y_preds)\nprint(\"AUC: {0:.05f}\".format(auc))\nprint(\"\")","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Visualize the Best Model"},{"metadata":{"trusted":true},"cell_type":"code","source":"print(\"Details of best model (validation AUC={0:.05f}):\".format(auc))\nprint(\"\")\nif isinstance(best_model, Pipeline):\n    print(best_model.named_steps[\"model\"])\nelse:\n    print(best_model)\nprint(\"\")\n\nprint(\"Best Grid Search parameters:\")\nfor k, v in best_params.items():\n    print(\"\\t\", k, v, sep=\"\\t\")\n\nprint(\"\")\n\nfeature_weights = pd.DataFrame(\n    data={\n        'importance': best_model.named_steps[\"model\"].feature_importances_,\n        'feature': best_model.named_steps[\"preprocessor\"].get_feature_names()}\n)\nfeature_weights.sort_values(by=\"importance\", ascending=False, inplace=True)\n\nst = sns.axes_style()\nsns.set_style(\n    \"whitegrid\",\n    {\n        \"axes.labelcolor\": \".99\",\n        \"axes.axisbelow\": False,\n    }\n)\n\ndef plot_feature_weights(fw: pd.DataFrame, title: str = \"Feature Importance\"):\n    plt.figure(figsize=(20, 10))\n\n    ax = sns.barplot(x=\"feature\", y=\"importance\", data=fw)\n    ax.set(xlabel='Feature', ylabel='Importance', title=\"Feature Importance\")\n    ax.set_xticklabels(ax.get_xticklabels(), rotation=45, horizontalalignment=\"right\")\n    plt.show()\n\nplot_feature_weights(feature_weights)\n\nif feature_weights.shape[0] > 15:\n    plot_feature_weights(feature_weights[:15], \"Top 15 Most Important Features\")\n\nprint()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Run a final training cycle on full data set\n\n### 🔴 Data Splits for Model Evaluation\nSplitting the data sets gets slightly complex so let's lay it all out here:\n\n1. **TRAIN_DF** is the full file provided as training data for the competition\n1. We split **TRAIN_DF** into **TRAIN** and **VALID**\n1. **TRAIN** is fed into the grid search\n1. Grid search uses k-fold cross validation, splitting **TRAIN** into **FOLD_TRAIN** and **FOLD_VALID**\n    * There is no contamination between **VALID** and **FOLD_VALID**\n    * **FOLD_VALID** is exclusively a subset of **TRAIN**.\n1. Grid search also uses **VALID** as an evaluation set for early stopping rounds in XGBoost.\n1. After grid search finds the best model, we run a prediction on **VALID** and report the auc.\n\n### 🔴 Data for Final Predictions\n\n1. **TEST_DF** is the full file provided as test data for the competition\n1. After the best parameters are identified, run one final training cycle, using the full **TRAIN_DF** (i.e. the full data set the competition provides in train.csv) and use the resulting model to make predictions on **TEST_DF**.\n"},{"metadata":{"trusted":true},"cell_type":"code","source":"t = print_stamp(\"Training best model...\")\nbest_model.fit(train_df.drop(label_column, axis=1), train_df[label_column])\nprint_stamp(\"Training complete\", t)\nprint(\"\")","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Make test predictions and submission file"},{"metadata":{"trusted":true},"cell_type":"code","source":"X_test = test_df\n\nt = print_stamp(\"Predicting on test set...\")\ntest_preds = best_model.predict(X_test)\nprint_stamp(\"Predicting complete\", t)\nprint(\"\")\n\nsubmission = pd.DataFrame({\"id\": X_test.index, label_column: test_preds})\n\n# sub_filename_pattern = \"preds-{0:s}-{1:0.5f}.csv\"\n# sub_file_date_pattern = \"%Y%m%d%H%M%S\"\nsub_filename = CTL.sub_filename_pattern\nif CTL.sub_file_date_pattern is not None:\n    sub_filename = CTL.sub_filename_pattern.format(datetime.now().strftime(CTL.sub_file_date_pattern), auc)\n    \nif CTL.output_dir is not None:\n    sub_filename = os.path.join(CTL.output_dir, sub_filename)\n\nprint(\"Submission file:\", sub_filename)\n\nsubmission.to_csv(sub_filename, index=False)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"name":"python","version":"3.6.4","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"}},"nbformat":4,"nbformat_minor":1}