{"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"codemirror_mode":{"name":"ipython","version":3},"file_extension":".py","mimetype":"text/x-python","name":"python","nbconvert_exporter":"python","pygments_lexer":"ipython3","version":"3.11.13"},"kaggle":{"accelerator":"gpu","dataSources":[{"sourceId":17000,"databundleVersionId":895292,"sourceType":"competition"}],"isInternetEnabled":false,"language":"python","sourceType":"notebook","isGpuEnabled":true}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# 📝 Kaggle Notebook – Categorical Feature Encoding Challenge II\n\nWelcome to this notebook!  \n\n## Objective\nThe goal of this notebook is to **participate in the Kaggle competition \"Categorical Feature Encoding Challenge II\"** by predicting the target variable using tabular data.  \nWe will preprocess the data, handle missing values, encode categorical features, train a model, and optimize its performance.\n\n## Notebook Overview\nThis notebook is structured as follows:\n\n1. **Libraries & Data Loading**  \n   Import all necessary Python libraries and load the competition datasets.\n\n2. **Feature Engineering & Preprocessing**  \n   - Handle missing values for numerical and categorical features.  \n   - Encode categorical variables using **One-Hot, Ordinal, and Target Encoding**.  \n   - Optional feature transformations (like cyclic encoding for dates).  \n\n3. **Modeling**  \n   Train a machine learning model (**XGBoost**) using a preprocessing pipeline.  \n   Address class imbalance and tune hyperparameters for optimal performance.\n\n4. **Randomized Hyperparameter Search**  \n   - Use **RandomizedSearchCV** to explore multiple hyperparameter combinations efficiently.  \n   - Optimize parameters like `n_estimators`, `learning_rate`, `max_depth`, `subsample`, etc.  \n   - Use **ROC-AUC** as the scoring metric to select the best model.  \n\n. **Evaluation**  \n   - Evaluate model performance using **ROC-AUC**.  \n   - Visualize results with a **ROC curve**.  \n   - **Validation ROC-AUC achieved:** `0.7793`\n\n. **Conclusion & Submission**  \n   - Summarize insights and predictions.  \n   - Prepare a CSV submission file for Kaggle.\n\n## Requirements\n- Python 3.x  \n- Libraries: `pandas`, `numpy`, `matplotlib`, `scikit-learn`, `category_encoders`, `xgboost`  \n\n> 💡 This notebook is designed to be **easy to follow** for anyone who wants to see a complete workflow for the Kaggle competition **\"Categorical Feature Encoding Challenge II\"**, from preprocessing to evaluation, including **hyperparameter optimization with Randomized Search** and **robust cross-validation** for reliable performance metrics.\n","metadata":{}},{"cell_type":"code","source":"# This Python 3 environment comes with many helpful analytics libraries installed\n# It is defined by the kaggle/python Docker image: https://github.com/kaggle/docker-python\n# For example, here's several helpful packages to load\n\nimport numpy as np # linear algebra\nimport pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\n\n# Input data files are available in the read-only \"../input/\" directory\n# For example, running this (by clicking run or pressing Shift+Enter) will list all files under the input directory\n\nimport os\nfor dirname, _, filenames in os.walk('/kaggle/input'):\n    for filename in filenames:\n        print(os.path.join(dirname, filename))\n\n# You can write up to 20GB to the current directory (/kaggle/working/) that gets preserved as output when you create a version using \"Save & Run All\" \n# You can also write temporary files to /kaggle/temp/, but they won't be saved outside of the current session","metadata":{"execution":{"iopub.execute_input":"2025-10-27T21:13:35.454925Z","iopub.status.busy":"2025-10-27T21:13:35.454685Z","iopub.status.idle":"2025-10-27T21:13:36.050305Z","shell.execute_reply":"2025-10-27T21:13:36.049503Z","shell.execute_reply.started":"2025-10-27T21:13:35.454901Z"},"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"# Load the dataset\n","metadata":{}},{"cell_type":"code","source":"train = pd.read_csv(\"/kaggle/input/cat-in-the-dat-ii/train.csv\")\ntest = pd.read_csv(\"/kaggle/input/cat-in-the-dat-ii/test.csv\")\nsample = pd.read_csv(\"/kaggle/input/cat-in-the-dat-ii/sample_submission.csv\")\n\nprint(train.shape, test.shape)","metadata":{"execution":{"iopub.execute_input":"2025-10-27T21:13:39.551051Z","iopub.status.busy":"2025-10-27T21:13:39.550385Z","iopub.status.idle":"2025-10-27T21:13:44.076245Z","shell.execute_reply":"2025-10-27T21:13:44.075437Z","shell.execute_reply.started":"2025-10-27T21:13:39.551023Z"},"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"code","source":"# To hide (disable) warnings\nimport seaborn as sns\nimport matplotlib.pyplot as plt\nimport warnings\nwarnings.filterwarnings(\"ignore\", category=RuntimeWarning)\n","metadata":{"execution":{"iopub.execute_input":"2025-10-27T21:13:45.539341Z","iopub.status.busy":"2025-10-27T21:13:45.538731Z","iopub.status.idle":"2025-10-27T21:13:46.262539Z","shell.execute_reply":"2025-10-27T21:13:46.261992Z","shell.execute_reply.started":"2025-10-27T21:13:45.539316Z"},"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"# Data Exploration\n","metadata":{}},{"cell_type":"code","source":"train.head()","metadata":{"execution":{"iopub.execute_input":"2025-10-27T20:00:12.868419Z","iopub.status.busy":"2025-10-27T20:00:12.868202Z","iopub.status.idle":"2025-10-27T20:00:12.895776Z","shell.execute_reply":"2025-10-27T20:00:12.895147Z","shell.execute_reply.started":"2025-10-27T20:00:12.868398Z"},"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"code","source":"train.info()","metadata":{"execution":{"iopub.execute_input":"2025-10-27T20:00:12.897851Z","iopub.status.busy":"2025-10-27T20:00:12.897651Z","iopub.status.idle":"2025-10-27T20:00:13.343179Z","shell.execute_reply":"2025-10-27T20:00:13.342459Z","shell.execute_reply.started":"2025-10-27T20:00:12.897829Z"},"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"code","source":"train.describe()","metadata":{"execution":{"iopub.execute_input":"2025-10-27T20:00:13.344034Z","iopub.status.busy":"2025-10-27T20:00:13.343857Z","iopub.status.idle":"2025-10-27T20:00:13.524690Z","shell.execute_reply":"2025-10-27T20:00:13.523843Z","shell.execute_reply.started":"2025-10-27T20:00:13.344020Z"},"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"code","source":"train.columns","metadata":{"execution":{"iopub.execute_input":"2025-10-27T20:00:13.525742Z","iopub.status.busy":"2025-10-27T20:00:13.525546Z","iopub.status.idle":"2025-10-27T20:00:13.530548Z","shell.execute_reply":"2025-10-27T20:00:13.529899Z","shell.execute_reply.started":"2025-10-27T20:00:13.525721Z"},"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"code","source":"counts = train['target'].value_counts()\n\nplt.pie(counts, labels=counts.index,\n    autopct='%1.1f%%', \n    startangle=90,\n    colors=['lightgreen', 'lightblue'])\n\nplt.title('Target Distribution')\nplt.axis('equal')  # make it a circle\nplt.show()\n\ncounts\n# unbalanced data","metadata":{"execution":{"iopub.execute_input":"2025-10-27T20:00:13.531421Z","iopub.status.busy":"2025-10-27T20:00:13.531213Z","iopub.status.idle":"2025-10-27T20:00:13.620775Z","shell.execute_reply":"2025-10-27T20:00:13.620055Z","shell.execute_reply.started":"2025-10-27T20:00:13.531406Z"},"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"code","source":"# number of unique values for every Feature\nfor col in train.columns:\n    unique_values = train[col].nunique()\n    print(f\"{col}: {unique_values}\")\n","metadata":{"execution":{"iopub.execute_input":"2025-10-27T20:00:13.621918Z","iopub.status.busy":"2025-10-27T20:00:13.621581Z","iopub.status.idle":"2025-10-27T20:00:14.003233Z","shell.execute_reply":"2025-10-27T20:00:14.002574Z","shell.execute_reply.started":"2025-10-27T20:00:13.621895Z"},"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"code","source":"#unique values for every Feature\nfor col in train.columns:\n    print(f\"{col} ==> {train[col].unique()}\")\n","metadata":{"execution":{"iopub.execute_input":"2025-10-27T20:00:14.004087Z","iopub.status.busy":"2025-10-27T20:00:14.003882Z","iopub.status.idle":"2025-10-27T20:00:14.377135Z","shell.execute_reply":"2025-10-27T20:00:14.376514Z","shell.execute_reply.started":"2025-10-27T20:00:14.004073Z"},"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"code","source":"#Distribute values for each Feature\nfor col in train.columns:\n    print(f\"\\n{col} value counts:\")\n    print(train[col].value_counts().head(10)) \n","metadata":{"execution":{"iopub.execute_input":"2025-10-27T20:00:14.378336Z","iopub.status.busy":"2025-10-27T20:00:14.377973Z","iopub.status.idle":"2025-10-27T20:00:15.061811Z","shell.execute_reply":"2025-10-27T20:00:15.061002Z","shell.execute_reply.started":"2025-10-27T20:00:14.378312Z"},"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"code","source":"train.duplicated().sum()","metadata":{"execution":{"iopub.execute_input":"2025-10-27T20:00:15.062960Z","iopub.status.busy":"2025-10-27T20:00:15.062670Z","iopub.status.idle":"2025-10-27T20:00:15.950550Z","shell.execute_reply":"2025-10-27T20:00:15.949723Z","shell.execute_reply.started":"2025-10-27T20:00:15.062933Z"},"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"code","source":"#counts of Missing values\nsummary = pd.DataFrame({\n    'Missing Count': train.isna().sum(),\n    'Unique Values': train.nunique(dropna=True)\n})\n\nprint(summary)\n","metadata":{"execution":{"iopub.execute_input":"2025-10-27T20:00:15.951784Z","iopub.status.busy":"2025-10-27T20:00:15.951415Z","iopub.status.idle":"2025-10-27T20:00:16.754796Z","shell.execute_reply":"2025-10-27T20:00:16.754134Z","shell.execute_reply.started":"2025-10-27T20:00:15.951757Z"},"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"code","source":"#Make Heatmap for Missing values\n\nplt.figure(figsize=(20,10))\nsns.heatmap(train.isnull(), cbar=False, cmap='gray')\nplt.title('Heatmap of Missing Data')\nplt.show()\n","metadata":{"execution":{"iopub.execute_input":"2025-10-27T20:00:16.757715Z","iopub.status.busy":"2025-10-27T20:00:16.757495Z","iopub.status.idle":"2025-10-27T20:00:29.488964Z","shell.execute_reply":"2025-10-27T20:00:29.488227Z","shell.execute_reply.started":"2025-10-27T20:00:16.757699Z"},"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"code","source":"# change values of binary object columns to 0,1 \ntrain['bin_3'] = train['bin_3'].map({'F': 0, 'T': 1})\ntrain['bin_4'] = train['bin_4'].map({'N': 0, 'Y': 1})\ntest['bin_3'] = train['bin_3'].map({'F': 0, 'T': 1})\ntest['bin_4'] = train['bin_4'].map({'N': 0, 'Y': 1})","metadata":{"execution":{"iopub.execute_input":"2025-10-27T21:13:56.546706Z","iopub.status.busy":"2025-10-27T21:13:56.546345Z","iopub.status.idle":"2025-10-27T21:13:56.769399Z","shell.execute_reply":"2025-10-27T21:13:56.768590Z","shell.execute_reply.started":"2025-10-27T21:13:56.546685Z"},"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"code","source":"train[['bin_3','bin_4']].info()","metadata":{"execution":{"iopub.execute_input":"2025-10-27T20:00:29.672050Z","iopub.status.busy":"2025-10-27T20:00:29.671830Z","iopub.status.idle":"2025-10-27T20:00:29.688703Z","shell.execute_reply":"2025-10-27T20:00:29.687952Z","shell.execute_reply.started":"2025-10-27T20:00:29.672034Z"},"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"code","source":"# Categorical Columns \ncat_vars = [var for var in train.columns if train[var].dtype == 'O']\nprint(len(cat_vars), cat_vars)","metadata":{"execution":{"iopub.execute_input":"2025-10-27T20:00:29.690171Z","iopub.status.busy":"2025-10-27T20:00:29.689623Z","iopub.status.idle":"2025-10-27T20:00:29.694844Z","shell.execute_reply":"2025-10-27T20:00:29.694000Z","shell.execute_reply.started":"2025-10-27T20:00:29.690148Z"},"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"code","source":"# Numerical Columns \ntarget = 'target'\nnum_vars = [var for var in train.columns if var not in cat_vars + [target] + ['id']]\nprint(len(num_vars), num_vars)\n\n","metadata":{"execution":{"iopub.execute_input":"2025-10-27T20:00:29.695709Z","iopub.status.busy":"2025-10-27T20:00:29.695479Z","iopub.status.idle":"2025-10-27T20:00:29.710982Z","shell.execute_reply":"2025-10-27T20:00:29.710306Z","shell.execute_reply.started":"2025-10-27T20:00:29.695689Z"},"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"# Exploratory Data Analysis (EDA)\n","metadata":{}},{"cell_type":"code","source":"corr_matrix = train[num_vars + ['target']].corr()\nplt.figure(figsize=(10,8))\nsns.heatmap(corr_matrix, annot=True, fmt=\".2f\", cmap=\"coolwarm\", cbar=True)\nplt.title(\"Correlation Heatmap\")\nplt.show()","metadata":{"execution":{"iopub.execute_input":"2025-10-27T20:00:29.711881Z","iopub.status.busy":"2025-10-27T20:00:29.711666Z","iopub.status.idle":"2025-10-27T20:00:30.199067Z","shell.execute_reply":"2025-10-27T20:00:30.198221Z","shell.execute_reply.started":"2025-10-27T20:00:29.711865Z"},"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"code","source":"id_test = test['id']\ntrain = train.drop(columns=['id'])\ntest = test.drop(columns=['id'])","metadata":{"execution":{"iopub.execute_input":"2025-10-27T21:14:06.737243Z","iopub.status.busy":"2025-10-27T21:14:06.736686Z","iopub.status.idle":"2025-10-27T21:14:06.899279Z","shell.execute_reply":"2025-10-27T21:14:06.898627Z","shell.execute_reply.started":"2025-10-27T21:14:06.737220Z"},"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"# Train-Validation Split\n","metadata":{}},{"cell_type":"code","source":"from sklearn.model_selection import train_test_split\n\nX = train.drop(columns=['target'])\ny = train['target']\n\nX_train, X_val, y_train, y_val = train_test_split(\n    X, y, test_size=0.2, random_state=42\n)","metadata":{"execution":{"iopub.execute_input":"2025-10-27T21:14:09.200796Z","iopub.status.busy":"2025-10-27T21:14:09.200063Z","iopub.status.idle":"2025-10-27T21:14:09.767151Z","shell.execute_reply":"2025-10-27T21:14:09.766189Z","shell.execute_reply.started":"2025-10-27T21:14:09.200768Z"},"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"# processing","metadata":{}},{"cell_type":"code","source":"# cyclical encoding For Day and Month (its like Normalization)\nimport numpy as np\nfrom sklearn.base import BaseEstimator, TransformerMixin\n\nclass CyclicDateEncoder(BaseEstimator, TransformerMixin):\n    def __init__(self, day_col='day', month_col='month'):\n        self.day_col = day_col\n        self.month_col = month_col\n\n    def fit(self, X, y=None):\n        return self\n\n    def transform(self, X):\n        \n        if isinstance(X, np.ndarray):\n            X = pd.DataFrame(X, columns=[self.day_col, self.month_col] + [f'col{i}' for i in range(X.shape[1]-2)])\n        else:\n            X = X.copy()\n\n       \n        if self.day_col in X.columns:\n            X['day_sin'] = np.sin(2 * np.pi * X[self.day_col] / 7)\n            X['day_cos'] = np.cos(2 * np.pi * X[self.day_col] / 7)\n            X.drop(self.day_col, axis=1, inplace=True)\n\n        if self.month_col in X.columns:\n            X['month_sin'] = np.sin(2 * np.pi * X[self.month_col] / 12)\n            X['month_cos'] = np.cos(2 * np.pi * X[self.month_col] / 12)\n            X.drop(self.month_col, axis=1, inplace=True)\n\n        return X.values  ","metadata":{},"outputs":[],"execution_count":null},{"cell_type":"code","source":"from sklearn.pipeline import Pipeline\nfrom sklearn.compose import ColumnTransformer\nfrom sklearn.preprocessing import OneHotEncoder, OrdinalEncoder\nfrom sklearn.impute import SimpleImputer                              # to fill missing values\nimport category_encoders as ce                                        # for target Encoding\n\n\nnumeric_features = ['bin_0','bin_1','bin_2','bin_3','bin_4','ord_0','day','month']    # fill missing with mode\nonehot_features = ['nom_0','nom_1','nom_2','nom_3','nom_4','ord_3','ord_4']           # fill missing with unknown + One-Hot encoding\nordinal_encoded_features = ['ord_1','ord_2']                                          # fill missing with mode + Ordinal Encoding\ntarget_encoded_features = ['nom_5','nom_6','nom_7','nom_8','nom_9','ord_5']          # fill missing with 'unknown' + Target Encoding\n\nordinal_order = [\n    ['Novice', 'Contributor', 'Expert', 'Master', 'Grandmaster'], \n    ['Freezing', 'Cold', 'Warm', 'Hot', 'Boiling Hot', 'Lava Hot']                    #to detect how the model understands the order\n]\n\nfor col in target_encoded_features:\n    for df in [X_train, X_val, test]:\n        if col in df.columns:\n            df[col] = df[col].fillna('Unknown')\n\ntarget_encoder = ce.TargetEncoder(cols=target_encoded_features)\nX_train[target_encoded_features] = target_encoder.fit_transform(X_train[target_encoded_features], y_train)\nX_val[target_encoded_features] = target_encoder.transform(X_val[target_encoded_features])\ntest[target_encoded_features] = target_encoder.transform(test[target_encoded_features])\n\nnumeric_transformer = Pipeline([\n    ('imputer', SimpleImputer(strategy='most_frequent')),        # 'most_frequent' its like 'mode\n    ('cyclic',CyclicDateEncoder(day_col='day', month_col='month'))    # cyclical encoding For Day and Month (its like Normalization)\n                                             \n])\n\nonehot_transformer = Pipeline([\n    ('imputer', SimpleImputer(strategy='constant', fill_value='Unknown')),\n    ('onehot', OneHotEncoder(handle_unknown='ignore'))   # (handle_unknown='ignore') is to ignore any new data in test data\n])\n\nordinal_transformer = Pipeline([\n    ('imputer', SimpleImputer(strategy='most_frequent')),\n    ('ordinal', OrdinalEncoder(categories=ordinal_order, handle_unknown='use_encoded_value', unknown_value=-1))\n])\n\npreprocessor = ColumnTransformer([\n    ('num', numeric_transformer, numeric_features),\n    ('onehot', onehot_transformer, onehot_features),\n    ('ordinal', ordinal_transformer, ordinal_encoded_features)\n], remainder='passthrough') \n\n","metadata":{"execution":{"iopub.execute_input":"2025-10-27T21:14:36.787569Z","iopub.status.busy":"2025-10-27T21:14:36.787270Z","iopub.status.idle":"2025-10-27T21:14:41.160441Z","shell.execute_reply":"2025-10-27T21:14:41.159615Z","shell.execute_reply.started":"2025-10-27T21:14:36.787548Z"},"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"# Select the model","metadata":{}},{"cell_type":"code","source":"from sklearn.linear_model import LogisticRegression\n\nfrom xgboost import XGBClassifier\nratio = (y_train == 0).sum() / (y_train == 1).sum()\n\nmodel = Pipeline([\n    ('preprocessor', preprocessor),\n    ('classifier',\n     \nXGBClassifier(\n    n_estimators=1500,  #Number of Decision Tree          \n    learning_rate=0.01,  # best one is 0,01  'alpha'       \n    max_depth=8,                   \n    min_child_weight=4,           \n    gamma=0.3,                     \n    subsample=0.11,                 \n    colsample_bytree=0.8,          \n    tree_method='gpu_hist',  # use GPU\n    predictor='gpu_predictor',  # use GPU in predict\n    grow_policy='lossguide',       \n    scale_pos_weight = ratio, #for Imbalanced Data            \n    eval_metric='auc',              \n    use_label_encoder=False,\n    n_jobs=-1,\n    random_state=42\n)\n     \n    )]\n                )\n","metadata":{"execution":{"iopub.execute_input":"2025-10-27T21:14:51.890707Z","iopub.status.busy":"2025-10-27T21:14:51.889990Z","iopub.status.idle":"2025-10-27T21:14:52.124706Z","shell.execute_reply":"2025-10-27T21:14:52.123912Z","shell.execute_reply.started":"2025-10-27T21:14:51.890686Z"},"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"# Train the Model","metadata":{}},{"cell_type":"code","source":"model.fit(X_train, y_train)","metadata":{"execution":{"iopub.execute_input":"2025-10-27T21:14:55.738431Z","iopub.status.busy":"2025-10-27T21:14:55.738149Z","iopub.status.idle":"2025-10-27T21:15:25.401709Z","shell.execute_reply":"2025-10-27T21:15:25.400894Z","shell.execute_reply.started":"2025-10-27T21:14:55.738411Z"},"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"# Evaluation","metadata":{}},{"cell_type":"code","source":"from sklearn.metrics import roc_curve, roc_auc_score\n\ny_val_prob = model.predict_proba(X_val)[:, 1]  # Probability of the positive class\nroc_auc = roc_auc_score(y_val, y_val_prob)\nprint(f'ROC-AUC: {roc_auc:.4f}')\n\n\nfpr, tpr, thresholds = roc_curve(y_val, y_val_prob)\nplt.figure(figsize=(8,6))\nplt.plot(fpr, tpr, label=f'ROC curve (AUC = {roc_auc:.4f})')\nplt.plot([0,1], [0,1], 'k--')  \nplt.xlabel('False Positive Rate')\nplt.ylabel('True Positive Rate')\nplt.title('ROC Curve')\nplt.legend(loc='lower right')\nplt.grid()\nplt.show()","metadata":{"execution":{"iopub.execute_input":"2025-10-27T21:18:23.926830Z","iopub.status.busy":"2025-10-27T21:18:23.926099Z","iopub.status.idle":"2025-10-27T21:18:24.905844Z","shell.execute_reply":"2025-10-27T21:18:24.905122Z","shell.execute_reply.started":"2025-10-27T21:18:23.926805Z"},"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"code","source":"'''\nfrom sklearn.model_selection import RandomizedSearchCV\n\nparam_grid = {\n    'classifier__n_estimators': [1000, 1500, 2000],\n    'classifier__learning_rate': [0.01, 0.02, 0.05],\n    'classifier__max_depth': [7, 9, 11],\n    'classifier__min_child_weight': [1, 3, 5],\n    'classifier__subsample': [0.7, 0.8, 0.9],\n    'classifier__colsample_bytree': [0.7, 0.8, 0.9],\n    'classifier__gamma': [0, 0.1, 0.2, 0.3]\n}\n\nsearch = RandomizedSearchCV(model, param_grid, n_iter=20, scoring='roc_auc', cv=, verbose=2)\nsearch.fit(X_train, y_train)\nprint(search.best_params_)\n'''\n\n","metadata":{"execution":{"iopub.execute_input":"2025-10-27T21:15:25.403262Z","iopub.status.busy":"2025-10-27T21:15:25.403014Z","iopub.status.idle":"2025-10-27T21:15:25.408402Z","shell.execute_reply":"2025-10-27T21:15:25.407791Z","shell.execute_reply.started":"2025-10-27T21:15:25.403245Z"},"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"code","source":"test_pred = model.predict(test)","metadata":{"execution":{"iopub.execute_input":"2025-10-27T21:18:32.780716Z","iopub.status.busy":"2025-10-27T21:18:32.780184Z","iopub.status.idle":"2025-10-27T21:18:34.882674Z","shell.execute_reply":"2025-10-27T21:18:34.881880Z","shell.execute_reply.started":"2025-10-27T21:18:32.780694Z"},"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"# submission","metadata":{}},{"cell_type":"code","source":"submission = pd.DataFrame({\n    'id': id_test,\n    'target': test_pred\n})\n\nsubmission.to_csv('test_predictions_Final.csv', index=False)","metadata":{"execution":{"iopub.execute_input":"2025-10-27T21:18:43.387705Z","iopub.status.busy":"2025-10-27T21:18:43.387430Z","iopub.status.idle":"2025-10-27T21:18:43.678433Z","shell.execute_reply":"2025-10-27T21:18:43.677844Z","shell.execute_reply.started":"2025-10-27T21:18:43.387685Z"},"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"code","source":"# to see how the  looks like after processing\n'''\nX_train_transformed = model.named_steps['preprocessor'].transform(X_train)\n\nX_train_transformed_df = pd.DataFrame(\n    X_train_transformed.toarray(),  \n    columns=model.named_steps['preprocessor'].get_feature_names_out()\n)\n\n'''\n","metadata":{"execution":{"iopub.execute_input":"2025-10-27T21:19:28.132794Z","iopub.status.busy":"2025-10-27T21:19:28.132158Z","iopub.status.idle":"2025-10-27T21:19:28.137711Z","shell.execute_reply":"2025-10-27T21:19:28.136936Z","shell.execute_reply.started":"2025-10-27T21:19:28.132772Z"},"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"code","source":"#X_train_transformed_df.columns","metadata":{"execution":{"iopub.status.busy":"2025-10-27T20:02:10.018179Z","iopub.status.idle":"2025-10-27T20:02:10.018422Z","shell.execute_reply":"2025-10-27T20:02:10.018317Z","shell.execute_reply.started":"2025-10-27T20:02:10.018304Z"},"trusted":true},"outputs":[],"execution_count":null}]}