{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"name":"python","version":"3.12.12","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"},"kaggle":{"accelerator":"none","dataSources":[{"sourceType":"competition","sourceId":114135,"databundleVersionId":14604229,"isSourceIdPinned":false}],"dockerImageVersionId":31234,"isInternetEnabled":true,"language":"python","sourceType":"notebook","isGpuEnabled":false}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# Competition Overview","metadata":{}},{"cell_type":"markdown","source":"The \"Detecting Reversal Points in US Equities\" Kaggle competition (September 13, 2025 – December 31, 2025) centers on identifying local and global reversal points—swing highs and lows—within anonymized US stock market data. Participants must classify timestamps into three distinct categories: H (High reversal), L (Low reversal), or None. \n\nThe dataset encompasses six anonymized tickers with approximately 2,760 samples and a massive feature space consisting of 68,504 mostly boolean signal descriptors.\n\nThe primary evaluation metric is Macro F1-score (equal importance across all classes despite severe imbalance: approx. 94% None, approx. 3% H, approx. 3% L). Tie-breakers include Macro Balanced Accuracy, multi-class Matthews Correlation Coefficient (MCC), and inference runtime. Total prize pool: $4,000 ($2,000 – 1st, $1,000 – 2nd, $500 – 3rd). The challenge tests the ability to detect rare but highly actionable market turning points in high-dimensional, imbalanced financial data.","metadata":{}},{"cell_type":"markdown","source":"[**Competition Link**](https://www.kaggle.com/competitions/detecting-reversal-points-in-us-equities/leaderboard)","metadata":{}},{"cell_type":"markdown","source":"# Introduction","metadata":{}},{"cell_type":"markdown","source":"Detecting reversal points (local highs (H) and lows (L)) is one of the most valuable yet difficult signals in financial markets. These swing points indicate potential trend changes and are foundational for trend analysis, chart pattern recognition, and algorithmic trading strategies. The \"Detecting Reversal Points in US Equities\" competition provides anonymized stock time-series data from 6 instruments (2023–2025) and challenges participants to classify each timestamp into one of three classes: H (High reversal), L (Low reversal), or None (no reversal). With 68,504 features (mostly boolean Signal Descriptors) and extreme class imbalance (~94% None), the task is both computationally and methodologically demanding. The provided code implements a complete end-to-end solution: advanced preprocessing, imbalance handling (ADASYN + class weights), feature selection, PCA reduction, rigorous model benchmarking, Optuna hyperparameter tuning, and a final weighted soft-voting ensemble of top tree-based models (ExtraTrees, RandomForest, LightGBM, XGBoost).","metadata":{}},{"cell_type":"markdown","source":"# Objective","metadata":{}},{"cell_type":"markdown","source":"The primary objective of this notebook is to build a high-performing, production-grade classification pipeline that maximizes Macro F1-score on the hidden test set while respecting inference time constraints. Specifically, the project aims to:\n\n- Perform robust time-aware preprocessing and feature engineering on high-dimensional, imbalanced financial time-series data.\n- Effectively address the severe class imbalance (~94% None) using ADASYN oversampling with balanced class weights.\n- Identify the best base learners through extensive 5-fold stratified OOF validation across 10 classifiers using Macro F1, Balanced Accuracy, MCC, and inference time.\n- Optimize hyperparameters of the top 4 models (ExtraTrees, RandomForest, XGBoost, LightGBM) via Optuna.\n- Train a weighted soft-voting ensemble that combines the strengths of the best models to achieve top-tier generalization and leaderboard performance. This solution serves as a reference for handling real-world imbalanced time-series classification problems in finance.","metadata":{}},{"cell_type":"markdown","source":"# Data Dictionary","metadata":{}},{"cell_type":"markdown","source":"**Dataset Overview: Financial Time Series Classification**\n\n- Total Samples: 2,760 (Train: 1,932 / Test for submission: 828)\n- Time Range: 2023-04-03 to 2025-01-31\n- Classification Type: 3-Class (Multiclass)\n- Primary Challenge: Severe class imbalance (approx. 94% of 'None' class)","metadata":{}},{"cell_type":"markdown","source":"**Data Dictionary**\n\n**Administrative & Metadata Columns**\n\nThese columns define the structure of the time series and are used for indexing or grouping.\n\n\n\n \n \n \n \n \n \n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n
Column NameData TypeDescriptionNotes
idIntegerUnique identifier for each sample.Only present in test.csv and sample_submission.csv.
ticker_idCategoricalAnonymous identifier for the financial instrument (e.g., 001, 002).6 unique tickers. Each has 460 samples.
tDatetimeTimestamp of the observation (YYYY-MM-DD).Data is chronologically ordered within each ticker.
\n\n**Feature Columns**\n\n\n\n \n \n \n \n \n \n\n \n \n \n \n \n \n\n
Column NameData TypeDescriptionNotes
feature_0 to feature_68503NumericTechnical and fundamental financial indicators.68,504 total features. Price, returns, and PnL data have been removed/anonymized.
\n\n**Target Variable**\n\n\n\n \n \n \n \n \n \n\n \n \n \n \n \n \n\n
Column NameData TypeDescriptionClass Mapping
class_labelCategoricalThe target pattern to be predicted.H: Higher Highs/Lower HighsL: Higher Lows/Lower LowsNone: No clear pattern/Missing
\n\n**Class Distribution Detail**\n\nThe dataset is highly imbalanced, which is typical for financial \"event\" detection.\n\n\n\n \n \n \n \n \n \n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n
Class LabelDescriptionSample CountPercentage
NoneNo clear pattern / Noise2,59794.10%
HHigh patterns (HH, LH)823.00%
LLow patterns (HL, LL)812.90%
","metadata":{}},{"cell_type":"markdown","source":"[**Dataset Link**](https://www.kaggle.com/competitions/detecting-reversal-points-in-us-equities/data)","metadata":{}},{"cell_type":"markdown","source":"# Pipeline Overview","metadata":{}},{"cell_type":"markdown","source":"The analytical pipeline is structured into several discrete stages to ensure data integrity and model performance:\n\n- Environment Setup: Installation of essential frameworks including imbalanced-learn, Optuna, XGBoost, LightGBM, and CatBoost.\n- Data Loading and Feature Engineering: Reading CSV files, ensuring chronological sorting by ticker, and generating datetime features (month, year, weekend indicators) along with ticker-specific one-hot encodings.\n- Temporal Validation Splitting: Executing an 80/20 chronological split per ticker to maintain the temporal order and prevent look-ahead bias.\n- Preprocessing and Sampling: Applying ticker-specific imputation (forward/backward filling) and scaling, followed by ADASYN oversampling to balance the training set.\n- Dimensionality Reduction: Filtering the top 26,000 boolean features using Mutual Information and applying PCA on numeric indicators to retain 80% variance.\n- Model Selection and Hyperparameter Tuning: Evaluating 10 classifiers through 5-fold stratified out-of-fold (OOF) validation and optimizing the top four models (ExtraTrees, RandomForest, XGBoost, and LightGBM) using Optuna.\n- Ensemble Model and Final Submission: Training a weighted soft-voting ensemble and generating the final submission.csv.","metadata":{}},{"cell_type":"markdown","source":"# Approach","metadata":{}},{"cell_type":"markdown","source":"A comprehensive methodology is employed to address the complexities of financial time-series classification:\n\n- Temporal Integrity: Chronological sorting and time-aware splitting are strictly enforced to prevent data leakage, ensuring the model reflects real-world trading conditions.\n- Imbalance Handling: The Adaptive Synthetic (ADASYN) algorithm generates minority samples in regions where the reversal points are hardest to learn, while balanced class weights penalize errors on the H and L classes during the training phase.\n- Feature Optimization: Mutual Information identifies the most relevant boolean signals among the 68,000 available features, significantly reducing noise. PCA further compresses numeric data to lower computational costs.\n- Model Selection: Classifiers are ranked by Macro F1 to ensure the model does not ignore the minority reversal classes. Tie-breakers of competition, it includes the Matthews Correlation Coefficient (MCC) and inference speed, the latter of which is vital for high-frequency financial applications.\n- Ensemble Strategy: A soft-voting ensemble combines predictions using specific weights based on their individual validation performance to maximize predictive stability.","metadata":{}},{"cell_type":"markdown","source":"# Environment and Configuration","metadata":{}},{"cell_type":"markdown","source":"## Packages Installation","metadata":{}},{"cell_type":"markdown","source":"Environment & Package Installation: Essential specialized libraries are installed to handle the specific challenges of the dataset. This includes imblearn for class imbalance, optuna for Bayesian optimization, and multiple gradient boosting frameworks like xgboost, lightgbm, and catboost.","metadata":{}},{"cell_type":"code","source":"%%capture\n!pip install imblearn\n!pip install --upgrade imbalanced-learn\n!pip install optuna\n!pip install xgboost","metadata":{"execution":{"iopub.execute_input":"2026-01-10T11:30:18.435817Z","iopub.status.busy":"2026-01-10T11:30:18.435544Z","iopub.status.idle":"2026-01-10T11:30:38.211163Z","shell.execute_reply":"2026-01-10T11:30:38.209708Z","shell.execute_reply.started":"2026-01-10T11:30:18.435790Z"},"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## Import Libraries","metadata":{}},{"cell_type":"markdown","source":"The environment is initialized by importing core data manipulation tools, a wide array of classifiers (from traditional trees to advanced boosting), and evaluation metrics like Macro F1 and Matthews Correlation Coefficient (MCC).","metadata":{}},{"cell_type":"code","source":"# Core Libraries\nimport os\nimport time\nimport shutil\nimport warnings\nimport tempfile\nimport numpy as np\nimport pandas as pd\nimport joblib\nfrom joblib import dump, load\n\n# Scikit-Learn (Preprocessing, Selection, Metrics)\nfrom sklearn.preprocessing import LabelEncoder, StandardScaler\nfrom sklearn.model_selection import train_test_split, StratifiedKFold\nfrom sklearn.feature_selection import SelectKBest, mutual_info_classif\nfrom sklearn.decomposition import PCA\nfrom sklearn.utils.class_weight import compute_class_weight\nfrom sklearn.metrics import (f1_score, accuracy_score, balanced_accuracy_score, \n matthews_corrcoef)\n\n# Models & Ensembles\nfrom sklearn.linear_model import LogisticRegression\nfrom sklearn.neighbors import KNeighborsClassifier\nfrom sklearn.tree import DecisionTreeClassifier\nfrom sklearn.ensemble import (RandomForestClassifier, ExtraTreesClassifier, \n GradientBoostingClassifier, VotingClassifier, \n StackingClassifier)\nfrom sklearn.svm import SVC\nfrom sklearn.naive_bayes import GaussianNB\nfrom sklearn.ensemble import VotingClassifier, StackingClassifier, ExtraTreesClassifier, GradientBoostingClassifier\n\n# Gradient Boosting\nimport xgboost as xgb\nfrom xgboost import XGBClassifier\nimport lightgbm as lgb\nfrom lightgbm import LGBMClassifier\nfrom catboost import CatBoostClassifier\n\n# Imbalanced Learning & Optimization\nimport optuna\nimport sklearn\nimport imblearn\nfrom imblearn.over_sampling import SMOTE, ADASYN\n\n# Suppression & Versions\nwarnings.filterwarnings(\"ignore\")\nprint(f\"scikit-learn: {sklearn.__version__} | imbalanced-learn: {imblearn.__version__}\")","metadata":{"execution":{"iopub.execute_input":"2026-01-10T11:30:38.214375Z","iopub.status.busy":"2026-01-10T11:30:38.213971Z","iopub.status.idle":"2026-01-10T11:30:49.737961Z","shell.execute_reply":"2026-01-10T11:30:49.736854Z","shell.execute_reply.started":"2026-01-10T11:30:38.214331Z"},"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## Configs and Paths Setup","metadata":{}},{"cell_type":"markdown","source":"Project organization is established by defining data paths and creating dedicated directories for storing preprocessing artifacts and trained model objects.","metadata":{}},{"cell_type":"code","source":"# PATHS & SETUP\nTRAIN_CSV = \"/kaggle/input/detecting-reversal-points-in-us-equities/new_comptetition_data/train.csv\"\nTEST_CSV = \"/kaggle/input/detecting-reversal-points-in-us-equities/new_comptetition_data/test.csv\"\nPREPROC_DIR = \"preproc_models\"\nos.makedirs(PREPROC_DIR, exist_ok=True)","metadata":{"execution":{"iopub.execute_input":"2026-01-10T11:30:49.742637Z","iopub.status.busy":"2026-01-10T11:30:49.742358Z","iopub.status.idle":"2026-01-10T11:30:49.748191Z","shell.execute_reply":"2026-01-10T11:30:49.746870Z","shell.execute_reply.started":"2026-01-10T11:30:49.742607Z"},"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"# Data Loading and Initial Preprocessing","metadata":{}},{"cell_type":"markdown","source":"Raw datasets are ingested and sorted chronologically by ticker and timestamp. This step enriches the data with temporal features—such as day of the week and weekend flags—and converts categorical ticker identifiers into numeric one-hot encoded columns.","metadata":{}},{"cell_type":"code","source":"# Load and Prepare Data (Enhanced: check for duplicates, basic stats)\ntrain = pd.read_csv(TRAIN_CSV, low_memory=False)\ntest = pd.read_csv(TEST_CSV, low_memory=False)\n\n# Drop unexpected columns\nif 'train_id' in train.columns:\n train = train.drop(columns=['train_id'])\n\n# Sort by ticker and time\ntrain = train.sort_values(['ticker_id', 't']).reset_index(drop=True)\ntest = test.sort_values(['ticker_id', 't']).reset_index(drop=True)\n\n# Convert timestamp\ntrain['t'] = pd.to_datetime(train['t'])\ntest['t'] = pd.to_datetime(test['t'])\n\n# Extract more datetime features (day, year, is_weekend)\ntrain['day_of_week'] = train['t'].dt.dayofweek\ntrain['month'] = train['t'].dt.month\ntrain['year'] = train['t'].dt.year\ntrain['is_weekend'] = train['day_of_week'].isin([5, 6]).astype(int)\ntest['day_of_week'] = test['t'].dt.dayofweek\ntest['month'] = test['t'].dt.month\ntest['year'] = test['t'].dt.year\ntest['is_weekend'] = test['day_of_week'].isin([5, 6]).astype(int)\n\n# One-hot encode ticker_id\ntrain = pd.concat([train, pd.get_dummies(train['ticker_id'], prefix='ticker')], axis=1)\ntest = pd.concat([test, pd.get_dummies(test['ticker_id'], prefix='ticker')], axis=1)\n\n# Align dummy columns\nfor col in train.columns:\n if col.startswith('ticker_') and col not in test.columns:\n test[col] = 0\nfor col in test.columns:\n if col.startswith('ticker_') and col not in train.columns:\n train[col] = 0\n\n# Save test ids\ntest_ids = test['id'].copy()\n\n# Metadata columns\nmeta_cols = ['id', 't', 'class_label', 'day_of_week', 'month', 'year', 'is_weekend']\nmeta_cols = [col for col in meta_cols if col in train.columns]\nfeatures = [col for col in train.columns if col not in meta_cols]","metadata":{"execution":{"iopub.execute_input":"2026-01-10T11:30:49.750081Z","iopub.status.busy":"2026-01-10T11:30:49.749704Z","iopub.status.idle":"2026-01-10T11:31:53.702761Z","shell.execute_reply":"2026-01-10T11:31:53.701517Z","shell.execute_reply.started":"2026-01-10T11:30:49.750026Z"},"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"# Train and Test Dataset Splitting","metadata":{}},{"cell_type":"markdown","source":"To simulate real-world trading and prevent data leakage, an 80/20 chronological split is applied to each ticker. The most recent 20% of data for every instrument is reserved strictly for validation.","metadata":{}},{"cell_type":"code","source":"# Time-aware Validation Split\ntrain['is_val'] = False\nfor ticker in train['ticker_id'].unique():\n ticker_mask = train['ticker_id'] == ticker\n idx = train[ticker_mask].index\n split_point = int(len(idx) * 0.8)\n if split_point < len(idx):\n train.loc[idx[split_point:], 'is_val'] = True\n\ntrain_set = train[~train['is_val']].copy()\nval_set = train[train['is_val']].copy()\n\nprint(f\"Train samples: {len(train_set)}, Val samples: {len(val_set)}\")","metadata":{"execution":{"iopub.execute_input":"2026-01-10T11:31:53.704455Z","iopub.status.busy":"2026-01-10T11:31:53.704063Z","iopub.status.idle":"2026-01-10T11:31:54.785715Z","shell.execute_reply":"2026-01-10T11:31:54.784568Z","shell.execute_reply.started":"2026-01-10T11:31:53.704416Z"},"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"# Data Preprocessing and Feature Engineering","metadata":{}},{"cell_type":"markdown","source":"## Target Encoding and Feature Preparation","metadata":{}},{"cell_type":"markdown","source":"The target labels (`H, L, None`) are converted into numeric format using `LabelEncoder`. Features are isolated from metadata, and test IDs are preserved to ensure the final submission aligns with competition requirements.","metadata":{}},{"cell_type":"code","source":"# Target Encoding\ny_train = train_set['class_label']\ny_val = val_set['class_label']\n\nprint(\"Class distribution in train:\\n\", y_train.value_counts())\n\nle = LabelEncoder()\ny_train_encoded = le.fit_transform(y_train)\ny_val_encoded = le.transform(y_val)\n\njoblib.dump(le, f\"{PREPROC_DIR}/label_encoder.pkl\")\n\n# Extract ticker for grouping\nticker_train = train_set['ticker_id']\nticker_val = val_set['ticker_id']\nticker_test = test['ticker_id']\n\n# Features without ticker_id\nfeatures_no_ticker = [f for f in features if f != 'ticker_id']\nX_train = train_set[features_no_ticker].copy()\nX_val = val_set[features_no_ticker].copy()\nX_test = test[features_no_ticker].copy()","metadata":{"execution":{"iopub.execute_input":"2026-01-10T11:31:54.789515Z","iopub.status.busy":"2026-01-10T11:31:54.789008Z","iopub.status.idle":"2026-01-10T11:31:55.229224Z","shell.execute_reply":"2026-01-10T11:31:55.228135Z","shell.execute_reply.started":"2026-01-10T11:31:54.789485Z"},"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## Missing Value Imputation and Data Scaling","metadata":{}},{"cell_type":"markdown","source":"Missing values are addressed using a ticker-grouped strategy (forward/backward fill followed by median fallback). Data is then normalized through ticker-specific `StandardScaler` to account for differing market volatilities.","metadata":{}},{"cell_type":"markdown","source":"**StandardScaler**\n\nStandardScaler is a fundamental preprocessing technique in machine learning used to standardize features by removing the mean and scaling to unit variance. It is part of a broader category of data preparation known as Feature Scaling.\n\nThe primary goal is to ensure that features with different magnitudes do not disproportionately influence a model. For instance, in financial data, a \"Volume\" feature (ranging in millions) would naturally overwhelm a \"Daily Return\" feature (ranging from -0.05 to 0.05) if left unscaled.\n\n**Underlying Theories and Assumptions of StandardScaler**\n\n**1. The Normal Distribution (Gaussian) Assumption**\n\nStandardScaler is most effective when the underlying data follows a Gaussian (Normal) Distribution. While it can be applied to non-normal data, the resulting mean of 0 and standard deviation of 1 may not represent the \"center\" as effectively if the data is heavily skewed or contains extreme outliers.\n\n**2. Sensitivity to Outliers**\n\nUnlike RobustScaler (which uses the median and interquartile range), StandardScaler is highly sensitive to outliers. Because the mean and standard deviation are calculated using all data points, a single extreme value can significantly inflate $\\sigma$, \"squashing\" the remaining data points into a very small range of Z-scores.\n\n**3. Distance-Based Algorithms**\n\nStandardization is theoretically required for algorithms that calculate distances between data points. Without scaling, the \"distance\" on the axis with the largest numbers would dominate the calculation.\n- K-Nearest Neighbors (KNN): Distance between neighbors.\n- Principal Component Analysis (PCA): Capturing maximum variance (features with higher scales would appear to have higher variance).\n- Support Vector Machines (SVM): Finding the optimal margin between classes.\n\n**4. Gradient Descent Convergence**\n\nIn models like Logistic Regression or Neural Networks, features with vastly different scales create an elongated, \"elliptical\" loss function. This forces the gradient descent algorithm to oscillate or take a very long path to the minimum. Standardization makes the loss function more spherical, allowing for faster and more stable convergence.\n\n**Mathematical Theory**\n\nStandardScaler follows the formula for calculating a Z-score. For every data point $x$ in a feature, the transformed value $z$ is calculated as:\n$$z = \\frac{x - \\mu}{\\sigma}$$\n\nWhere:\n- $\\mu$: The mean of the feature samples.\n- $\\sigma$: The standard deviation of the feature samples.\n\n**The Transformation Result**\n\nAfter applying StandardScaler:\n- Mean ($\\mu$) = 0: The distribution is \"centered\" around zero.\n- Standard Deviation ($\\sigma$) = 1: The data is \"spread\" such that the majority of values fall within the range of -1 to 1 (for normally distributed data).\n\n**Implementation: Fit vs. Transform**\n\nIn a competition or production pipeline, it is critical to adhere to the theory of Data Leakage prevention:\n- `.fit()`: Calculated only on the training set. This computes the $\\mu$ and $\\sigma$ of the training data.\n- `.transform()`: Applied to both the training and test sets using the $\\mu$ and $\\sigma$ derived from the training set.Using the mean or standard deviation of the test set to scale the test data is a theoretical error, as it \"leaks\" information from the future/unseen data into the model's environment.","metadata":{}},{"cell_type":"code","source":"# Imputation & Scaling \nfor X, ticker_series in [(X_train, ticker_train), (X_val, ticker_val), (X_test, ticker_test)]:\n grouped = X.groupby(ticker_series)\n X[:] = grouped.transform(lambda x: x.fillna(x.median() if x.dtype.kind in 'biufc' else (x.mode()[0] if not x.mode().empty else 0)))\n X[:] = grouped.ffill().bfill().fillna(0)\n\n# Ticker-specific scaling + global scaling for comparison\nscaler = StandardScaler()\nnumeric_cols = X_train.select_dtypes(include=[np.number]).columns\n\nfor ticker in train['ticker_id'].unique():\n mask_tr = ticker_train == ticker\n mask_val = ticker_val == ticker\n mask_te = ticker_test == ticker\n if mask_tr.any():\n X_train.loc[mask_tr, numeric_cols] = scaler.fit_transform(X_train.loc[mask_tr, numeric_cols])\n if mask_val.any():\n X_val.loc[mask_val, numeric_cols] = scaler.transform(X_val.loc[mask_val, numeric_cols])\n if mask_te.any():\n X_test.loc[mask_te, numeric_cols] = scaler.transform(X_test.loc[mask_te, numeric_cols])\n\n# Global scaling as additional technique\nglobal_scaler = StandardScaler()\nX_train[numeric_cols] = global_scaler.fit_transform(X_train[numeric_cols])\nX_val[numeric_cols] = global_scaler.transform(X_val[numeric_cols])\nX_test[numeric_cols] = global_scaler.transform(X_test[numeric_cols])","metadata":{"execution":{"iopub.execute_input":"2026-01-10T11:31:55.230783Z","iopub.status.busy":"2026-01-10T11:31:55.230469Z","iopub.status.idle":"2026-01-10T11:35:47.929305Z","shell.execute_reply":"2026-01-10T11:35:47.928167Z","shell.execute_reply.started":"2026-01-10T11:31:55.230755Z"},"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## Features Selection","metadata":{}},{"cell_type":"markdown","source":"Given the high dimensionality, boolean features are isolated. `SelectKBest` with mutual information is used to retain only the top 26,000 most informative signals, significantly reducing computational load while preserving predictive power.","metadata":{}},{"cell_type":"markdown","source":"### Mutual Information (MI)","metadata":{}},{"cell_type":"markdown","source":"Mutual Information is a measure from information theory that quantifies the amount of information obtained about one random variable through the observation of another. In the context of your equities competition, it is used to measure how much knowing the state of a \"Signal Descriptor\" reduces the uncertainty regarding whether a timestamp is a reversal point (H or L).Unlike correlation, which only measures linear relationships, Mutual Information captures any kind of statistical dependence, including non-linear, periodic, or irregular patterns.\n\n**Mathematical Foundation**\n\nThe theory of MI is built upon the concept of Shannon Entropy ($H$), which represents the average amount of \"uncertainty\" in a variable.\n\n**The Entropy Formula**\n\nFor a discrete variable $X$, entropy is defined as:\n\n$$H(X) = -\\sum_{x \\in \\mathcal{X}} p(x) \\log p(x)$$\n\nThe Mutual Information Formula\n\nThe MI between two variables $X$ and $Y$ is the difference between the initial uncertainty and the remaining uncertainty after the other variable is known:\n\n$$I(X; Y) = H(X) - H(X|Y)$$\n\nEquivalently, it measures the \"distance\" between the joint probability distribution and the assumption of independence:\n\n$$I(X; Y) = \\sum_{x,y} p(x,y) \\log \\left( \\frac{p(x,y)}{p(x)p(y)} \\right)$$\n\n**Concepts**\n\n**Reduction of Uncertainty**\n\nMI is often called Information Gain. If $I(X; Y) = 0$, the variables are strictly independent; knowing $X$ tells you nothing about $Y$. A high MI indicates that $X$ and $Y$ share significant information, meaning $X$ is a strong predictor for $Y$.\n\n**Model Agnosticism**\n\nBecause MI relies on probability distributions rather than functional forms, it is \"model-agnostic.\" It can identify a relationship even if that relationship is a complex U-shape or a step function that a linear correlation coefficient (like Pearson's $r$) would miss entirely.\n\n**Symmetry**\n\nOne of the core properties of MI is that it is symmetric:$$I(X; Y) = I(Y; X)$$This means the information that a feature provides about the target is exactly equal to the information the target provides about the feature.\n\n**The \"Information Bottleneck\" Principle**\n\nIn deep learning and complex pipelines, MI is used to understand how much information from the input is preserved through different layers. In your pipeline, the SelectKBest step uses MI to ensure that only features with high \"shared information\" with the reversal labels are passed to the classifiers, effectively acting as a filter to remove noise.\n\n**Application in High-Dimensional Data**\n\nIn your competition, with 68,504 features, calculating MI for every feature is a \"filter method\" of feature selection. It is computationally efficient because it evaluates features individually (univariate) before training expensive models like XGBoost. This ensures the models focus on the subset of data that actually contains predictive \"signals\" rather than \"noise.\"","metadata":{}},{"cell_type":"code","source":"# Feature Selection\nbool_features = [f for f in X_train.columns if X_train[f].nunique() <= 2]\nnon_bool_features = [f for f in X_train.columns if f not in bool_features]\n\nprint(f\"Boolean features: {len(bool_features)}, Non-boolean: {len(non_bool_features)}\")\n\nk_bool = 26000 # Can increased for better coverage \nselector = SelectKBest(mutual_info_classif, k=min(k_bool, len(bool_features)))\nselector.fit(X_train[bool_features], y_train_encoded)\nselected_bool = np.array(bool_features)[selector.get_support()]\n\njoblib.dump(selector, f\"{PREPROC_DIR}/bool_selector.pkl\")\n\nfinal_features = non_bool_features + list(selected_bool)\nprint(f\"Final selected features: {len(final_features)}\")\njoblib.dump(final_features, f\"{PREPROC_DIR}/final_features.pkl\")\n\nX_train = X_train[final_features]\nX_val = X_val[final_features]\nX_test = X_test[final_features]","metadata":{"execution":{"iopub.execute_input":"2026-01-10T11:35:47.930762Z","iopub.status.busy":"2026-01-10T11:35:47.930430Z","iopub.status.idle":"2026-01-10T11:48:30.480522Z","shell.execute_reply":"2026-01-10T11:48:30.478949Z","shell.execute_reply.started":"2026-01-10T11:35:47.930735Z"},"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## Oversampling (ADASYN) and Dimensionality Reduction (PCA)","metadata":{}},{"cell_type":"markdown","source":"**Class Weights & ADASYN Oversampling**: To combat the 94% \"None\" class dominance, the ADASYN (Adaptive Synthetic) algorithm generates synthetic samples for the minority \"H\" and \"L\" classes. Balanced class weights are also computed to further penalize misclassifications of rare events during training.\n\n**PCA Dimensionality Reduction**: For numeric features, Principal Component Analysis (PCA) is applied to the resampled data. By retaining 80% of the variance (condensed into two components), noise is filtered out while the core signal remains.","metadata":{}},{"cell_type":"markdown","source":"**Adaptive Synthetic (ADASYN) Sampling**\n\nADASYN is an advanced oversampling technique designed to address class imbalance. It is an evolution of the SMOTE (Synthetic Minority Over-sampling Technique) algorithm, specifically focused on the \"hard-to-learn\" examples.\n\n**Concepts**\n\n- Density Distribution: Unlike standard oversampling, which generates samples uniformly, ADASYN uses a weighted distribution for different minority class examples. It calculates the \"density\" of majority class neighbors for each minority point.\n- Adaptive Learning: The primary theory behind ADASYN is that the model should generate more synthetic data in regions where the minority class is most overwhelmed by the majority class. If a minority point is surrounded by many majority points, it is considered \"hard to learn,\" and ADASYN assigns it a higher weight for sample generation.\n\n**Mathematical Mechanism**\n\nThe number of synthetic samples to be generated for a specific minority point $x_i$ is determined by:\n- Finding Neighbors: For each minority sample, find its $K$ nearest neighbors.\n- Calculating the Ratio ($r_i$): Determine how many of those $K$ neighbors belong to the majority class.\n$$r_i = \\frac{\\Delta_i}{K}$$\nwhere $\\Delta_i$ is the count of majority neighbors.\n- Density Distribution ($\\hat{r}_i$): Normalize these ratios so they sum to 1.\n- Generation: The number of synthetic points generated around $x_i$ is $g_i = \\hat{r}_i \\times G$, where $G$ is the total number of samples needed to reach balance.\n\n**Key Advantage**\n\nThe theoretical benefit of ADASYN is that it shifts the decision boundary of the classifier toward the difficult-to-learn samples. By focusing on high-density majority areas, it helps the model distinguish subtle patterns in reversal points (H and L) that might otherwise be treated as noise (None).","metadata":{}},{"cell_type":"markdown","source":"**Dimensionality Reduction (PCA)**\n\nPrincipal Component Analysis (PCA) is an unsupervised linear transformation technique used for feature extraction and dimensionality reduction.\n\n**Concepts**\n\n- Variance Maximization: PCA operates on the theory that the most important \"information\" in a dataset is contained in the features (or combinations of features) that show the highest variance.\n- Orthogonality: PCA transforms the original correlated features into a new set of uncorrelated variables called Principal Components (PCs). These components are orthogonal (at 90-degree angles) to each other, meaning they represent completely independent pieces of information.\n- Information Compression: By keeping only the first few PCs that explain the majority of the variance (e.g., 80%), one can reduce the size of the data while losing very little predictive power.\n\n**Mathematical Mechanism**\n\nThe transformation involves several linear algebra steps:\n\n- Standardization: The data must be centered (mean = 0) to ensure the first PC captures the direction of maximum variance.\n- Covariance Matrix: A matrix is constructed to show how all features vary together.\n- Eigen-Decomposition: The algorithm calculates Eigenvectors (the directions of the new axes) and Eigenvalues (the magnitude/importance of those directions).\n- Feature Projection: The original data is projected onto these new axes (PCs).\n\n**PCA in this Pipeline**\n\nIn this pipeline, PCA is applied to numeric features to retain 80% variance, which is condensed into just two components.\n\n- Noise Reduction: By discarding components with low eigenvalues, the model ignores small fluctuations in financial data that are likely random noise.\n- Multicollinearity: PCA removes the correlation between numeric features, which helps stabilize models like Logistic Regression or Naive Bayes that assume feature independence.","metadata":{}},{"cell_type":"code","source":"# Class Weights & ADASYN Oversampling\nprint(\"Original training class distribution:\")\nprint(pd.Series(y_train_encoded).value_counts().sort_index())\n\nclass_weights = compute_class_weight('balanced', classes=np.unique(y_train_encoded), y=y_train_encoded)\nclass_weight_dict_full = dict(enumerate(class_weights))\nprint(\"\\nComputed class weights (3-class):\", class_weight_dict_full)\n\nprint(\"\\nApplying ADASYN oversampling...\")\nadasyn = ADASYN(sampling_strategy='auto', random_state=42, n_neighbors=5) \nX_train_res, y_train_res = adasyn.fit_resample(X_train, y_train_encoded)\n\nX_train_res = pd.DataFrame(X_train_res, columns=X_train.columns)\ny_train_res = pd.Series(y_train_res)\n\nprint(f\"After ADASYN: {len(X_train_res)} samples\")\nprint(pd.Series(y_train_res).value_counts().sort_index())\n\n# Identify numeric columns from original X_train (before oversampling)\npca_numeric_cols = X_train.select_dtypes(include=[np.number]).columns.tolist()\n\n# Use 80% variance (as in your code)\npca = PCA(n_components=0.8, svd_solver='full', random_state=42)\n\n# Fit PCA on oversampled training data (X_train_res)\nX_train_res_pca = pd.DataFrame(\n pca.fit_transform(X_train_res[pca_numeric_cols]),\n index=X_train_res.index,\n columns=[f'pca_{i}' for i in range(pca.n_components_)]\n)\n\n# Transform validation and test sets using the same PCA\nX_val_pca = pd.DataFrame(\n pca.transform(X_val[pca_numeric_cols]),\n index=X_val.index,\n columns=[f'pca_{i}' for i in range(pca.n_components_)]\n)\n\nX_test_pca = pd.DataFrame(\n pca.transform(X_test[pca_numeric_cols]),\n index=X_test.index,\n columns=[f'pca_{i}' for i in range(pca.n_components_)]\n)\n\nprint(f\"PCA reduced {len(pca_numeric_cols)} numeric features to {pca.n_components_} components (90% variance)\")\n\n# Replace numeric columns with PCA components\nX_train_res = pd.concat([X_train_res.drop(columns=pca_numeric_cols), X_train_res_pca], axis=1)\nX_val = pd.concat([X_val.drop(columns=pca_numeric_cols), X_val_pca], axis=1)\nX_test = pd.concat([X_test.drop(columns=pca_numeric_cols), X_test_pca], axis=1)\n\n# Standardize all column names to strings\nX_train_res.columns = X_train_res.columns.astype(str)\nX_val.columns = X_val.columns.astype(str)\nX_test.columns = X_test.columns.astype(str)\n\nprint(\"\\nPCA applied after oversampling — Finish...\")\n\nclass_weights = compute_class_weight('balanced', classes=np.unique(y_train_res), y=y_train_res)\nclass_weight_dict_full = dict(enumerate(class_weights))\nprint(\"\\nComputed class weights (3-class):\", class_weight_dict_full)","metadata":{"execution":{"iopub.execute_input":"2026-01-10T11:48:30.483460Z","iopub.status.busy":"2026-01-10T11:48:30.482945Z","iopub.status.idle":"2026-01-10T11:48:43.065685Z","shell.execute_reply":"2026-01-10T11:48:43.064702Z","shell.execute_reply.started":"2026-01-10T11:48:30.483424Z"},"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"# Model Training","metadata":{}},{"cell_type":"markdown","source":"## Weak Learner Model Selection","metadata":{}},{"cell_type":"markdown","source":"Weak Learner Model Selection (5-fold Stratified OOF): Conducts a comprehensive evaluation of 10 diverse classifiers. Using Out-of-Fold (OOF) validation, models are ranked by their ability to achieve a high Macro F1-score and low inference latency.","metadata":{}},{"cell_type":"code","source":"# Extended model list\nml_models = [\n ('Logistic Regression', LogisticRegression(max_iter=1000, random_state=42)),\n ('KNN', KNeighborsClassifier()),\n ('Decision Tree', DecisionTreeClassifier(random_state=42)),\n ('Random Forest', RandomForestClassifier(random_state=42)),\n ('Extra Trees', ExtraTreesClassifier(random_state=42)),\n ('Gradient Boosting', GradientBoostingClassifier(random_state=42)),\n ('XGBoost', xgb.XGBClassifier(use_label_encoder=False, eval_metric='mlogloss', random_state=42)),\n ('LightGBM', lgb.LGBMClassifier(verbose=-1, random_state=42)),\n ('CatBoost', CatBoostClassifier(verbose=False, random_state=42)),\n ('Naive Bayes', GaussianNB())\n]\n\n# OOF setup\nn_folds = 5 # Can increase for better result\nskf = StratifiedKFold(n_splits=n_folds, shuffle=True, random_state=42)\nmodel_results = {}\ntest_oof_all = {} # Store test predictions from each model\n\nprint(\"\\nStarting model selection with 5-fold Stratified OOF (Macro F1, Balanced Accuracy, MCC, Inference Time)...\\n\")\n\nfor name, model in ml_models:\n print(f\"Evaluating {name}...\")\n \n # OOF arrays\n oof_proba = np.zeros((len(X_train_res), 3))\n oof_true = np.zeros(len(X_train_res), dtype=int)\n fold_f1_scores = []\n fold_bal_acc_scores = []\n fold_mcc_scores = []\n fold_inference_times = []\n \n # Test predictions accumulation\n test_proba = np.zeros((len(X_test), 3))\n \n for fold, (tr_idx, val_idx) in enumerate(skf.split(X_train_res, y_train_res)):\n X_tr = X_train_res.iloc[tr_idx]\n y_tr = y_train_res.iloc[tr_idx]\n X_va = X_train_res.iloc[val_idx]\n y_va = y_train_res.iloc[val_idx]\n \n # Fit model\n model.fit(X_tr, y_tr)\n \n # Validation prediction with inference time\n start_time = time.time()\n val_proba = model.predict_proba(X_va)\n inference_time = time.time() - start_time\n val_pred = np.argmax(val_proba, axis=1)\n \n # Metrics\n fold_f1 = f1_score(y_va, val_pred, average='macro')\n fold_bal_acc = balanced_accuracy_score(y_va, val_pred)\n fold_mcc = matthews_corrcoef(y_va, val_pred)\n \n fold_f1_scores.append(fold_f1)\n fold_bal_acc_scores.append(fold_bal_acc)\n fold_mcc_scores.append(fold_mcc)\n fold_inference_times.append(inference_time)\n \n # Store OOF\n oof_proba[val_idx] = val_proba\n oof_true[val_idx] = y_va\n \n # Accumulate test prediction\n test_proba += model.predict_proba(X_test) / n_folds\n \n # Overall OOF performance\n oof_pred = np.argmax(oof_proba, axis=1)\n overall_f1 = f1_score(oof_true, oof_pred, average='macro')\n overall_bal_acc = balanced_accuracy_score(oof_true, oof_pred)\n overall_mcc = matthews_corrcoef(oof_true, oof_pred)\n \n model_results[name] = {\n 'mean_f1': np.mean(fold_f1_scores),\n 'std_f1': np.std(fold_f1_scores),\n 'overall_f1': overall_f1,\n 'mean_bal_acc': np.mean(fold_bal_acc_scores),\n 'std_bal_acc': np.std(fold_bal_acc_scores),\n 'overall_bal_acc': overall_bal_acc,\n 'mean_mcc': np.mean(fold_mcc_scores),\n 'std_mcc': np.std(fold_mcc_scores),\n 'overall_mcc': overall_mcc,\n 'mean_inference_time': np.mean(fold_inference_times),\n 'std_inference_time': np.std(fold_inference_times),\n 'oof_proba': oof_proba\n }\n test_oof_all[name] = test_proba\n \n print(f\"{name}: Mean Macro F1 = {np.mean(fold_f1_scores):.5f} ± {np.std(fold_f1_scores):.4f} | OOF F1 = {overall_f1:.5f}\")\n print(f\" Mean Balanced Acc = {np.mean(fold_bal_acc_scores):.5f} ± {np.std(fold_bal_acc_scores):.4f} | OOF Bal Acc = {overall_bal_acc:.5f}\")\n print(f\" Mean MCC = {np.mean(fold_mcc_scores):.5f} ± {np.std(fold_mcc_scores):.4f} | OOF MCC = {overall_mcc:.5f}\")\n print(f\" Mean Inference Time = {np.mean(fold_inference_times):.5f}s ± {np.std(fold_inference_times):.4f}s\")\n\n\n# Model Ranking & Selection\nprint(\"\\n\" + \"=\"*90)\nprint(\"MODEL SELECTION RESULTS (Primary: Macro F1, Tie-breakers: Bal Acc, MCC, Inference Time)\")\nprint(\"=\"*90)\nranking = sorted(model_results.items(), key=lambda x: (\n x[1]['overall_f1'], # Primary metric\n x[1]['overall_bal_acc'], # Tie-breaker 1\n x[1]['overall_mcc'], # Tie-breaker 2\n -x[1]['mean_inference_time'] # Tie-breaker 3 (lower is better)\n), reverse=True)\n\nfor i, (name, res) in enumerate(ranking):\n print(f\"{i+1:2d}. {name:<25} | OOF Macro F1: {res['overall_f1']:.5f} | CV Mean F1: {res['mean_f1']:.5f} ± {res['std_f1']:.4f}\")\n print(f\" OOF Bal Acc: {res['overall_bal_acc']:.5f} | CV Mean Bal Acc: {res['mean_bal_acc']:.5f} ± {res['std_bal_acc']:.4f}\")\n print(f\" OOF MCC: {res['overall_mcc']:.5f} | CV Mean MCC: {res['mean_mcc']:.5f} ± {res['std_mcc']:.4f}\")\n print(f\" Mean Inference Time: {res['mean_inference_time']:.5f}s ± {res['std_inference_time']:.4f}s\")\n print(\"-\"*90)\n\nprint(\"=\"*90)\n\n# Select top 5 models for final ensemble\ntop_k = 5\nselected_models = ranking[:top_k]\nprint(f\"\\nSelected top {top_k} models for final ensemble:\")\nfor i, (name, _) in enumerate(selected_models):\n print(f\" {i+1}. {name}\")","metadata":{"execution":{"iopub.execute_input":"2026-01-09T14:07:03.625162Z","iopub.status.busy":"2026-01-09T14:07:03.623987Z","iopub.status.idle":"2026-01-09T14:59:24.659330Z","shell.execute_reply":"2026-01-09T14:59:24.658053Z","shell.execute_reply.started":"2026-01-09T14:07:03.625111Z"},"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"According to the result, Extra Trees, XGBoost, LightGBM, and Random Forest ware selected as weak learner model, but CatBoost was not slected because it will take long time for training and hyperparameters tuning.","metadata":{}},{"cell_type":"markdown","source":"**Random Forest (RF)**\n\nRandom Forest is an ensemble learning method based on Bagging (Bootstrap Aggregating). It builds multiple decision trees independently using random subsets of the training data (drawn with replacement). To increase diversity, it also selects a random subset of features at each split. The final prediction is the majority vote (classification) or average (regression) of all trees.\n\n**Important Hyperparameters:**\n\n- `n_estimators`: Number of trees in the forest. More trees generally improve stability but increase training time.\n- `max_depth`: The maximum depth of each tree. Controlling this helps prevent trees from becoming too complex and overfitting.\n- `max_features`: The number of random features to consider at each split (e.g., sqrt or log2).\n- `min_samples_leaf`: The minimum number of samples required to be at a leaf node; higher values smooth the model.\n\n**Extra Trees (Extremely Randomized Trees)**\n\nSimilar to Random Forest, but introduces a higher level of randomness. While RF searches for the optimum split threshold for each feature, Extra Trees chooses a random threshold. Additionally, it typically uses the entire dataset rather than bootstrap samples. This significantly reduces training time and often reduces variance further than RF.\n\n**Important Hyperparameters:**\n\n- `n_estimators`: Total number of randomized trees.\n- `max_features`: Number of features to randomly sample for each split.\n- `bootstrap`: Boolean (default False). Whether to use the whole dataset or samples with replacement.\n- `min_samples_split`: The minimum number of samples required to split an internal node.\n\n**XGBoost (eXtreme Gradient Boosting)**\n\nA powerful implementation of Gradient Boosting that builds trees sequentially. Each new tree attempts to correct the errors (residuals) of the previous ones using gradient descent. It uses Level-Wise Growth (splitting level by level) and incorporates L1/L2 regularization directly into the objective function to handle overfitting.\n\n**Important Hyperparameters:**\n\n- `learning_rate (eta)`: Scales the contribution of each tree. Lower values (e.g., 0.01) require more trees but lead to better generalization.\n- `max_depth`: Limits the complexity of individual trees.\n- `gamma`: The minimum loss reduction required to make a further split; higher values make the model more conservative.\n- `lambda (L2) and alpha (L1)`: Regularization terms on weights to penalize complexity.\n\n**LightGBM (Light Gradient Boosting Machine)**\n\nA high-speed boosting framework developed by Microsoft. Its primary differentiator is Leaf-Wise Growth, where it splits the node that results in the greatest loss reduction, regardless of depth. It also uses Histogram-based binning and GOSS (Gradient-based One-Side Sampling) to handle large datasets with significantly less memory and time.\n\n**Important Hyperparameters:**\n\n- `num_leaves`: The most important parameter for LightGBM; it controls tree complexity (should be less than $2^{\\text{max\\_depth}}$).\n- `learning_rate`: Similar to XGBoost, dictates the step size of the optimization.\n- `min_data_in_leaf`: Prevents overfitting by ensuring each leaf has enough supporting samples.\n- `feature_fraction`: Randomly selects a subset of features on each iteration (similar to colsample_bytree).","metadata":{}},{"cell_type":"markdown","source":"## Ensemble Model Selection","metadata":{}},{"cell_type":"markdown","source":"A Voting Ensemble is a meta-modeling approach that combines the predictions from multiple independent machine learning models to improve overall performance. The core theory is based on the Condorcet Jury Theorem, which suggests that a group of independent \"voters\" (models) is more likely to arrive at the correct decision than any single individual voter, provided each voter performs better than random chance.\n\n**Concepts**\n\n**Wisdom of the Crowd**\n\nThe primary motivation for using a Voting Ensemble is to reduce variance and bias. Individual models may make errors on specific data points due to their unique biases or sensitivities to noise. By aggregating their outputs, the ensemble \"smooths out\" individual errors. If the errors made by the models are uncorrelated, the ensemble will significantly outperform its components.\n\n**Diversity is Key**\n\nFor a Voting Ensemble to be effective, the base models must be diverse. If you combine five identical models, the ensemble will simply repeat the same errors. Diversity is typically achieved by:\n- Using different algorithms (e.g., combining a Tree-based model with a Linear model).\n- Training on different subsets of data.\n- Using different feature engineering approaches.\n\n**Types of Voting Mechanisms**\n\n**Hard Voting (Majority Class Voting)**\n\nIn Hard Voting, the ensemble predicts the class that received the most \"votes\" from the individual models.\n- Concept: It follows a simple democratic process.\n- Calculation: If Model A predicts \"H,\" Model B predicts \"L,\" and Model C predicts \"H,\" the ensemble output is \"H.\"\n- Use Case: Best used when models do not output well-calibrated probabilities.\n\n**Soft Voting (Average Probability Voting)**\n\nIn Soft Voting, the ensemble calculates the average predicted probability (class membership) across all models for each class.\n- Concept: It gives more weight to models that are highly \"confident\" in their prediction.\n- Calculation: If Model A predicts an 80% chance of \"H\" and Model B predicts a 40% chance of \"H,\" the average probability is 60%.\n- Use Case: Generally superior to Hard Voting because it utilizes more information (confidence levels) rather than just the final label.\n\n**Weighted Soft Voting**\n\nWeighted voting refines the process further by assigning a weight ($w$) to each model based on its individual performance (e.g., its Macro F1-score on a validation set).\n- Formula:$$P(\\text{class}) = \\frac{\\sum (w_i \\times p_i)}{\\sum w_i}$$\n\nIn the provided equities pipeline, ExtraTrees is given a higher weight (32%) than XGBoost (16%) because it demonstrated higher reliability during the benchmarking phase.\n\n**Why Ensembling Works?**\n\nThe Statistical PerspectiveEnsembles work because they navigate the Bias-Variance Tradeoff more effectively:\n- Averaging Out Errors: In high-dimensional financial data, one model might overfit to a specific noise pattern in Ticker A. Another model might overfit to a pattern in Ticker B. The ensemble averages these out, leaving only the \"consensus\" signal.\n- Increased Stability: Financial markets are non-stationary (patterns change over time). An ensemble of diverse models is often more robust to these shifts than a single \"tuned\" model.","metadata":{}},{"cell_type":"markdown","source":"**Ensemble Model Selection Implementation**\n\nTwo strategies are compared: simple soft voting and a weighted soft-voting approach. The weighted ensemble of the top four models is selected for its superior Macro F1 performance.","metadata":{}},{"cell_type":"code","source":"# Define ensemble models using the top 4 weak learners\nensemble_models = [\n ('Voting Top 4 (Soft)', VotingClassifier(\n estimators=[\n ('Extra Trees', ExtraTreesClassifier()),\n ('Random Forest', RandomForestClassifier()),\n ('XGBoost', xgb.XGBClassifier()),\n ('LightGBM', lgb.LGBMClassifier(verbose=-1, random_state=42))\n ],\n voting='soft',\n n_jobs=-1\n )),\n ('Weighted Voting Top 4', VotingClassifier(\n estimators=[\n ('Extra Trees', ExtraTreesClassifier()),\n ('Random Forest', RandomForestClassifier()),\n ('XGBoost', xgb.XGBClassifier()),\n ('LightGBM', lgb.LGBMClassifier(verbose=-1, random_state=42))\n ],\n voting='soft',\n weights=[0.32, 0.25, 0.22, 0.21], # Adjusted for highest F1: ET > RF > XGB > LGBM\n n_jobs=-1\n )),\n ('Blending Manual Top 4', 'manual_blend') # Placeholder for manual weighted average blending (implemented later)\n]\n\n# OOF setup\nn_folds = 3\nskf = StratifiedKFold(n_splits=n_folds, shuffle=True, random_state=42)\nensemble_results = {}\ntest_oof_all = {}\nprint(\"\\nStarting ensemble model selection with 5-fold Stratified OOF (Macro F1, Balanced Accuracy, MCC, Inference Time)...\\n\")\nfor name, model in ensemble_models:\n if name == 'Blending Manual Top 4':\n # Skip in OOF loop — manual blending done separately after all models trained\n continue\n \n print(f\"Evaluating {name}...\")\n \n oof_proba = np.zeros((len(X_train_res), 3))\n oof_true = np.zeros(len(X_train_res), dtype=int)\n fold_f1_scores = []\n fold_bal_acc_scores = []\n fold_mcc_scores = []\n fold_inference_times = []\n \n test_proba = np.zeros((len(X_test), 3))\n \n for fold, (tr_idx, val_idx) in enumerate(skf.split(X_train_res, y_train_res)):\n X_tr = X_train_res.iloc[tr_idx]\n y_tr = y_train_res.iloc[tr_idx]\n X_va = X_train_res.iloc[val_idx]\n y_va = y_train_res.iloc[val_idx]\n \n model.fit(X_tr, y_tr)\n \n start_time = time.time()\n val_proba = model.predict_proba(X_va)\n inference_time = time.time() - start_time\n val_pred = np.argmax(val_proba, axis=1)\n \n fold_f1 = f1_score(y_va, val_pred, average='macro')\n fold_bal_acc = balanced_accuracy_score(y_va, val_pred)\n fold_mcc = matthews_corrcoef(y_va, val_pred)\n \n fold_f1_scores.append(fold_f1)\n fold_bal_acc_scores.append(fold_bal_acc)\n fold_mcc_scores.append(fold_mcc)\n fold_inference_times.append(inference_time)\n \n oof_proba[val_idx] = val_proba\n oof_true[val_idx] = y_va\n \n test_proba += model.predict_proba(X_test) / n_folds\n \n oof_pred = np.argmax(oof_proba, axis=1)\n overall_f1 = f1_score(oof_true, oof_pred, average='macro')\n overall_bal_acc = balanced_accuracy_score(oof_true, oof_pred)\n overall_mcc = matthews_corrcoef(oof_true, oof_pred)\n \n ensemble_results[name] = {\n 'mean_f1': np.mean(fold_f1_scores),\n 'std_f1': np.std(fold_f1_scores),\n 'overall_f1': overall_f1,\n 'mean_bal_acc': np.mean(fold_bal_acc_scores),\n 'std_bal_acc': np.std(fold_bal_acc_scores),\n 'overall_bal_acc': overall_bal_acc,\n 'mean_mcc': np.mean(fold_mcc_scores),\n 'std_mcc': np.std(fold_mcc_scores),\n 'overall_mcc': overall_mcc,\n 'mean_inference_time': np.mean(fold_inference_times),\n 'std_inference_time': np.std(fold_inference_times),\n 'oof_proba': oof_proba,\n 'test_proba': test_proba\n }\n \n print(f\"{name}: Mean Macro F1 = {np.mean(fold_f1_scores):.5f} ± {np.std(fold_f1_scores):.4f} | OOF F1 = {overall_f1:.5f}\")\n print(f\" Mean Balanced Acc = {np.mean(fold_bal_acc_scores):.5f} ± {np.std(fold_bal_acc_scores):.4f} | OOF Bal Acc = {overall_bal_acc:.5f}\")\n print(f\" Mean MCC = {np.mean(fold_mcc_scores):.5f} ± {np.std(fold_mcc_scores):.4f} | OOF MCC = {overall_mcc:.5f}\")\n print(f\" Mean Inference Time = {np.mean(fold_inference_times):.5f}s ± {np.std(fold_inference_times):.4f}s\")\n\n# Manual Blending (Top 4) - Uses previously trained individual models\n# Weights based on individual OOF Macro F1 (adjusted for highest F1: ET > RF > XGB > LGBM)\nblend_weights = [0.32, 0.25, 0.22, 0.21] # Extra Trees, RF, XGB, LGBM\n\n# Retrain top 4 on full resampled data\nmanual_blend_test_proba = (\n 0.32 * test_oof_all.get('Extra Trees', np.zeros((len(X_test), 3))) +\n 0.25 * test_oof_all.get('Random Forest', np.zeros((len(X_test), 3))) +\n 0.22 * test_oof_all.get('XGBoost', np.zeros((len(X_test), 3))) +\n 0.21 * test_oof_all.get('LightGBM', np.zeros((len(X_test), 3)))\n)\nensemble_results['Blending Manual Top 4'] = {\n 'test_proba': manual_blend_test_proba\n # No OOF for manual blend unless computed separately\n}\n\n# Ensemble Model Ranking & Selection\nprint(\"\\n\" + \"=\"*90)\nprint(\"ENSEMBLE MODEL SELECTION RESULTS (Primary: Macro F1, Tie-breakers: Bal Acc, MCC, Inference Time)\")\nprint(\"=\"*90)\nranking = sorted([ (name, res) for name, res in ensemble_results.items() if 'overall_f1' in res ],\n key=lambda x: (\n x[1]['overall_f1'],\n x[1]['overall_bal_acc'],\n x[1]['overall_mcc'],\n -x[1]['mean_inference_time']\n ), reverse=True)\nfor i, (name, res) in enumerate(ranking):\n print(f\"{i+1:2d}. {name:<35} | OOF Macro F1: {res['overall_f1']:.5f} | CV Mean F1: {res['mean_f1']:.5f} ± {res['std_f1']:.4f}\")\n print(f\" OOF Bal Acc: {res['overall_bal_acc']:.5f} | CV Mean Bal Acc: {res['mean_bal_acc']:.5f} ± {res['std_bal_acc']:.4f}\")\n print(f\" OOF MCC: {res['overall_mcc']:.5f} | CV Mean MCC: {res['mean_mcc']:.5f} ± {res['std_mcc']:.4f}\")\n print(f\" Mean Inference Time: {res['mean_inference_time']:.5f}s ± {res['std_inference_time']:.4f}s\")\n print(\"-\"*90)\nprint(\"=\"*90)\n\n# Best ensemble\nbest_ensemble_name = ranking[0][0]\nbest_test_proba = ensemble_results[best_ensemble_name]['test_proba']\ntest_pred_labels = np.argmax(best_test_proba, axis=1)\ntest_pred_str = le.inverse_transform(test_pred_labels)\nsubmission = pd.DataFrame({'id': test_ids, 'class_label': test_pred_str})\nsubmission['class_label'] = submission['class_label'].fillna('None')\nsubmission.to_csv(\"submission.csv\", index=False)\nprint(f\"\\nBest ensemble: {best_ensemble_name}\")\nprint(\"Submission saved successfully!\")","metadata":{"execution":{"iopub.execute_input":"2026-01-10T11:48:43.069914Z","iopub.status.busy":"2026-01-10T11:48:43.069616Z","iopub.status.idle":"2026-01-10T12:02:22.895672Z","shell.execute_reply":"2026-01-10T12:02:22.894279Z","shell.execute_reply.started":"2026-01-10T11:48:43.069889Z"},"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"Weighted Voting was selected because it achieved higher OOF Macro F1 than Voting (Soft).","metadata":{}},{"cell_type":"markdown","source":"# Hyperparameter Tuning","metadata":{}},{"cell_type":"markdown","source":"Hyperparameter Tuning (Optuna): The top four waek learner models—ExtraTrees, RandomForest, XGBoost, and LightGBM—are optimized via Optuna. This Bayesian search fine-tunes parameters like tree depth and learning rates to maximize model efficiency.","metadata":{}},{"cell_type":"markdown","source":"### Optuna\n\nOptuna is an open-source hyperparameter optimization (HPO) framework that shifts the paradigm from traditional \"static\" grid searches to \"dynamic\" automated searches. It is built on several key information-theoretic and probabilistic concepts.\n\n**1. Core Structural Concepts**\n\nOptuna organizes optimization into a clear hierarchy:\n- Study: The overall optimization task (e.g., \"Minimize the error of my XGBoost model\").\n- Trial: A single execution of the objective function with one specific set of hyperparameters.\n- Objective Function: A user-defined Python function that Optuna calls repeatedly. It takes a trial object as input and returns a numerical value (the score) to be optimized.\n\n**2. Theoretical Foundations: \"Define-by-Run\"**\n\nUnlike other frameworks that require you to pre-define a static configuration dictionary, Optuna uses a Define-by-Run API.\n- The Theory: The search space is constructed dynamically as the code executes.\n- The Concept: This allows for Conditional Hyperparameters. For example, the search can suggest a \"Kernel Type\" for an SVM, and only if the kernel is \"Polynomial\" does it then suggest a \"Degree\" parameter. This mirrors the logic of a human researcher.\n\n**3. Sampling Algorithms (Search Strategies)**\n\nOptuna uses \"Samplers\" to decide which hyperparameter values to try next based on past results.\n\n**Tree-structured Parzen Estimator (TPE) — Defaul**t\n \n- Theory: Instead of modeling the objective function directly (like Gaussian Processes), TPE models the distribution of hyperparameters.\n- Concept: It splits previous trials into two groups: \"Good\" (top performing) and \"Bad.\" It then models two probability densities: $l(x)$ for the good group and $g(x)$ for the bad group. It samples new points that maximize the ratio $l(x)/g(x)$, essentially looking for values that are likely to be \"good\" and unlikely to be \"bad.\"\n \n**CMA-ES (Covariance Matrix Adaptation Evolution Strategy)**\n \n- Theory: An evolutionary algorithm for continuous, non-linear optimization.\n- Concept: It maintains a multivariate normal distribution over the search space. In each generation, it \"evolves\" by shifting the mean toward better results and adapting the covariance matrix to follow the \"path\" of steepest improvement.\n\n**4. Pruning Theory (Automated Early Stopping)**\n\nPruning is the process of killing a \"bad\" trial before it finishes training to save time and compute.\n- Median Pruner: Theoretically simple; it stops a trial if its intermediate result (e.g., validation loss at epoch 5) is worse than the median of previous trials at the same step.\n- Hyperband / Successive Halving (SHA):\n - Theory: Budget-based allocation.\n - Concept: Start many trials with a tiny budget (e.g., 1 epoch). Keep the top 25%, give them more budget, and repeat. This ensures that only the most promising \"survivors\" reach the final training stage.\n\n**5. Hyperparameter Importance: fANOVA**\n\nOptuna can calculate which hyperparameters actually matter using Functional Analysis of Variance (fANOVA).\n- Theory: It decomposes the variance of the model's performance.\n- Concept: If changing the \"Learning Rate\" causes a 50% change in accuracy, but changing \"Batch Size\" only causes a 1% change, fANOVA identifies the Learning Rate as the most critical parameter to focus on.\n\n**The Optimization Cycle**\n\nThe process follows a feedback loop:\n\n- Suggest: The Sampler picks a value based on the Study's history.\n- Evaluate: The Trial runs the objective function.\n- Report: The trial sends intermediate results back for potential Pruning.\n- Update: The Sampler updates its internal probability model (TPE) with the final result.","metadata":{}},{"cell_type":"markdown","source":"## ExtraTree Classifier","metadata":{}},{"cell_type":"markdown","source":"**Extra Trees (Extremely Randomized Trees)**\n\nSimilar to Random Forest, but introduces a higher level of randomness. While RF searches for the optimum split threshold for each feature, Extra Trees chooses a random threshold. Additionally, it typically uses the entire dataset rather than bootstrap samples. This significantly reduces training time and often reduces variance further than RF.\n\n**Important Hyperparameters:**\n\n- `n_estimators`: Total number of randomized trees.\n- `max_features`: Number of features to randomly sample for each split.\n- `bootstrap`: Boolean (default False). Whether to use the whole dataset or samples with replacement.\n- `min_samples_split`: The minimum number of samples required to split an internal node.","metadata":{}},{"cell_type":"code","source":"MODEL_NAME = \"extra_trees\"\nTUNING_FILE = f\"best_{MODEL_NAME}_params.pkl\"\n\n# Read-only input DB (if exists)\nREADONLY_DB = \"/kaggle/input/optuna-extra-trees-1/scikitlearn/default/1/optuna_extra_trees (1).db\"\n\n# Writable DB in working directory\nWRITABLE_DB = f\"/kaggle/working/optuna_{MODEL_NAME}.db\"\nSTUDY_NAME = f\"reversal_{MODEL_NAME}\"\n\n# Copy readonly DB to writable location if it exists\nif os.path.exists(READONLY_DB):\n print(\"Copying readonly Extra Trees Optuna database to writable location...\")\n shutil.copy(READONLY_DB, WRITABLE_DB)\n print(\"Database copied successfully.\")\nelse:\n print(\"No existing Extra Trees database found. Starting fresh.\")\n\n# Use writable DB\nDB_FILE = WRITABLE_DB\n\n# Load previous best if exists\nbest_score = 0.0\nif os.path.exists(TUNING_FILE):\n loaded = load(TUNING_FILE)\n best_score = loaded.get('score', 0.0)\n print(f\"Previous best {MODEL_NAME} OOF Macro F1: {best_score:.5f}\")\n\nn_folds = 3 #Increase for better hyperparameter tuning validation accuracy (less bias score), I use 3 fold for illustration\nskf = StratifiedKFold(n_splits=n_folds, shuffle=True, random_state=42)\n\ndef objective(trial):\n print(f\"\\n=== {MODEL_NAME.upper()} - Trial {trial.number} ===\")\n \n params = {\n 'n_estimators': trial.suggest_int('n_estimators', 100, 1500),\n 'max_depth': trial.suggest_categorical('max_depth', [None, 20, 40, 60, 80, 100]),\n 'min_samples_split': trial.suggest_int('min_samples_split', 2, 20),\n 'min_samples_leaf': trial.suggest_int('min_samples_leaf', 1, 10),\n 'max_features': trial.suggest_categorical('max_features', ['sqrt', 'log2', 0.6, 0.8, None]),\n 'class_weight': class_weight_dict_full,\n 'n_jobs': -1,\n 'random_state': 42\n }\n \n print(\"Params:\", {k: v for k, v in params.items() if k not in ['class_weight', 'n_jobs', 'random_state']})\n \n model = ExtraTreesClassifier(**params)\n scores = []\n for tr_idx, val_idx in skf.split(X_train_res, y_train_res):\n X_tr, X_va = X_train_res.iloc[tr_idx], X_train_res.iloc[val_idx]\n y_tr, y_va = y_train_res.iloc[tr_idx], y_train_res.iloc[val_idx]\n model.fit(X_tr, y_tr)\n pred = model.predict(X_va)\n scores.append(f1_score(y_va, pred, average='macro'))\n \n mean_score = np.mean(scores)\n print(f\"Trial {trial.number} OOF Macro F1: {mean_score:.5f}\")\n return mean_score\n\nstudy = optuna.create_study(direction=\"maximize\", study_name=STUDY_NAME, storage=f\"sqlite:///{DB_FILE}\", load_if_exists=True)\nprint(f\"Starting/continuing {MODEL_NAME.upper()} tuning...\")\nstudy.optimize(objective, n_trials=3, timeout=None) #Increase for better score, I use 3 iteration for illustration\n\nbest_score = study.best_value\nbest_params = study.best_params\ndump({'score': best_score, 'params': best_params}, TUNING_FILE)\nprint(f\"\\n{MODEL_NAME.upper()} tuning complete! Best OOF Macro F1: {best_score:.5f}\")\nprint(f\"Best params saved to {TUNING_FILE}\")","metadata":{"execution":{"iopub.execute_input":"2026-01-09T18:59:42.447303Z","iopub.status.busy":"2026-01-09T18:59:42.446848Z","iopub.status.idle":"2026-01-09T20:42:37.375819Z","shell.execute_reply":"2026-01-09T20:42:37.373077Z","shell.execute_reply.started":"2026-01-09T18:59:42.447271Z"},"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## RandomForest Classifier","metadata":{}},{"cell_type":"markdown","source":"**Random Forest (RF)**\n\nRandom Forest is an ensemble learning method based on Bagging (Bootstrap Aggregating). It builds multiple decision trees independently using random subsets of the training data (drawn with replacement). To increase diversity, it also selects a random subset of features at each split. The final prediction is the majority vote (classification) or average (regression) of all trees.\n\n**Important Hyperparameters:**\n\n- `n_estimators`: Number of trees in the forest. More trees generally improve stability but increase training time.\n- `max_depth`: The maximum depth of each tree. Controlling this helps prevent trees from becoming too complex and overfitting.\n- `max_features`: The number of random features to consider at each split (e.g., sqrt or log2).\n- `min_samples_leaf`: The minimum number of samples required to be at a leaf node; higher values smooth the model.","metadata":{}},{"cell_type":"code","source":"MODEL_NAME = \"random_forest\"\nTUNING_FILE = f\"best_{MODEL_NAME}_params.pkl\"\n\n# Read-only input DB (if exists)\nREADONLY_DB = \"/kaggle/input/optuna-random-forest-1/scikitlearn/default/1/optuna_random_forest (1).db\"\n\n# Writable DB in working directory\nWRITABLE_DB = f\"/kaggle/working/optuna_{MODEL_NAME}.db\"\nSTUDY_NAME = f\"reversal_{MODEL_NAME}\"\n\n# Copy readonly DB to writable location if it exists\nif os.path.exists(READONLY_DB):\n print(\"Copying readonly Random Forest Optuna database to writable location...\")\n shutil.copy(READONLY_DB, WRITABLE_DB)\n print(\"Database copied successfully.\")\nelse:\n print(\"No existing Random Forest database found. Starting fresh.\")\n\n# Use writable DB\nDB_FILE = WRITABLE_DB\n\n# Load previous best if exists\nbest_score = 0.0\nif os.path.exists(TUNING_FILE):\n loaded = load(TUNING_FILE)\n best_score = loaded.get('score', 0.0)\n print(f\"Previous best {MODEL_NAME} OOF Macro F1: {best_score:.5f}\")\n\nn_folds = 3 #Increase for better hyperparameter tuning validation accuracy (less bias score), I use 3 fold for illustration\nskf = StratifiedKFold(n_splits=n_folds, shuffle=True, random_state=42)\n\ndef objective(trial):\n print(f\"\\n=== {MODEL_NAME.upper()} - Trial {trial.number} ===\")\n \n params = {\n 'n_estimators': trial.suggest_int('n_estimators', 100, 1500),\n 'max_depth': trial.suggest_categorical('max_depth', [None, 20, 40, 60, 80, 100]),\n 'min_samples_split': trial.suggest_int('min_samples_split', 2, 20),\n 'min_samples_leaf': trial.suggest_int('min_samples_leaf', 1, 10),\n 'max_features': trial.suggest_categorical('max_features', ['sqrt', 'log2', 0.6, 0.8, None]),\n 'class_weight': class_weight_dict_full,\n 'n_jobs': -1,\n 'random_state': 42\n }\n \n print(\"Params:\", {k: v for k, v in params.items() if k not in ['class_weight', 'n_jobs', 'random_state']})\n \n model = RandomForestClassifier(**params)\n scores = []\n for tr_idx, val_idx in skf.split(X_train_res, y_train_res):\n X_tr, X_va = X_train_res.iloc[tr_idx], X_train_res.iloc[val_idx]\n y_tr, y_va = y_train_res.iloc[tr_idx], y_train_res.iloc[val_idx]\n model.fit(X_tr, y_tr)\n pred = model.predict(X_va)\n scores.append(f1_score(y_va, pred, average='macro'))\n \n mean_score = np.mean(scores)\n print(f\"Trial {trial.number} OOF Macro F1: {mean_score:.5f}\")\n return mean_score\n\nstudy = optuna.create_study(direction=\"maximize\", study_name=STUDY_NAME, storage=f\"sqlite:///{DB_FILE}\", load_if_exists=True)\nprint(f\"Starting/continuing {MODEL_NAME.upper()} tuning...\")\nstudy.optimize(objective, n_trials=3, timeout=None) #Increase for better score, I use 3 iteration for illustration\n\nbest_score = study.best_value\nbest_params = study.best_params\ndump({'score': best_score, 'params': best_params}, TUNING_FILE)\nprint(f\"\\n{MODEL_NAME.upper()} tuning complete! Best OOF Macro F1: {best_score:.5f}\")\nprint(f\"Best params saved to {TUNING_FILE}\")","metadata":{"execution":{"iopub.execute_input":"2026-01-09T20:42:37.384008Z","iopub.status.busy":"2026-01-09T20:42:37.382662Z","iopub.status.idle":"2026-01-09T21:37:54.625413Z","shell.execute_reply":"2026-01-09T21:37:54.623945Z","shell.execute_reply.started":"2026-01-09T20:42:37.383948Z"},"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## XGBoost Classifier","metadata":{}},{"cell_type":"markdown","source":"**XGBoost (eXtreme Gradient Boosting)**\n\nA powerful implementation of Gradient Boosting that builds trees sequentially. Each new tree attempts to correct the errors (residuals) of the previous ones using gradient descent. It uses Level-Wise Growth (splitting level by level) and incorporates L1/L2 regularization directly into the objective function to handle overfitting.\n\n**Important Hyperparameters:**\n\n- `learning_rate (eta)`: Scales the contribution of each tree. Lower values (e.g., 0.01) require more trees but lead to better generalization.\n- `max_depth`: Limits the complexity of individual trees.\n- `gamma`: The minimum loss reduction required to make a further split; higher values make the model more conservative.\n- `lambda (L2) and alpha (L1)`: Regularization terms on weights to penalize complexity.","metadata":{}},{"cell_type":"code","source":"MODEL_NAME = \"xgboost\"\nTUNING_FILE = f\"best_{MODEL_NAME}_params.pkl\"\nDB_FILE_ORIGINAL = \"/kaggle/input/optuna-xgboost-1/scikitlearn/default/1/optuna_xgboost (1).db\"\nSTUDY_NAME = f\"reversal_{MODEL_NAME}\"\n\n# Copy the read-only DB to a writable location (Kaggle working dir)\nWORKING_DIR = \"/kaggle/working\"\nDB_FILE = os.path.join(WORKING_DIR, f\"optuna_{MODEL_NAME}.db\")\n\nif os.path.exists(DB_FILE_ORIGINAL):\n shutil.copy(DB_FILE_ORIGINAL, DB_FILE)\n print(f\"Copied input DB to writable location: {DB_FILE}\")\nelse:\n print(\"No input DB found — starting fresh study.\")\n\n# Load previous best if exists\nbest_score = 0.0\nif os.path.exists(TUNING_FILE):\n print(\"Loading previous best tuning parameters...\")\n loaded = load(TUNING_FILE)\n best_score = loaded.get('score', 0.0)\n print(f\"Previous best {MODEL_NAME.upper()} OOF Macro F1: {best_score:.5f}\")\n\nn_folds = 3 #Increase for better hyperparameter tuning validation accuracy (less bias score), I use 3 fold for illustration\nskf = StratifiedKFold(n_splits=n_folds, shuffle=True, random_state=42)\n\ndef objective(trial):\n print(f\"\\n=== {MODEL_NAME.upper()} - Trial {trial.number} ===\")\n \n params = {\n 'n_estimators': trial.suggest_int('n_estimators', 100, 1500),\n 'learning_rate': trial.suggest_float('learning_rate', 0.005, 0.15, log=True),\n 'max_depth': trial.suggest_int('max_depth', 3, 15),\n 'min_child_weight': trial.suggest_int('min_child_weight', 1, 15),\n 'subsample': trial.suggest_float('subsample', 0.5, 1.0),\n 'colsample_bytree': trial.suggest_float('colsample_bytree', 0.5, 1.0),\n 'gamma': trial.suggest_float('gamma', 0.0, 1.0),\n 'reg_alpha': trial.suggest_float('reg_alpha', 0.0, 5.0),\n 'reg_lambda': trial.suggest_float('reg_lambda', 0.0, 5.0),\n 'objective': 'multi:softprob',\n 'num_class': 3,\n 'eval_metric': 'mlogloss',\n 'random_state': 42,\n 'n_jobs': -1\n }\n \n print(\"Params:\", {k: v for k, v in params.items() if k not in ['objective', 'num_class', 'eval_metric', 'random_state', 'n_jobs']})\n \n model = xgb.XGBClassifier(**params)\n scores = []\n for tr_idx, val_idx in skf.split(X_train_res, y_train_res):\n X_tr, X_va = X_train_res.iloc[tr_idx], X_train_res.iloc[val_idx]\n y_tr, y_va = y_train_res.iloc[tr_idx], y_train_res.iloc[val_idx]\n model.fit(X_tr, y_tr)\n pred = model.predict(X_va)\n scores.append(f1_score(y_va, pred, average='macro'))\n \n mean_score = np.mean(scores)\n print(f\"Trial {trial.number} OOF Macro F1: {mean_score:.5f}\")\n return mean_score\n\n# Create/load study with writable DB\nstudy = optuna.create_study(\n direction=\"maximize\",\n study_name=STUDY_NAME,\n storage=f\"sqlite:///{DB_FILE}\",\n load_if_exists=True\n)\n\nprint(f\"Starting/continuing {MODEL_NAME.upper()} tuning...\")\nprint(f\"Current trials in study: {len(study.trials)}\")\nstudy.optimize(objective, n_trials=3, timeout=None) #Increase for better score, I use 3 iteration for illustration\n\nbest_score = study.best_value\nbest_params = study.best_params\n\ndump({'score': best_score, 'params': best_params}, TUNING_FILE)\nprint(f\"\\n{MODEL_NAME.upper()} tuning complete! Best OOF Macro F1: {best_score:.5f}\")\nprint(f\"Best params saved to {TUNING_FILE}\")","metadata":{"execution":{"iopub.execute_input":"2026-01-09T21:37:54.628999Z","iopub.status.busy":"2026-01-09T21:37:54.628618Z","iopub.status.idle":"2026-01-09T22:22:42.367618Z","shell.execute_reply":"2026-01-09T22:22:42.366166Z","shell.execute_reply.started":"2026-01-09T21:37:54.628970Z"},"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## LightGBM Classifier","metadata":{}},{"cell_type":"markdown","source":"**LightGBM (Light Gradient Boosting Machine)**\n\nA high-speed boosting framework developed by Microsoft. Its primary differentiator is Leaf-Wise Growth, where it splits the node that results in the greatest loss reduction, regardless of depth. It also uses Histogram-based binning and GOSS (Gradient-based One-Side Sampling) to handle large datasets with significantly less memory and time.\n\n**Important Hyperparameters:**\n\n- `num_leaves`: The most important parameter for LightGBM; it controls tree complexity (should be less than $2^{\\text{max\\_depth}}$).\n- `learning_rate`: Similar to XGBoost, dictates the step size of the optimization.\n- `min_data_in_leaf`: Prevents overfitting by ensuring each leaf has enough supporting samples.\n- `feature_fraction`: Randomly selects a subset of features on each iteration (similar to colsample_bytree).","metadata":{}},{"cell_type":"code","source":"MODEL_NAME = \"lightgbm\"\nTUNING_FILE = f\"best_{MODEL_NAME}_params.pkl\"\nDB_FILE_ORIGINAL = \"/kaggle/input/optuna-lightgbm-1/scikitlearn/default/1/optuna_lightgbm (1).db\"\nSTUDY_NAME = f\"reversal_{MODEL_NAME}\"\n\n# Copy the read-only DB to writable working directory\nWORKING_DIR = \"/kaggle/working\"\nDB_FILE = os.path.join(WORKING_DIR, f\"optuna_{MODEL_NAME}.db\")\n\nif os.path.exists(DB_FILE_ORIGINAL):\n shutil.copy(DB_FILE_ORIGINAL, DB_FILE)\n print(f\"Copied input DB to writable location: {DB_FILE}\")\nelse:\n print(\"No input DB found — starting fresh study.\")\n\n# Load previous best if exists\nbest_score = 0.0\nif os.path.exists(TUNING_FILE):\n print(\"Loading previous best tuning parameters...\")\n loaded = load(TUNING_FILE)\n best_score = loaded.get('score', 0.0)\n print(f\"Previous best {MODEL_NAME.upper()} OOF Macro F1: {best_score:.5f}\")\n\nn_folds = 3 #Increase for better hyperparameter tuning validation accuracy (less bias score), I use 3 fold for illustration\nskf = StratifiedKFold(n_splits=n_folds, shuffle=True, random_state=42)\n\ndef objective(trial):\n print(f\"\\n=== {MODEL_NAME.upper()} - Trial {trial.number} ===\")\n \n params = {\n 'n_estimators': trial.suggest_int('n_estimators', 100, 2000),\n 'learning_rate': trial.suggest_float('learning_rate', 0.005, 0.15, log=True),\n 'max_depth': trial.suggest_int('max_depth', -1, 40),\n 'num_leaves': trial.suggest_int('num_leaves', 31, 512),\n 'min_data_in_leaf': trial.suggest_int('min_data_in_leaf', 10, 200),\n 'subsample': trial.suggest_float('subsample', 0.5, 1.0),\n 'colsample_bytree': trial.suggest_float('colsample_bytree', 0.5, 1.0),\n 'reg_alpha': trial.suggest_float('reg_alpha', 0.0, 10.0),\n 'reg_lambda': trial.suggest_float('reg_lambda', 0.0, 10.0),\n 'objective': 'multiclass',\n 'num_class': 3,\n 'class_weight': 'balanced', \n 'random_state': 42,\n 'verbose': -1,\n 'n_jobs': -1\n }\n \n print(\"Params:\", {k: v for k, v in params.items() if k not in ['objective', 'num_class', 'class_weight', 'random_state', 'verbose', 'n_jobs']})\n \n model = lgb.LGBMClassifier(**params)\n scores = []\n for tr_idx, val_idx in skf.split(X_train_res, y_train_res):\n X_tr, X_va = X_train_res.iloc[tr_idx], X_train_res.iloc[val_idx]\n y_tr, y_va = y_train_res.iloc[tr_idx], y_train_res.iloc[val_idx]\n model.fit(X_tr, y_tr)\n pred = model.predict(X_va)\n scores.append(f1_score(y_va, pred, average='macro'))\n \n mean_score = np.mean(scores)\n print(f\"Trial {trial.number} OOF Macro F1: {mean_score:.5f}\")\n return mean_score\n\n# Create/load study with writable DB\nstudy = optuna.create_study(\n direction=\"maximize\",\n study_name=STUDY_NAME,\n storage=f\"sqlite:///{DB_FILE}\",\n load_if_exists=True\n)\n\nprint(f\"Starting/continuing {MODEL_NAME.upper()} tuning...\")\nprint(f\"Current trials in study: {len(study.trials)}\")\nstudy.optimize(objective, n_trials=3, timeout=None) #Increase for better score, I use 3 iteration for illustration\n\nbest_score = study.best_value\nbest_params = study.best_params\n\ndump({'score': best_score, 'params': best_params}, TUNING_FILE)\nprint(f\"\\n{MODEL_NAME.upper()} tuning complete! Best OOF Macro F1: {best_score:.5f}\")\nprint(f\"Best params saved to {TUNING_FILE}\")","metadata":{"execution":{"iopub.execute_input":"2026-01-09T22:22:42.371893Z","iopub.status.busy":"2026-01-09T22:22:42.371403Z","iopub.status.idle":"2026-01-09T22:24:31.632441Z","shell.execute_reply":"2026-01-09T22:24:31.631145Z","shell.execute_reply.started":"2026-01-09T22:22:42.371864Z"},"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"# Final Ensemble Model Training and Submission Generation","metadata":{}},{"cell_type":"markdown","source":"The tuned models are integrated into a final weighted soft-voting ensemble. Weights are distributed based on individual model performance. The pipeline generates final predictions, maps them back to the original labels (H, L, None), and exports the results to submission.csv.","metadata":{}},{"cell_type":"code","source":"# Extra Trees \net_best = ExtraTreesClassifier(\n n_estimators=115,\n max_depth=100,\n min_samples_split=15,\n min_samples_leaf=1,\n max_features='sqrt',\n class_weight=class_weight_dict_full,\n n_jobs=-1,\n random_state=42\n)\n\n# Random Forest \nrf_best = RandomForestClassifier(\n n_estimators=853,\n max_depth=None,\n min_samples_split=8,\n min_samples_leaf=1,\n max_features='log2',\n class_weight=class_weight_dict_full,\n n_jobs=-1,\n random_state=42\n)\n\n# XGBoost \nxgb_best = xgb.XGBClassifier(\n n_estimators=5294,\n learning_rate=0.0507,\n max_depth=12,\n min_child_weight=5,\n subsample=0.7659,\n colsample_bytree=0.613,\n gamma=0.46586,\n reg_alpha=1.180167,\n reg_lambda=0.2799,\n objective='multi:softprob',\n num_class=3,\n eval_metric='mlogloss',\n random_state=42,\n n_jobs=-1\n)\n\n# LightGBM \nlgb_best = lgb.LGBMClassifier(\n n_estimators=1280,\n learning_rate=0.01533360874282631,\n max_depth=27,\n num_leaves=461,\n min_data_in_leaf=30,\n subsample=0.5876046117077005,\n colsample_bytree=0.5799546893007104,\n reg_alpha=0.0550092101268719,\n reg_lambda=8.675002657367935,\n objective='multiclass',\n num_class=3,\n class_weight=class_weight_dict_full,\n random_state=42,\n verbose=-1,\n n_jobs=-1\n)\n\n# Optimized Weights for Highest F1 (based on trial performance & blending experiments)\n# Weights: ExtraTrees (highest) > RF > LGBM > XGB\nblend_weights = [0.32, 0.28, 0.24, 0.16] # ET, RF, LGBM, XGB\n\n# Manual Weighted Soft Voting Ensemble \nprint(\"\\nTraining final weighted soft voting ensemble (ET + RF + LGBM + XGB)...\")\nensemble = VotingClassifier(\n estimators=[\n ('ExtraTrees', et_best),\n ('RandomForest', rf_best),\n ('LightGBM', lgb_best),\n ('XGBoost', xgb_best)\n ],\n voting='soft',\n weights=blend_weights,\n n_jobs=-1\n)\n\n# Train on full resampled data\nensemble.fit(X_train_res, y_train_res)\n\n# Validation Performance\nval_proba = ensemble.predict_proba(X_val)\nval_pred = np.argmax(val_proba, axis=1)\nval_pred_str = le.inverse_transform(val_pred)\n\nmacro_f1 = f1_score(y_val_encoded, val_pred, average='macro')\nbal_acc = balanced_accuracy_score(y_val_encoded, val_pred)\nmcc = matthews_corrcoef(y_val_encoded, val_pred)\n\nprint(f\"\\nValidation Macro F1: {macro_f1:.5f}\")\nprint(f\"Validation Balanced Acc: {bal_acc:.5f}\")\nprint(f\"Validation MCC: {mcc:.5f}\")\n\n# Test Prediction & Submission\ntest_proba = ensemble.predict_proba(X_test)\ntest_pred = np.argmax(test_proba, axis=1)\ntest_pred_str = le.inverse_transform(test_pred)\n\nsubmission = pd.DataFrame({'id': test_ids, 'class_label': test_pred_str})\nsubmission['class_label'] = submission['class_label'].fillna('None')\nsubmission.to_csv(\"submission.csv\", index=False)\n\nprint(\"\\nSubmission saved successfully!\")\nprint(\"Best ensemble: Weighted Voting (ET + RF + LGBM + XGB)\")","metadata":{"execution":{"iopub.execute_input":"2026-01-10T12:02:22.897747Z","iopub.status.busy":"2026-01-10T12:02:22.897333Z","iopub.status.idle":"2026-01-10T12:57:20.416489Z","shell.execute_reply":"2026-01-10T12:57:20.415233Z","shell.execute_reply.started":"2026-01-10T12:02:22.897715Z"},"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"code","source":"submission[\"class_label\"].value_counts(normalize=True*100)","metadata":{"execution":{"iopub.execute_input":"2026-01-10T13:05:09.236850Z","iopub.status.busy":"2026-01-10T13:05:09.236462Z","iopub.status.idle":"2026-01-10T13:05:09.246715Z","shell.execute_reply":"2026-01-10T13:05:09.245410Z","shell.execute_reply.started":"2026-01-10T13:05:09.236821Z"},"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"# Conclusion","metadata":{}},{"cell_type":"markdown","source":"The developed solution provides a complete, high-quality framework for detecting US equity reversals. By synthesizing time-aware preprocessing, aggressive imbalance management, and Bayesian-tuned ensembles, the pipeline achieves a high Macro F1 performance while maintaining the inference efficiency required for real-world production. This project effectively navigates the challenges of high dimensionality and temporal dependencies, serving as a reliable blueprint for imbalanced time-series classification tasks in the financial domain.","metadata":{}},{"cell_type":"markdown","source":"# References","metadata":{}},{"cell_type":"markdown","source":"- [Standard Scaler sk-learn](https://scikit-learn.org/stable/modules/generated/sklearn.preprocessing.StandardScaler.html)\n- [Standard Scaler Greeksforgreeks](https://www.geeksforgeeks.org/machine-learning/standardscaler-minmaxscaler-and-robustscaler-techniques-ml/)\n- [ADASYN Imblearn](https://imbalanced-learn.org/stable/references/generated/imblearn.over_sampling.ADASYN.html)\n- [PCA Wikipedia](https://en.wikipedia.org/wiki/Principal_component_analysis)\n- [PCA sk-learn](https://scikit-learn.org/stable/modules/generated/sklearn.decomposition.PCA.html)\n- [PCA Greeksforgreeks](https://www.geeksforgeeks.org/data-analysis/principal-component-analysis-pca/)\n- [Voting Classifier sk-learn](https://scikit-learn.org/stable/modules/generated/sklearn.ensemble.VotingClassifier.html)\n- [Voting Classifier Greeksforgreek](https://www.geeksforgeeks.org/machine-learning/voting-classifier/)\n- [Optuna Hyperparameter Tuning](https://optuna.org/)\n- [ExtraTree Classifier](https://scikit-learn.org/stable/modules/generated/sklearn.ensemble.ExtraTreesClassifier.html)\n- [RandomForest Classifier sk-learn](https://scikit-learn.org/stable/modules/generated/sklearn.ensemble.RandomForestClassifier.html)\n- [RandomForest Classifier Greeksforgreeks](https://www.geeksforgeeks.org/dsa/random-forest-classifier-using-scikit-learn/)\n- [XGBoost Classifier](https://xgboost.readthedocs.io/en/stable/)\n- [XGBoost Classifier Greeksforgreeks](https://www.geeksforgeeks.org/machine-learning/xgboost/)\n- [LightGBM Classifier](https://lightgbm.readthedocs.io/en/stable/)\n- [LightGBM Greeksforgreeks](https://www.geeksforgeeks.org/machine-learning/lightgbm-light-gradient-boosting-machine/)","metadata":{}}]}