This repository contains my solution for the Electricity Trading Challenge (ensimag-if-2025).
The task was approached primarily as an economic forecasting problem, rather than as a conventional regression problem. The objective was not simply to minimize RMSE/MAE, but to produce a directionally useful forecast whose errors translated into favorable trading PnL under the competition's fixed-position mechanics.
\nThe final pipeline combines:
\nThe final solution deliberately avoids several seemingly promising approaches that failed to transfer robustly to the leaderboard.
\nThe target is the German/DE-LU electricity price spread:
\nspread = imbalance_price - day_ahead_price\nThe competition converts the forecast direction into a fixed trading position.
\nConsequently, the important distinction is:
\nprediction accuracy\n ≠\ntrading profitability\nA model can improve RMSE while producing worse trading decisions, while a model with modest statistical correlation can produce substantially better PnL.
\nFor that reason, model selection throughout the project was increasingly centered on chronological validation PnL, while MAE, RMSE, Pearson correlation, Spearman correlation and directional hit rate were retained as diagnostics.
\nThe training timeline covers approximately:
\n2020-01-01 → 2024-09-08\nwith 15-minute observations.
\nThe feature pipeline combines several information sources.
\nExamples include:
\nThe model uses variables such as:
\nExamples include:
\nThe pipeline includes:
\nImportant historical spread/system variables include:
\nAll lagged features were constructed chronologically to avoid using future target information.
\nLeakage prevention was treated as a primary design constraint.
\nThe validation procedure uses chronological splits rather than random train/test splitting.
\nThe pipeline produces expanding-window out-of-fold predictions so that each validation observation is predicted using information available before that observation.
\nThis is particularly important for electricity markets because:
\nThe goal was therefore not:
\n\n\n\"How well can the model fit the historical dataset?\"
\n
but:
\n\n\n\"How well does a model trained on the past perform on a genuinely later period?\"
\n
The project went through many feature-engineering iterations.
\nAn important lesson was that more features did not necessarily mean a better leaderboard result.
\nSeveral large feature expansions were tested and subsequently removed when they failed to demonstrate stable out-of-sample improvement.
\nIn particular, the final solution does not include every experimental feature introduced during development.
\nThis was intentional.
\nThe pipeline initially constructs a relatively large feature set.
\nFeature reduction includes numerical sanity checks and correlation/importance-based pruning.
\nThe purpose is to remove:
\nHowever, economically meaningful features were protected where appropriate.
\nThe final feature set was substantially smaller than some of the experimental feature pools.
\nThis turned out to be preferable to continually adding engineered variables.
\nOne of the more useful experiments was to evaluate feature families economically, rather than judging every feature solely by generic model importance.
\nFeature families included concepts such as:
\nEach ablation was given its own Optuna search rather than comparing models using identical frozen hyperparameters.
\nThis matters because removing a feature family can change the optimal model complexity.
\nOne experiment produced the following result:
\nBaseline: €584,381 CV PnL\n\nGRID_RAMP removed:\n €476,167\n Δ -€108,214\n\nWIND_REGIME removed:\n €602,212\n Δ +€17,831\n\nGAS_MARKET removed:\n €557,930\n Δ -€26,451\nThe interpretation was not that every positive CV difference would necessarily improve the leaderboard.
\nInstead, it provided evidence for which economic information was potentially redundant.
\nThe final experiment therefore tested removal of the WIND_REGIME family while preserving the underlying physical variables used to construct other system features.
\nLightGBM became the strongest individual production model during the experiments.
\nThe optimization target was not ordinary squared-error loss alone.
\nOptuna searched model parameters using a chronological validation framework with PnL as the primary economic objective.
\nA representative strong parameter region was:
\nlearning_rate ≈ 0.0112\nnum_leaves ≈ 26\nmax_depth 8\nsubsample ≈ 0.668\ncolsample_bytree ≈ 0.629\nmin_child_samples 65\nreg_alpha ≈ 4.26\nreg_lambda ≈ 0.073\nThe exact optimum was allowed to vary with the feature set because fixing parameters while changing the feature pool would make the feature-ablation experiment less meaningful.
\nA custom JAX-based gradient-boosting model was developed as a complementary learner.
\nThe model was not simply introduced because it improved conventional regression metrics.
\nIts second head was designed around the economic objective.
\nThe production configuration used in the final experiments was:
\nJAXGBMRegressor(\n n_estimators=30,\n learning_rate=0.01,\n max_depth=6,\n batch_size=2000,\n num_thresholds=100,\n alpha=0.75,\n k=0.2\n)\nA key part of the development process was recognizing that increasing model complexity was not consistently beneficial.
\nDifferent chronological folds produced very different optimal stopping points.
\nFor example, some folds reached their best PnL after only a handful of trees, while another fold continued improving much longer.
\nThat behavior was one reason I ultimately avoided making another large JAXGBM complexity change immediately before the final submission.
\nThe final production prediction used a deliberately simple blend:
\n80% LightGBM\n20% JAXGBM\nThis consistently proved more useful than many of the more complicated routing schemes tested.
\nThe reason for retaining JAXGBM was not that it was always the strongest standalone model.
\nInstead, it provided a complementary prediction surface that sometimes improved the economic result when combined with LightGBM.
\nTCN and N-BEATS models were also investigated as residual/error detectors.
\nThe motivation was to determine whether neural sequence models could identify situations in which the primary ensemble was likely to make a poor prediction.
\nAlthough these models produced interesting diagnostics, they did not provide sufficiently stable leaderboard evidence to justify replacing the simpler production ensemble.
\nThey therefore remained experimental rather than becoming a mandatory component of the final trading signal.
\nAnother experiment classified predictions according to:
\nusing a 5×5 matrix.
\nThis revealed economically interesting regions.
\nFor example, one diagnostic produced cells with substantially different average PnL per trade, demonstrating that model confidence and reconstruction error were related to trading outcomes.
\nHowever, this approach introduced an important methodological danger:
\n\n\nselecting a matrix cell using realized target information can turn the analysis into an oracle.
\n
Therefore matrix/carry/pyramid logic was not used blindly in the final submission.
\nThe analysis was retained as a diagnostic rather than being treated as automatically deployable alpha.
\nA one-step carry/pyramid strategy was also investigated.
\nThe idea was to use the previous prediction and its regime classification to influence the next position.
\nThe experiments showed that this could generate attractive historical diagnostics.
\nHowever, because the matrix itself could be selected using realized future information, the apparent improvement was not sufficiently clean to justify deployment.
\nThis was another case where a complicated strategy looked promising in-sample but had weaker evidence than the simpler unrestricted ensemble.
\nA particularly interesting experiment tested restricting execution to selected UTC hours.
\nThe historical/OOT diagnostic initially looked very promising.
\nFor example:
\nAll hours:\nPnL/trade €21.99\nHit rate 55.08%\n\n09–15 UTC:\nPnL/trade €33.88\nHit rate 58.20%\nThis suggested a strong concentration of historical model edge during the middle of the day.
\nHowever, the actual leaderboard result provided a much stronger test.
\nThe 09–15 UTC submission executed only:
\n6,032 / 24,138 = 25%\nof possible orders and achieved approximately:
\n€3.91M\nThis was dramatically below the established champion.
\nThe time filter was therefore rejected.
\nThis experiment was particularly useful because it demonstrated why historical PnL concentration is not sufficient evidence for a hard execution rule in a non-stationary electricity market.
\nA further experiment introduced:
\nsolar_share_load\nwind_share_load\nlignite_share_load\nhard_coal_share_load\nThe motivation was to normalize generation against contemporaneous demand.
\nAlthough physically plausible, the resulting experiment produced weaker economic CV results than the stronger preceding configuration.
\nThe feature expansion was therefore not carried into the final last-run configuration.
\nThis reinforced an important principle from the project:
\n\n\nA physically sensible feature is not automatically a useful predictive feature.
\n
The strongest configuration was surprisingly conservative compared with many of the experimental variants.
\nThe final approach was approximately:
\nHistorical electricity data\n │\n ├── weather\n ├── renewable generation\n ├── load / net load\n ├── market variables\n ├── calendar / Fourier\n ├── spread lags\n └── forecast-error features\n │\n ▼\n Feature selection\n │\n ├──────────────┐\n ▼ ▼\n LightGBM JAXGBM\n │ │\n └──────┬───────┘\n ▼\n 80 / 20\n ensemble\n │\n ▼\n final direction\nThe final production logic deliberately remained simple.
\nNo hard UTC time filter.
\nNo aggressive pyramid.
\nNo oracle-derived matrix routing.
\nNo wholesale feature expansion.
\nDuring development, the strongest leaderboard submissions reached approximately the €18M level, substantially outperforming many of the individual experimental models.
\nRepresentative experiments included:
\n| Configuration | \nApprox. result | \n
|---|---|
| LGBM | \n€16.3M | \n
| JAXGBM | \n€11.1M | \n
| 80/20 LGBM + JAXGBM | \n€17.2M | \n
| ET | \n€14.4M | \n
| Complex ensemble variants | \ngenerally below the strongest LGBM-dominant blend | \n
| 09–15 UTC restriction | \n€3.91M | \n
The exact leaderboard result depends on the submitted version and competition evaluation.
\nThe main conclusion was that a strong LightGBM model with a relatively small JAXGBM contribution was more robust than increasingly complicated trading logic.
\nIf the competition rewards trading PnL, conventional RMSE optimization is insufficient.
\nRandom cross-validation can give a misleading impression of robustness in electricity markets.
\nSeveral plausible physical features reduced performance.
\nRemoving an entire redundant family can sometimes improve generalization.
\nMatrix-based routing, carry rules and neural switches need extremely strong out-of-sample evidence.
\nThe 09–15 experiment was a particularly clear demonstration of regime dependence.
\nJAXGBM did not need to beat LightGBM independently to be useful in the final blend.
\nA large amount of experimentation ultimately led back toward a relatively compact production system.
\nThis project was developed with substantial use of OpenAI's ChatGPT as an AI-assisted research and engineering tool.
\nChatGPT was used for tasks including:
\nThe experimental direction, datasets, codebase, validation methodology, acceptance/rejection of experiments, and final submission decisions remained under my control.
\nI treated AI suggestions as hypotheses to test rather than as authoritative results.
\nThis distinction was particularly important during the final stages, where several plausible-looking ideas failed on the actual leaderboard.
\nThe repository contains the final training/inference implementation and supporting diagnostics.
\nThe exact environment may affect:
\nFor a reproducible run, the data sources and package versions should therefore be pinned where possible.
\nThe biggest lesson from this competition was not a particular model architecture.
\nIt was the importance of controlled experimentation.
\nMany ideas produced better-looking local metrics:
\nbut the strongest leaderboard behavior came from preserving a robust core model and changing only components supported by genuinely out-of-sample evidence.
\nThe final solution therefore favors:
\nchronological validation + economic optimization + physical electricity features + LightGBM/JAXGBM complementarity + disciplined rejection of unstable ideas.
", "comments": [], "comment_count": 0, "messages": [ { "id": 3527220, "postDate": "2026-09-23T07:20:32.613Z", "content": "This repository contains my solution for the Electricity Trading Challenge (ensimag-if-2025).
The task was approached primarily as an economic forecasting problem, rather than as a conventional regression problem. The objective was not simply to minimize RMSE/MAE, but to produce a directionally useful forecast whose errors translated into favorable trading PnL under the competition's fixed-position mechanics.
\nThe final pipeline combines:
\nThe final solution deliberately avoids several seemingly promising approaches that failed to transfer robustly to the leaderboard.
\nThe target is the German/DE-LU electricity price spread:
\nspread = imbalance_price - day_ahead_price\nThe competition converts the forecast direction into a fixed trading position.
\nConsequently, the important distinction is:
\nprediction accuracy\n ≠\ntrading profitability\nA model can improve RMSE while producing worse trading decisions, while a model with modest statistical correlation can produce substantially better PnL.
\nFor that reason, model selection throughout the project was increasingly centered on chronological validation PnL, while MAE, RMSE, Pearson correlation, Spearman correlation and directional hit rate were retained as diagnostics.
\nThe training timeline covers approximately:
\n2020-01-01 → 2024-09-08\nwith 15-minute observations.
\nThe feature pipeline combines several information sources.
\nExamples include:
\nThe model uses variables such as:
\nExamples include:
\nThe pipeline includes:
\nImportant historical spread/system variables include:
\nAll lagged features were constructed chronologically to avoid using future target information.
\nLeakage prevention was treated as a primary design constraint.
\nThe validation procedure uses chronological splits rather than random train/test splitting.
\nThe pipeline produces expanding-window out-of-fold predictions so that each validation observation is predicted using information available before that observation.
\nThis is particularly important for electricity markets because:
\nThe goal was therefore not:
\n\n\n\"How well can the model fit the historical dataset?\"
\n
but:
\n\n\n\"How well does a model trained on the past perform on a genuinely later period?\"
\n
The project went through many feature-engineering iterations.
\nAn important lesson was that more features did not necessarily mean a better leaderboard result.
\nSeveral large feature expansions were tested and subsequently removed when they failed to demonstrate stable out-of-sample improvement.
\nIn particular, the final solution does not include every experimental feature introduced during development.
\nThis was intentional.
\nThe pipeline initially constructs a relatively large feature set.
\nFeature reduction includes numerical sanity checks and correlation/importance-based pruning.
\nThe purpose is to remove:
\nHowever, economically meaningful features were protected where appropriate.
\nThe final feature set was substantially smaller than some of the experimental feature pools.
\nThis turned out to be preferable to continually adding engineered variables.
\nOne of the more useful experiments was to evaluate feature families economically, rather than judging every feature solely by generic model importance.
\nFeature families included concepts such as:
\nEach ablation was given its own Optuna search rather than comparing models using identical frozen hyperparameters.
\nThis matters because removing a feature family can change the optimal model complexity.
\nOne experiment produced the following result:
\nBaseline: €584,381 CV PnL\n\nGRID_RAMP removed:\n €476,167\n Δ -€108,214\n\nWIND_REGIME removed:\n €602,212\n Δ +€17,831\n\nGAS_MARKET removed:\n €557,930\n Δ -€26,451\nThe interpretation was not that every positive CV difference would necessarily improve the leaderboard.
\nInstead, it provided evidence for which economic information was potentially redundant.
\nThe final experiment therefore tested removal of the WIND_REGIME family while preserving the underlying physical variables used to construct other system features.
\nLightGBM became the strongest individual production model during the experiments.
\nThe optimization target was not ordinary squared-error loss alone.
\nOptuna searched model parameters using a chronological validation framework with PnL as the primary economic objective.
\nA representative strong parameter region was:
\nlearning_rate ≈ 0.0112\nnum_leaves ≈ 26\nmax_depth 8\nsubsample ≈ 0.668\ncolsample_bytree ≈ 0.629\nmin_child_samples 65\nreg_alpha ≈ 4.26\nreg_lambda ≈ 0.073\nThe exact optimum was allowed to vary with the feature set because fixing parameters while changing the feature pool would make the feature-ablation experiment less meaningful.
\nA custom JAX-based gradient-boosting model was developed as a complementary learner.
\nThe model was not simply introduced because it improved conventional regression metrics.
\nIts second head was designed around the economic objective.
\nThe production configuration used in the final experiments was:
\nJAXGBMRegressor(\n n_estimators=30,\n learning_rate=0.01,\n max_depth=6,\n batch_size=2000,\n num_thresholds=100,\n alpha=0.75,\n k=0.2\n)\nA key part of the development process was recognizing that increasing model complexity was not consistently beneficial.
\nDifferent chronological folds produced very different optimal stopping points.
\nFor example, some folds reached their best PnL after only a handful of trees, while another fold continued improving much longer.
\nThat behavior was one reason I ultimately avoided making another large JAXGBM complexity change immediately before the final submission.
\nThe final production prediction used a deliberately simple blend:
\n80% LightGBM\n20% JAXGBM\nThis consistently proved more useful than many of the more complicated routing schemes tested.
\nThe reason for retaining JAXGBM was not that it was always the strongest standalone model.
\nInstead, it provided a complementary prediction surface that sometimes improved the economic result when combined with LightGBM.
\nTCN and N-BEATS models were also investigated as residual/error detectors.
\nThe motivation was to determine whether neural sequence models could identify situations in which the primary ensemble was likely to make a poor prediction.
\nAlthough these models produced interesting diagnostics, they did not provide sufficiently stable leaderboard evidence to justify replacing the simpler production ensemble.
\nThey therefore remained experimental rather than becoming a mandatory component of the final trading signal.
\nAnother experiment classified predictions according to:
\nusing a 5×5 matrix.
\nThis revealed economically interesting regions.
\nFor example, one diagnostic produced cells with substantially different average PnL per trade, demonstrating that model confidence and reconstruction error were related to trading outcomes.
\nHowever, this approach introduced an important methodological danger:
\n\n\nselecting a matrix cell using realized target information can turn the analysis into an oracle.
\n
Therefore matrix/carry/pyramid logic was not used blindly in the final submission.
\nThe analysis was retained as a diagnostic rather than being treated as automatically deployable alpha.
\nA one-step carry/pyramid strategy was also investigated.
\nThe idea was to use the previous prediction and its regime classification to influence the next position.
\nThe experiments showed that this could generate attractive historical diagnostics.
\nHowever, because the matrix itself could be selected using realized future information, the apparent improvement was not sufficiently clean to justify deployment.
\nThis was another case where a complicated strategy looked promising in-sample but had weaker evidence than the simpler unrestricted ensemble.
\nA particularly interesting experiment tested restricting execution to selected UTC hours.
\nThe historical/OOT diagnostic initially looked very promising.
\nFor example:
\nAll hours:\nPnL/trade €21.99\nHit rate 55.08%\n\n09–15 UTC:\nPnL/trade €33.88\nHit rate 58.20%\nThis suggested a strong concentration of historical model edge during the middle of the day.
\nHowever, the actual leaderboard result provided a much stronger test.
\nThe 09–15 UTC submission executed only:
\n6,032 / 24,138 = 25%\nof possible orders and achieved approximately:
\n€3.91M\nThis was dramatically below the established champion.
\nThe time filter was therefore rejected.
\nThis experiment was particularly useful because it demonstrated why historical PnL concentration is not sufficient evidence for a hard execution rule in a non-stationary electricity market.
\nA further experiment introduced:
\nsolar_share_load\nwind_share_load\nlignite_share_load\nhard_coal_share_load\nThe motivation was to normalize generation against contemporaneous demand.
\nAlthough physically plausible, the resulting experiment produced weaker economic CV results than the stronger preceding configuration.
\nThe feature expansion was therefore not carried into the final last-run configuration.
\nThis reinforced an important principle from the project:
\n\n\nA physically sensible feature is not automatically a useful predictive feature.
\n
The strongest configuration was surprisingly conservative compared with many of the experimental variants.
\nThe final approach was approximately:
\nHistorical electricity data\n │\n ├── weather\n ├── renewable generation\n ├── load / net load\n ├── market variables\n ├── calendar / Fourier\n ├── spread lags\n └── forecast-error features\n │\n ▼\n Feature selection\n │\n ├──────────────┐\n ▼ ▼\n LightGBM JAXGBM\n │ │\n └──────┬───────┘\n ▼\n 80 / 20\n ensemble\n │\n ▼\n final direction\nThe final production logic deliberately remained simple.
\nNo hard UTC time filter.
\nNo aggressive pyramid.
\nNo oracle-derived matrix routing.
\nNo wholesale feature expansion.
\nDuring development, the strongest leaderboard submissions reached approximately the €18M level, substantially outperforming many of the individual experimental models.
\nRepresentative experiments included:
\n| Configuration | \nApprox. result | \n
|---|---|
| LGBM | \n€16.3M | \n
| JAXGBM | \n€11.1M | \n
| 80/20 LGBM + JAXGBM | \n€17.2M | \n
| ET | \n€14.4M | \n
| Complex ensemble variants | \ngenerally below the strongest LGBM-dominant blend | \n
| 09–15 UTC restriction | \n€3.91M | \n
The exact leaderboard result depends on the submitted version and competition evaluation.
\nThe main conclusion was that a strong LightGBM model with a relatively small JAXGBM contribution was more robust than increasingly complicated trading logic.
\nIf the competition rewards trading PnL, conventional RMSE optimization is insufficient.
\nRandom cross-validation can give a misleading impression of robustness in electricity markets.
\nSeveral plausible physical features reduced performance.
\nRemoving an entire redundant family can sometimes improve generalization.
\nMatrix-based routing, carry rules and neural switches need extremely strong out-of-sample evidence.
\nThe 09–15 experiment was a particularly clear demonstration of regime dependence.
\nJAXGBM did not need to beat LightGBM independently to be useful in the final blend.
\nA large amount of experimentation ultimately led back toward a relatively compact production system.
\nThis project was developed with substantial use of OpenAI's ChatGPT as an AI-assisted research and engineering tool.
\nChatGPT was used for tasks including:
\nThe experimental direction, datasets, codebase, validation methodology, acceptance/rejection of experiments, and final submission decisions remained under my control.
\nI treated AI suggestions as hypotheses to test rather than as authoritative results.
\nThis distinction was particularly important during the final stages, where several plausible-looking ideas failed on the actual leaderboard.
\nThe repository contains the final training/inference implementation and supporting diagnostics.
\nThe exact environment may affect:
\nFor a reproducible run, the data sources and package versions should therefore be pinned where possible.
\nThe biggest lesson from this competition was not a particular model architecture.
\nIt was the importance of controlled experimentation.
\nMany ideas produced better-looking local metrics:
\nbut the strongest leaderboard behavior came from preserving a robust core model and changing only components supported by genuinely out-of-sample evidence.
\nThe final solution therefore favors:
\nchronological validation + economic optimization + physical electricity features + LightGBM/JAXGBM complementarity + disciplined rejection of unstable ideas.
", "rawMarkdown": "# Electricity Trading Challenge — Solution\n\n## Overview\n\nThis repository contains my solution for the **Electricity Trading Challenge (`ensimag-if-2025`)**.\n\nThe task was approached primarily as an **economic forecasting problem**, rather than as a conventional regression problem. The objective was not simply to minimize RMSE/MAE, but to produce a directionally useful forecast whose errors translated into favorable trading PnL under the competition's fixed-position mechanics.\n\nThe final pipeline combines:\n\n* chronological feature engineering\n* German electricity-system fundamentals\n* weather and renewable-generation information\n* day-ahead market context\n* commodity variables\n* autoregressive spread features\n* Fourier/calendar features\n* automated feature selection\n* Optuna optimization using a PnL-oriented objective\n* LightGBM\n* a custom JAXGBM model with a PnL-oriented second head\n* chronological out-of-fold predictions\n* a final LightGBM/JAXGBM ensemble\n\nThe final solution deliberately avoids several seemingly promising approaches that failed to transfer robustly to the leaderboard.\n\n---\n\n## 1. Problem Formulation\n\nThe target is the German/DE-LU electricity price spread:\n\n```text\nspread = imbalance_price - day_ahead_price\n```\n\nThe competition converts the forecast direction into a fixed trading position.\n\nConsequently, the important distinction is:\n\n```text\nprediction accuracy\n ≠\ntrading profitability\n```\n\nA model can improve RMSE while producing worse trading decisions, while a model with modest statistical correlation can produce substantially better PnL.\n\nFor that reason, model selection throughout the project was increasingly centered on **chronological validation PnL**, while MAE, RMSE, Pearson correlation, Spearman correlation and directional hit rate were retained as diagnostics.\n\n---\n\n## 2. Data\n\nThe training timeline covers approximately:\n\n```text\n2020-01-01 → 2024-09-08\n```\n\nwith 15-minute observations.\n\nThe feature pipeline combines several information sources.\n\n### Electricity-system variables\n\nExamples include:\n\n* load\n* wind\n* solar\n* net load\n* net-load ramps\n* expected net load\n* expected net-load ramps\n* renewable ratios\n* ENTSO-E generation data\n* hydro generation\n* lignite generation\n* hard-coal generation\n* onshore/offshore wind\n\n### Weather\n\nThe model uses variables such as:\n\n* temperature\n* precipitation\n* cloud cover\n* wind speed\n* solar radiation\n* direct solar radiation\n* heating degree days\n* cooling degree days\n\n### Market / commodity variables\n\nExamples include:\n\n* natural gas\n* WTI\n* gas/oil relationships\n* spark-spread proxies\n\n### Temporal structure\n\nThe pipeline includes:\n\n* hour\n* minute\n* day of week\n* month\n* Fourier daily components\n* weekend/event-style calendar indicators\n\n### Autoregressive information\n\nImportant historical spread/system variables include:\n\n* 24-hour spread lag\n* 1-week spread lag\n* 24-hour rolling spread mean\n* 24-hour net-load lag\n* 1-week net-load lag\n* forecast-error lags\n* forecast-bias variables\n\nAll lagged features were constructed chronologically to avoid using future target information.\n\n---\n\n## 3. Leakage Prevention\n\nLeakage prevention was treated as a primary design constraint.\n\nThe validation procedure uses chronological splits rather than random train/test splitting.\n\nThe pipeline produces expanding-window out-of-fold predictions so that each validation observation is predicted using information available before that observation.\n\nThis is particularly important for electricity markets because:\n\n* seasonality is strong\n* weather regimes change\n* generation mixes change\n* market conditions are non-stationary\n* random CV can allow information from future regimes to influence model selection\n\nThe goal was therefore not:\n\n> \"How well can the model fit the historical dataset?\"\n\nbut:\n\n> \"How well does a model trained on the past perform on a genuinely later period?\"\n\n---\n\n# 4. Feature Engineering\n\nThe project went through many feature-engineering iterations.\n\nAn important lesson was that **more features did not necessarily mean a better leaderboard result**.\n\nSeveral large feature expansions were tested and subsequently removed when they failed to demonstrate stable out-of-sample improvement.\n\nIn particular, the final solution does not include every experimental feature introduced during development.\n\nThis was intentional.\n\n---\n\n## 5. Automated Feature Selection\n\nThe pipeline initially constructs a relatively large feature set.\n\nFeature reduction includes numerical sanity checks and correlation/importance-based pruning.\n\nThe purpose is to remove:\n\n* redundant variables\n* highly correlated duplicates\n* features with consistently weak model contribution\n\nHowever, economically meaningful features were protected where appropriate.\n\nThe final feature set was substantially smaller than some of the experimental feature pools.\n\nThis turned out to be preferable to continually adding engineered variables.\n\n---\n\n# 6. Economic Feature Ablation\n\nOne of the more useful experiments was to evaluate **feature families economically**, rather than judging every feature solely by generic model importance.\n\nFeature families included concepts such as:\n\n* GRID_RAMP\n* WIND_REGIME\n* GAS_MARKET\n* SPREAD_AR\n* DA_FORECAST_ERROR\n* WEATHER_DEMAND\n* COMMODITY\n* CALENDAR\n* FOURIER\n* RAW_SYSTEM_INPUTS\n\nEach ablation was given its own Optuna search rather than comparing models using identical frozen hyperparameters.\n\nThis matters because removing a feature family can change the optimal model complexity.\n\nOne experiment produced the following result:\n\n```text\nBaseline: €584,381 CV PnL\n\nGRID_RAMP removed:\n €476,167\n Δ -€108,214\n\nWIND_REGIME removed:\n €602,212\n Δ +€17,831\n\nGAS_MARKET removed:\n €557,930\n Δ -€26,451\n```\n\nThe interpretation was not that every positive CV difference would necessarily improve the leaderboard.\n\nInstead, it provided evidence for which economic information was potentially redundant.\n\nThe final experiment therefore tested removal of the WIND_REGIME family while preserving the underlying physical variables used to construct other system features.\n\n---\n\n# 7. LightGBM\n\nLightGBM became the strongest individual production model during the experiments.\n\nThe optimization target was not ordinary squared-error loss alone.\n\nOptuna searched model parameters using a chronological validation framework with **PnL as the primary economic objective**.\n\nA representative strong parameter region was:\n\n```text\nlearning_rate ≈ 0.0112\nnum_leaves ≈ 26\nmax_depth 8\nsubsample ≈ 0.668\ncolsample_bytree ≈ 0.629\nmin_child_samples 65\nreg_alpha ≈ 4.26\nreg_lambda ≈ 0.073\n```\n\nThe exact optimum was allowed to vary with the feature set because fixing parameters while changing the feature pool would make the feature-ablation experiment less meaningful.\n\n---\n\n# 8. JAXGBM\n\nA custom JAX-based gradient-boosting model was developed as a complementary learner.\n\nThe model was not simply introduced because it improved conventional regression metrics.\n\nIts second head was designed around the economic objective.\n\nThe production configuration used in the final experiments was:\n\n```python\nJAXGBMRegressor(\n n_estimators=30,\n learning_rate=0.01,\n max_depth=6,\n batch_size=2000,\n num_thresholds=100,\n alpha=0.75,\n k=0.2\n)\n```\n\nA key part of the development process was recognizing that increasing model complexity was not consistently beneficial.\n\nDifferent chronological folds produced very different optimal stopping points.\n\nFor example, some folds reached their best PnL after only a handful of trees, while another fold continued improving much longer.\n\nThat behavior was one reason I ultimately avoided making another large JAXGBM complexity change immediately before the final submission.\n\n---\n\n# 9. Ensemble\n\nThe final production prediction used a deliberately simple blend:\n\n```text\n80% LightGBM\n20% JAXGBM\n```\n\nThis consistently proved more useful than many of the more complicated routing schemes tested.\n\nThe reason for retaining JAXGBM was not that it was always the strongest standalone model.\n\nInstead, it provided a complementary prediction surface that sometimes improved the economic result when combined with LightGBM.\n\n---\n\n# 10. Neural Error Detectors\n\nTCN and N-BEATS models were also investigated as residual/error detectors.\n\nThe motivation was to determine whether neural sequence models could identify situations in which the primary ensemble was likely to make a poor prediction.\n\nAlthough these models produced interesting diagnostics, they did not provide sufficiently stable leaderboard evidence to justify replacing the simpler production ensemble.\n\nThey therefore remained experimental rather than becoming a mandatory component of the final trading signal.\n\n---\n\n# 11. PnL Matrix / Regime Analysis\n\nAnother experiment classified predictions according to:\n\n1. prediction magnitude\n2. reconstruction/error magnitude\n\nusing a 5×5 matrix.\n\nThis revealed economically interesting regions.\n\nFor example, one diagnostic produced cells with substantially different average PnL per trade, demonstrating that model confidence and reconstruction error were related to trading outcomes.\n\nHowever, this approach introduced an important methodological danger:\n\n> selecting a matrix cell using realized target information can turn the analysis into an oracle.\n\nTherefore matrix/carry/pyramid logic was not used blindly in the final submission.\n\nThe analysis was retained as a diagnostic rather than being treated as automatically deployable alpha.\n\n---\n\n# 12. Carry / Pyramid Experiments\n\nA one-step carry/pyramid strategy was also investigated.\n\nThe idea was to use the previous prediction and its regime classification to influence the next position.\n\nThe experiments showed that this could generate attractive historical diagnostics.\n\nHowever, because the matrix itself could be selected using realized future information, the apparent improvement was not sufficiently clean to justify deployment.\n\nThis was another case where a complicated strategy looked promising in-sample but had weaker evidence than the simpler unrestricted ensemble.\n\n---\n\n# 13. Time-of-Day Experiment\n\nA particularly interesting experiment tested restricting execution to selected UTC hours.\n\nThe historical/OOT diagnostic initially looked very promising.\n\nFor example:\n\n```text\nAll hours:\nPnL/trade €21.99\nHit rate 55.08%\n\n09–15 UTC:\nPnL/trade €33.88\nHit rate 58.20%\n```\n\nThis suggested a strong concentration of historical model edge during the middle of the day.\n\nHowever, the actual leaderboard result provided a much stronger test.\n\nThe 09–15 UTC submission executed only:\n\n```text\n6,032 / 24,138 = 25%\n```\n\nof possible orders and achieved approximately:\n\n```text\n€3.91M\n```\n\nThis was dramatically below the established champion.\n\nThe time filter was therefore rejected.\n\nThis experiment was particularly useful because it demonstrated why **historical PnL concentration is not sufficient evidence for a hard execution rule in a non-stationary electricity market**.\n\n---\n\n# 14. Generation-Share Experiment\n\nA further experiment introduced:\n\n```text\nsolar_share_load\nwind_share_load\nlignite_share_load\nhard_coal_share_load\n```\n\nThe motivation was to normalize generation against contemporaneous demand.\n\nAlthough physically plausible, the resulting experiment produced weaker economic CV results than the stronger preceding configuration.\n\nThe feature expansion was therefore not carried into the final last-run configuration.\n\nThis reinforced an important principle from the project:\n\n> A physically sensible feature is not automatically a useful predictive feature.\n\n---\n\n# 15. What Ultimately Worked\n\nThe strongest configuration was surprisingly conservative compared with many of the experimental variants.\n\nThe final approach was approximately:\n\n```text\nHistorical electricity data\n │\n ├── weather\n ├── renewable generation\n ├── load / net load\n ├── market variables\n ├── calendar / Fourier\n ├── spread lags\n └── forecast-error features\n │\n ▼\n Feature selection\n │\n ├──────────────┐\n ▼ ▼\n LightGBM JAXGBM\n │ │\n └──────┬───────┘\n ▼\n 80 / 20\n ensemble\n │\n ▼\n final direction\n```\n\nThe final production logic deliberately remained simple.\n\nNo hard UTC time filter.\n\nNo aggressive pyramid.\n\nNo oracle-derived matrix routing.\n\nNo wholesale feature expansion.\n\n---\n\n# 16. Results\n\nDuring development, the strongest leaderboard submissions reached approximately the **€18M level**, substantially outperforming many of the individual experimental models.\n\nRepresentative experiments included:\n\n| Configuration | Approx. result |\n| ------------------------- | ------------------------------------------------: |\n| LGBM | €16.3M |\n| JAXGBM | €11.1M |\n| 80/20 LGBM + JAXGBM | €17.2M |\n| ET | €14.4M |\n| Complex ensemble variants | generally below the strongest LGBM-dominant blend |\n| 09–15 UTC restriction | **€3.91M** |\n\nThe exact leaderboard result depends on the submitted version and competition evaluation.\n\nThe main conclusion was that **a strong LightGBM model with a relatively small JAXGBM contribution was more robust than increasingly complicated trading logic**.\n\n---\n\n# 17. Lessons Learned\n\n### 1. Optimize the metric that matters\n\nIf the competition rewards trading PnL, conventional RMSE optimization is insufficient.\n\n### 2. Chronology matters\n\nRandom cross-validation can give a misleading impression of robustness in electricity markets.\n\n### 3. More features are not necessarily better\n\nSeveral plausible physical features reduced performance.\n\n### 4. Economic feature-family ablation was more informative than blindly adding variables\n\nRemoving an entire redundant family can sometimes improve generalization.\n\n### 5. Complex routing is easy to overfit\n\nMatrix-based routing, carry rules and neural switches need extremely strong out-of-sample evidence.\n\n### 6. Historical hourly edge can disappear\n\nThe 09–15 experiment was a particularly clear demonstration of regime dependence.\n\n### 7. Ensemble diversity matters\n\nJAXGBM did not need to beat LightGBM independently to be useful in the final blend.\n\n### 8. The final solution should be simpler than the research process\n\nA large amount of experimentation ultimately led back toward a relatively compact production system.\n\n---\n\n# 18. AI-Assisted Development\n\nThis project was developed with substantial use of **OpenAI's ChatGPT** as an AI-assisted research and engineering tool.\n\nChatGPT was used for tasks including:\n\n* reviewing model architecture\n* inspecting and debugging Python/JAX code\n* proposing controlled experiments\n* analyzing validation and leaderboard diagnostics\n* checking potential leakage\n* comparing feature-engineering alternatives\n* identifying implementation bugs\n* designing ablation experiments\n* generating and reviewing code patches\n* interpreting PnL matrices and chronological validation results\n\nThe experimental direction, datasets, codebase, validation methodology, acceptance/rejection of experiments, and final submission decisions remained under my control.\n\nI treated AI suggestions as hypotheses to test rather than as authoritative results.\n\nThis distinction was particularly important during the final stages, where several plausible-looking ideas failed on the actual leaderboard.\n\n---\n\n# 19. Reproducibility\n\nThe repository contains the final training/inference implementation and supporting diagnostics.\n\nThe exact environment may affect:\n\n* LightGBM versions\n* JAX/XLA behavior\n* numerical results\n* external weather/commodity data availability\n* Prophet behavior\n\nFor a reproducible run, the data sources and package versions should therefore be pinned where possible.\n\n---\n\n## Final Takeaway\n\nThe biggest lesson from this competition was not a particular model architecture.\n\nIt was the importance of **controlled experimentation**.\n\nMany ideas produced better-looking local metrics:\n\n* additional feature families\n* generation ratios\n* time filters\n* neural routers\n* matrix policies\n* carry/pyramiding\n* increasingly complex ensembles\n\nbut the strongest leaderboard behavior came from preserving a robust core model and changing only components supported by genuinely out-of-sample evidence.\n\nThe final solution therefore favors:\n\n**chronological validation + economic optimization + physical electricity features + LightGBM/JAXGBM complementarity + disciplined rejection of unstable ideas.**\n" } ], "complete": true, "source": "live", "fetched_at": "2026-10-19T19:21:08Z", "competition_id": 107843, "competition_slug": "ensimag-if-2025", "team_id": 15571016, "solution_url": "/competitions/ensimag-if-2025/writeups/ensimag-extended-electricity-spread-energy-trader" }