# 3d place solution

Cyber-Physical Anomaly Detection for DER Systems — Winning Solution (Private LB: 0.91190 Public: 0.91149)
Summary
Our winning approach combined a
LightGBM + CatBoost ensemble
with domain-driven feature engineering tailored to the physics of Distributed Energy Resource (DER) systems. The core ideas were:
Aggressive column pruning
— profiling all 725 columns on a sample and discarding noise before training
Domain-specific feature engineering
— capacity utilization ratios, power factor anomalies, frequency deviations, control aggregations, and
per-device z-score normalization
to account for fundamentally different device scales
Gain-based feature selection
— reducing to the top 120 most informative features
Multi-seed LightGBM
(2 seeds × 5 folds) for stable probability estimates
CatBoost
as a secondary learner with ordered boosting for ensemble diversity
Optimized LGB/CB blending
with fine-grained F2 threshold tuning
Private leaderboard F2:
0.91190
.
1. Data Understanding
The dataset contains
~2.36M training rows
and
~1.01M test rows
with
725 columns
of cyber-physical features drawn from SunSpec Modbus register models of a DER simulator.
Key column families include:
common[0].*
— device metadata (serial number, model)
DERMeasureAC[0].*
— AC-side electrical measurements (W, VA, Var, PF, Hz, voltages, currents)
DERMeasureDC[0].*
— DC-side measurements
DERCapacity[0].*
— operational and rated capacity limits (WMax, VAMax, etc.)
DERCtlAC[0].*
— control setpoints
DEREnterService[0].*
— enter-service voltage/frequency limits
DERVoltVar[0].*
— Volt-Var curve parameters
The target
Label
is binary (0 = normal, 1 = anomaly) and the evaluation metric is
F2-score
, which weights recall 4× higher than precision — reflecting the real-world priority of catching attacks over avoiding false alarms.
Two physically distinct DER devices are present in the data, identifiable via
common[0].SN
:
Device A
(
SN-Three-Phase
): ~1.5M rows, strong anomaly signal
Device B
(
1100058974
): ~820K rows, much weaker signal with very different measurement scales (e.g., W std ~2,500 vs ~24,000)
This scale difference motivated per-device normalization features.
2. Preprocessing
2.1 Column Profiling & Filtering
Before loading the full dataset, we profiled every column on a 50K-row sample:
skip if: null_frac == 1.0          # entirely empty
         nunique <= 1               # constant
         null_frac > 0.95           # nearly empty
This reduced
725 → ~260 usable columns
, cutting I/O and memory roughly in half.
2.2 Label Encoding
All object/string columns were label-encoded. Unseen test categories were mapped to a synthetic
__MISSING__
class to prevent encoding failures.
2.3 Duplicate Column Removal
We hashed column values on a random 50K-row subsample and dropped any column whose hash collided with an earlier column — removing exact duplicates cheaply.
2.4 Memory Downcasting
Every
float64
was cast to
float32
and
int64
to the smallest integer dtype that could hold the data. This kept the full ~3.4M-row combined dataset in memory on a single machine.
3. Feature Engineering
We created
~20 lean, domain-grounded features
on top of the ~260 surviving raw columns:
Feature Group
Features Created
Rationale
Null count
null_count
— row-level missing-value count
Missing data is itself an attack signature
AC aggregations
ac_mean
,
ac_std
,
ac_min
,
ac_max
,
ac_range
,
ac_skew
Summarize the AC electrical state across all
DERMeasureAC[0].*
columns
DC aggregations
dc_mean
,
dc_std
DC-side summary
Power ratios
W_VA_ratio
,
W_Var_ratio
Detect power-factor manipulation
Capacity utilization
W_utilization
,
W_utilization_rtg
,
VA_utilization
,
Var_util_inj
,
Var_util_abs
Ratio of measured value to operational/rated capacity — flags out-of-bounds operation
Capacity override detection
W_cap_ratio
,
VA_cap_ratio
,
VarInj_cap_ratio
,
VarAbs_cap_ratio
+ their diffs
Operational-vs-rated capacity ratios flag tampered capacity settings
Frequency deviation
hz_dev_60
,
hz_dev_60_abs
US grid runs at 60 Hz — deviations indicate anomalies
Control aggregations
ctl_mean
,
ctl_std
Summarize
DERCtlAC[0].*
control commands
Per-device z-scores
Z-score of W, VA, Var, PF, Hz, WMax, VAMax grouped by device SN
The two devices operate at completely different scales; z-scoring removes this confound and lets the model compare within-device deviations
The per-device z-score features were a key addition. Without them, a model trained globally would learn thresholds dominated by Device A's larger magnitudes and struggle to detect anomalies in Device B's smaller-scale signals.
4. Feature Selection
After engineering, we had ~280 total features. Many were noise. We ran a
quick LightGBM importance scan
to prune them:
Train a fast model (500 rounds, LR=0.1) on an 80/20 split
Rank all features by gain importance
Keep only the
top 120 features
with importance > 0
This step improved generalization noticeably — removing low-importance features reduced overfitting and sped up the main training loop.
5. Modeling
5.1 LightGBM (Multi-Seed)
We trained LightGBM with
2 random seeds
(42, 2024), each running
5-fold stratified CV
for a total of 10 models. Predictions were averaged across all seeds and folds.
Hyperparameters:
objective:        binary
boosting:         gbdt
learning_rate:    0.03
num_leaves:       255
min_child_samples: 100
feature_fraction: 0.6
bagging_fraction: 0.75
bagging_freq:     5
reg_alpha:        0.5
reg_lambda:       3.0
min_gain_to_split: 0.02
path_smooth:      1.0
is_unbalance:     True
max boost rounds: 10,000 (early stopping at 200)
Multi-seed averaging smoothed out the decision boundary, which was particularly helpful near the classification threshold where small probability shifts flip predictions.
5.2 CatBoost
CatBoost was chosen for
ensemble diversity
— its ordered boosting and symmetric tree structure make it learn different decision boundaries from LightGBM on the same data.
Hyperparameters:
iterations:         5,000 (early stopping at 200)
learning_rate:      0.05
depth:              8
l2_leaf_reg:        5.0
random_strength:    1.0
bagging_temperature: 0.8
border_count:       254
auto_class_weights: Balanced
Categorical features (integer columns with <20 unique values) were passed natively to CatBoost via
cat_features
, letting it use its ordered target-encoding rather than plain label encoding.
CatBoost was trained with the same 5-fold stratified CV as LightGBM.
5.3 Ensemble Blending
We optimized the LGB/CB blend weight and threshold jointly on the OOF predictions:
for w in np.arange(0.40, 1.01, 0.05):        # LGB weight
    for thresh in np.arange(0.05, 0.60, 0.005):
        oof_blend = w * oof_lgb + (1 - w) * oof_cb
        f2 = fbeta_score(y, (oof_blend > thresh).astype(int), beta=2)
The optimizer selected the best
(w, threshold)
pair that maximized OOF F2. A fallback check ensured the blend was only used if it strictly beat LGB-only — if not, pure LGB predictions were used instead.
6. Threshold Optimization
Since F2 heavily penalizes missed anomalies, the probability-to-binary threshold was critical. We performed a fine-grained grid search over the final OOF predictions:
Range:
0.05 to 0.60
Step size:
0.001
The optimal threshold was applied to the blended test probabilities. We also generated
alternate submissions
at ±0.005 and ±0.01 from the optimal threshold to hedge against train/test distribution shift.
7. Full Pipeline
Raw Data (725 columns, ~3.4M rows)
    │
    ├── Column profiling on 50K sample
    ├── Drop useless columns (all-null, constant, >95% missing) → ~260 cols
    ├── Label-encode categoricals (handle unseen with __MISSING__)
    ├── Drop duplicate columns via hash comparison
    ├── Downcast dtypes (float64→32, int64→smallest)
    │
    ├── Feature Engineering (~20 new features)
    │     ├── Null count, AC/DC aggregations
    │     ├── Power ratios, capacity utilization
    │     ├── Capacity override ratios + diffs
    │     ├── Frequency deviation
    │     ├── Control aggregations
    │     └── Per-device z-score normalization
    │
    ├── Feature Selection (quick LGB → top 120 by gain)
    │
    ├── LightGBM × 2 seeds × 5 folds → averaged OOF & test probs
    ├── CatBoost × 5 folds → OOF & test probs
    │
    ├── Blend Optimization (LGB weight, threshold) on OOF
    │     └── Fallback: use LGB-only if blend doesn't improve F2
    │
    └── Apply threshold → binary submission
8. What Worked
Technique
Impact
Per-device z-scores
Normalized the vastly different scales between the two DER devices, letting the model learn device-agnostic anomaly patterns
Capacity utilization features
W/WMax and similar were consistently top-importance — directly capture out-of-bounds DER operation
Feature selection (top 120)
Removed noise features that hurt generalization
CatBoost blending
Ordered boosting captures patterns LGB misses; blend outperformed either model alone
Multi-seed averaging
Smoothed decision boundary, reduced variance
F2 threshold optimization
F2's recall emphasis means the optimal threshold is much lower than 0.5 — fine-grained search was essential
9. What Didn't Work / We Tried
Approach
Outcome
XGBoost in ensemble (v2)
Hurt LB score — too similar to LGB, added no diversity
Massive feature engineering (v2, ~60 features)
Overfit; lean feature set (v3/v4) generalized better
Single global model without device normalization
Good on Device A, struggled on Device B
More than 120 selected features
Marginal gains in CV, slight LB degradation
10. Key Takeaways
Profile before you load.
With 725 columns, blindly loading everything wastes memory and adds noise. A 50K-row profile pass let us cut the feature space in half before training.
Domain features beat volume.
Twenty carefully designed features (capacity utilization, frequency deviation, per-device z-scores) outperformed 60+ kitchen-sink features from earlier iterations.
Ensemble diversity matters more than ensemble size.
LGB + CatBoost (different boosting algorithms) was more effective than LGB + XGB (too similar).
Per-device normalization is critical in multi-population data.
The two DER devices had wildly different measurement scales. Z-scoring within device let the model focus on
relative
anomalies rather than
absolute
magnitudes.
F2 threshold tuning is not optional.
The default 0.5 threshold is far from optimal when recall is weighted 4× over precision. Fine-grained search over OOF predictions found the sweet spot.
Reproducibility
Environment:
Python 3.x, LightGBM, CatBoost, scikit-learn, pandas, NumPy
Runtime:
Single machine, CPU only. Full pipeline completes in ~30 minutes on a modern laptop. My computer is equipped with a  Ryzen 9 9950x 16 cores, so i trained quite fast locally.
Code:
Attachted the Kaggle Notebook :)
Private LB F2: 0.91190