# 8th place solution Long read ahead)))- many personal thoughts and emotions. All below is only my personal opinion and view, and when I say 'you should' or 'you must,' of course it's just my view. Thanks Kaggle team and CMI for such great competition - the data is exceptional and was so much fun to work with it! When you start working on competition, you should think from the start about several aspects: The code organization Versioning Experiments tracking Results repeatability Code organization Normally my ML experiments follow such a structure (that is enough and good for me). data/ : Contains all data files raw/ : Original, unmodified data files processed/ : Data after various processing steps base/ : Basic processed data (e.g., cleaned, filtered) fe/ : Feature-engineered data temp/ : Temporary data files that don't need to be version controlled checkpoits/ : To store weights and runs notebooks/ : Jupyter notebooks public/ : Notebooks that were publicly shared (of course you should check what other people did) eda/ : Data analysis (complex or single shot data checks) old/ : My TRASH folder where I drop copy of copy of copy notebooks models/ : Main folder where experiments live separated by models types lgb/ : lstm/ : xxx/ : src/ : Source code organized by functionality data/ : Code for data loading and processing fe/ : Feature engineering code models/ : Isolated code separated by model type lgb/ : LightGBM models lstm/ : LSTM model files pca/ : PCA model files temp/ : Temporary code files and test scripts utils/ : Utility functions and helper modules history/ : Historical project data and records / if no git in project doing src dumps here tb/ : Tensorboard tracking logs tests/ : Unit tests and integration tests I really advise you to use some version control (git). It will greatly help you to navigate through your thoughts during the competition 'marathon' (normally 3 months). The Plan So you downloaded the data - so did I. Did some minimal data checks and started thinking about the plan. The goal was GOLD - any other result would be a failure)))). We have 2 types of Data, so I will need good models for each branch IMU and TOF. IMU is easy just 7 features - so will start with it. It's a solo competition, so I need to Ensemble in the end with different architectures and model types. Will need NNs for sure - any gradient boosting will be not enough. Post-processing? Sure. Almost any competition needs PP. Also, F1 metric is good for hacking (some thresholds optimizations?). Inference time limit. Data is sequence per sequence and small - time should not be an issue. Models types - We have a temporal component (sequence) that should be used - lstm / gru? We have 18 classes but also orientation / phase, and the main target that could split by components 'Forehead/zone' - 'pull hairline/action' - needs to be used for sure. Just 9k sequences - Augmentation we need First steps I decided to start with lgbm (surprise)))) The previous CMI competition (that we won with [@trasibulo](https://www.kaggle.com/trasibulo) ) showed that gradient models could be competitive to NN in some tasks. Did few experiments that show interesting details. Model overfits A LOT! It's relatively easy to predict Gesture / Non gesture History window of 15 steps is enough for the model to catch up Leaky std means normalization by subject helps a lot - so there are some subject's device differences (calibration / demographics data?). ROC AUC very close to 1 - 0.998+ - so we need more data - augmentation for sure (can we generate our own data? or just augment what we have? or predict not on sequence level but on point-wise level N-predictions looking to the past?) Looked at a competition prediction process and tried to submit lgbm very basic predictions - Baseline v0 · mo ago . / . Ok it works. Think dude, think, - do I have something to improve in the process (not talking about the scores - was obvious that it's not even close to be good). I can use Different models for each target - 1 for multiclass and one for binary. [ADDed to plan] - Make two different models and use both if inference time and memory permit so (and if binary model would be better) LGBM still but with multiclass problems - I thought to switch to catboost or xgb, but it was obvious that gradient boosting is a dead-end. I kept lgbm flow to test model reaction to FE to decide what to keep and what to extend (was useless in the end as no matter how good the feature was, NN treats it differently, and almost every custom feature made NN results worse) Also, was thinking to do PCA / Autoencoder for gradient boosting but decided to switch to pure NN. TARGETs | What we are predicting? Ok, we have a baseline with almost no feature engineering; it's time to go deeeeeper. Let's check what our targets are. .drop_duplicates(subset=['sequence_id'])['gesture'].value_counts() - pull hairline - pinch skin phone - scratch - scratch - pull hair ear - pull hair - pull hair - pinch skin hello name in air air toward your face around in tray and pull out an object name leg knee/leg skin knee/leg skin from bottle/cup / : count, dtype: int64 Stop WHAT? Is each person doing exactly the same gestures? Above ear - pull hair - pinch skin - pull hair - pull hair - pull hairline - scratch - pinch skin - scratch phone air toward your face hello name in air from bottle/cup around in tray and pull out an object / knee/leg skin knee/leg skin name leg Above ear - pull hair - pinch skin - pull hair - pull hair - pull hairline - scratch - pinch skin - scratch phone air toward your face hello name in air from bottle/cup around in tray and pull out an object / knee/leg skin knee/leg skin name leg Ok - THIS is will be our Post-processing! - don't know what exactly and how, but here is the key! NN experiments Tried different NNs configurations (thanks to Junie it was fast). Decided to go with different 'branches' for each modality and customizable heads. Junie talks here: Our solution is a neural network tailored for gesture recognition from heterogeneous time‑series features. The architecture is assembled dynamically from a configuration: for each feature group (e.g., raw IMU, gravity‑removed signals, engineered features, ToF‑derived D maps), the model a dedicated encoder fixes its output for later fusion. The encoded features are concatenated passed through two independent fusion with modality dropout. The first fusion stream feeds the main classifier over all gesture non‑gesture categories, while the second stream feeds an auxiliary detector (gesture vs non‑gesture). Optionally, the second stream can also produce a small auxiliary head a lightweight regression head . Each active feature group uses a processor chosen configuration. Supported processors include temporal encoders for D sequences, a compact `dense` processor, several ToF D Only features with `use_feat=True` are Every encoder projects inputs to a fixed `output_dim`, these embeddings are concatenated fusion. This modular design allows easy ablations expansions of the feature set without changing the training loop. We use a unified fusion module with configurable width dropout, plus modality dropout that stochastically drops entire modality embeddings to improve robustness. fusion modules receive the same concatenated representation specialize during training: the first targets fine‑grained the second refines a clean gesture signal. The main head outputs logits over all classes Training couples the main objective with the auxiliary objective using a family of combined losses selected configuration with optional mixup support. When auxiliary heads are active, we a cross‑entropy term for the auxiliary target an MSE term for the regression head, with small weights. The loop supports per‑learning‑rate optional EMA of model weights, an optional dropout annealer to gradually adjust regularization. This setup stabilizes optimization across long sequences heterogeneous modalities. inference, the model encodes each active feature group, concatenates embeddings, runs fusion pathways to produce predictions. For validation, we compute metrics aligned with the competition goals: ROC AUC F1 for gesture vs non‑gesture - Modular encoders let us integrate signals architectures per modality without touching the core training loop. - Dual fusion streams decouple fine‑grained classification from robust gesture detection, improving stability overall performance. - Auxiliary heads provide targeted regularization richer supervision. - Modality dropout enhances resilience when some feature channels are missing noisy. For the code agent (Junie) I've created a special folder and some instructions. It will be his personal sandbox to play around and not 'make sh.t files' in other places. The directory is organized to track model experiments: : Contains experiment proposals in markdown format Each file represents a single experiment proposal Naming convention: where XXX is a sequential number : Contains experiments that are currently in progress Experiments moved here from when they are being implemented May include additional files related to the experiment implementation : Contains completed experiment results Final outcomes, metrics, and analysis of experiments Should reference the original proposal Each experiment proposal follows a standardized format with sections for: Objective Hypothesis Methodology Evaluation Metrics Expected Outcome Resources Required References The experiment workflow involves: Creating a proposal in the directory Moving it to during implementation Documenting results in the directory Using findings to inform future experiments In the end I came to such modalities and features (features are added / turned off / turned on in different models): = [ , , , , , , , , , , , , ] = [ , , , , , , , , ] = [ , , , , , , , , , , , , , ] = [c for c in list(base_df) if in c] = [ , , , ,, , ] = [ , , , , , , , , , , , ... ] = [ , , , , , , , ... ] = [ , , , , , , , ] FE and data cleaning I've tried different approaches like flipping hand / small rotation / TTA - spent too much time with it without success. Simple things were left as base: left handed flip 'acc_x' 'rot_y' 'rot_z'] | *-1 wierd 'SUBJ_045235', 'SUBJ_019262' where acc_y seems to be flipped - *-1 slightly improved cv FE Gravity removal - 2 versions Velocity Distance Angles and projections Magnitudes **2 / **3 all and just pairs Lags / Difs / Ratios No fancy staff here Added custom naming for consistency across different versions like 'raw__|rot_x__dif__|lag-1' - was simple to add new features and experiment Pause and Return 11 of August I had local validation 0.805- 0.815 for IMU data and stopped working on competition after 3 weeks. Completely stopped kaggling as I started new work at JetBrains and all time was dedicated to it (another reason was that I thought that my model is not good at all for gold zone). Friday night 29 August - good Portuguese wine in my glass, working day came to an end / 20.00 time / no hobby - of course let's check what I did for competition and review with clean eyes. WHAT!??? Error? Huge error on competition metric. I was experimenting on TTA, and my final metric was a bit weird (forgot to change back to normal))) Simplification example: f1 on mask = binary_predictions > 0.5 np.clip(np.argmax(values[mask], axis=1), 0, 7) + f1 on np.argmax(values, axis=1)<=7 / 2 Of course, it's not a competition metric - immediately changed to np.clip(np.argmax(.values, axis=1), 0, 8) and tadaaam I have 0.82-0.824. So I have a competitive single model - THM and ToF should give + 0.03 0.85 + ensembl - 0.86 + PP 0.87 - I can challenge the gold zone The config During the weekend I rewrote all my models and prepared unified config to start making ensemble ALWAYS use Tensorboard (or equivalent) - because overwise you'll not be able to make good comparisons. Unified config looks like this base_params = { : , : , : , : , : , : , : , : , : , : , : , : , : {}, : {}, : , : , : , : , : , : * , : , : , : , : , : , : , : , : , : , : , : , : , : , : , : {}, : { : , : [, , , , , ], : (, ), : , : , : (, ), : , : , : (, ), : ., : ., : , : , : ., : , : ., }, : , : , : , : , : , : , : { : { : , : , : , : , : { : { : , : [, ], : [, ], : , : , : , : }, : { : , : [, , ], : [, , ], : , : , : , : } } }, ... : { : , : , : , : [, , ], : , : , : , : }, : { : , : , : , : , : , : , }, : , } Such a config permitted changing params on the fly and track any changes and rerun model inference on kaggle with stable results. Good models were uploaded to the Kaggle dataset, and I made several submissions. Monday - Thursday I slowly trained IMU models as didn't 100% believe in success. 25 August night submission of 9 / 5 folded IMU models gave 0.838 on LB. Junie talks here: is a compact sequence encoder that combines convolutional feature extraction, temporal modeling, and attention-based pooling to produce a fixed-size representation per sequence. It takes an input tensor shaped (batch, time, features), extracts local temporal patterns with residual 1D CNNs, models longer-range dependencies with a bidirectional GRU, and compresses the sequence to a single vector via learnable attention before projecting to a fixed . Two residual CNN blocks () operate on the transposed signal : Block 1 adapts channel width from and uses a 3-layer or 2-layer 1D CNN core. Block 2 continues feature refinement with its own core. Each block includes: Channel attention via a Squeeze-and-Excitation to reweight feature maps. Residual addition, ReLU activation, downsampling, and dropout. This stage increases the receptive field and reduces sequence length (two pooling steps) while maintaining stable gradients through residual connections. The convolved output is transposed back to and fed to a bidirectional GRU: equals the final CNN channel count (from Block 2). , , , . A dropout layer follows the GRU to regularize temporal representations. A learnable consumes the GRU outputs and returns a weighted context vector , focusing the model on the most informative time steps. A final linear layer maps this vector to , yielding a per-sequence embedding suitable for downstream fusion and classification. Key arguments: : number of input channels per time step. : dictionary that configures channels, kernel sizes, pooling, dropout, SE reduction, and per block. Defaults provide a strong general-purpose setup: Block 1: , , . Block 2: , , . : GRU output dropout; each block has its own dropout. : final embedding size exported to the fusion module. Shapes: Input: After CNN blocks: downsampled in time, expanded in channels After BiGRU: Output: (sequence-level vector) Residual CNNs with SE attention capture multi-scale local dynamics and suppress noisy channels early. Max pooling reduces computational load and encourages invariance to minor timing shifts. The bidirectional GRU complements CNNs by modeling longer contexts beyond the convolutional receptive field. Attention pooling provides a flexible, data-driven way to summarize variable-length sequences. The fixed makes this processor plug-and-play within the multi-modal fusion framework, allowing different feature groups to contribute comparably sized embeddings. Last Days Ok, now I had and option or FULL IN or just STOP . Danila Savenkov (JB side, you may know him as [@daniel89](https://www.kaggle.com/daniel89) ) permitted me to spend 3 days on Kaggle (Friday / Monday / Tuesday + Weekend, of course). Decision made - FULL IN . What I need: GPU - lighting AI - 1 own gpu is not enough ToF model Postprocessing Better IMU model LUCK))) Carefull LB probing Not f.ck up with CV as had no chance to correct anything Thursday Night / Friday LB probing with ensemble setup - average / weighted average / number models / full data or 5 folds - 0.838 still Decided to go with many week 5 folded models - this was the only chance as had no option to probe separate single model on full data Let's make ToF model - 0.855 with IMU mix wih 1 ToF I need more ToF models and more IMU - started mass training Satruday I need post-process - NO way 1st place has such a difference in score with models only First PP version - 0.862 - a good start, but I need more More ToF models - 5 ready and validated - 0.871 with PP (no submissions left to probe without) Fixed EMA model - was cloning trainable param but not buffers - fix made EMA works Sunday Switch to EMA only models Changed LR scheduler Retrained IMU Models Made Massive probing (to make sure all components work as intended) Single IMU only no PP - (0.823 / 0.834) 6 IMU models no PP weighted average - (0.837 / 0.845) 6 IMU models with PP weighted average - (0.840 / 0.853) 6 IMU + old ToF no PP (0.851 / 0.868) 6 IMU + old ToF with PP (0.863 / 0.876) Did few changes to Post-processing but didn't manage to improve much the output my local CV showed +0.01 +/- 0.0015 std / max-min 0.006 Monday New ToF models no PP (0.853 / 0.872) with PP (0.868 / 0.883) Tuesday - Final day Timing should be very precise All new models scores should be ready till 22.00 London time to submit a final solution New Imu models 6 runs - 1.5H each on 3 gpu should be ready till 16 00 - 2h inference - 19 00 scores maximum New ToF models - 3h each run - will be able to make 3 models more - should be ready till 19 00 - 21 00 scores maximum 1h to prepare final mix with pp and submit till 22 00 CMI - Final 5.0 - Final candidate 2! Succeeded · Tue Sep 02 2025 21:58:40 GMT+0100 (0.860 / 0.877) CMI - Final 4.0 - Final Candidate 1! Succeeded · Tue Sep 02 2025 21:57:42 GMT+0100 (0.858 / 0.880) Done - choose Candidate 1 and previous best lb model - probably 0.858 would be enough for gold also but we will never know))) Models changes experiments features different CNN block - 'cnn', 'inception', 'gated_dilated', 'convnext', 'miniconformer', 'fft' conv layers output dims extrapolated window LR dropouts fusion blocks aux targets loss functions Had no option to test if my weights are right - probably a simple average mix with cv score cutoff would perform better. Mix was made with Dirichlet search with oof predictions (too risky if CV is wrong and not aligned with LB) - cutoff low weights and normalized back IMU Final metric (rounded+pruned): 0.843523 | 0.854 with pp TOF Final metric (rounded+pruned): 0.891330 | 0.903 with pp ################################# # Models Weights ################################# # IMU : np.float64(), : np.float64(), : np.float64(), : np.float64(), : np.float64(), : np.float64(), : np.float64(), : np.float64(), : np.float64(), : np.float64(), : np.float64(), : np.float64(), : np.float64() } # TOF {: np.float64(), : np.float64(), : np.float64(), : np.float64(), : np.float64(), : np.float64(), : np.float64(), : np.float64(), : np.float64(), : np.float64(), : np.float64(), : np.float64(), : np.float64(), : np.float64(), : np.float64(), : np.float64(), : np.float64(), : np.float64() } After words. I joined Kaggle almost 7 years ago. Never thought that I would be a triple Grandmaster - till today I had the shame even to call myself a GM as only competition title matters. The road to the last medal was long))) It's not the end - I'll compete (when will have more free time) and definitely will compete always when tabular data appears (not stock market)). So many good memories come to me: My first competitions - top 50 places when I believed that I could make some good models First Team with [@simpletonwang](https://www.kaggle.com/simpletonwang) and [@xkagjaeo05](https://www.kaggle.com/xkagjaeo05) (still friends btw) and how we missed our gold (due to bad time management) 1st place in IEEE Fraud competition - with one and only [@cdeotte](https://www.kaggle.com/cdeotte) [https://www.kaggle.com/competitions/ieee-fraud-detection/discussion/111284](https://www.kaggle.com/competitions/ieee-fraud-detection/discussion/111284) [https://www.kaggle.com/competitions/ieee-fraud-detection/writeups/fraudsquad-1st-place-solution-part-2](https://www.kaggle.com/competitions/ieee-fraud-detection/writeups/fraudsquad-1st-place-solution-part-2) Dream team with [@johnpateha](https://www.kaggle.com/johnpateha) [@headsortails](https://www.kaggle.com/headsortails) [@cdeotte](https://www.kaggle.com/cdeotte) and our fail on M5 competition (with guessing distribution on private test part) - [https://www.kaggle.com/competitions/m5-forecasting-accuracy/discussion/163621](https://www.kaggle.com/competitions/m5-forecasting-accuracy/discussion/163621) AMEX competition - how kaggle noobs took 2nd place))) 1st place with [@dott1718](https://www.kaggle.com/dott1718) on AMP Parkinson's Disease Progression Prediction (when I had a privilege to work with a pure genius) [@trasibuloon](https://www.kaggle.com/trasibuloon) first CMI competition and how we were trying to solve metric puzzle)) Emotional discussions / I would call them FIGHTS)))) with [@bakeryproducts](https://www.kaggle.com/bakeryproducts) (we met and worked together because of Kaggle), who taught me A LOT! I found so many friends because of Kaggle - not only during competitions but also social networks, meetups, conferences! Without you guys - your mentoring, long conversations, your example, I wouldn't achieve all these. And of course the Kaggle community - the support, knowledge sharing, gentle teaching - there is no other place like this. THANK YOU!