Long read ahead)))- many personal thoughts and emotions.
\nAll below is only my personal opinion and view, and when I say 'you should' or 'you must,' of course it's just my view.
Thanks Kaggle team and CMI for such great competition - the data is exceptional and was so much fun to work with it!
\nWhen you start working on competition, you should think from the start about several aspects:
\nNormally my ML experiments follow such a structure (that is enough and good for me).
\ndata/: Contains all data files
raw/: Original, unmodified data filesprocessed/: Data after various processing stepsbase/: Basic processed data (e.g., cleaned, filtered)fe/: Feature-engineered datatemp/: Temporary data files that don't need to be version controlledcheckpoits/: To store weights and runs
notebooks/: Jupyter notebooks
public/: Notebooks that were publicly shared (of course you should check what other people did)eda/: Data analysis (complex or single shot data checks)old/: My TRASH folder where I drop copy of copy of copy notebooksmodels/: Main folder where experiments live separated by models typeslgb/:lstm/:xxx/:src/: Source code organized by functionality
data/: Code for data loading and processingfe/: Feature engineering codemodels/: Isolated code separated by model typelgb/: LightGBM modelslstm/: LSTM model filespca/: PCA model filestemp/: Temporary code files and test scriptsutils/: Utility functions and helper moduleshistory/: Historical project data and records / if no git in project doing src dumps here
tb/: Tensorboard tracking logs
tests/: Unit tests and integration tests
I really advise you to use some version control (git). It will greatly help you to navigate through your thoughts during the competition 'marathon' (normally 3 months).
\nSo you downloaded the data - so did I. Did some minimal data checks and started thinking about the plan.
\nThe goal was GOLD - any other result would be a failure)))).
I decided to start with lgbm (surprise)))) The previous CMI competition (that we won with @trasibulo) showed that gradient models could be competitive to NN in some tasks.
\nDid few experiments that show interesting details.
Looked at a competition prediction process and tried to submit lgbm very basic predictions
\n - Baseline v0\n · mo ago \n. / .\nOk it works. Think dude, think, - do I have something to improve in the process (not talking about the scores - was obvious that it's not even close to be good).
\nI can use Different models for each target - 1 for multiclass and one for binary.
LGBM still but with multiclass problems - I thought to switch to catboost or xgb, but it was obvious that gradient boosting is a dead-end.
\nI kept lgbm flow to test model reaction to FE to decide what to keep and what to extend (was useless in the end as no matter how good the feature was, NN treats it differently, and
\nalmost every custom feature made NN results worse)
Also, was thinking to do PCA / Autoencoder for gradient boosting but decided to switch to pure NN.
\nOk, we have a baseline with almost no feature engineering; it's time to go deeeeeper.
\nLet's check what our targets are.
\n.drop_duplicates(subset=['sequence_id'])['gesture'].value_counts()\n\n - pull hairline \n - pinch skin \n phone \n - scratch \n - scratch \n - pull hair \n ear - pull hair \n - pull hair \n - pinch skin \n hello \n name in air \n air toward your face \n around in tray and pull out an object \n name leg \n knee/leg skin \n knee/leg skin \n from bottle/cup \n / \n: count, dtype: int64\nStop WHAT? Is each person doing exactly the same gestures?
\n Above ear - pull hair \n - pinch skin \n - pull hair \n - pull hair \n - pull hairline \n - scratch \n - pinch skin \n - scratch \n phone \n air toward your face \n hello \n name in air \n from bottle/cup \n around in tray and pull out an object \n / \n knee/leg skin \n knee/leg skin \n name leg \n Above ear - pull hair \n - pinch skin \n - pull hair \n - pull hair \n - pull hairline \n - scratch \n - pinch skin \n - scratch \n phone \n air toward your face \n hello \n name in air \n from bottle/cup \n around in tray and pull out an object \n / \n knee/leg skin \n knee/leg skin \n name leg \nOk - THIS is will be our Post-processing! - don't know what exactly and how, but here is the key!
\nTried different NNs configurations (thanks to Junie it was fast). Decided to go with different 'branches' for each modality and customizable heads.
\nJunie talks here:
\n\n\nOur solution is a neural network tailored for gesture recognition from heterogeneous time‑series features. The architecture is assembled dynamically from a configuration: for each feature group (e.g., raw IMU, gravity‑removed signals, engineered features, ToF‑derived D maps), the model a dedicated encoder fixes its\noutput for later fusion. The encoded features are concatenated passed through two independent fusion with modality dropout. The first fusion stream feeds the main classifier over all gesture non‑gesture categories, while the second stream feeds an auxiliary detector (gesture vs non‑gesture). Optionally, the second stream can also produce a small auxiliary head a lightweight regression head .\n\n\n\nEach active feature group uses a processor chosen configuration. Supported processors include temporal encoders for D sequences, a compact `dense` processor, several ToF D Only features with `use_feat=True` are Every encoder projects inputs to a fixed `output_dim`, these embeddings are concatenated fusion. This modular design allows easy ablations expansions of the feature set without changing the training loop.\n\n\n\nWe use a unified fusion module with configurable width dropout, plus modality dropout that stochastically drops entire modality embeddings to improve robustness. fusion modules receive the same concatenated representation specialize during training: the first targets fine‑grained the second refines a clean gesture signal. The main head outputs logits over all classes\n\n\n\nTraining couples the main objective with the auxiliary objective using a family of combined losses selected configuration with optional mixup support. When auxiliary heads are active, we a cross‑entropy term for the auxiliary target an MSE term for the regression head, with small weights. The loop supports per‑learning‑rate optional EMA of model weights, an optional dropout annealer to gradually adjust regularization. This setup stabilizes optimization across long sequences heterogeneous modalities.\n\n\n\n inference, the model encodes each active feature group, concatenates embeddings, runs fusion pathways to produce predictions. For validation, we compute metrics aligned with the competition goals: ROC AUC F1 for gesture vs non‑gesture\n\n\n\n- Modular encoders let us integrate signals architectures per modality without touching the core training loop.\n- Dual fusion streams decouple fine‑grained classification from robust gesture detection, improving stability overall performance.\n- Auxiliary heads provide targeted regularization richer supervision.\n- Modality dropout enhances resilience when some feature channels are missing noisy.\nFor the code agent (Junie) I've created a special folder and some instructions. It will be his personal sandbox to play around and not 'make sh.t files' in other places.
\n\n\nThe directory is organized to track model experiments:\n\n : Contains experiment proposals in markdown format\n Each file represents a single experiment proposal\n Naming convention: where XXX is a sequential number\n\n : Contains experiments that are currently in progress\n Experiments moved here from when they are being implemented\n May include additional files related to the experiment implementation\n\n : Contains completed experiment results\n Final outcomes, metrics, and analysis of experiments\n Should reference the original proposal\n\nEach experiment proposal follows a standardized format with sections for:\n\n Objective\n Hypothesis\n Methodology\n Evaluation Metrics\n Expected Outcome\n Resources Required\n References\n\nThe experiment workflow involves:\n\n Creating a proposal in the directory\n Moving it to during implementation\n Documenting results in the directory\n Using findings to inform future experiments\nIn the end I came to such modalities and features (features are added / turned off / turned on in different models):
\n\n = [\n , , , ,\n , , , ,\n , , , ,\n]\n\n\n = [\n , , , ,\n , , , ,\n]\n\n\n = [\n , , , , ,\n , , , ,\n , , , , \n]\n\n\n = [c for c in list(base_df) if in c]\n\n\n = [\n , , ,\n ,, ,\n]\n\n = [\n , , , ,\n , ,\n , , ,\n , ,\n ...\n]\n\n\n = [\n , , ,\n , , , ,\n ...\n]\n\n\n\n = [\n , , , , , , ,\n]\nI've tried different approaches like flipping hand / small rotation / TTA - spent too much time with it without success.
\nSimple things were left as base:
FE
\nNo fancy staff here
\n11 of August I had local validation 0.805- 0.815 for IMU data and stopped working on competition after 3 weeks. Completely stopped kaggling as I started new work at JetBrains and all time was dedicated to it (another reason was that I thought that my model is not good at all for gold zone).
\nFriday night 29 August - good Portuguese wine in my glass, working day came to an end / 20.00 time / no hobby - of course let's check what I did for competition and review with clean eyes.
\nWHAT!??? Error? Huge error on competition metric. I was experimenting on TTA, and my final metric was a bit weird (forgot to change back to normal)))
\nSimplification example:
\nf1 on
\nmask = binary_predictions > 0.5
\nnp.clip(np.argmax(values[mask], axis=1), 0, 7)
\n+
\nf1 on
\nnp.argmax(values, axis=1)<=7
/ 2
\nOf course, it's not a competition metric - immediately changed to np.clip(np.argmax(.values, axis=1), 0, 8)
\nand tadaaam I have 0.82-0.824.
\nSo I have a competitive single model - THM and ToF should give + 0.03
\n0.85 + ensembl - 0.86 + PP 0.87 - I can challenge the gold zone
During the weekend I rewrote all my models and prepared unified config to start making ensemble
\nALWAYS use Tensorboard (or equivalent) - because overwise you'll not be able to make good comparisons.
\nUnified config looks like this
\nbase_params = {\n : ,\n\n \n : ,\n : , \n : ,\n : ,\n : , \n : , \n : ,\n\n : , \n : ,\n\n : , \n : ,\n\n : {}, \n : {}, \n : ,\n : ,\n : ,\n : ,\n\n \n : ,\n : * ,\n\n \n : , \n\n \n : , \n : , \n : ,\n : ,\n\n \n : ,\n : , \n : ,\n : ,\n\n \n : ,\n : ,\n : ,\n : ,\n : ,\n : {},\n : {\n : ,\n : [, , , , , ],\n : (, ),\n : ,\n : ,\n : (, ),\n : ,\n : ,\n : (, ),\n : .,\n : .,\n : ,\n : ,\n : .,\n : ,\n : .,\n },\n\n \n : ,\n\n \n : ,\n : ,\n\n \n : ,\n : ,\n : ,\n\n : {\n : {\n : ,\n : ,\n : ,\n : ,\n : {\n : {\n : ,\n : [, ],\n : [, ],\n : ,\n : ,\n : ,\n : \n },\n : {\n : ,\n : [, , ],\n : [, , ],\n : ,\n : ,\n : ,\n : \n }\n }\n },\n ...\n : {\n : ,\n : ,\n : ,\n : [, , ],\n : ,\n : ,\n : ,\n : \n },\n : {\n : ,\n : ,\n : ,\n : ,\n : ,\n : ,\n },\n\n \n : ,\n}\nSuch a config permitted changing params on the fly and track any changes and rerun model inference on kaggle with stable results.
\nGood models were uploaded to the Kaggle dataset, and I made several submissions. Monday - Thursday I slowly trained IMU models as didn't 100% believe in success.
\n25 August night submission of 9 / 5 folded IMU models gave 0.838 on LB.
Junie talks here:
\n\n\n is a compact sequence encoder that combines convolutional feature extraction, temporal modeling, and attention-based pooling to produce a fixed-size representation per sequence. It takes an input tensor shaped (batch, time, features), extracts local temporal patterns with residual 1D CNNs, models longer-range dependencies with a bidirectional GRU, and compresses the sequence to a single vector via learnable attention before projecting to a fixed .\n\n\n Two residual CNN blocks () operate on the transposed signal :\n Block 1 adapts channel width from and uses a 3-layer or 2-layer 1D CNN core.\n Block 2 continues feature refinement with its own core.\n Each block includes:\n Channel attention via a Squeeze-and-Excitation to reweight feature maps.\n Residual addition, ReLU activation, downsampling, and dropout.\n This stage increases the receptive field and reduces sequence length (two pooling steps) while maintaining stable gradients through residual connections.\n\n\n The convolved output is transposed back to and fed to a bidirectional GRU:\n equals the final CNN channel count (from Block 2).\n , , , .\n A dropout layer follows the GRU to regularize temporal representations.\n\n\n A learnable consumes the GRU outputs and returns a weighted context vector , focusing the model on the most informative time steps.\n A final linear layer maps this vector to , yielding a per-sequence embedding suitable for downstream fusion and classification.\n\n\n Key arguments:\n : number of input channels per time step.\n : dictionary that configures channels, kernel sizes, pooling, dropout, SE reduction, and per block. Defaults provide a strong general-purpose setup:\n Block 1: , , .\n Block 2: , , .\n : GRU output dropout; each block has its own dropout.\n : final embedding size exported to the fusion module.\n Shapes:\n Input: \n After CNN blocks: downsampled in time, expanded in channels\n After BiGRU: \n Output: (sequence-level vector)\n\n\n Residual CNNs with SE attention capture multi-scale local dynamics and suppress noisy channels early.\n Max pooling reduces computational load and encourages invariance to minor timing shifts.\n The bidirectional GRU complements CNNs by modeling longer contexts beyond the convolutional receptive field.\n Attention pooling provides a flexible, data-driven way to summarize variable-length sequences.\n The fixed makes this processor plug-and-play within the multi-modal fusion framework, allowing different feature groups to contribute comparably sized embeddings.\nOk, now I had and option or FULL IN or just STOP. Danila Savenkov (JB side, you may know him as @daniel89) permitted me to spend 3 days on Kaggle (Friday / Monday / Tuesday + Weekend, of course).
\nWhat I need:
\nSwitch to EMA only models
Changed LR scheduler
Retrained IMU Models
Made Massive probing (to make sure all components work as intended)
Single IMU only no PP - (0.823 / 0.834)
6 IMU models no PP weighted average - (0.837 / 0.845)
6 IMU models with PP weighted average - (0.840 / 0.853)
6 IMU + old ToF no PP (0.851 / 0.868)
6 IMU + old ToF with PP (0.863 / 0.876)
Did few changes to Post-processing but didn't manage to improve much the output my local CV showed +0.01 +/- 0.0015 std / max-min 0.006
CMI - Final 5.0 - Final candidate 2!
\nSucceeded · Tue Sep 02 2025 21:58:40 GMT+0100 (0.860 / 0.877)
CMI - Final 4.0 - Final Candidate 1!
\nSucceeded · Tue Sep 02 2025 21:57:42 GMT+0100 (0.858 / 0.880)
Had no option to test if my weights are right - probably a simple average mix with cv score cutoff would perform better.
\nMix was made with Dirichlet search with oof predictions (too risky if CV is wrong and not aligned with LB) - cutoff low weights and normalized back
\n#################################\n# Models Weights\n#################################\n\n# IMU\n: np.float64(),\n : np.float64(),\n : np.float64(),\n : np.float64(),\n : np.float64(),\n : np.float64(),\n : np.float64(),\n : np.float64(),\n : np.float64(),\n : np.float64(),\n : np.float64(),\n : np.float64(),\n : np.float64()\n}\n\n# TOF\n{: np.float64(),\n : np.float64(),\n : np.float64(),\n : np.float64(),\n : np.float64(),\n : np.float64(),\n : np.float64(),\n : np.float64(),\n : np.float64(),\n : np.float64(),\n : np.float64(),\n : np.float64(),\n : np.float64(),\n : np.float64(),\n : np.float64(),\n : np.float64(),\n : np.float64(),\n : np.float64()\n}\nI joined Kaggle almost 7 years ago. Never thought that I would be a triple Grandmaster - till today I had the shame even to call myself a GM as only competition title matters.
\nThe road to the last medal was long))) It's not the end - I'll compete (when will have more free time) and definitely will compete always when tabular data appears (not stock market)).
\nSo many good memories come to me:
Without you guys - your mentoring, long conversations, your example, I wouldn't achieve all these.
\nAnd of course the Kaggle community - the support, knowledge sharing, gentle teaching - there is no other place like this.
THANK YOU!
", "messages": [ { "id": "3281533", "postDate": "09/04/2025 13:32:50", "content": "Long read ahead)))- many personal thoughts and emotions.
\nAll below is only my personal opinion and view, and when I say 'you should' or 'you must,' of course it's just my view.
Thanks Kaggle team and CMI for such great competition - the data is exceptional and was so much fun to work with it!
\nWhen you start working on competition, you should think from the start about several aspects:
\nNormally my ML experiments follow such a structure (that is enough and good for me).
\ndata/: Contains all data files
raw/: Original, unmodified data filesprocessed/: Data after various processing stepsbase/: Basic processed data (e.g., cleaned, filtered)fe/: Feature-engineered datatemp/: Temporary data files that don't need to be version controlledcheckpoits/: To store weights and runs
notebooks/: Jupyter notebooks
public/: Notebooks that were publicly shared (of course you should check what other people did)eda/: Data analysis (complex or single shot data checks)old/: My TRASH folder where I drop copy of copy of copy notebooksmodels/: Main folder where experiments live separated by models typeslgb/:lstm/:xxx/:src/: Source code organized by functionality
data/: Code for data loading and processingfe/: Feature engineering codemodels/: Isolated code separated by model typelgb/: LightGBM modelslstm/: LSTM model filespca/: PCA model filestemp/: Temporary code files and test scriptsutils/: Utility functions and helper moduleshistory/: Historical project data and records / if no git in project doing src dumps here
tb/: Tensorboard tracking logs
tests/: Unit tests and integration tests
I really advise you to use some version control (git). It will greatly help you to navigate through your thoughts during the competition 'marathon' (normally 3 months).
\nSo you downloaded the data - so did I. Did some minimal data checks and started thinking about the plan.
\nThe goal was GOLD - any other result would be a failure)))).
I decided to start with lgbm (surprise)))) The previous CMI competition (that we won with @trasibulo) showed that gradient models could be competitive to NN in some tasks.
\nDid few experiments that show interesting details.
Looked at a competition prediction process and tried to submit lgbm very basic predictions
\n - Baseline v0\n · mo ago \n. / .\nOk it works. Think dude, think, - do I have something to improve in the process (not talking about the scores - was obvious that it's not even close to be good).
\nI can use Different models for each target - 1 for multiclass and one for binary.
LGBM still but with multiclass problems - I thought to switch to catboost or xgb, but it was obvious that gradient boosting is a dead-end.
\nI kept lgbm flow to test model reaction to FE to decide what to keep and what to extend (was useless in the end as no matter how good the feature was, NN treats it differently, and
\nalmost every custom feature made NN results worse)
Also, was thinking to do PCA / Autoencoder for gradient boosting but decided to switch to pure NN.
\nOk, we have a baseline with almost no feature engineering; it's time to go deeeeeper.
\nLet's check what our targets are.
\n.drop_duplicates(subset=['sequence_id'])['gesture'].value_counts()\n\n - pull hairline \n - pinch skin \n phone \n - scratch \n - scratch \n - pull hair \n ear - pull hair \n - pull hair \n - pinch skin \n hello \n name in air \n air toward your face \n around in tray and pull out an object \n name leg \n knee/leg skin \n knee/leg skin \n from bottle/cup \n / \n: count, dtype: int64\nStop WHAT? Is each person doing exactly the same gestures?
\n Above ear - pull hair \n - pinch skin \n - pull hair \n - pull hair \n - pull hairline \n - scratch \n - pinch skin \n - scratch \n phone \n air toward your face \n hello \n name in air \n from bottle/cup \n around in tray and pull out an object \n / \n knee/leg skin \n knee/leg skin \n name leg \n Above ear - pull hair \n - pinch skin \n - pull hair \n - pull hair \n - pull hairline \n - scratch \n - pinch skin \n - scratch \n phone \n air toward your face \n hello \n name in air \n from bottle/cup \n around in tray and pull out an object \n / \n knee/leg skin \n knee/leg skin \n name leg \nOk - THIS is will be our Post-processing! - don't know what exactly and how, but here is the key!
\nTried different NNs configurations (thanks to Junie it was fast). Decided to go with different 'branches' for each modality and customizable heads.
\nJunie talks here:
\n\n\nOur solution is a neural network tailored for gesture recognition from heterogeneous time‑series features. The architecture is assembled dynamically from a configuration: for each feature group (e.g., raw IMU, gravity‑removed signals, engineered features, ToF‑derived D maps), the model a dedicated encoder fixes its\noutput for later fusion. The encoded features are concatenated passed through two independent fusion with modality dropout. The first fusion stream feeds the main classifier over all gesture non‑gesture categories, while the second stream feeds an auxiliary detector (gesture vs non‑gesture). Optionally, the second stream can also produce a small auxiliary head a lightweight regression head .\n\n\n\nEach active feature group uses a processor chosen configuration. Supported processors include temporal encoders for D sequences, a compact `dense` processor, several ToF D Only features with `use_feat=True` are Every encoder projects inputs to a fixed `output_dim`, these embeddings are concatenated fusion. This modular design allows easy ablations expansions of the feature set without changing the training loop.\n\n\n\nWe use a unified fusion module with configurable width dropout, plus modality dropout that stochastically drops entire modality embeddings to improve robustness. fusion modules receive the same concatenated representation specialize during training: the first targets fine‑grained the second refines a clean gesture signal. The main head outputs logits over all classes\n\n\n\nTraining couples the main objective with the auxiliary objective using a family of combined losses selected configuration with optional mixup support. When auxiliary heads are active, we a cross‑entropy term for the auxiliary target an MSE term for the regression head, with small weights. The loop supports per‑learning‑rate optional EMA of model weights, an optional dropout annealer to gradually adjust regularization. This setup stabilizes optimization across long sequences heterogeneous modalities.\n\n\n\n inference, the model encodes each active feature group, concatenates embeddings, runs fusion pathways to produce predictions. For validation, we compute metrics aligned with the competition goals: ROC AUC F1 for gesture vs non‑gesture\n\n\n\n- Modular encoders let us integrate signals architectures per modality without touching the core training loop.\n- Dual fusion streams decouple fine‑grained classification from robust gesture detection, improving stability overall performance.\n- Auxiliary heads provide targeted regularization richer supervision.\n- Modality dropout enhances resilience when some feature channels are missing noisy.\nFor the code agent (Junie) I've created a special folder and some instructions. It will be his personal sandbox to play around and not 'make sh.t files' in other places.
\n\n\nThe directory is organized to track model experiments:\n\n : Contains experiment proposals in markdown format\n Each file represents a single experiment proposal\n Naming convention: where XXX is a sequential number\n\n : Contains experiments that are currently in progress\n Experiments moved here from when they are being implemented\n May include additional files related to the experiment implementation\n\n : Contains completed experiment results\n Final outcomes, metrics, and analysis of experiments\n Should reference the original proposal\n\nEach experiment proposal follows a standardized format with sections for:\n\n Objective\n Hypothesis\n Methodology\n Evaluation Metrics\n Expected Outcome\n Resources Required\n References\n\nThe experiment workflow involves:\n\n Creating a proposal in the directory\n Moving it to during implementation\n Documenting results in the directory\n Using findings to inform future experiments\nIn the end I came to such modalities and features (features are added / turned off / turned on in different models):
\n\n = [\n , , , ,\n , , , ,\n , , , ,\n]\n\n\n = [\n , , , ,\n , , , ,\n]\n\n\n = [\n , , , , ,\n , , , ,\n , , , , \n]\n\n\n = [c for c in list(base_df) if in c]\n\n\n = [\n , , ,\n ,, ,\n]\n\n = [\n , , , ,\n , ,\n , , ,\n , ,\n ...\n]\n\n\n = [\n , , ,\n , , , ,\n ...\n]\n\n\n\n = [\n , , , , , , ,\n]\nI've tried different approaches like flipping hand / small rotation / TTA - spent too much time with it without success.
\nSimple things were left as base:
FE
\nNo fancy staff here
\n11 of August I had local validation 0.805- 0.815 for IMU data and stopped working on competition after 3 weeks. Completely stopped kaggling as I started new work at JetBrains and all time was dedicated to it (another reason was that I thought that my model is not good at all for gold zone).
\nFriday night 29 August - good Portuguese wine in my glass, working day came to an end / 20.00 time / no hobby - of course let's check what I did for competition and review with clean eyes.
\nWHAT!??? Error? Huge error on competition metric. I was experimenting on TTA, and my final metric was a bit weird (forgot to change back to normal)))
\nSimplification example:
\nf1 on
\nmask = binary_predictions > 0.5
\nnp.clip(np.argmax(values[mask], axis=1), 0, 7)
\n+
\nf1 on
\nnp.argmax(values, axis=1)<=7
/ 2
\nOf course, it's not a competition metric - immediately changed to np.clip(np.argmax(.values, axis=1), 0, 8)
\nand tadaaam I have 0.82-0.824.
\nSo I have a competitive single model - THM and ToF should give + 0.03
\n0.85 + ensembl - 0.86 + PP 0.87 - I can challenge the gold zone
During the weekend I rewrote all my models and prepared unified config to start making ensemble
\nALWAYS use Tensorboard (or equivalent) - because overwise you'll not be able to make good comparisons.
\nUnified config looks like this
\nbase_params = {\n : ,\n\n \n : ,\n : , \n : ,\n : ,\n : , \n : , \n : ,\n\n : , \n : ,\n\n : , \n : ,\n\n : {}, \n : {}, \n : ,\n : ,\n : ,\n : ,\n\n \n : ,\n : * ,\n\n \n : , \n\n \n : , \n : , \n : ,\n : ,\n\n \n : ,\n : , \n : ,\n : ,\n\n \n : ,\n : ,\n : ,\n : ,\n : ,\n : {},\n : {\n : ,\n : [, , , , , ],\n : (, ),\n : ,\n : ,\n : (, ),\n : ,\n : ,\n : (, ),\n : .,\n : .,\n : ,\n : ,\n : .,\n : ,\n : .,\n },\n\n \n : ,\n\n \n : ,\n : ,\n\n \n : ,\n : ,\n : ,\n\n : {\n : {\n : ,\n : ,\n : ,\n : ,\n : {\n : {\n : ,\n : [, ],\n : [, ],\n : ,\n : ,\n : ,\n : \n },\n : {\n : ,\n : [, , ],\n : [, , ],\n : ,\n : ,\n : ,\n : \n }\n }\n },\n ...\n : {\n : ,\n : ,\n : ,\n : [, , ],\n : ,\n : ,\n : ,\n : \n },\n : {\n : ,\n : ,\n : ,\n : ,\n : ,\n : ,\n },\n\n \n : ,\n}\nSuch a config permitted changing params on the fly and track any changes and rerun model inference on kaggle with stable results.
\nGood models were uploaded to the Kaggle dataset, and I made several submissions. Monday - Thursday I slowly trained IMU models as didn't 100% believe in success.
\n25 August night submission of 9 / 5 folded IMU models gave 0.838 on LB.
Junie talks here:
\n\n\n is a compact sequence encoder that combines convolutional feature extraction, temporal modeling, and attention-based pooling to produce a fixed-size representation per sequence. It takes an input tensor shaped (batch, time, features), extracts local temporal patterns with residual 1D CNNs, models longer-range dependencies with a bidirectional GRU, and compresses the sequence to a single vector via learnable attention before projecting to a fixed .\n\n\n Two residual CNN blocks () operate on the transposed signal :\n Block 1 adapts channel width from and uses a 3-layer or 2-layer 1D CNN core.\n Block 2 continues feature refinement with its own core.\n Each block includes:\n Channel attention via a Squeeze-and-Excitation to reweight feature maps.\n Residual addition, ReLU activation, downsampling, and dropout.\n This stage increases the receptive field and reduces sequence length (two pooling steps) while maintaining stable gradients through residual connections.\n\n\n The convolved output is transposed back to and fed to a bidirectional GRU:\n equals the final CNN channel count (from Block 2).\n , , , .\n A dropout layer follows the GRU to regularize temporal representations.\n\n\n A learnable consumes the GRU outputs and returns a weighted context vector , focusing the model on the most informative time steps.\n A final linear layer maps this vector to , yielding a per-sequence embedding suitable for downstream fusion and classification.\n\n\n Key arguments:\n : number of input channels per time step.\n : dictionary that configures channels, kernel sizes, pooling, dropout, SE reduction, and per block. Defaults provide a strong general-purpose setup:\n Block 1: , , .\n Block 2: , , .\n : GRU output dropout; each block has its own dropout.\n : final embedding size exported to the fusion module.\n Shapes:\n Input: \n After CNN blocks: downsampled in time, expanded in channels\n After BiGRU: \n Output: (sequence-level vector)\n\n\n Residual CNNs with SE attention capture multi-scale local dynamics and suppress noisy channels early.\n Max pooling reduces computational load and encourages invariance to minor timing shifts.\n The bidirectional GRU complements CNNs by modeling longer contexts beyond the convolutional receptive field.\n Attention pooling provides a flexible, data-driven way to summarize variable-length sequences.\n The fixed makes this processor plug-and-play within the multi-modal fusion framework, allowing different feature groups to contribute comparably sized embeddings.\nOk, now I had and option or FULL IN or just STOP. Danila Savenkov (JB side, you may know him as @daniel89) permitted me to spend 3 days on Kaggle (Friday / Monday / Tuesday + Weekend, of course).
\nWhat I need:
\nSwitch to EMA only models
Changed LR scheduler
Retrained IMU Models
Made Massive probing (to make sure all components work as intended)
Single IMU only no PP - (0.823 / 0.834)
6 IMU models no PP weighted average - (0.837 / 0.845)
6 IMU models with PP weighted average - (0.840 / 0.853)
6 IMU + old ToF no PP (0.851 / 0.868)
6 IMU + old ToF with PP (0.863 / 0.876)
Did few changes to Post-processing but didn't manage to improve much the output my local CV showed +0.01 +/- 0.0015 std / max-min 0.006
CMI - Final 5.0 - Final candidate 2!
\nSucceeded · Tue Sep 02 2025 21:58:40 GMT+0100 (0.860 / 0.877)
CMI - Final 4.0 - Final Candidate 1!
\nSucceeded · Tue Sep 02 2025 21:57:42 GMT+0100 (0.858 / 0.880)
Had no option to test if my weights are right - probably a simple average mix with cv score cutoff would perform better.
\nMix was made with Dirichlet search with oof predictions (too risky if CV is wrong and not aligned with LB) - cutoff low weights and normalized back
\n#################################\n# Models Weights\n#################################\n\n# IMU\n: np.float64(),\n : np.float64(),\n : np.float64(),\n : np.float64(),\n : np.float64(),\n : np.float64(),\n : np.float64(),\n : np.float64(),\n : np.float64(),\n : np.float64(),\n : np.float64(),\n : np.float64(),\n : np.float64()\n}\n\n# TOF\n{: np.float64(),\n : np.float64(),\n : np.float64(),\n : np.float64(),\n : np.float64(),\n : np.float64(),\n : np.float64(),\n : np.float64(),\n : np.float64(),\n : np.float64(),\n : np.float64(),\n : np.float64(),\n : np.float64(),\n : np.float64(),\n : np.float64(),\n : np.float64(),\n : np.float64(),\n : np.float64()\n}\nI joined Kaggle almost 7 years ago. Never thought that I would be a triple Grandmaster - till today I had the shame even to call myself a GM as only competition title matters.
\nThe road to the last medal was long))) It's not the end - I'll compete (when will have more free time) and definitely will compete always when tabular data appears (not stock market)).
\nSo many good memories come to me:
Without you guys - your mentoring, long conversations, your example, I wouldn't achieve all these.
\nAnd of course the Kaggle community - the support, knowledge sharing, gentle teaching - there is no other place like this.
THANK YOU!
", "rawMarkdown": "Long read ahead)))- many personal thoughts and emotions.\nAll below is only my personal opinion and view, and when I say 'you should' or 'you must,' of course it's just my view.\n\nThanks Kaggle team and CMI for such great competition - the data is exceptional and was so much fun to work with it!\n\n---\n\nWhen you start working on competition, you should think from the start about several aspects:\n\n- The code organization\n- Versioning\n- Experiments tracking\n- Results repeatability\n\n# Code organization\n\nNormally my ML experiments follow such a structure (that is enough and good for me).\n\n- `data/`: Contains all data files\n - `raw/`: Original, unmodified data files\n - `processed/`: Data after various processing steps\n - `base/`: Basic processed data (e.g., cleaned, filtered)\n - `fe/`: Feature-engineered data\n - `temp/`: Temporary data files that don't need to be version controlled\n\n- `checkpoits/`: To store weights and runs\n\n- `notebooks/`: Jupyter notebooks\n - `public/`: Notebooks that were publicly shared (of course you should check what other people did)\n - `eda/`: Data analysis (complex or single shot data checks)\n - `old/`: My TRASH folder where I drop copy of copy of copy notebooks\n - `models/`: Main folder where experiments live separated by models types\n - `lgb/`:\n - `lstm/`:\n - `xxx/`:\n\n- `src/`: Source code organized by functionality\n - `data/`: Code for data loading and processing\n - `fe/`: Feature engineering code\n - `models/`: Isolated code separated by model type\n - `lgb/`: LightGBM models\n - `lstm/`: LSTM model files\n - `pca/`: PCA model files\n - `temp/`: Temporary code files and test scripts\n - `utils/`: Utility functions and helper modules\n\n- `history/`: Historical project data and records / if no git in project doing src dumps here\n- `tb/`: Tensorboard tracking logs\n- `tests/`: Unit tests and integration tests\n\nI really advise you to use some version control (git). It will greatly help you to navigate through your thoughts during the competition 'marathon' (normally 3 months).\n\n\n# The Plan\n\nSo you downloaded the data - so did I. Did some minimal data checks and started thinking about the plan.\nThe goal was GOLD - any other result would be a failure)))).\n\n- We have 2 types of Data, so I will need good models for each branch IMU and TOF. IMU is easy just 7 features - so will start with it.\n- It's a solo competition, so I need to Ensemble in the end with different architectures and model types.\n- Will need NNs for sure - any gradient boosting will be not enough.\n- Post-processing? Sure. Almost any competition needs PP. Also, F1 metric is good for hacking (some thresholds optimizations?).\n- Inference time limit. Data is sequence per sequence and small - time should not be an issue.\n- Models types - We have a temporal component (sequence) that should be used - lstm / gru?\n- We have 18 classes but also orientation / phase, and the main target that could split by components 'Forehead/zone' - 'pull hairline/action' - needs to be used for sure.\n- Just 9k sequences - Augmentation we need\n\n# First steps\n\nI decided to start with lgbm (surprise)))) The previous CMI competition (that we won with @trasibulo) showed that gradient models could be competitive to NN in some tasks.\nDid few experiments that show interesting details.\n\n- Model overfits A LOT!\n- It's relatively easy to predict Gesture / Non gesture\n- History window of 15 steps is enough for the model to catch up\n- Leaky std means normalization by subject helps a lot - so there are some subject's device differences (calibration / demographics data?).\n- ROC AUC very close to 1 - 0.998+ - so we need more data - augmentation for sure (can we generate our own data? or just augment what we have? or predict not on sequence level but on point-wise level N-predictions looking to the past?)\n\nLooked at a competition prediction process and tried to submit lgbm very basic predictions\n\n```\nCMI - Baseline v0\nSucceeded · 2mo ago \n0.704 / 0.722\n```\n\nOk it works. Think dude, think, - do I have something to improve in the process (not talking about the scores - was obvious that it's not even close to be good).\nI can use Different models for each target - 1 for multiclass and one for binary.\n\n- [ADDed to plan] - Make two different models and use both if inference time and memory permit so (and if binary model would be better)\n\nLGBM still but with multiclass problems - I thought to switch to catboost or xgb, but it was obvious that gradient boosting is a dead-end.\n\nI kept lgbm flow to test model reaction to FE to decide what to keep and what to extend (was useless in the end as no matter how good the feature was, NN treats it differently, and\nalmost every custom feature made NN results worse)\n\nAlso, was thinking to do PCA / Autoencoder for gradient boosting but decided to switch to pure NN.\n\n# TARGETs | What we are predicting?\n\nOk, we have a baseline with almost no feature engineering; it's time to go deeeeeper.\n\nLet's check what our targets are.\n\n```\ndf.drop_duplicates(subset=['sequence_id'])['gesture'].value_counts()\ngesture\nForehead - pull hairline 640\nNeck - pinch skin 640\nText on phone 640\nNeck - scratch 640\nForehead - scratch 640\nEyelash - pull hair 640\nAbove ear - pull hair 638\nEyebrow - pull hair 638\nCheek - pinch skin 637\nWave hello 478\nWrite name in air 477\nPull air toward your face 477\nFeel around in tray and pull out an object 161\nWrite name on leg 161\nPinch knee/leg skin 161\nScratch knee/leg skin 161\nDrink from bottle/cup 161\nGlasses on/off 161\nName: count, dtype: int64\n```\n\nStop WHAT? Is each person doing exactly the same gestures?\n\n```\nSUBJ_000206 Above ear - pull hair 8\n Cheek - pinch skin 8\n Eyebrow - pull hair 8\n Eyelash - pull hair 8\n Forehead - pull hairline 8\n Forehead - scratch 8\n Neck - pinch skin 8\n Neck - scratch 8\n Text on phone 8\n Pull air toward your face 6\n Wave hello 6\n Write name in air 6\n Drink from bottle/cup 2\n Feel around in tray and pull out an object 2\n Glasses on/off 2\n Pinch knee/leg skin 2\n Scratch knee/leg skin 2\n Write name on leg 2\nSUBJ_001430 Above ear - pull hair 8\n Cheek - pinch skin 8\n Eyebrow - pull hair 8\n Eyelash - pull hair 8\n Forehead - pull hairline 8\n Forehead - scratch 8\n Neck - pinch skin 8\n Neck - scratch 8\n Text on phone 8\n Pull air toward your face 6\n Wave hello 6\n Write name in air 6\n Drink from bottle/cup 2\n Feel around in tray and pull out an object 2\n Glasses on/off 2\n Pinch knee/leg skin 2\n Scratch knee/leg skin 2\n Write name on leg 2\n```\n\nOk - THIS is will be our Post-processing! - don't know what exactly and how, but here is the key!\n\n# NN experiments\n\nTried different NNs configurations (thanks to Junie it was fast). Decided to go with different 'branches' for each modality and customizable heads.\n\n**Junie talks here:**\n\n```\n### Model Overview\n\nOur solution is a multi‑modal neural network tailored for gesture recognition from heterogeneous time‑series features. The architecture is assembled dynamically from a configuration: for each feature group (e.g., raw IMU, gravity‑removed signals, engineered features, ToF‑derived 2D maps), the model instantiates a dedicated encoder and fixes its\noutput dimension for later fusion. The encoded features are concatenated and passed through two independent fusion blocks with modality dropout. The first fusion stream feeds the main multiclass classifier over all gesture and non‑gesture categories, while the second stream feeds an auxiliary binary detector (gesture vs non‑gesture). Optionally, the second stream can also produce a small auxiliary multiclass head and a lightweight regression head .\n\n### Modality‑Specific Encoders\n\nEach active feature group uses a processor chosen by configuration. Supported processors include temporal encoders for 1D sequences, a compact `dense` processor, and several ToF 2D backbones. Only features with `use_feat=True` are instantiated. Every encoder projects inputs to a fixed `output_dim`, and these embeddings are concatenated before fusion. This modular design allows easy ablations or expansions of the feature set without changing the training loop.\n\n### Fusion and Classification Heads\n\nWe use a unified fusion module with configurable width and dropout, plus modality dropout that stochastically drops entire modality embeddings to improve robustness. Both fusion modules receive the same concatenated representation but specialize during training: the first targets fine‑grained multiclass discrimination, and the second refines a clean binary gesture signal. The main head outputs logits over all classes; the auxiliary head outputs a single sigmoid probability for gesture vs non‑gesture. When enabled, the second stream can also drive an auxiliary multiclass classifier and a simple regression head to act as inductive regularizers.\n\n### Training Objectives and Optimization\n\nTraining couples the main multiclass objective with the auxiliary binary objective using a family of combined losses selected by configuration with optional mixup support. When auxiliary heads are active, we add a cross‑entropy term for the auxiliary multiclass target and an MSE term for the regression head, both with small weights. The loop supports per‑batch learning‑rate scheduling, optional EMA of model weights, and an optional dropout annealer to gradually adjust regularization. This setup stabilizes optimization across long sequences and heterogeneous modalities.\n\n### Inference and Evaluation\n\nAt inference, the model encodes each active feature group, concatenates embeddings, and runs both fusion pathways to produce predictions. For validation, we compute metrics aligned with the competition goals: binary ROC AUC and F1 for gesture vs non‑gesture; mean ROC AUC across the first eight gesture classes; macro‑F1 over the full label space; and a clipped gesture‑only macro‑F1. The reported competition metric is the average of the binary F1 and the gesture macro‑F1, balancing detection quality with class discrimination. A simple TTA mode can aggregate predictions at the sequence level for additional robustness.\n\n### Practical Advantages\n\n- Modular encoders let us integrate diverse signals and swap architectures per modality without touching the core training loop.\n- Dual fusion streams decouple fine‑grained classification from robust gesture detection, improving stability and overall performance.\n- Auxiliary heads provide targeted regularization and richer supervision.\n- Modality dropout enhances resilience when some feature channels are missing or noisy.\n```\n\nFor the code agent (Junie) I've created a special folder and some instructions. It will be his personal sandbox to play around and not 'make sh.t files' in other places.\n\n```\n## Experiments Structure and Organization\n\nThe `src/experiments` directory is organized to track model experiments:\n\n- `proposal/`: Contains experiment proposals in markdown format\n - Each file represents a single experiment proposal\n - Naming convention: `exp_XXX_short_description.md` where XXX is a sequential number\n\n- `running/`: Contains experiments that are currently in progress\n - Experiments moved here from `proposal/` when they are being implemented\n - May include additional files related to the experiment implementation\n\n- `results/`: Contains completed experiment results\n - Final outcomes, metrics, and analysis of experiments\n - Should reference the original proposal\n\nEach experiment proposal follows a standardized format with sections for:\n\n- Objective\n- Hypothesis\n- Methodology\n- Evaluation Metrics\n- Expected Outcome\n- Resources Required\n- References\n\nThe experiment workflow involves:\n\n1. Creating a proposal in the `proposal/` directory\n2. Moving it to `running/` during implementation\n3. Documenting results in the `results/` directory\n4. Using findings to inform future experiments\n```\n\nIn the end I came to such modalities and features (features are added / turned off / turned on in different models):\n\n```\n# 1d Temporal\nraw_acc = [\n 'raw__|acc_x', 'raw__|acc_y', 'raw__|acc_z', 'raw__|acc_magnitude_all',\n 'raw__|acc_x__dif__|lag-1', 'raw__|acc_y__dif__|lag-1', 'raw__|acc_z__dif__|lag-1', 'raw__|acc_magnitude_all__dif__|lag-1',\n 'raw__|acc_x__ratio__|lag-1', 'raw__|acc_y__ratio__|lag-1', 'raw__|acc_z__ratio__|lag-1', 'raw__|acc_magnitude_all__ratio__|lag-1',\n]\n\n# 1d Temporal\nraw_rot = [\n 'raw__|rot_x', 'raw__|rot_y', 'raw__|rot_z', 'raw__|rot_w',\n 'raw__|rot_x__dif__|lag-1', 'raw__|rot_y__dif__|lag-1', 'raw__|rot_z__dif__|lag-1', 'raw__|rot_w__dif__|lag-1',\n]\n\n# 1d Temporal\nraw_thm = [\n 'thm_1', 'thm_2', 'thm_3', 'thm_4', 'thm_5',\n 'thm__global__|mean', 'thm__global__|std', 'thm__global__|min', 'thm__global__|max',\n 'tof__global__|mean', 'tof__global__|std', 'tof__global__|min', 'tof__global__|max', \n]\n\n# 2d Temporal\nraw_tof = [c for c in list(base_df) if 'tof_v' in c]\n\n# 1d Temporal\nraw_vel = [\n 'ang__|velocity_acc_x', 'ang__|velocity_acc_y', 'ang__|velocity_acc_z',\n 'ang__|velocity_acc_x__dif__|lag-1','ang__|velocity_acc_y__dif__|lag-1', 'ang__|velocity_acc_z__dif__|lag-1',\n]\n\nraw_custom = [\n 'raw__|acc_magnitude_all', 'raw__|acc_magnitude_xy', 'raw__|acc_magnitude_zx', 'raw__|acc_magnitude_zy',\n 'raw__|rot_angle', 'raw__|rot_angle__dif__|lag-1',\n 'raw__|acc_xy_angle', 'raw__|acc_zx_angle', 'raw__|acc_zy_angle',\n 'no_grav__|acc_magnitude_all', 'no_grav__|acc_magnitude_all__dif__|lag-1',\n ...\n]\n\n# Dense\nraw_global = [\n 'raw__|acc_x__global__|std', 'raw__|acc_y__global__|std', 'raw__|acc_z__global__|std',\n 'raw__|rot_w__global__|std', 'raw__|rot_x__global__|std', 'raw__|rot_y__global__|std', 'raw__|rot_z__global__|std',\n ...\n]\n\n# Dense\n\nraw_demo = [\n 'adult_child', 'sex', 'handedness', 'age', 'height_cm', 'shoulder_to_wrist_cm', 'elbow_to_wrist_cm',\n]\n```\n\n# FE and data cleaning\n\nI've tried different approaches like flipping hand / small rotation / TTA - spent too much time with it without success.\nSimple things were left as base:\n- left handed flip 'acc_x' 'rot_y' 'rot_z'] | *-1\n- wierd 'SUBJ_045235', 'SUBJ_019262' where acc_y seems to be flipped - *-1 slightly improved cv\n\nFE\n- Gravity removal - 2 versions\n- Velocity \n- Distance\n- Angles and projections\n- Magnitudes **2 / **3 all and just pairs\n- Lags / Difs / Ratios\n\nNo fancy staff here\n- Added custom naming for consistency across different versions like 'raw__|rot_x__dif__|lag-1' - was simple to add new features and experiment \n\n# Pause and Return\n\n11 of August I had local validation 0.805- 0.815 for IMU data and stopped working on competition after 3 weeks. Completely stopped kaggling as I started new work at JetBrains and all time was dedicated to it (another reason was that I thought that my model is not good at all for gold zone). \n\nFriday night 29 August - good Portuguese wine in my glass, working day came to an end / 20.00 time / no hobby - of course let's check what I did for competition and review with clean eyes.\n\nWHAT!??? Error? Huge error on competition metric. I was experimenting on TTA, and my final metric was a bit weird (forgot to change back to normal)))\n\nSimplification example:\nf1 on \nmask = binary_predictions > 0.5\nnp.clip(np.argmax(values[mask], axis=1), 0, 7)\n+\nf1 on\nnp.argmax(values, axis=1)<=7\n\n/ 2\n\nOf course, it's not a competition metric - immediately changed to np.clip(np.argmax(.values, axis=1), 0, 8)\nand tadaaam I have 0.82-0.824.\nSo I have a competitive single model - THM and ToF should give + 0.03 \n0.85 + ensembl - 0.86 + PP 0.87 - I can challenge the gold zone\n\n# The config\n\nDuring the weekend I rewrote all my models and prepared unified config to start making ensemble\n\nALWAYS use Tensorboard (or equivalent) - because overwise you'll not be able to make good comparisons.\n\nUnified config looks like this\n\n```\nbase_params = {\n 'GLOBAL_VERBOSE': False,\n\n # Dataset Params\n 'WINDOW_SIZE': 140,\n 'OUT_WINDOW_SIZE': 386, \n 'CUR_FOLD': 0,\n 'TRAIN_OFFSET': 0,\n 'MAIN_TARGET': 'target_b', # MAIN multiclass\n 'AUX_TARGET': 'target_a', # Aux binary\n 'AUX_TARGET_WEIGHT': 0.3,\n\n 'AUX_TARGET_2': 'target_c', # Aux multiclass\n 'AUX2_NUM_CLASSES': 4,\n\n 'AUX_TARGET_3': True, # Regression target\n 'AUX3_WEIGHT': 0.05,\n\n 'FEATURES': {}, # added on fly\n 'DIMS': {}, # added on fly\n 'NUM_CLASSES': 18,\n 'FBFILL': False,\n 'FILLNA_VALUE': 0,\n 'USE_SCALER': True,\n\n # Batch params\n 'BATCH_SIZE': 64,\n 'EVAL_BATCH_SIZE': 256 * 8,\n\n # Feature Engineering\n 'MAKE_FE': False, # Runtime fe\n\n # Model params\n 'LEARNING_RATE': 0.0005, \n 'NUM_EPOCHS': 30, \n 'WEIGHT_DECAY': 0.0001,\n 'EMA_DECAY': 0.999,\n\n # Scheduler params\n 'SCHEDULER': 'cosine_warmup_decay',\n 'WARMUP_EPOCHS': 5, \n 'WARMUP_LR': 0.0001,\n 'MIN_LR': 0.00008,\n\n # Augmentations\n 'USE_TTA': False,\n 'DS_AUG': 10,\n 'MIXUP': True,\n 'MIXUP_PROB': 0.8,\n 'MIXUP_ALFA': 0.5,\n 'RAW_AUG': {},\n 'TS_AUG': {\n 'enable': True,\n 'apply_to': ['raw_acc', 'raw_rot', 'raw_vel', 'raw_thm', 'raw_tof', 'raw_custom'],\n 'time_stretch_range': (0.8, 1.2),\n 'time_shift_range': 0.1,\n 'noise_std': 0.02,\n 'magnitude_scale_range': (0.9, 1.1),\n 'rotation_angle_range': 0.1,\n 'mask_ratio': 0.1,\n 'freq_filter_range': (0.1, 0.9),\n 'p_stretch': 0.,\n 'p_shift': 0.,\n 'p_noise': 0.5,\n 'p_mag': 0.1,\n 'p_rotate': 0.,\n 'p_mask': 0.1,\n 'p_freq': 0.,\n },\n\n # Other\n 'SEED': 18,\n\n # losss\n 'LOSS_NAME': 'v1',\n 'NEGATIVE_PROBS': False,\n\n # Layers params\n 'fusion_channels': 512,\n 'fusion_dropout': 0.5,\n 'fusion_modality_dropout': 0.2,\n\n 'PROCESSOR_CONFIGS': {\n \"raw_acc\": {\n \"processor_type\": \"temporal_v2\",\n \"use_feat\": True,\n \"output_dim\": 512,\n \"dropout\": 0.5,\n \"res_blocks\": {\n \"block1\": {\n \"in_channels\": None,\n \"channels\": [64, 128],\n \"ks\": [3, 5],\n \"dropout\": 0.5,\n \"reduction\": 8,\n \"pool_size\": 2,\n \"core_type\": \"cnn2\"\n },\n \"block2\": {\n \"in_channels\": 256,\n \"channels\": [128, 256, 256],\n \"ks\": [3, 5, 5],\n \"dropout\": 0.5,\n \"reduction\": 8,\n \"pool_size\": 2,\n \"core_type\": \"cnn3\"\n }\n }\n },\n ...\n 'raw_tof': {\n 'use_feat': True,\n 'processor_type': 'tof_conv2d',\n 'output_dim': 64,\n 'conv_channels': [8, 16, 32],\n 'kernel_size': 3,\n 'dropout': 0.5,\n 'pool_size': 2,\n 'temporal_dim': 512\n },\n 'raw_global': {\n 'processor_type': 'dense',\n 'use_feat': False,\n 'output_dim': 32,\n 'hidden_dim': 64,\n 'dropout': 0.7,\n 'use_bn': True,\n },\n\n # Dropout shdeuler\n 'USE_DROPOUT_SCHEDULER': False,\n}\n\n```\n\nSuch a config permitted changing params on the fly and track any changes and rerun model inference on kaggle with stable results.\n\nGood models were uploaded to the Kaggle dataset, and I made several submissions. Monday - Thursday I slowly trained IMU models as didn't 100% believe in success.\n25 August night submission of 9 / 5 folded IMU models gave 0.838 on LB.\n\n**Junie talks here:**\n```\n### TemporalProcessor\n\n`TemporalProcessor` is a compact sequence encoder that combines convolutional feature extraction, temporal modeling, and attention-based pooling to produce a fixed-size representation per sequence. It takes an input tensor shaped `[B, T, F]` (batch, time, features), extracts local temporal patterns with residual 1D CNNs, models longer-range dependencies with a bidirectional GRU, and compresses the sequence to a single vector via learnable attention before projecting to a fixed `output_dim`.\n\n#### Convolutional front-end (local pattern extraction)\n- Two residual CNN blocks (`ResidualCNNBlock`) operate on the transposed signal `[B, F, T]`:\n - Block 1 adapts channel width from `in_features` and uses a 3-layer or 2-layer 1D CNN core.\n - Block 2 continues feature refinement with its own core.\n- Each block includes:\n - Channel attention via a Squeeze-and-Excitation `SEBlock` to reweight feature maps.\n - Residual addition, ReLU activation, `MaxPool1d` downsampling, and dropout.\n- This stage increases the receptive field and reduces sequence length (two pooling steps) while maintaining stable gradients through residual connections.\n\n#### Temporal backbone (sequence modeling)\n- The convolved output is transposed back to `[B, T', C]` and fed to a bidirectional GRU:\n - `input_size = C` equals the final CNN channel count (from Block 2).\n - `hidden_size = C`, `bidirectional=True`, `batch_first=True`, `bias=False`.\n - A dropout layer follows the GRU to regularize temporal representations.\n\n#### Attention pooling and projection\n- A learnable `AttentionLayer` consumes the GRU outputs `[B, T', 2C]` and returns a weighted context vector `[B, 2C]`, focusing the model on the most informative time steps.\n- A final linear layer maps this vector to `output_dim`, yielding a per-sequence embedding suitable for downstream fusion and classification.\n\n#### Configurability and I/O\n- Key arguments:\n - `in_features`: number of input channels per time step.\n - `res_blocks`: dictionary that configures channels, kernel sizes, pooling, dropout, SE reduction, and `core_type` per block. Defaults provide a strong general-purpose setup:\n - Block 1: `channels=[64,128]`, `ks=[3,5]`, `core_type='cnn2'`.\n - Block 2: `channels=[128,128,256]`, `ks=[3,5,5]`, `core_type='cnn3'`.\n - `dropout`: GRU output dropout; each block has its own dropout.\n - `output_dim`: final embedding size exported to the fusion module.\n- Shapes:\n - Input: `[B, T, in_features]`\n - After CNN blocks: downsampled in time, expanded in channels\n - After BiGRU: `[B, T', 2C]`\n - Output: `[B, output_dim]` (sequence-level vector)\n\n#### Design rationale\n- Residual CNNs with SE attention capture multi-scale local dynamics and suppress noisy channels early.\n- Max pooling reduces computational load and encourages invariance to minor timing shifts.\n- The bidirectional GRU complements CNNs by modeling longer contexts beyond the convolutional receptive field.\n- Attention pooling provides a flexible, data-driven way to summarize variable-length sequences.\n- The fixed `output_dim` makes this processor plug-and-play within the multi-modal fusion framework, allowing different feature groups to contribute comparably sized embeddings.\n```\n\n# Last Days\n\nOk, now I had and option or **FULL IN** or just **STOP**. Danila Savenkov (JB side, you may know him as @daniel89) permitted me to spend 3 days on Kaggle (Friday / Monday / Tuesday + Weekend, of course).\n\n## Decision made - **FULL IN**.\n\n**What I need:**\n- GPU - lighting AI - 1 own gpu is not enough\n- ToF model\n- Postprocessing\n- Better IMU model\n- LUCK)))\n- Carefull LB probing\n- Not f.ck up with CV as had no chance to correct anything\n\n### Thursday Night / Friday\n- LB probing with ensemble setup - average / weighted average / number models / full data or 5 folds - 0.838 still\n- Decided to go with many week 5 folded models - this was the only chance as had no option to probe separate single model on full data\n- Let's make ToF model - 0.855 with IMU mix wih 1 ToF \n- I need more ToF models and more IMU - started mass training\n\n### Satruday\n- I need post-process - NO way 1st place has such a difference in score with models only\n- First PP version - 0.862 - a good start, but I need more\n- More ToF models - 5 ready and validated - 0.871 with PP (no submissions left to probe without)\n- Fixed EMA model - was cloning trainable param but not buffers - fix made EMA works\n\n### Sunday\n- Switch to EMA only models\n- Changed LR scheduler \n- Retrained IMU Models\n- Made Massive probing (to make sure all components work as intended)\n\n- Single IMU only no PP - (0.823 / 0.834)\n- 6 IMU models no PP weighted average - (0.837 / 0.845)\n- 6 IMU models with PP weighted average - (0.840 / 0.853)\n- 6 IMU + old ToF no PP (0.851 / 0.868)\n- 6 IMU + old ToF with PP (0.863 / 0.876) \n\n- Did few changes to Post-processing but didn't manage to improve much the output my local CV showed +0.01 +/- 0.0015 std / max-min 0.006 \n\n### Monday\n- New ToF models\n- no PP (0.853 / 0.872)\n- with PP (0.868 / 0.883)\n\n### Tuesday - Final day \n- Timing should be very precise \n- All new models scores should be ready till 22.00 London time to submit a final solution\n- New Imu models 6 runs - 1.5H each on 3 gpu should be ready till 16 00 - 2h inference - 19 00 scores maximum\n- New ToF models - 3h each run - will be able to make 3 models more - should be ready till 19 00 - 21 00 scores maximum\n- 1h to prepare final mix with pp and submit till 22 00 \n\n**CMI - Final 5.0 - Final candidate 2!**\nSucceeded · Tue Sep 02 2025 21:58:40 GMT+0100 (0.860 / 0.877)\n\n**CMI - Final 4.0 - Final Candidate 1!**\nSucceeded · Tue Sep 02 2025 21:57:42 GMT+0100 (0.858 / 0.880) \n- Done - choose Candidate 1 and previous best lb model - probably 0.858 would be enough for gold also but we will never know)))\n\n#### Models changes experiments\n- features\n- different CNN block - 'cnn', 'inception', 'gated_dilated', 'convnext', 'miniconformer', 'fft' \n- conv layers \n- output dims\n- extrapolated window\n- LR\n- dropouts\n- fusion blocks\n- aux targets\n- loss functions\n\nHad no option to test if my weights are right - probably a simple average mix with cv score cutoff would perform better.\n\nMix was made with Dirichlet search with oof predictions (too risky if CV is wrong and not aligned with LB) - cutoff low weights and normalized back\n\n# IMU Final metric (rounded+pruned): 0.843523 | 0.854 with pp\n# TOF Final metric (rounded+pruned): 0.891330 | 0.903 with pp\n\n```\n#################################\n# Models Weights\n#################################\n\n# IMU\n'test_run_-_2025-09-02_102121': np.float64(0.04577114427860696),\n 'test_run_-_2025-09-02_104733': np.float64(0.07462686567164178),\n 'test_run_-_2025-09-02_100513': np.float64(0.13532338308457711),\n 'test_run_-_2025-09-02_100715': np.float64(0.04676616915422885),\n 'test_run_-_2025-09-01_232203': np.float64(0.07462686567164178),\n 'test_run_-_2025-08-31_182223': np.float64(0.16417910447761194),\n 'test_run_-_2025-08-31_182238': np.float64(0.04577114427860696),\n 'test_run_-_2025-08-31_182155': np.float64(0.055721393034825865),\n 'test_run_-_2025-08-30_144324': np.float64(0.04577114427860696),\n 'test_run_-_2025-08-26_125131': np.float64(0.04676616915422885),\n 'test_run_-_2025-08-26_095603': np.float64(0.1024875621890547),\n 'test_run_-_2025-08-26_055724': np.float64(0.05771144278606965),\n 'test_run_-_2025-08-26_055618': np.float64(0.10447761194029849)\n}\n\n# TOF\n{'test_run_-_2025-09-01_174406': np.float64(0.08208884884305259),\n 'test_run_-_2025-09-01_095855': np.float64(0.027909755890227466),\n 'test_run_-_2025-09-01_022627': np.float64(0.15450960916030498),\n 'test_run_-_2025-08-30_112546': np.float64(0.029019711064357332),\n 'test_run_-_2025-08-29_151211': np.float64(0.028584738288600937),\n 'test_run_-_2025-08-28_124723': np.float64(0.009465796363042244),\n 'test_run_-_2025-08-28_124649': np.float64(0.07006594305155826),\n 'test_run_-_2025-08-29_091104': np.float64(0.08299200257345174),\n 'test_run_-_2025-08-28_154159': np.float64(0.13787604157789557),\n 'test_run_-_2025-08-27_205748': np.float64(0.028090284333857785),\n 'test_run_-_2025-08-27_141837': np.float64(0.12763658458693833),\n 'test_run_-_2025-08-31_182223': np.float64(0.11118294788297756),\n 'test_run_-_2025-08-31_182142': np.float64(0.005057595225800813),\n 'test_run_-_2025-08-31_182238': np.float64(0.06738590961562396),\n 'test_run_-_2025-08-31_182155': np.float64(0.00539168026876913),\n 'test_run_-_2025-08-26_170459': np.float64(0.010176717301051534),\n 'test_run_-_2025-08-26_095603': np.float64(0.010695373071077567),\n 'test_run_-_2025-08-26_055724': np.float64(0.01187046090141201)\n}\n```\n\n---------------\n\n# After words.\n\nI joined Kaggle almost 7 years ago. Never thought that I would be a triple Grandmaster - till today I had the shame even to call myself a GM as only competition title matters.\nThe road to the last medal was long))) It's not the end - I'll compete (when will have more free time) and definitely will compete always when tabular data appears (not stock market)).\nSo many good memories come to me:\n- My first competitions - top 50 places when I believed that I could make some good models\n- First Team with @simpletonwang and @xkagjaeo05 (still friends btw) and how we missed our gold (due to bad time management)\n- 1st place in IEEE Fraud competition - with one and only @cdeotte https://www.kaggle.com/competitions/ieee-fraud-detection/discussion/111284 https://www.kaggle.com/competitions/ieee-fraud-detection/writeups/fraudsquad-1st-place-solution-part-2 \n- Dream team with @johnpateha @headsortails @cdeotte and our fail on M5 competition (with guessing distribution on private test part) - https://www.kaggle.com/competitions/m5-forecasting-accuracy/discussion/163621\n- AMEX competition - how kaggle noobs took 2nd place)))\n- 1st place with @dott1718 on AMP Parkinson's Disease Progression Prediction (when I had a privilege to work with a pure genius)\n- @trasibuloon first CMI competition and how we were trying to solve metric puzzle))\n- Emotional discussions / I would call them FIGHTS)))) with @bakeryproducts (we met and worked together because of Kaggle), who taught me A LOT!\n- I found so many friends because of Kaggle - not only during competitions but also social networks, meetups, conferences!\n\nWithout you guys - your mentoring, long conversations, your example, I wouldn't achieve all these. \nAnd of course the Kaggle community - the support, knowledge sharing, gentle teaching - there is no other place like this.\n\n**THANK YOU!**", "votes": null }, { "id": "3281592", "postDate": "09/04/2025 14:43:12", "content": "Dataset and model both accepts unified config - all configuration is done in a single place.
\nCustomizable Temporal block look like this:
\nall params comes from config and permits do many experiments without changing NN at all
\n 'in_channels': 128,
\n 'channels': [128, 256, 256],
\n 'ks': [3, 5, 5],
\n 'dropout': 0.3,
\n 'reduction': 16,
\n 'pool_size': 2,
\n 'core_type': 'cnn3',
(nn.Module):\n \n\n ():\n ().__init__()\n .core_type = core_type\n\n channels :\n channels = [, , ]\n ks :\n ks = [, , ]\n\n core_type == :\n .core = nn.Sequential(\n nn.Conv1d(in_channels, channels[], ks[], padding=ks[] // , bias=),\n nn.BatchNorm1d(channels[]),\n nn.ReLU(inplace=),\n\n nn.Conv1d(channels[], channels[], ks[], padding=ks[] // , bias=),\n nn.BatchNorm1d(channels[]),\n nn.ReLU(inplace=),\n )\n out_channels = channels[]\n\n core_type == :\n .core = nn.Sequential(\n nn.Conv1d(in_channels, channels[], ks[], padding=ks[] // , bias=),\n nn.BatchNorm1d(channels[]),\n nn.ReLU(inplace=),\n\n nn.Conv1d(channels[], channels[], ks[], padding=ks[] // , bias=),\n nn.BatchNorm1d(channels[]),\n nn.ReLU(inplace=),\n\n nn.Conv1d(channels[], channels[], ks[], padding=ks[] // , bias=),\n nn.BatchNorm1d(channels[]),\n nn.ReLU(inplace=),\n )\n out_channels = channels[]\n\n :\n ValueError()\n\n .channel_attention = SEBlock(out_channels, reduction=reduction)\n\n .shortcut_connection = nn.Sequential()\n in_channels != out_channels:\n .shortcut_connection = nn.Sequential(\n nn.Conv1d(in_channels, out_channels, , bias=),\n nn.BatchNorm1d(out_channels)\n )\n\n .pooling = nn.MaxPool1d(pool_size)\n .dropout_layer = nn.Dropout(dropout)\n .act = nn.ReLU(inplace=)\n\n ():\n residual = .shortcut_connection(x)\n out = .core(x)\n out = .channel_attention(out)\n out = out + residual\n out = .act(out)\n out = .pooling(out)\n out = .dropout_layer(out)\n out\n\n\n (nn.Module):\n ():\n ().__init__()\n\n res_blocks :\n res_blocks = {\n : {\n : [, ],\n : [, ],\n : ,\n : ,\n : ,\n : ,\n },\n : {\n : ,\n : [, , ],\n : [, , ],\n : ,\n : ,\n : ,\n : ,\n },\n }\n\n \n res_blocks[][] = in_features\n\n .block1 = ResidualCNNBlock_v2(**res_blocks[])\n .block2 = ResidualCNNBlock_v2(**res_blocks[])\n\n ():\n ct = cfg.get(, )\n ch = cfg[]\n ch[] ct == ch[]\n\n cnn_out_features = _out_channels(res_blocks[])\n\n .temporal = nn.GRU(input_size=cnn_out_features,\n hidden_size=cnn_out_features,\n bias=, bidirectional=, batch_first=)\n .temporal_dropout = nn.Dropout(dropout)\n\n .attention = AttentionLayer(cnn_out_features * )\n .output_projection = nn.Linear(cnn_out_features * , output_dim)\n\n ():\n x = x.transpose(, ) \n x = .block1(x)\n x = .block2(x)\n x = x.transpose(, ) \n\n .temporal.flatten_parameters()\n x, _ = .temporal(x)\n x = .temporal_dropout(x)\n\n x = .attention(x)\n x = .output_projection(x)\n x\nAnd here is a customizable model:
\n (nn.Module):\n ():\n (GestureModel, ).__init__()\n\n \n .all_params = all_params\n .num_classes = all_params[]\n\n \n .features = all_params[]\n .dims = all_params[]\n\n \n .processors = nn.ModuleDict()\n .processor_out_dims = {}\n\n \n feat_name, feat_data .features.items():\n feat_data (feat_data) > :\n dim_key = \n dim_key .dims:\n ._create_processor(feat_name, .dims[dim_key])\n\n \n ._create_fusion_processors(all_params)\n\n \n .classifier = nn.Linear(.fusion_1.output_dim, .num_classes)\n .binary_classifier = nn.Linear(.fusion_2.output_dim, )\n\n \n .aux2_target = all_params.get(, )\n .aux2_num_classes = all_params.get(, )\n .use_aux2 = .aux2_target \n .use_aux2:\n .aux2_classifier = nn.Linear(.fusion_2.output_dim, .aux2_num_classes)\n\n \n .aux3_target = all_params.get(, )\n .use_aux3 = .aux3_target\n .use_aux3:\n .aux3_classifier = nn.Linear(.fusion_2.output_dim, )\n\n ():\n \n\n \n .fusion_channels = all_params.get(, )\n .fusion_dropout = all_params.get(, )\n .fusion_modality_dropout = all_params.get(, )\n\n \n active_features = []\n split_dims = []\n\n feat_name .features:\n feat_name .processor_out_dims:\n active_features.append(feat_name)\n split_dims.append(.processor_out_dims[feat_name])\n\n total_input_dim = (split_dims)\n\n total_input_dim == :\n ValueError()\n\n \n .fusion_1 = FusionProcessorNew(\n input_dim=total_input_dim,\n fusion_channels=.fusion_channels,\n dropout=.fusion_dropout,\n split_dims=(split_dims),\n modality_dropout=.fusion_modality_dropout\n )\n\n .fusion_2 = FusionProcessorNew(\n input_dim=total_input_dim,\n fusion_channels=.fusion_channels,\n dropout=.fusion_dropout,\n split_dims=(split_dims),\n modality_dropout=.fusion_modality_dropout\n )\n\n ():\n \n\n \n processor_configs = .all_params.get(, {})\n\n \n config = \n config_key processor_configs:\n config_key feat_name:\n config = processor_configs[config_key].copy()\n \n\n \n config :\n (.(feat_name))\n \n\n config.get(, ):\n ()\n \n\n (.(feat_name))\n\n \n config_for_processor = config.copy()\n config_for_processor.pop(, )\n\n config_for_processor[] == :\n config_for_processor.pop(, )\n \n .processors[feat_name] = TemporalProcessor(\n in_features=input_dim,\n **config_for_processor\n )\n config_for_processor[] == :\n config_for_processor.pop(, )\n \n .processors[feat_name] = TemporalProcessor_v2(\n in_features=input_dim,\n **config_for_processor\n )\n config_for_processor[] == :\n config_for_processor.pop(, )\n .processors[feat_name] = TripleTemporalProcessor(\n in_features=input_dim,\n **config_for_processor\n )\n config_for_processor[] == :\n config_for_processor.pop(, )\n .processors[feat_name] = DenseProcessor(\n in_features=input_dim,\n **config_for_processor\n )\n config_for_processor[] == :\n config_for_processor.pop(, )\n .processors[feat_name] = TofConv2DProcessor(\n in_features=input_dim,\n **config_for_processor\n )\n config_for_processor[] == :\n config_for_processor.pop(, )\n .processors[feat_name] = TofConv2DProcessor_SensorAsChannel(\n in_features=input_dim,\n **config_for_processor\n )\n config_for_processor[] == :\n config_for_processor.pop(, )\n .processors[feat_name] = TofConv2DProcessor_InceptionCBAM(\n in_features=input_dim,\n **config_for_processor\n )\n config_for_processor[] == :\n config_for_processor.pop(, )\n .processors[feat_name] = TofConv2DProcessor_AxialTCN(\n in_features=input_dim,\n **config_for_processor\n )\n config_for_processor[] == :\n config_for_processor.pop(, )\n .processors[feat_name] = TofConv2DProcessor_CrossSensorAttn(\n in_features=input_dim,\n **config_for_processor\n )\n\n .processor_out_dims[feat_name] = config[]\n\n ():\n\n processed_features = []\n\n \n feat_name .features:\n feat_key = \n feat_key features_dict feat_name .processors:\n feat_tensor = features_dict[feat_key]\n processed_feat = .processors[feat_name](feat_tensor)\n processed_features.append(processed_feat)\n\n \n processed_features:\n combined_features = torch.cat(processed_features, dim=)\n :\n ValueError()\n\n \n fused_features_1 = .fusion_1(combined_features)\n fused_features_2 = .fusion_2(combined_features)\n\n class_logits = .classifier(fused_features_1)\n binary_aux = torch.sigmoid(.binary_classifier(fused_features_2)).squeeze(-)\n\n (, , ) (, , ):\n aux2_logits = .aux2_classifier(fused_features_2)\n aux3_output = .aux3_classifier(fused_features_2).squeeze(-) \n class_logits, binary_aux, aux2_logits, aux3_output\n (, , ):\n aux2_logits = .aux2_classifier(fused_features_2)\n class_logits, binary_aux, aux2_logits\n (, , ):\n aux3_output = .aux3_classifier(fused_features_2).squeeze(-)\n class_logits, binary_aux, aux3_output\n :\n class_logits, binary_aux\nGreat write-up, Konstantin. I found it very engaging to read. I’m so glad we connected through the first CMI competition, and I really admire your skills with tabular data.
", "rawMarkdown": "Great write-up, Konstantin. I found it very engaging to read. I’m so glad we connected through the first CMI competition, and I really admire your skills with tabular data.", "votes": null }, { "id": "3281752", "postDate": "09/04/2025 20:41:17", "content": "Really admire your style of storytelling - it’s one of the best write-ups I’ve seen here. The way you share your journey from your own perspective, with such depth and empathy, makes readers feel as if they’re walking your path alongside you. Even the technical parts about NNs, paired with Junie’s description, come across as almost poetic.
", "rawMarkdown": "Really admire your style of storytelling - it’s one of the best write-ups I’ve seen here. The way you share your journey from your own perspective, with such depth and empathy, makes readers feel as if they’re walking your path alongside you. Even the technical parts about NNs, paired with Junie’s description, come across as almost poetic.", "votes": null }, { "id": "3282635", "postDate": "09/06/2025 15:47:19", "content": "To keep all in one place
\nPost-processing
\nhttps://www.kaggle.com/code/kyakovlev/cmi-8th-place-postprocessing
hi man, it's a touchy read. Only people finish the read can relate.
\nbtw, how do you organise all the changes in like an IDE format? did you run the code locally or you have your local setup connect to Kaggle VM? How does the whole setup work to allow fast training and experiment at the same time
", "rawMarkdown": "hi man, it's a touchy read. Only people finish the read can relate.\n\nbtw, how do you organise all the changes in like an IDE format? did you run the code locally or you have your local setup connect to Kaggle VM? How does the whole setup work to allow fast training and experiment at the same time", "votes": null }, { "id": "3284701", "postDate": "09/08/2025 14:01:34", "content": "I have a single local GPU and majority of changes were tested on single fold locally (in the notebook). Also had end to end \"pipeline\" but notebooks are more convenient for me.
\nFor bulk experiments I've used lightning.ai where I clone \"src\" folder and notebook that accepts config (tensorboard logs, oofs and weights downloaded to local machine for evaluation and comparison)
\ncandidate_name = \ntb_dir = \n\n\nwriter = SummaryWriter(log_dir=) \n\n k, v validation_results.items():\n (v, (, )):\n writer.add_scalar(, v, epoch)\n\nvalidation_results = {\n : val_loss,\n : gesture_f1,\n : gesture_roc_auc,\n : gesture_f1_mean,\n : gesture_roc_auc_mean,\n\n : g8_mean_roc_auc,\n\n : g8_gesture_f1,\n : g8_gesture_f1_clipped,\n : mean_match,\n : competition_metric,\n }\nthanks for replying. Let's say you finished running your notebook on local GPU, how do you submit to Kaggle then? Do you upload it to kaggle & rerun?
\nBesides, since you run this locally, does that mean for all competitions you take in you download the data set to your laptop? What happens to very huge data sets?
", "rawMarkdown": "thanks for replying. Let's say you finished running your notebook on local GPU, how do you submit to Kaggle then? Do you upload it to kaggle & rerun?\n\nBesides, since you run this locally, does that mean for all competitions you take in you download the data set to your laptop? What happens to very huge data sets?", "votes": null }, { "id": "3286048", "postDate": "09/09/2025 06:54:28", "content": "You can create dataset on kaggle - upload your weights and make just inference.
\nIt’s interesting that in “code competitions” you have limited submission run time, but nobody asking you to train the model on fly - so you can train huge number of models locally and then upload to kaggle - during inference I’ve used 100+ models (if we consider each model fold as a separate model) and didn’t hit nor the memory limit nor time limit.
\n——
\nHuge datasets.
\nYes, I download original data, then carefully downcast data types and save in some format that is more compact and loads faster - parquet / feather / pickle (don’t use pickle). After such operations even the biggest kaggle datasets don’t consume more than 100mb - I don’t really remember any competition where the data size was a problem.
amazing thanks!
", "rawMarkdown": "amazing thanks!", "votes": null }, { "id": "3286072", "postDate": "09/09/2025 07:50:58", "content": "btw, this organization is what I imagine a GM could be, pretty amazing. Do you have github where you use this structure that I can refer to? great learning for me
", "rawMarkdown": "btw, this organization is what I imagine a GM could be, pretty amazing. Do you have github where you use this structure that I can refer to? great learning for me", "votes": null }, { "id": "3286104", "postDate": "09/09/2025 08:27:23", "content": "I'm shamed to publish my code as I see how ugly it is - sorry no public github repo.
", "rawMarkdown": "I'm shamed to publish my code as I see how ugly it is - sorry no public github repo.", "votes": null }, { "id": "3286140", "postDate": "09/09/2025 09:50:55", "content": "very humble, if you don't mind sharing with me, this is my git
\nhttps://github.com/MJeremy2017
Dataset and model both accepts unified config - all configuration is done in a single place.
\nCustomizable Temporal block look like this:
\nall params comes from config and permits do many experiments without changing NN at all
\n 'in_channels': 128,
\n 'channels': [128, 256, 256],
\n 'ks': [3, 5, 5],
\n 'dropout': 0.3,
\n 'reduction': 16,
\n 'pool_size': 2,
\n 'core_type': 'cnn3',
(nn.Module):\n \n\n ():\n ().__init__()\n .core_type = core_type\n\n channels :\n channels = [, , ]\n ks :\n ks = [, , ]\n\n core_type == :\n .core = nn.Sequential(\n nn.Conv1d(in_channels, channels[], ks[], padding=ks[] // , bias=),\n nn.BatchNorm1d(channels[]),\n nn.ReLU(inplace=),\n\n nn.Conv1d(channels[], channels[], ks[], padding=ks[] // , bias=),\n nn.BatchNorm1d(channels[]),\n nn.ReLU(inplace=),\n )\n out_channels = channels[]\n\n core_type == :\n .core = nn.Sequential(\n nn.Conv1d(in_channels, channels[], ks[], padding=ks[] // , bias=),\n nn.BatchNorm1d(channels[]),\n nn.ReLU(inplace=),\n\n nn.Conv1d(channels[], channels[], ks[], padding=ks[] // , bias=),\n nn.BatchNorm1d(channels[]),\n nn.ReLU(inplace=),\n\n nn.Conv1d(channels[], channels[], ks[], padding=ks[] // , bias=),\n nn.BatchNorm1d(channels[]),\n nn.ReLU(inplace=),\n )\n out_channels = channels[]\n\n :\n ValueError()\n\n .channel_attention = SEBlock(out_channels, reduction=reduction)\n\n .shortcut_connection = nn.Sequential()\n in_channels != out_channels:\n .shortcut_connection = nn.Sequential(\n nn.Conv1d(in_channels, out_channels, , bias=),\n nn.BatchNorm1d(out_channels)\n )\n\n .pooling = nn.MaxPool1d(pool_size)\n .dropout_layer = nn.Dropout(dropout)\n .act = nn.ReLU(inplace=)\n\n ():\n residual = .shortcut_connection(x)\n out = .core(x)\n out = .channel_attention(out)\n out = out + residual\n out = .act(out)\n out = .pooling(out)\n out = .dropout_layer(out)\n out\n\n\n (nn.Module):\n ():\n ().__init__()\n\n res_blocks :\n res_blocks = {\n : {\n : [, ],\n : [, ],\n : ,\n : ,\n : ,\n : ,\n },\n : {\n : ,\n : [, , ],\n : [, , ],\n : ,\n : ,\n : ,\n : ,\n },\n }\n\n \n res_blocks[][] = in_features\n\n .block1 = ResidualCNNBlock_v2(**res_blocks[])\n .block2 = ResidualCNNBlock_v2(**res_blocks[])\n\n ():\n ct = cfg.get(, )\n ch = cfg[]\n ch[] ct == ch[]\n\n cnn_out_features = _out_channels(res_blocks[])\n\n .temporal = nn.GRU(input_size=cnn_out_features,\n hidden_size=cnn_out_features,\n bias=, bidirectional=, batch_first=)\n .temporal_dropout = nn.Dropout(dropout)\n\n .attention = AttentionLayer(cnn_out_features * )\n .output_projection = nn.Linear(cnn_out_features * , output_dim)\n\n ():\n x = x.transpose(, ) \n x = .block1(x)\n x = .block2(x)\n x = x.transpose(, ) \n\n .temporal.flatten_parameters()\n x, _ = .temporal(x)\n x = .temporal_dropout(x)\n\n x = .attention(x)\n x = .output_projection(x)\n x\nAnd here is a customizable model:
\n (nn.Module):\n ():\n (GestureModel, ).__init__()\n\n \n .all_params = all_params\n .num_classes = all_params[]\n\n \n .features = all_params[]\n .dims = all_params[]\n\n \n .processors = nn.ModuleDict()\n .processor_out_dims = {}\n\n \n feat_name, feat_data .features.items():\n feat_data (feat_data) > :\n dim_key = \n dim_key .dims:\n ._create_processor(feat_name, .dims[dim_key])\n\n \n ._create_fusion_processors(all_params)\n\n \n .classifier = nn.Linear(.fusion_1.output_dim, .num_classes)\n .binary_classifier = nn.Linear(.fusion_2.output_dim, )\n\n \n .aux2_target = all_params.get(, )\n .aux2_num_classes = all_params.get(, )\n .use_aux2 = .aux2_target \n .use_aux2:\n .aux2_classifier = nn.Linear(.fusion_2.output_dim, .aux2_num_classes)\n\n \n .aux3_target = all_params.get(, )\n .use_aux3 = .aux3_target\n .use_aux3:\n .aux3_classifier = nn.Linear(.fusion_2.output_dim, )\n\n ():\n \n\n \n .fusion_channels = all_params.get(, )\n .fusion_dropout = all_params.get(, )\n .fusion_modality_dropout = all_params.get(, )\n\n \n active_features = []\n split_dims = []\n\n feat_name .features:\n feat_name .processor_out_dims:\n active_features.append(feat_name)\n split_dims.append(.processor_out_dims[feat_name])\n\n total_input_dim = (split_dims)\n\n total_input_dim == :\n ValueError()\n\n \n .fusion_1 = FusionProcessorNew(\n input_dim=total_input_dim,\n fusion_channels=.fusion_channels,\n dropout=.fusion_dropout,\n split_dims=(split_dims),\n modality_dropout=.fusion_modality_dropout\n )\n\n .fusion_2 = FusionProcessorNew(\n input_dim=total_input_dim,\n fusion_channels=.fusion_channels,\n dropout=.fusion_dropout,\n split_dims=(split_dims),\n modality_dropout=.fusion_modality_dropout\n )\n\n ():\n \n\n \n processor_configs = .all_params.get(, {})\n\n \n config = \n config_key processor_configs:\n config_key feat_name:\n config = processor_configs[config_key].copy()\n \n\n \n config :\n (.(feat_name))\n \n\n config.get(, ):\n ()\n \n\n (.(feat_name))\n\n \n config_for_processor = config.copy()\n config_for_processor.pop(, )\n\n config_for_processor[] == :\n config_for_processor.pop(, )\n \n .processors[feat_name] = TemporalProcessor(\n in_features=input_dim,\n **config_for_processor\n )\n config_for_processor[] == :\n config_for_processor.pop(, )\n \n .processors[feat_name] = TemporalProcessor_v2(\n in_features=input_dim,\n **config_for_processor\n )\n config_for_processor[] == :\n config_for_processor.pop(, )\n .processors[feat_name] = TripleTemporalProcessor(\n in_features=input_dim,\n **config_for_processor\n )\n config_for_processor[] == :\n config_for_processor.pop(, )\n .processors[feat_name] = DenseProcessor(\n in_features=input_dim,\n **config_for_processor\n )\n config_for_processor[] == :\n config_for_processor.pop(, )\n .processors[feat_name] = TofConv2DProcessor(\n in_features=input_dim,\n **config_for_processor\n )\n config_for_processor[] == :\n config_for_processor.pop(, )\n .processors[feat_name] = TofConv2DProcessor_SensorAsChannel(\n in_features=input_dim,\n **config_for_processor\n )\n config_for_processor[] == :\n config_for_processor.pop(, )\n .processors[feat_name] = TofConv2DProcessor_InceptionCBAM(\n in_features=input_dim,\n **config_for_processor\n )\n config_for_processor[] == :\n config_for_processor.pop(, )\n .processors[feat_name] = TofConv2DProcessor_AxialTCN(\n in_features=input_dim,\n **config_for_processor\n )\n config_for_processor[] == :\n config_for_processor.pop(, )\n .processors[feat_name] = TofConv2DProcessor_CrossSensorAttn(\n in_features=input_dim,\n **config_for_processor\n )\n\n .processor_out_dims[feat_name] = config[]\n\n ():\n\n processed_features = []\n\n \n feat_name .features:\n feat_key = \n feat_key features_dict feat_name .processors:\n feat_tensor = features_dict[feat_key]\n processed_feat = .processors[feat_name](feat_tensor)\n processed_features.append(processed_feat)\n\n \n processed_features:\n combined_features = torch.cat(processed_features, dim=)\n :\n ValueError()\n\n \n fused_features_1 = .fusion_1(combined_features)\n fused_features_2 = .fusion_2(combined_features)\n\n class_logits = .classifier(fused_features_1)\n binary_aux = torch.sigmoid(.binary_classifier(fused_features_2)).squeeze(-)\n\n (, , ) (, , ):\n aux2_logits = .aux2_classifier(fused_features_2)\n aux3_output = .aux3_classifier(fused_features_2).squeeze(-) \n class_logits, binary_aux, aux2_logits, aux3_output\n (, , ):\n aux2_logits = .aux2_classifier(fused_features_2)\n class_logits, binary_aux, aux2_logits\n (, , ):\n aux3_output = .aux3_classifier(fused_features_2).squeeze(-)\n class_logits, binary_aux, aux3_output\n :\n class_logits, binary_aux\nGreat write-up, Konstantin. I found it very engaging to read. I’m so glad we connected through the first CMI competition, and I really admire your skills with tabular data.
", "votes": null, "replies": [] }, { "id": 3281752, "author_name": "yekenot", "author_url": "", "post_date": "09/04/2025 20:41:17", "content": "Really admire your style of storytelling - it’s one of the best write-ups I’ve seen here. The way you share your journey from your own perspective, with such depth and empathy, makes readers feel as if they’re walking your path alongside you. Even the technical parts about NNs, paired with Junie’s description, come across as almost poetic.
", "votes": null, "replies": [] }, { "id": 3282635, "author_name": "kyakovlev", "author_url": "", "post_date": "09/06/2025 15:47:19", "content": "To keep all in one place
\nPost-processing
\nhttps://www.kaggle.com/code/kyakovlev/cmi-8th-place-postprocessing
hi man, it's a touchy read. Only people finish the read can relate.
\nbtw, how do you organise all the changes in like an IDE format? did you run the code locally or you have your local setup connect to Kaggle VM? How does the whole setup work to allow fast training and experiment at the same time
", "votes": null, "replies": [ { "id": 3284701, "author_name": "kyakovlev", "author_url": "", "post_date": "09/08/2025 14:01:34", "content": "I have a single local GPU and majority of changes were tested on single fold locally (in the notebook). Also had end to end \"pipeline\" but notebooks are more convenient for me.
\nFor bulk experiments I've used lightning.ai where I clone \"src\" folder and notebook that accepts config (tensorboard logs, oofs and weights downloaded to local machine for evaluation and comparison)
\ncandidate_name = \ntb_dir = \n\n\nwriter = SummaryWriter(log_dir=) \n\n k, v validation_results.items():\n (v, (, )):\n writer.add_scalar(, v, epoch)\n\nvalidation_results = {\n : val_loss,\n : gesture_f1,\n : gesture_roc_auc,\n : gesture_f1_mean,\n : gesture_roc_auc_mean,\n\n : g8_mean_roc_auc,\n\n : g8_gesture_f1,\n : g8_gesture_f1_clipped,\n : mean_match,\n : competition_metric,\n }\nthanks for replying. Let's say you finished running your notebook on local GPU, how do you submit to Kaggle then? Do you upload it to kaggle & rerun?
\nBesides, since you run this locally, does that mean for all competitions you take in you download the data set to your laptop? What happens to very huge data sets?
", "votes": null, "replies": [ { "id": 3286048, "author_name": "kyakovlev", "author_url": "", "post_date": "09/09/2025 06:54:28", "content": "You can create dataset on kaggle - upload your weights and make just inference.
\nIt’s interesting that in “code competitions” you have limited submission run time, but nobody asking you to train the model on fly - so you can train huge number of models locally and then upload to kaggle - during inference I’ve used 100+ models (if we consider each model fold as a separate model) and didn’t hit nor the memory limit nor time limit.
\n——
\nHuge datasets.
\nYes, I download original data, then carefully downcast data types and save in some format that is more compact and loads faster - parquet / feather / pickle (don’t use pickle). After such operations even the biggest kaggle datasets don’t consume more than 100mb - I don’t really remember any competition where the data size was a problem.
amazing thanks!
", "votes": null, "replies": [ { "id": 3286072, "author_name": "zhangyue199", "author_url": "", "post_date": "09/09/2025 07:50:58", "content": "btw, this organization is what I imagine a GM could be, pretty amazing. Do you have github where you use this structure that I can refer to? great learning for me
", "votes": null, "replies": [ { "id": 3286104, "author_name": "kyakovlev", "author_url": "", "post_date": "09/09/2025 08:27:23", "content": "I'm shamed to publish my code as I see how ugly it is - sorry no public github repo.
", "votes": null, "replies": [ { "id": 3286140, "author_name": "zhangyue199", "author_url": "", "post_date": "09/09/2025 09:50:55", "content": "very humble, if you don't mind sharing with me, this is my git
\nhttps://github.com/MJeremy2017