![\"GRU_model\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F20605011%2F7c556f5732790869f2d634d4cd8b1c55%2Fcmi2025_1.drawio.png?generation=1756909002205113&alt=media\")
First of all, thank you for organizing such a fantastic competition! Kudos to the organizers and all the participants for their hard work!
\nHere’s a brief overview of my approach.
\nI used an ensemble of transformer, GRU, and squeezeformer models. The models were divided into two categories: IMU only and Full data. For Full data, I used only GRU models. Below is the structure of the GRU model (other models are quite similar).
\n
A key feature was the addition of a 1D CNN layer after the input, rather than using GRU alone. This significantly boosted the cross-validation (CV) score.
\nIMU only: 2 transformer models + 2 GRU models + 1 squeezeformer model
\nFull data: 10 GRU models
\nSince this is time-series data, I mainly added features that incorporate information beyond the current row. For example, applying a rolling standard deviation to acc_x improved CV. I also included centered rolling statistics:
\ntrain_df[] = train_df.groupby()[].transform( x: x.rolling(window=, min_periods=).std())\ntrain_df[] = train_df.groupby()[].transform( x: x.rolling(window=, center=, min_periods=).std())\nInterestingly, something unexpected happened: I accidentally added a feature from thm twice, and for some reason, CV improved by 0.01. Although the reason is unclear, I kept it since it enhanced performance.
\nIncorporating EMA helped improve CV while also enhancing generalization.
\nThe tof data contained a fair amount of missing values, so proper null handling was essential. I used linear interpolation and median imputation to fill in missing values smoothly. Here’s the code used for null handling during training:
\n group_prefix tof_groups:\n group_cols = [col col train_df.columns col.startswith(group_prefix)]\n train_df[group_cols] = train_df[group_cols].replace(-, np.nan)\n train_df[group_cols] = train_df[group_cols].apply(\n row: row.interpolate(method=, limit_direction=, limit=), axis=\n )\n train_df[group_cols] = train_df[group_cols].apply(\n row: row.fillna(row.median()), axis=\n )\n train_df[group_cols] = train_df[group_cols].fillna(-)\nI set the mixup alpha to 1.0. This was intended to improve generalization, and it did lead to better CV.
\nModels with early stopping were treated as “best models,” while those without were considered “final models.” Both types were used in the ensemble. Here’s the breakdown:
\nIMU only:
\n2 transformer models (best & final)
\n2 GRU models (best & final)
\n1 squeezeformer model (final only)
\nFull data:
\n10 GRU models (2 versions × 5 seeds: best & final)
\nI used simple averaging for the ensemble.
\nAuxiliary Loss for orientation: It slightly improved CV but worsened the leaderboard (LB) score, so I didn’t adopt it.
\nThanks for reading!
", "messages": [ { "id": 3281048, "postDate": "2025-09-03T14:17:07.437Z", "content": "First of all, thank you for organizing such a fantastic competition! Kudos to the organizers and all the participants for their hard work!
\nHere’s a brief overview of my approach.
\nI used an ensemble of transformer, GRU, and squeezeformer models. The models were divided into two categories: IMU only and Full data. For Full data, I used only GRU models. Below is the structure of the GRU model (other models are quite similar).
\n
A key feature was the addition of a 1D CNN layer after the input, rather than using GRU alone. This significantly boosted the cross-validation (CV) score.
\nIMU only: 2 transformer models + 2 GRU models + 1 squeezeformer model
\nFull data: 10 GRU models
\nSince this is time-series data, I mainly added features that incorporate information beyond the current row. For example, applying a rolling standard deviation to acc_x improved CV. I also included centered rolling statistics:
\ntrain_df[] = train_df.groupby()[].transform( x: x.rolling(window=, min_periods=).std())\ntrain_df[] = train_df.groupby()[].transform( x: x.rolling(window=, center=, min_periods=).std())\nInterestingly, something unexpected happened: I accidentally added a feature from thm twice, and for some reason, CV improved by 0.01. Although the reason is unclear, I kept it since it enhanced performance.
\nIncorporating EMA helped improve CV while also enhancing generalization.
\nThe tof data contained a fair amount of missing values, so proper null handling was essential. I used linear interpolation and median imputation to fill in missing values smoothly. Here’s the code used for null handling during training:
\n group_prefix tof_groups:\n group_cols = [col col train_df.columns col.startswith(group_prefix)]\n train_df[group_cols] = train_df[group_cols].replace(-, np.nan)\n train_df[group_cols] = train_df[group_cols].apply(\n row: row.interpolate(method=, limit_direction=, limit=), axis=\n )\n train_df[group_cols] = train_df[group_cols].apply(\n row: row.fillna(row.median()), axis=\n )\n train_df[group_cols] = train_df[group_cols].fillna(-)\nI set the mixup alpha to 1.0. This was intended to improve generalization, and it did lead to better CV.
\nModels with early stopping were treated as “best models,” while those without were considered “final models.” Both types were used in the ensemble. Here’s the breakdown:
\nIMU only:
\n2 transformer models (best & final)
\n2 GRU models (best & final)
\n1 squeezeformer model (final only)
\nFull data:
\n10 GRU models (2 versions × 5 seeds: best & final)
\nI used simple averaging for the ensemble.
\nAuxiliary Loss for orientation: It slightly improved CV but worsened the leaderboard (LB) score, so I didn’t adopt it.
\nThanks for reading!
", "rawMarkdown": "First of all, thank you for organizing such a fantastic competition! Kudos to the organizers and all the participants for their hard work!\n\nHere’s a brief overview of my approach.\n\n## Models Used\nI used an ensemble of transformer, GRU, and squeezeformer models. The models were divided into two categories: IMU only and Full data. For Full data, I used only GRU models. Below is the structure of the GRU model (other models are quite similar).\n\n\n\nA key feature was the addition of a 1D CNN layer after the input, rather than using GRU alone. This significantly boosted the cross-validation (CV) score.\n\n### Best Submission (Overview)\nIMU only: 2 transformer models + 2 GRU models + 1 squeezeformer model\n\nFull data: 10 GRU models\n\n## Feature Engineering\nSince this is time-series data, I mainly added features that incorporate information beyond the current row. For example, applying a rolling standard deviation to acc_x improved CV. I also included centered rolling statistics:\n\n```python\ntrain_df['acc_x_std'] = train_df.groupby('sequence_id')['acc_x'].transform(lambda x: x.rolling(window=5, min_periods=1).std())\ntrain_df['acc_x_center_std'] = train_df.groupby('sequence_id')['acc_x'].transform(lambda x: x.rolling(window=5, center=True, min_periods=1).std())\n```\n\nInterestingly, something unexpected happened: I accidentally added a feature from thm twice, and for some reason, CV improved by 0.01. Although the reason is unclear, I kept it since it enhanced performance.\n\n## EMA\nIncorporating EMA helped improve CV while also enhancing generalization.\n\n## Null Handling\nThe tof data contained a fair amount of missing values, so proper null handling was essential. I used linear interpolation and median imputation to fill in missing values smoothly. Here’s the code used for null handling during training:\n\n```python\nfor group_prefix in tof_groups:\n group_cols = [col for col in train_df.columns if col.startswith(group_prefix)]\n train_df[group_cols] = train_df[group_cols].replace(-1, np.nan)\n train_df[group_cols] = train_df[group_cols].apply(\n lambda row: row.interpolate(method='linear', limit_direction='both', limit=5), axis=1\n )\n train_df[group_cols] = train_df[group_cols].apply(\n lambda row: row.fillna(row.median()), axis=1\n )\n train_df[group_cols] = train_df[group_cols].fillna(-1)\n```\n\n## Mixup\nI set the mixup alpha to 1.0. This was intended to improve generalization, and it did lead to better CV.\n\n## Best Models and Final Models\nModels with early stopping were treated as “best models,” while those without were considered “final models.” Both types were used in the ensemble. Here’s the breakdown:\n\nIMU only:\n\n2 transformer models (best & final)\n\n2 GRU models (best & final)\n\n1 squeezeformer model (final only)\n\nFull data:\n\n10 GRU models (2 versions × 5 seeds: best & final)\n\n## Ensemble Method\nI used simple averaging for the ensemble.\n\n## What Didn’t Work\nAuxiliary Loss for orientation: It slightly improved CV but worsened the leaderboard (LB) score, so I didn’t adopt it.\n\n\n\nThanks for reading!\n", "votes": 10 } ], "comments": [], "raw_markdown_by_id": { "3281048": "First of all, thank you for organizing such a fantastic competition! Kudos to the organizers and all the participants for their hard work!\n\nHere’s a brief overview of my approach.\n\n## Models Used\nI used an ensemble of transformer, GRU, and squeezeformer models. The models were divided into two categories: IMU only and Full data. For Full data, I used only GRU models. Below is the structure of the GRU model (other models are quite similar).\n\n\n\nA key feature was the addition of a 1D CNN layer after the input, rather than using GRU alone. This significantly boosted the cross-validation (CV) score.\n\n### Best Submission (Overview)\nIMU only: 2 transformer models + 2 GRU models + 1 squeezeformer model\n\nFull data: 10 GRU models\n\n## Feature Engineering\nSince this is time-series data, I mainly added features that incorporate information beyond the current row. For example, applying a rolling standard deviation to acc_x improved CV. I also included centered rolling statistics:\n\n```python\ntrain_df['acc_x_std'] = train_df.groupby('sequence_id')['acc_x'].transform(lambda x: x.rolling(window=5, min_periods=1).std())\ntrain_df['acc_x_center_std'] = train_df.groupby('sequence_id')['acc_x'].transform(lambda x: x.rolling(window=5, center=True, min_periods=1).std())\n```\n\nInterestingly, something unexpected happened: I accidentally added a feature from thm twice, and for some reason, CV improved by 0.01. Although the reason is unclear, I kept it since it enhanced performance.\n\n## EMA\nIncorporating EMA helped improve CV while also enhancing generalization.\n\n## Null Handling\nThe tof data contained a fair amount of missing values, so proper null handling was essential. I used linear interpolation and median imputation to fill in missing values smoothly. Here’s the code used for null handling during training:\n\n```python\nfor group_prefix in tof_groups:\n group_cols = [col for col in train_df.columns if col.startswith(group_prefix)]\n train_df[group_cols] = train_df[group_cols].replace(-1, np.nan)\n train_df[group_cols] = train_df[group_cols].apply(\n lambda row: row.interpolate(method='linear', limit_direction='both', limit=5), axis=1\n )\n train_df[group_cols] = train_df[group_cols].apply(\n lambda row: row.fillna(row.median()), axis=1\n )\n train_df[group_cols] = train_df[group_cols].fillna(-1)\n```\n\n## Mixup\nI set the mixup alpha to 1.0. This was intended to improve generalization, and it did lead to better CV.\n\n## Best Models and Final Models\nModels with early stopping were treated as “best models,” while those without were considered “final models.” Both types were used in the ensemble. Here’s the breakdown:\n\nIMU only:\n\n2 transformer models (best & final)\n\n2 GRU models (best & final)\n\n1 squeezeformer model (final only)\n\nFull data:\n\n10 GRU models (2 versions × 5 seeds: best & final)\n\n## Ensemble Method\nI used simple averaging for the ensemble.\n\n## What Didn’t Work\nAuxiliary Loss for orientation: It slightly improved CV but worsened the leaderboard (LB) score, so I didn’t adopt it.\n\n\n\nThanks for reading!\n" } }