{ "id": 459715, "title": "1st place solution", "url": "/competitions/child-mind-institute-detect-sleep-states/writeups/shimacos-vs-sakami-vs-kami-1st-place-solution", "author_name": "", "post_date": "2024-01-06T07:47:16.310Z", "votes": 168, "comment_count": 33, "views": 0, "content": "

First of all, I would like to express gratitude to all participants and the competition host. It was a challenging competition, but I am pleased with the positive outcome and feel relieved.

\n

Here is a brief summary of our solution.
\nYou can check our code here.

\n

Single model

\n

Here's a log on how to improve the CV score after the summary. The final scores were: CV: 0.8206, public LB: 0.768, private LB: 0.829 (equivalent to 9th place).

\n

Summary

\n

Model structure

\n

The model structure is primarily based on this amazing notebook, with a structure comprising:
\nCNN (down sample) → Residual GRU → CNN (up sample)

\n

\"\"

\n\n

For input scaling, SEModule was utilized. (https://arxiv.org/abs/1709.01507)

\n
 (nn.Module):\n    () -> :\n       ().__init__()\n       self.fc1 = nn.Linear(ch, r)\n       self.fc2 = nn.Linear(r, ch)\n\n    () -> torch.FloatTensor:\n       h = self.fc1(x)\n       h = F.relu(h)\n       h = self.fc2(h).sigmoid()\n        h * x\n
\n\n

As noted in several discussions and notebooks, there was a bias in the minute when ground truth events occurred. To account for this, features related to minutes were concatenated separately in the final layer.

\n
    () -> torch.FloatTensor:\n       cat_embeddings = [embedding(cat_x[:, :, i])  i, embedding  (self.category_embeddings)]\n       num_x = self.numerical_linear(num_x)\n\n       x = torch.cat([num_x] + cat_embeddings, dim=)\n       x = self.input_linear(x)\n       x = self.conv(x.transpose(-, -)).transpose(-, -)\n\n        gru  self.gru_layers:\n           x, _ = gru(x)\n\n       x = self.dconv(x.transpose(-, -)).transpose(-, -)\n       minute_embedding = self.minute_embedding(cat_x[:, :, ])\n       x = self.output_linear(torch.cat([x, minute_embedding], dim=))\n        x\n
\n

Data Preparation

\n

Each series of data is divided into daily chunks, offset by 0.35 days.

\n
       train_df = train_df.with_columns(pl.arange(, pl.count()).alias())\n       series_row_ids = (train_df.group_by().agg().rows())\n\n       series_chunk_ids = []  \n       series_chunk_row_ids = []  \n        series_id, row_ids  tqdm(series_row_ids.items(), desc=):\n            start_idx  (, (row_ids), (config.stride_size / config.epoch_sample_rate)):\n                start_idx + config.chunk_size <= (row_ids):\n                   chunk_row_ids = row_ids[start_idx : start_idx + config.chunk_size]\n                   series_chunk_ids.append(series_id)\n                   series_chunk_row_ids.append(np.array(chunk_row_ids))\n               :\n                   chunk_row_ids = row_ids[-config.chunk_size :]\n                   series_chunk_ids.append(series_id)\n                   series_chunk_row_ids.append(np.array(chunk_row_ids))\n                   \n
\n

During training, half of each chunk is used in every epoch.

\n
               sampled_train_idx = train_idx[epoch % config.epoch_sample_rate :: config.epoch_sample_rate]\n
\n

For evaluation, overlapping sections are averaged, and the ends of each chunk are trimmed by 30 minutes.

\n

Target

\n

A decaying target is created based on the distance from the ground truth event, with diminishing values as the distance increases.

\n

\"\"

\n
tolerance_steps = [, , , , , , , , , ]\ntarget_columns = [, ]\n\ntrain_df = (\n   train_df.join(train_events_df.select([, , ]), on=[, ], how=)\n   .to_dummies(columns=[])\n   .with_columns(\n       pl.max_horizontal(\n           pl.col(target_columns)\n           .rolling_max(window_size *  - , min_periods=, center=)\n           .over()\n           * ( - i / (tolerance_steps))\n            i, window_size  (tolerance_steps)\n       )\n   )\n)\n
\n

The target is updated each epoch to decay further.

\n
               \n               targets = np.where(targets == , , (targets - ( / config.n_epochs)).clip(=))\n
\n

By attenuating the target, the range of predicted values narrows, allowing for the detection of finer peaks.

\n

\"\"

\n

Periodicity Filter

\n

As discussed in here, there is a daily periodicity in the data when the measuring device is removed. This is leveraged to predict these periods rule-based and used as a filter for inputs and predictions.

\n

\"\"

\n

Input Features

\n\n

Change logs

\n\n
\n

Post Processing

\n

This post-processing creates a submission DataFrame to optimize the evaluation metrics. With this post-processing method, we significantly improved our scores (public: 0.768 → 0.790, private: 0.829 → 0.852 !!!).

\n

This was a complex procedure, which I will explain step by step.

\n
    \n

  1. Data Characteristics

    \n

    First, let's discuss the characteristics of the data. As noted in several discussions and notebooks, the second of the target events was always set to zero.
    \nThe competition's evaluation metric doesn't differentiate predictions within a 30-second range from the ground truth event. So, whether the submission timestamp's seconds are 5, 10, 15, 20, … 25, the same score is returned.

    \n

    \"\"

    2. \n

  2. Creation of the 2nd Level Model

    \n

    The 1st level model's predictions were trained to recognize events within a certain range from the ground truth as positive. However, the 2nd level model transforms these into probabilities of a ground truth event existing for each minute.
    \nThe output of the 1st level model was at seconds 0, 5, 10, …, but the 2nd level model aggregates these to always be at second 0. Specifically, it inputs aggregated features around hh:mm:00 and learns to predict 1 only at the exact time of an event, otherwise 0. Details of the 2nd level model will be described later.

    \n

    \"\"

    3. \n

  3. Score Calculation for Each Point

    \n

    As explained earlier, submitting any second within the same minute yields the same score. Therefore, we estimate the score at the 15 and 45 second points of each minute, and submit the one with the highest value, effectively submitting the highest score for all points. The method of score estimation is as follows:

    \n

    For instance, let's estimate the score at 10:00:15.

    \n

    \"\"

    \n

    First, we create a window of 12 steps from the point of interest and sum the predictions of the 2nd level model within this window to calculate the tolerance_12_score.

    \n

    \"\"

    \n

    Similarly, we calculate tolerance_36_score, tolerance_60_score, …, for the respective tolerances used in the evaluation, and the sum of these scores is considered the score for the point of interest.

    \n

    \"\"

    \n

    We perform this calculation for all points, and for each series, we adopt the point with the highest score and add it to the submission DataFrame.

    \n

    \"\"

    4. \n

  4. Score Recalculation

    \n

    Next, we recalculate the score to determine the next point to be adopted. For example, suppose the point 09:59:15 was chosen.

    \n

    First, consider updating the tolerance_12_score. Events within tolerance 12 of the adopted point cannot match overlappingly with the next point to be submitted.

    \n

    \"\"

    \n

    Therefore, when calculating the tolerance_12_score for the next point to be adopted, it's necessary to discount the prediction values within 12 steps of the currently adopted point.

    \n

    \"\"

    \n

    Likewise, for tolerance_36_score, tolerance_60_score, …, we recalculate by discounting the prediction values within 36, 60, …, steps of the adopted point.

    \n

    \"\"

    \n

    With the updated scores calculated, we again adopt the highest scoring point for each series and add it to the submission dataframe.

    \n

    \"\"

    5. \n

  5. Creating Submissions

    \n

    We repeat the above step 4 to extract a sufficient number of submission points, then compile these into a DataFrame to create the submission file.

    6. \n
\n

Additional Techniques

\n

Several other techniques were employed to make the post-processing work effectively:

\n\n

Details of the 2nd Level Model

\n\n

\"\"

\n\n

\"\"

\n\n

\"\"

", "messages": [ { "id": "2550872", "postDate": "12/06/2023 11:01:43", "content": "

First of all, I would like to express gratitude to all participants and the competition host. It was a challenging competition, but I am pleased with the positive outcome and feel relieved.

\n

Here is a brief summary of our solution.
\nYou can check our code here.

\n

Single model

\n

Here's a log on how to improve the CV score after the summary. The final scores were: CV: 0.8206, public LB: 0.768, private LB: 0.829 (equivalent to 9th place).

\n

Summary

\n

Model structure

\n

The model structure is primarily based on this amazing notebook, with a structure comprising:
\nCNN (down sample) → Residual GRU → CNN (up sample)

\n

\"\"

\n\n

For input scaling, SEModule was utilized. (https://arxiv.org/abs/1709.01507)

\n
 (nn.Module):\n    () -> :\n       ().__init__()\n       self.fc1 = nn.Linear(ch, r)\n       self.fc2 = nn.Linear(r, ch)\n\n    () -> torch.FloatTensor:\n       h = self.fc1(x)\n       h = F.relu(h)\n       h = self.fc2(h).sigmoid()\n        h * x\n
\n\n

As noted in several discussions and notebooks, there was a bias in the minute when ground truth events occurred. To account for this, features related to minutes were concatenated separately in the final layer.

\n
    () -> torch.FloatTensor:\n       cat_embeddings = [embedding(cat_x[:, :, i])  i, embedding  (self.category_embeddings)]\n       num_x = self.numerical_linear(num_x)\n\n       x = torch.cat([num_x] + cat_embeddings, dim=)\n       x = self.input_linear(x)\n       x = self.conv(x.transpose(-, -)).transpose(-, -)\n\n        gru  self.gru_layers:\n           x, _ = gru(x)\n\n       x = self.dconv(x.transpose(-, -)).transpose(-, -)\n       minute_embedding = self.minute_embedding(cat_x[:, :, ])\n       x = self.output_linear(torch.cat([x, minute_embedding], dim=))\n        x\n
\n

Data Preparation

\n

Each series of data is divided into daily chunks, offset by 0.35 days.

\n
       train_df = train_df.with_columns(pl.arange(, pl.count()).alias())\n       series_row_ids = (train_df.group_by().agg().rows())\n\n       series_chunk_ids = []  \n       series_chunk_row_ids = []  \n        series_id, row_ids  tqdm(series_row_ids.items(), desc=):\n            start_idx  (, (row_ids), (config.stride_size / config.epoch_sample_rate)):\n                start_idx + config.chunk_size <= (row_ids):\n                   chunk_row_ids = row_ids[start_idx : start_idx + config.chunk_size]\n                   series_chunk_ids.append(series_id)\n                   series_chunk_row_ids.append(np.array(chunk_row_ids))\n               :\n                   chunk_row_ids = row_ids[-config.chunk_size :]\n                   series_chunk_ids.append(series_id)\n                   series_chunk_row_ids.append(np.array(chunk_row_ids))\n                   \n
\n

During training, half of each chunk is used in every epoch.

\n
               sampled_train_idx = train_idx[epoch % config.epoch_sample_rate :: config.epoch_sample_rate]\n
\n

For evaluation, overlapping sections are averaged, and the ends of each chunk are trimmed by 30 minutes.

\n

Target

\n

A decaying target is created based on the distance from the ground truth event, with diminishing values as the distance increases.

\n

\"\"

\n
tolerance_steps = [, , , , , , , , , ]\ntarget_columns = [, ]\n\ntrain_df = (\n   train_df.join(train_events_df.select([, , ]), on=[, ], how=)\n   .to_dummies(columns=[])\n   .with_columns(\n       pl.max_horizontal(\n           pl.col(target_columns)\n           .rolling_max(window_size *  - , min_periods=, center=)\n           .over()\n           * ( - i / (tolerance_steps))\n            i, window_size  (tolerance_steps)\n       )\n   )\n)\n
\n

The target is updated each epoch to decay further.

\n
               \n               targets = np.where(targets == , , (targets - ( / config.n_epochs)).clip(=))\n
\n

By attenuating the target, the range of predicted values narrows, allowing for the detection of finer peaks.

\n

\"\"

\n

Periodicity Filter

\n

As discussed in here, there is a daily periodicity in the data when the measuring device is removed. This is leveraged to predict these periods rule-based and used as a filter for inputs and predictions.

\n

\"\"

\n

Input Features

\n\n

Change logs

\n\n
\n

Post Processing

\n

This post-processing creates a submission DataFrame to optimize the evaluation metrics. With this post-processing method, we significantly improved our scores (public: 0.768 → 0.790, private: 0.829 → 0.852 !!!).

\n

This was a complex procedure, which I will explain step by step.

\n
    \n

  1. Data Characteristics

    \n

    First, let's discuss the characteristics of the data. As noted in several discussions and notebooks, the second of the target events was always set to zero.
    \nThe competition's evaluation metric doesn't differentiate predictions within a 30-second range from the ground truth event. So, whether the submission timestamp's seconds are 5, 10, 15, 20, … 25, the same score is returned.

    \n

    \"\"

    2. \n

  2. Creation of the 2nd Level Model

    \n

    The 1st level model's predictions were trained to recognize events within a certain range from the ground truth as positive. However, the 2nd level model transforms these into probabilities of a ground truth event existing for each minute.
    \nThe output of the 1st level model was at seconds 0, 5, 10, …, but the 2nd level model aggregates these to always be at second 0. Specifically, it inputs aggregated features around hh:mm:00 and learns to predict 1 only at the exact time of an event, otherwise 0. Details of the 2nd level model will be described later.

    \n

    \"\"

    3. \n

  3. Score Calculation for Each Point

    \n

    As explained earlier, submitting any second within the same minute yields the same score. Therefore, we estimate the score at the 15 and 45 second points of each minute, and submit the one with the highest value, effectively submitting the highest score for all points. The method of score estimation is as follows:

    \n

    For instance, let's estimate the score at 10:00:15.

    \n

    \"\"

    \n

    First, we create a window of 12 steps from the point of interest and sum the predictions of the 2nd level model within this window to calculate the tolerance_12_score.

    \n

    \"\"

    \n

    Similarly, we calculate tolerance_36_score, tolerance_60_score, …, for the respective tolerances used in the evaluation, and the sum of these scores is considered the score for the point of interest.

    \n

    \"\"

    \n

    We perform this calculation for all points, and for each series, we adopt the point with the highest score and add it to the submission DataFrame.

    \n

    \"\"

    4. \n

  4. Score Recalculation

    \n

    Next, we recalculate the score to determine the next point to be adopted. For example, suppose the point 09:59:15 was chosen.

    \n

    First, consider updating the tolerance_12_score. Events within tolerance 12 of the adopted point cannot match overlappingly with the next point to be submitted.

    \n

    \"\"

    \n

    Therefore, when calculating the tolerance_12_score for the next point to be adopted, it's necessary to discount the prediction values within 12 steps of the currently adopted point.

    \n

    \"\"

    \n

    Likewise, for tolerance_36_score, tolerance_60_score, …, we recalculate by discounting the prediction values within 36, 60, …, steps of the adopted point.

    \n

    \"\"

    \n

    With the updated scores calculated, we again adopt the highest scoring point for each series and add it to the submission dataframe.

    \n

    \"\"

    5. \n

  5. Creating Submissions

    \n

    We repeat the above step 4 to extract a sufficient number of submission points, then compile these into a DataFrame to create the submission file.

    6. \n
\n

Additional Techniques

\n

Several other techniques were employed to make the post-processing work effectively:

\n\n

Details of the 2nd Level Model

\n\n

\"\"

\n\n

\"\"

\n\n

\"\"

", "rawMarkdown": "First of all, I would like to express gratitude to all participants and the competition host. It was a challenging competition, but I am pleased with the positive outcome and feel relieved.\n\nHere is a brief summary of our solution.\nYou can check our code [here](https://github.com/sakami0000/child-mind-institute-detect-sleep-states-1st-place).\n\n## Single model\n\nHere's a log on how to improve the CV score after the summary. The final scores were: CV: 0.8206, public LB: 0.768, private LB: 0.829 (equivalent to 9th place).\n\n### Summary\n\n#### Model structure\n\nThe model structure is primarily based on [this amazing notebook](https://www.kaggle.com/code/danielphalen/cmss-grunet-train), with a structure comprising:\nCNN (down sample) → Residual GRU → CNN (up sample)\n\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F2133818%2Fcc6a10ba62b3a3d9361b36e8b162ca5d%2Fmodel_structure.jpeg?generation=1701857831021267&alt=media)\n\n- SEScale\n\nFor input scaling, SEModule was utilized. (https://arxiv.org/abs/1709.01507)\n```python\nclass SEScale(nn.Module):\n def __init__(self, ch: int, r: int) -> None:\n super().__init__()\n self.fc1 = nn.Linear(ch, r)\n self.fc2 = nn.Linear(r, ch)\n\n def forward(self, x: torch.FloatTensor) -> torch.FloatTensor:\n h = self.fc1(x)\n h = F.relu(h)\n h = self.fc2(h).sigmoid()\n return h * x\n```\n\n- Minute connection\n\nAs noted in several discussions and notebooks, there was a bias in the minute when ground truth events occurred. To account for this, features related to minutes were concatenated separately in the final layer.\n\n```python\n def forward(self, num_x: torch.FloatTensor, cat_x: torch.LongTensor) -> torch.FloatTensor:\n cat_embeddings = [embedding(cat_x[:, :, i]) for i, embedding in enumerate(self.category_embeddings)]\n num_x = self.numerical_linear(num_x)\n\n x = torch.cat([num_x] + cat_embeddings, dim=2)\n x = self.input_linear(x)\n x = self.conv(x.transpose(-1, -2)).transpose(-1, -2)\n\n for gru in self.gru_layers:\n x, _ = gru(x)\n\n x = self.dconv(x.transpose(-1, -2)).transpose(-1, -2)\n minute_embedding = self.minute_embedding(cat_x[:, :, 0])\n x = self.output_linear(torch.cat([x, minute_embedding], dim=2))\n return x\n```\n\n#### Data Preparation\n\nEach series of data is divided into daily chunks, offset by 0.35 days.\n\n```python\n train_df = train_df.with_columns(pl.arange(0, pl.count()).alias(\"row_id\"))\n series_row_ids = dict(train_df.group_by(\"series_id\").agg(\"row_id\").rows())\n\n series_chunk_ids = [] # list[str]\n series_chunk_row_ids = [] # list[list[int]]\n for series_id, row_ids in tqdm(series_row_ids.items(), desc=\"split into chunks\"):\n for start_idx in range(0, len(row_ids), int(config.stride_size / config.epoch_sample_rate)):\n if start_idx + config.chunk_size <= len(row_ids):\n chunk_row_ids = row_ids[start_idx : start_idx + config.chunk_size]\n series_chunk_ids.append(series_id)\n series_chunk_row_ids.append(np.array(chunk_row_ids))\n else:\n chunk_row_ids = row_ids[-config.chunk_size :]\n series_chunk_ids.append(series_id)\n series_chunk_row_ids.append(np.array(chunk_row_ids))\n break\n```\n\nDuring training, half of each chunk is used in every epoch.\n\n```python\n sampled_train_idx = train_idx[epoch % config.epoch_sample_rate :: config.epoch_sample_rate]\n```\n\nFor evaluation, overlapping sections are averaged, and the ends of each chunk are trimmed by 30 minutes.\n\n#### Target\n\nA decaying target is created based on the distance from the ground truth event, with diminishing values as the distance increases.\n\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F2133818%2Fc6d809492da742bbcb9252615375e76d%2Ftarget.png?generation=1701858069231326&alt=media)\n\n```python\ntolerance_steps = [12, 36, 60, 90, 120, 150, 180, 240, 300, 360]\ntarget_columns = [\"event_onset\", \"event_wakeup\"]\n\ntrain_df = (\n train_df.join(train_events_df.select([\"series_id\", \"step\", \"event\"]), on=[\"series_id\", \"step\"], how=\"left\")\n .to_dummies(columns=[\"event\"])\n .with_columns(\n pl.max_horizontal(\n pl.col(target_columns)\n .rolling_max(window_size * 2 - 1, min_periods=1, center=True)\n .over(\"series_id\")\n * (1 - i / len(tolerance_steps))\n for i, window_size in enumerate(tolerance_steps)\n )\n )\n)\n```\n\nThe target is updated each epoch to decay further.\n\n```python\n # update target\n targets = np.where(targets == 1.0, 1.0, (targets - (1.0 / config.n_epochs)).clip(min=0.0))\n```\n\nBy attenuating the target, the range of predicted values narrows, allowing for the detection of finer peaks.\n\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F2133818%2F002ebac2c4da25cdde7a81b27f1ec492%2F2023-12-09%200.15.31.png?generation=1702048562641170&alt=media)\n\n#### Periodicity Filter\n\nAs discussed in [here](https://www.kaggle.com/competitions/child-mind-institute-detect-sleep-states/discussion/456177), there is a daily periodicity in the data when the measuring device is removed. This is leveraged to predict these periods rule-based and used as a filter for inputs and predictions.\n\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F2133818%2Fbd12584427bc9e2e628e5a095419ba2f%2F2023-12-06%2015.11.06.png?generation=1701858205750939&alt=media)\n\n#### Input Features\n\n- categorical features\n - hour\n - minute\n - weekday\n - periodicity flag\n- numerical features\n - anglez / 45\n - enmo.log1p().clip_max(1.0) / 0.1\n - anglez, enmo 12 steps rolling_mean, rolling_std, rolling_max\n - anglez_diff_abs 5 min rolling median\n\n### Change logs\n\n- baseline model (cv: 0.7510) - public: 0.728\n- Add a process to decay the target every epoch (cv: 0.7699, +19pt)\n- Add a periodicity filter to the output (cv: 0.7807, +11pt)\n- Add a periodicity flag to the input as well (cv: 0.7870, +6pt) - public: 0.739\n- batch_size: 16 → 4, hidden_size: 128 → 64, num_layers: 2 → 8 (cv: 0.7985, +11pt) - public: 0.755\n- Normalize the score in the submission file by the daily score sum (cv: 0.8044, +6pt)\n- Remove month and day from the input (cv: 0.8117, +7pt)\n- Trim the edges of the chunk by 30 minutes on both sides (cv: 0.8142, +4pt) - public: 0.765\n- Modify to concatenate the minute features to the final layer (cv: 0.8206, +6pt) - public: 0.768\n\n----\n\n## Post Processing\n\nThis post-processing creates a submission DataFrame to optimize the evaluation metrics. With this post-processing method, we significantly improved our scores (public: 0.768 → **0.790**, private: 0.829 → **0.852** !!!).\n\nThis was a complex procedure, which I will explain step by step.\n\n1. **Data Characteristics**\n\n First, let's discuss the characteristics of the data. As noted in several discussions and notebooks, the second of the target events was always set to zero.\n The competition's evaluation metric doesn't differentiate predictions within a 30-second range from the ground truth event. So, whether the submission timestamp's seconds are 5, 10, 15, 20, ... 25, the same score is returned.\n\n \"\"\n\n2. **Creation of the 2nd Level Model**\n\n The 1st level model's predictions were trained to recognize events within a certain range from the ground truth as positive. However, the 2nd level model transforms these into probabilities of a ground truth event existing for each minute.\n The output of the 1st level model was at seconds 0, 5, 10, ..., but the 2nd level model aggregates these to always be at second 0. Specifically, it inputs aggregated features around hh:mm:00 and learns to predict 1 only at the exact time of an event, otherwise 0. Details of the 2nd level model will be described later.\n\n \"\"\n\n3. **Score Calculation for Each Point**\n\n As explained earlier, submitting any second within the same minute yields the same score. Therefore, we estimate the score at the 15 and 45 second points of each minute, and submit the one with the highest value, effectively submitting the highest score for all points. The method of score estimation is as follows:\n\n For instance, let's estimate the score at 10:00:15.\n\n \"\"\n\n First, we create a window of 12 steps from the point of interest and sum the predictions of the 2nd level model within this window to calculate the `tolerance_12_score`.\n\n \"\"\n\n Similarly, we calculate `tolerance_36_score`, `tolerance_60_score`, ..., for the respective tolerances used in the evaluation, and the sum of these scores is considered the score for the point of interest.\n\n \"\"\n\n We perform this calculation for all points, and for each series, we adopt the point with the highest score and add it to the submission DataFrame.\n\n \"\"\n\n4. **Score Recalculation**\n\n Next, we recalculate the score to determine the next point to be adopted. For example, suppose the point 09:59:15 was chosen.\n\n First, consider updating the `tolerance_12_score`. Events within tolerance 12 of the adopted point cannot match overlappingly with the next point to be submitted.\n\n \"\"\n\n Therefore, when calculating the `tolerance_12_score` for the next point to be adopted, it's necessary to discount the prediction values within 12 steps of the currently adopted point.\n\n \"\"\n\n Likewise, for `tolerance_36_score`, `tolerance_60_score`, ..., we recalculate by discounting the prediction values within 36, 60, ..., steps of the adopted point.\n\n \"\"\n\n With the updated scores calculated, we again adopt the highest scoring point for each series and add it to the submission dataframe.\n\n \"\"\n\n5. **Creating Submissions**\n\n We repeat the above step 4 to extract a sufficient number of submission points, then compile these into a DataFrame to create the submission file.\n\n### Additional Techniques\n\nSeveral other techniques were employed to make the post-processing work effectively:\n\n- Normalize the predictions of the 2nd level model daily.\n- When recalculating the score, calculate the difference from the previous score to reduce the computation.\n- Speed up the above calculations using JIT compilation.\n\n### Details of the 2nd Level Model\n\n- The 2nd level model starts by averaging the 1st level model's predictions on a per-minute basis and then detecting peaks in these averages using `find_peaks` with a height of 0.001 and a distance of 8.\n- Based on the detected peaks, chunks are created from the original time series, capturing 8 minutes before and after each peak. (Recall: 0.9845)\n - This step_size was crucial because the ratio of positive to negative examples changes depending on how many steps are included, affecting the accuracy of subsequent stages. Therefore, we tuned the number of steps for optimal performance.\n - If chunks are connected, they are treated as a single chunk.\n\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F2133818%2F907332d70498f46059a2935461f69941%2F2023-12-06%2019.42.55.png?generation=1701859390836314&alt=media)\n\n- For each chunk, we aggregated features from the 1st model's predictions and other features like anglez and enmo. These aggregated features were then used to train models such as LightGBM and CatBoost.\n- Additionally, we treated each chunk as a sequence for training CNN-RNN, CNN, and Transformer models. As a result, we developed a model that could account for minute biases not fully addressed by the 1st level model.\n\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F2133818%2F4b8ec9d1aecb6172e700a8ce485d226f%2F2023-12-06%2019.43.55.png?generation=1701859451118326&alt=media)\n\n- The predictions of the 2nd level model were sufficiently calibrated, so there was no need for further transformation.\n\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F2133818%2Fad0665bfff9828ebc6516a54e4e20276%2F2024-01-06%2016.46.51.png?generation=1704527223913793&alt=media)", "votes": null }, { "id": "2550900", "postDate": "12/06/2023 11:41:24", "content": "

The winning model is from @danielphalen 's notebook! Amazing work!

", "rawMarkdown": "The winning model is from @danielphalen 's notebook! Amazing work!", "votes": null }, { "id": "2550915", "postDate": "12/06/2023 11:53:00", "content": "
\n

As noted in several discussions and notebooks, there was a bias in the minute when ground truth events occurred.

\n
\n

Could you please comment about the bias more?

", "rawMarkdown": ">As noted in several discussions and notebooks, there was a bias in the minute when ground truth events occurred.\n\nCould you please comment about the bias more?", "votes": null }, { "id": "2550926", "postDate": "12/06/2023 12:03:48", "content": "

Please refer to this discussion.

", "rawMarkdown": "Please refer to [this discussion](https://www.kaggle.com/competitions/child-mind-institute-detect-sleep-states/discussion/444374).", "votes": null }, { "id": "2550987", "postDate": "12/06/2023 12:49:16", "content": "

@sakami . Amazing work . Congratulations on winning the comp .

", "rawMarkdown": "sakami . Amazing work . Congratulations on winning the comp .", "votes": null }, { "id": "2551264", "postDate": "12/06/2023 15:38:28", "content": "

All I can say is wow! I will have to spend more time on this one, incredible solution. A solo model at 9th is honestly just as impressive as the win!

", "rawMarkdown": "All I can say is wow! I will have to spend more time on this one, incredible solution. A solo model at 9th is honestly just as impressive as the win!", "votes": null }, { "id": "2551286", "postDate": "12/06/2023 15:48:50", "content": "

Thank you. I get it.

", "rawMarkdown": "Thank you. I get it.", "votes": null }, { "id": "2551322", "postDate": "12/06/2023 16:18:45", "content": "

Great Work! Did you adjust the model to add the skip connections? Also, did you keep the KLDivLoss, or did you use a different loss function?

", "rawMarkdown": "Great Work! Did you adjust the model to add the skip connections? Also, did you keep the KLDivLoss, or did you use a different loss function?", "votes": null }, { "id": "2551375", "postDate": "12/06/2023 16:57:42", "content": "

Having started with the same public notebook, really cool how this progressed! Congratulations

", "rawMarkdown": "Having started with the same public notebook, really cool how this progressed! Congratulations", "votes": null }, { "id": "2551633", "postDate": "12/06/2023 20:48:00", "content": "

Congrats! Now I'm thinking that step function makes more sense than the Gaussian target because of the metric.

", "rawMarkdown": "Congrats! Now I'm thinking that step function makes more sense than the Gaussian target because of the metric.", "votes": null }, { "id": "2551806", "postDate": "12/07/2023 01:46:39", "content": "
\n

Did you adjust the model to add the skip connections?

\n
\n

Yes. Additionally, I have added several more layers, but the underlying architecture remains unchanged.

\n
\n

Also, did you keep the KLDivLoss, or did you use a different loss function?

\n
\n

In my case, using BCEWithLogitsLoss worked better.

\n

I found your notebook to be very helpful. Thank you!!

", "rawMarkdown": "> Did you adjust the model to add the skip connections?\n\nYes. Additionally, I have added several more layers, but the underlying architecture remains unchanged.\n\n> Also, did you keep the KLDivLoss, or did you use a different loss function?\n\nIn my case, using BCEWithLogitsLoss worked better.\n\nI found your notebook to be very helpful. Thank you!!", "votes": null }, { "id": "2551864", "postDate": "12/07/2023 03:02:10", "content": "

I am glad it got used! Really I just like the name…

\n

Congratulations again. Your post processing is some nice detailed work, and I think the real difference.

", "rawMarkdown": "I am glad it got used! Really I just like the name...\n\nCongratulations again. Your post processing is some nice detailed work, and I think the real difference.", "votes": null }, { "id": "2551944", "postDate": "12/07/2023 04:40:33", "content": "

GPU deep learning

", "rawMarkdown": "GPU deep learning", "votes": null }, { "id": "2552191", "postDate": "12/07/2023 09:25:01", "content": "

Congratulations on top position in this competition. Thanks for sharing details of your solution.

", "rawMarkdown": "Congratulations on top position in this competition. Thanks for sharing details of your solution.", "votes": null }, { "id": "2552308", "postDate": "12/07/2023 11:23:15", "content": "

Nice work. Its really interesting

", "rawMarkdown": "Nice work. Its really interesting", "votes": null }, { "id": "2553580", "postDate": "12/08/2023 11:26:00", "content": "

Congratulations on the top position in this competition.

", "rawMarkdown": "Congratulations on the top position in this competition.", "votes": null }, { "id": "2554597", "postDate": "12/09/2023 08:34:13", "content": "

Awesome !!

", "rawMarkdown": "Awesome !!", "votes": null }, { "id": "2556102", "postDate": "12/10/2023 13:16:16", "content": "

Very Nice!!

", "rawMarkdown": "Very Nice!!", "votes": null }, { "id": "2556723", "postDate": "12/11/2023 00:35:58", "content": "

Conngratulations team on your amazing 1st place cash gold finish. I like your technique of reducing the size of target \"gaussian\" during training. That is very ingenious. And your post process is fantastic!

", "rawMarkdown": "Conngratulations team on your amazing 1st place cash gold finish. I like your technique of reducing the size of target \"gaussian\" during training. That is very ingenious. And your post process is fantastic!", "votes": null }, { "id": "2556871", "postDate": "12/11/2023 04:06:12", "content": "

Congratulations on your amazing achievement.

", "rawMarkdown": "Congratulations on your amazing achievement.", "votes": null }, { "id": "2560017", "postDate": "12/13/2023 07:55:02", "content": "

Congratulations !

\n

Your work is very impressive i don't understand everything but i still being impressed.

", "rawMarkdown": "Congratulations !\n\nYour work is very impressive i don't understand everything but i still being impressed.", "votes": null }, { "id": "2560592", "postDate": "12/13/2023 20:00:01", "content": "

All I can say is… WOW!!! Excellent work, guys. Simply breathtaking processing and clear explanation behind key processes. The implementation details will definitely be something to see.

", "rawMarkdown": "All I can say is... WOW!!! Excellent work, guys. Simply breathtaking processing and clear explanation behind key processes. The implementation details will definitely be something to see.", "votes": null }, { "id": "2560852", "postDate": "12/14/2023 03:51:55", "content": "

Congratulations on your accomplishment!

", "rawMarkdown": "Congratulations on your accomplishment!", "votes": null }, { "id": "2567493", "postDate": "12/19/2023 17:06:46", "content": "

You can check our code here.

", "rawMarkdown": "You can check our code [here](https://github.com/sakami0000/child-mind-institute-detect-sleep-states-1st-place).", "votes": null }, { "id": "2567792", "postDate": "12/20/2023 00:49:05", "content": "

@sakami
\nCongratulation for the 1st place. Very delicate handling of submission. Impressed.

\n

I have some questions.

\n
    \n
  1. In step 3, why two points of interest are chosen per each minutes (i.e. +-15 of 0 sec points)? Is it better to simply apply step 4-5 on 0 sec points?
    2. \n
  2. What is the target (i.e. loss function and labels) of second level model?
    3. \n
", "rawMarkdown": "sakami \nCongratulation for the 1st place. Very delicate handling of submission. Impressed.\n\nI have some questions.\n\n1. In step 3, why two points of interest are chosen per each minutes (i.e. +-15 of 0 sec points)? Is it better to simply apply step 4-5 on 0 sec points?\n2. What is the target (i.e. loss function and labels) of second level model?", "votes": null }, { "id": "2567867", "postDate": "12/20/2023 04:17:39", "content": "

Thank you!!

\n
    \n

  1. This is determined by the evaluation metrics of this competition. The score is calculated based on whether events submitted fall within a specified tolerance of the Ground Truth events, irrespective of their proximity.

    \n\n

    Therefore, regardless of whether the submitted events are at 5, 10, …, 25 seconds, the same score is returned. Similarly, events at 35, 40, …, 55 seconds returns the same score. To maximize the score for all points, representative points are extracted at 15-second and 45-second intervals, and the score is computed.

    2. \n

  2. The target of 2nd level model is a binary flag indicating whether the ground truth event exists at the point.

    3. \n
\n
\n

Specifically, it inputs aggregated features around hh:mm:00 and learns to predict 1 only at the exact time of an event, otherwise 0.

\n
", "rawMarkdown": "Thank you!!\n\n1. This is determined by the evaluation metrics of this competition. The score is calculated based on whether events submitted fall within a specified tolerance of the Ground Truth events, irrespective of their proximity.\n\n - All Ground Truth events must be at 0 seconds.\n - The smallest tolerance width is in 30-second intervals.\n\n Therefore, regardless of whether the submitted events are at 5, 10, ..., 25 seconds, the same score is returned. Similarly, events at 35, 40, ..., 55 seconds returns the same score. To maximize the score for all points, representative points are extracted at 15-second and 45-second intervals, and the score is computed.\n\n2. The target of 2nd level model is a binary flag indicating whether the ground truth event exists at the point.\n\n > Specifically, it inputs aggregated features around hh:mm:00 and learns to predict 1 only at the exact time of an event, otherwise 0.", "votes": null }, { "id": "2567917", "postDate": "12/20/2023 05:39:28", "content": "
\n

The smallest tolerance width is in 30-second intervals.

\n
\n

According to the reference implementation of this competition metric[1], it seems the finest tolerance interval is the open interval (-60sec, +60sec). Isn't that all points [-55, …, -5, 0, 5, …, 55] are candidate for matching GTs at 0sec? However, later I come to think we should avoid both 0sec because these points are edge of tolerance.

\n

For example, if we choose 0sec for representative point, it doesn't match to GTs at +-1min points. However, if we choose -15sec or 15 sec, it will mach to GTs at -1min and 1min respectively. Thus, these points has more chance to mach candidate GTs (i.e, twice as likely than 0sec points).

\n

-> It seems the last statement may not correct.

\n

(Feel free to point out if I the above explanation is wrong.)

\n", "rawMarkdown": "> The smallest tolerance width is in 30-second intervals.\n\nAccording to the reference implementation of this competition metric[1], it seems the finest tolerance interval is the open interval (-60sec, +60sec). Isn't that all points [-55, ..., -5, 0, 5, ..., 55] are candidate for matching GTs at 0sec? However, later I come to think we should avoid both 0sec ~~and 30sec~~ because these points are edge of tolerance.\n\nFor example, if we choose 0sec for representative point, it doesn't match to GTs at +-1min points. However, if we choose -15sec or 15 sec, it will mach to GTs at -1min and 1min respectively. Thus, these points has more chance to mach candidate GTs (i.e, twice as likely than 0sec points).\n\n~~And if we choose 30sec for representative point, maybe it is problematic for the step4-5 in your team's solution, because they are edge point from +-30sec points. E.g., it is hard to determine whether to include probabilities at 0sec and 60sec from the candidate point at 30sec.~~ -> It seems the last statement may not correct.\n\n(Feel free to point out if I the above explanation is wrong.)\n\n- [1] https://www.kaggle.com/code/metric/event-detection-ap/notebook", "votes": null }, { "id": "2567975", "postDate": "12/20/2023 06:54:38", "content": "

Yes, what you're saying is correct.

\n

The reason we are only calculating for two points within a minute is to reduce computational complexity. Calculating for all candidate points (5, 10, …, 55 seconds) or using our method should yield the same score.

\n

For example, if the tolerance for calculating evaluation metrics is only 1 minute, then submitting any second within the same minute, such as 5, 10, …, 55 seconds, would result in the same matching ground truth, leading to the same score. Therefore, considering candidate points that represent, for instance, hh:mm:30, would be equivalent to optimizing for all points.

\n

However, in this calculation, tolerances like 7.5 minutes or 12.5 minutes, in 30-second intervals, are included. As a result, using hh:mm:15 and hh:mm:45 as representative points will yield different scores. Thus, adopting points at 15 seconds and 45 seconds as representative will be equivalent to optimizing for all points.

", "rawMarkdown": "Yes, what you're saying is correct.\n\nThe reason we are only calculating for two points within a minute is to reduce computational complexity. Calculating for all candidate points (5, 10, ..., 55 seconds) or using our method should yield the same score.\n\nFor example, if the tolerance for calculating evaluation metrics is only 1 minute, then submitting any second within the same minute, such as 5, 10, ..., 55 seconds, would result in the same matching ground truth, leading to the same score. Therefore, considering candidate points that represent, for instance, hh:mm:30, would be equivalent to optimizing for all points.\n\nHowever, in this calculation, tolerances like 7.5 minutes or 12.5 minutes, in 30-second intervals, are included. As a result, using hh:mm:15 and hh:mm:45 as representative points will yield different scores. Thus, adopting points at 15 seconds and 45 seconds as representative will be equivalent to optimizing for all points.", "votes": null }, { "id": "2568101", "postDate": "12/20/2023 09:02:42", "content": "
\n

However, in this calculation, tolerances like 7.5 minutes or 12.5 minutes, in 30-second intervals, are included.

\n
\n

Yeah, I missed that point. Makes sense. Thanks.

", "rawMarkdown": "> However, in this calculation, tolerances like 7.5 minutes or 12.5 minutes, in 30-second intervals, are included. \n\nYeah, I missed that point. Makes sense. Thanks.", "votes": null }, { "id": "2570195", "postDate": "12/22/2023 02:18:54", "content": "

@sakami I have one more question.

\n

Can I ask the gain by using step3-5 solely, i.e. not using 2nd level model?

\n

I implemented a slightly modified version of this algorithm, but the gain on my local validation score is not huge (0.8103-> 0.8155; +0.005) as you shared on the solution (private: 0.829 -> 0.852; +0.023).

\n

Does this technique mainly helps 2nd level model, rather it has score gain by itself?

", "rawMarkdown": "sakami I have one more question.\n\nCan I ask the gain by using step3-5 solely, i.e. not using 2nd level model?\n\nI implemented a slightly modified version of this algorithm, but the gain on my local validation score is not huge (0.8103-> 0.8155; +0.005) as you shared on the solution (private: 0.829 -> 0.852; +0.023).\n\nDoes this technique mainly helps 2nd level model, rather it has score gain by itself?", "votes": null }, { "id": "2570318", "postDate": "12/22/2023 05:31:00", "content": "

In our case, performance did not improve with post-processing alone. (w/o daily normalization of the score)

\n

The 2nd level model and daily normalization of the score were crucial elements in improving performance.

", "rawMarkdown": "In our case, performance did not improve with post-processing alone. (w/o daily normalization of the score)\n\nThe 2nd level model and daily normalization of the score were crucial elements in improving performance.", "votes": null }, { "id": "2570401", "postDate": "12/22/2023 07:13:00", "content": "

I see. That is aline with my observation. Thanks.

", "rawMarkdown": "I see. That is aline with my observation. Thanks.", "votes": null }, { "id": "2629913", "postDate": "02/01/2024 04:06:28", "content": "

Explained super well

", "rawMarkdown": "Explained super well", "votes": null }, { "id": "3042195", "postDate": "11/11/2024 09:55:58", "content": "

Awesome !!

", "rawMarkdown": "Awesome !!", "votes": null } ], "comments": [ { "id": 2550900, "author_name": "dongyk", "author_url": "", "post_date": "12/06/2023 11:41:24", "content": "

The winning model is from @danielphalen 's notebook! Amazing work!

", "votes": null, "replies": [] }, { "id": 2550915, "author_name": "dongyk", "author_url": "", "post_date": "12/06/2023 11:53:00", "content": "
\n

As noted in several discussions and notebooks, there was a bias in the minute when ground truth events occurred.

\n
\n

Could you please comment about the bias more?

", "votes": null, "replies": [ { "id": 2550926, "author_name": "sakami", "author_url": "", "post_date": "12/06/2023 12:03:48", "content": "

Please refer to this discussion.

", "votes": null, "replies": [ { "id": 2551286, "author_name": "dongyk", "author_url": "", "post_date": "12/06/2023 15:48:50", "content": "

Thank you. I get it.

", "votes": null, "replies": [ { "id": 2552308, "author_name": "kaitalemuhammad", "author_url": "", "post_date": "12/07/2023 11:23:15", "content": "

Nice work. Its really interesting

", "votes": null, "replies": [] } ] } ] } ] }, { "id": 2550987, "author_name": "soumendraprasad", "author_url": "", "post_date": "12/06/2023 12:49:16", "content": "

@sakami . Amazing work . Congratulations on winning the comp .

", "votes": null, "replies": [] }, { "id": 2551264, "author_name": "cody11null", "author_url": "", "post_date": "12/06/2023 15:38:28", "content": "

All I can say is wow! I will have to spend more time on this one, incredible solution. A solo model at 9th is honestly just as impressive as the win!

", "votes": null, "replies": [] }, { "id": 2551322, "author_name": "danielphalen", "author_url": "", "post_date": "12/06/2023 16:18:45", "content": "

Great Work! Did you adjust the model to add the skip connections? Also, did you keep the KLDivLoss, or did you use a different loss function?

", "votes": null, "replies": [ { "id": 2551806, "author_name": "sakami", "author_url": "", "post_date": "12/07/2023 01:46:39", "content": "
\n

Did you adjust the model to add the skip connections?

\n
\n

Yes. Additionally, I have added several more layers, but the underlying architecture remains unchanged.

\n
\n

Also, did you keep the KLDivLoss, or did you use a different loss function?

\n
\n

In my case, using BCEWithLogitsLoss worked better.

\n

I found your notebook to be very helpful. Thank you!!

", "votes": null, "replies": [ { "id": 2551864, "author_name": "danielphalen", "author_url": "", "post_date": "12/07/2023 03:02:10", "content": "

I am glad it got used! Really I just like the name…

\n

Congratulations again. Your post processing is some nice detailed work, and I think the real difference.

", "votes": null, "replies": [] } ] } ] }, { "id": 2551375, "author_name": "exjustice", "author_url": "", "post_date": "12/06/2023 16:57:42", "content": "

Having started with the same public notebook, really cool how this progressed! Congratulations

", "votes": null, "replies": [] }, { "id": 2551633, "author_name": "tolgadincer", "author_url": "", "post_date": "12/06/2023 20:48:00", "content": "

Congrats! Now I'm thinking that step function makes more sense than the Gaussian target because of the metric.

", "votes": null, "replies": [] }, { "id": 2551944, "author_name": "mdnuruzzamankallal", "author_url": "", "post_date": "12/07/2023 04:40:33", "content": "

GPU deep learning

", "votes": null, "replies": [] }, { "id": 2552191, "author_name": "crsuthikshnkumar", "author_url": "", "post_date": "12/07/2023 09:25:01", "content": "

Congratulations on top position in this competition. Thanks for sharing details of your solution.

", "votes": null, "replies": [] }, { "id": 2553580, "author_name": "deepakdhull", "author_url": "", "post_date": "12/08/2023 11:26:00", "content": "

Congratulations on the top position in this competition.

", "votes": null, "replies": [] }, { "id": 2554597, "author_name": "pranavnv", "author_url": "", "post_date": "12/09/2023 08:34:13", "content": "

Awesome !!

", "votes": null, "replies": [] }, { "id": 2556102, "author_name": "syedusmanahmad", "author_url": "", "post_date": "12/10/2023 13:16:16", "content": "

Very Nice!!

", "votes": null, "replies": [] }, { "id": 2556723, "author_name": "cdeotte", "author_url": "", "post_date": "12/11/2023 00:35:58", "content": "

Conngratulations team on your amazing 1st place cash gold finish. I like your technique of reducing the size of target \"gaussian\" during training. That is very ingenious. And your post process is fantastic!

", "votes": null, "replies": [] }, { "id": 2556871, "author_name": "indrasn0wal", "author_url": "", "post_date": "12/11/2023 04:06:12", "content": "

Congratulations on your amazing achievement.

", "votes": null, "replies": [] }, { "id": 2560017, "author_name": "antoinetheissen75", "author_url": "", "post_date": "12/13/2023 07:55:02", "content": "

Congratulations !

\n

Your work is very impressive i don't understand everything but i still being impressed.

", "votes": null, "replies": [] }, { "id": 2560592, "author_name": "aidlre001", "author_url": "", "post_date": "12/13/2023 20:00:01", "content": "

All I can say is… WOW!!! Excellent work, guys. Simply breathtaking processing and clear explanation behind key processes. The implementation details will definitely be something to see.

", "votes": null, "replies": [] }, { "id": 2560852, "author_name": "romadlon", "author_url": "", "post_date": "12/14/2023 03:51:55", "content": "

Congratulations on your accomplishment!

", "votes": null, "replies": [] }, { "id": 2567493, "author_name": "sakami", "author_url": "", "post_date": "12/19/2023 17:06:46", "content": "

You can check our code here.

", "votes": null, "replies": [] }, { "id": 2567792, "author_name": "tatamikenn", "author_url": "", "post_date": "12/20/2023 00:49:05", "content": "

@sakami
\nCongratulation for the 1st place. Very delicate handling of submission. Impressed.

\n

I have some questions.

\n
    \n
  1. In step 3, why two points of interest are chosen per each minutes (i.e. +-15 of 0 sec points)? Is it better to simply apply step 4-5 on 0 sec points?
    2. \n
  2. What is the target (i.e. loss function and labels) of second level model?
    3. \n
", "votes": null, "replies": [ { "id": 2567867, "author_name": "sakami", "author_url": "", "post_date": "12/20/2023 04:17:39", "content": "

Thank you!!

\n
    \n

  1. This is determined by the evaluation metrics of this competition. The score is calculated based on whether events submitted fall within a specified tolerance of the Ground Truth events, irrespective of their proximity.

    \n\n

    Therefore, regardless of whether the submitted events are at 5, 10, …, 25 seconds, the same score is returned. Similarly, events at 35, 40, …, 55 seconds returns the same score. To maximize the score for all points, representative points are extracted at 15-second and 45-second intervals, and the score is computed.

    2. \n

  2. The target of 2nd level model is a binary flag indicating whether the ground truth event exists at the point.

    3. \n
\n
\n

Specifically, it inputs aggregated features around hh:mm:00 and learns to predict 1 only at the exact time of an event, otherwise 0.

\n
", "votes": null, "replies": [ { "id": 2567917, "author_name": "tatamikenn", "author_url": "", "post_date": "12/20/2023 05:39:28", "content": "
\n

The smallest tolerance width is in 30-second intervals.

\n
\n

According to the reference implementation of this competition metric[1], it seems the finest tolerance interval is the open interval (-60sec, +60sec). Isn't that all points [-55, …, -5, 0, 5, …, 55] are candidate for matching GTs at 0sec? However, later I come to think we should avoid both 0sec because these points are edge of tolerance.

\n

For example, if we choose 0sec for representative point, it doesn't match to GTs at +-1min points. However, if we choose -15sec or 15 sec, it will mach to GTs at -1min and 1min respectively. Thus, these points has more chance to mach candidate GTs (i.e, twice as likely than 0sec points).

\n

-> It seems the last statement may not correct.

\n

(Feel free to point out if I the above explanation is wrong.)

\n", "votes": null, "replies": [ { "id": 2567975, "author_name": "sakami", "author_url": "", "post_date": "12/20/2023 06:54:38", "content": "

Yes, what you're saying is correct.

\n

The reason we are only calculating for two points within a minute is to reduce computational complexity. Calculating for all candidate points (5, 10, …, 55 seconds) or using our method should yield the same score.

\n

For example, if the tolerance for calculating evaluation metrics is only 1 minute, then submitting any second within the same minute, such as 5, 10, …, 55 seconds, would result in the same matching ground truth, leading to the same score. Therefore, considering candidate points that represent, for instance, hh:mm:30, would be equivalent to optimizing for all points.

\n

However, in this calculation, tolerances like 7.5 minutes or 12.5 minutes, in 30-second intervals, are included. As a result, using hh:mm:15 and hh:mm:45 as representative points will yield different scores. Thus, adopting points at 15 seconds and 45 seconds as representative will be equivalent to optimizing for all points.

", "votes": null, "replies": [ { "id": 2568101, "author_name": "tatamikenn", "author_url": "", "post_date": "12/20/2023 09:02:42", "content": "
\n

However, in this calculation, tolerances like 7.5 minutes or 12.5 minutes, in 30-second intervals, are included.

\n
\n

Yeah, I missed that point. Makes sense. Thanks.

", "votes": null, "replies": [ { "id": 2570195, "author_name": "tatamikenn", "author_url": "", "post_date": "12/22/2023 02:18:54", "content": "

@sakami I have one more question.

\n

Can I ask the gain by using step3-5 solely, i.e. not using 2nd level model?

\n

I implemented a slightly modified version of this algorithm, but the gain on my local validation score is not huge (0.8103-> 0.8155; +0.005) as you shared on the solution (private: 0.829 -> 0.852; +0.023).

\n

Does this technique mainly helps 2nd level model, rather it has score gain by itself?

", "votes": null, "replies": [ { "id": 2570318, "author_name": "sakami", "author_url": "", "post_date": "12/22/2023 05:31:00", "content": "

In our case, performance did not improve with post-processing alone. (w/o daily normalization of the score)

\n

The 2nd level model and daily normalization of the score were crucial elements in improving performance.

", "votes": null, "replies": [ { "id": 2570401, "author_name": "tatamikenn", "author_url": "", "post_date": "12/22/2023 07:13:00", "content": "

I see. That is aline with my observation. Thanks.

", "votes": null, "replies": [] } ] } ] } ] } ] } ] } ] } ] }, { "id": 2629913, "author_name": "aleksanderklimczyk", "author_url": "", "post_date": "02/01/2024 04:06:28", "content": "

Explained super well

", "votes": null, "replies": [] }, { "id": 3042195, "author_name": "rominh", "author_url": "", "post_date": "11/11/2024 09:55:58", "content": "

Awesome !!

", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "2550872": "First of all, I would like to express gratitude to all participants and the competition host. It was a challenging competition, but I am pleased with the positive outcome and feel relieved.\n\nHere is a brief summary of our solution.\nYou can check our code [here](https://github.com/sakami0000/child-mind-institute-detect-sleep-states-1st-place).\n\n## Single model\n\nHere's a log on how to improve the CV score after the summary. The final scores were: CV: 0.8206, public LB: 0.768, private LB: 0.829 (equivalent to 9th place).\n\n### Summary\n\n#### Model structure\n\nThe model structure is primarily based on [this amazing notebook](https://www.kaggle.com/code/danielphalen/cmss-grunet-train), with a structure comprising:\nCNN (down sample) → Residual GRU → CNN (up sample)\n\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F2133818%2Fcc6a10ba62b3a3d9361b36e8b162ca5d%2Fmodel_structure.jpeg?generation=1701857831021267&alt=media)\n\n- SEScale\n\nFor input scaling, SEModule was utilized. (https://arxiv.org/abs/1709.01507)\n```python\nclass SEScale(nn.Module):\n def __init__(self, ch: int, r: int) -> None:\n super().__init__()\n self.fc1 = nn.Linear(ch, r)\n self.fc2 = nn.Linear(r, ch)\n\n def forward(self, x: torch.FloatTensor) -> torch.FloatTensor:\n h = self.fc1(x)\n h = F.relu(h)\n h = self.fc2(h).sigmoid()\n return h * x\n```\n\n- Minute connection\n\nAs noted in several discussions and notebooks, there was a bias in the minute when ground truth events occurred. To account for this, features related to minutes were concatenated separately in the final layer.\n\n```python\n def forward(self, num_x: torch.FloatTensor, cat_x: torch.LongTensor) -> torch.FloatTensor:\n cat_embeddings = [embedding(cat_x[:, :, i]) for i, embedding in enumerate(self.category_embeddings)]\n num_x = self.numerical_linear(num_x)\n\n x = torch.cat([num_x] + cat_embeddings, dim=2)\n x = self.input_linear(x)\n x = self.conv(x.transpose(-1, -2)).transpose(-1, -2)\n\n for gru in self.gru_layers:\n x, _ = gru(x)\n\n x = self.dconv(x.transpose(-1, -2)).transpose(-1, -2)\n minute_embedding = self.minute_embedding(cat_x[:, :, 0])\n x = self.output_linear(torch.cat([x, minute_embedding], dim=2))\n return x\n```\n\n#### Data Preparation\n\nEach series of data is divided into daily chunks, offset by 0.35 days.\n\n```python\n train_df = train_df.with_columns(pl.arange(0, pl.count()).alias(\"row_id\"))\n series_row_ids = dict(train_df.group_by(\"series_id\").agg(\"row_id\").rows())\n\n series_chunk_ids = [] # list[str]\n series_chunk_row_ids = [] # list[list[int]]\n for series_id, row_ids in tqdm(series_row_ids.items(), desc=\"split into chunks\"):\n for start_idx in range(0, len(row_ids), int(config.stride_size / config.epoch_sample_rate)):\n if start_idx + config.chunk_size <= len(row_ids):\n chunk_row_ids = row_ids[start_idx : start_idx + config.chunk_size]\n series_chunk_ids.append(series_id)\n series_chunk_row_ids.append(np.array(chunk_row_ids))\n else:\n chunk_row_ids = row_ids[-config.chunk_size :]\n series_chunk_ids.append(series_id)\n series_chunk_row_ids.append(np.array(chunk_row_ids))\n break\n```\n\nDuring training, half of each chunk is used in every epoch.\n\n```python\n sampled_train_idx = train_idx[epoch % config.epoch_sample_rate :: config.epoch_sample_rate]\n```\n\nFor evaluation, overlapping sections are averaged, and the ends of each chunk are trimmed by 30 minutes.\n\n#### Target\n\nA decaying target is created based on the distance from the ground truth event, with diminishing values as the distance increases.\n\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F2133818%2Fc6d809492da742bbcb9252615375e76d%2Ftarget.png?generation=1701858069231326&alt=media)\n\n```python\ntolerance_steps = [12, 36, 60, 90, 120, 150, 180, 240, 300, 360]\ntarget_columns = [\"event_onset\", \"event_wakeup\"]\n\ntrain_df = (\n train_df.join(train_events_df.select([\"series_id\", \"step\", \"event\"]), on=[\"series_id\", \"step\"], how=\"left\")\n .to_dummies(columns=[\"event\"])\n .with_columns(\n pl.max_horizontal(\n pl.col(target_columns)\n .rolling_max(window_size * 2 - 1, min_periods=1, center=True)\n .over(\"series_id\")\n * (1 - i / len(tolerance_steps))\n for i, window_size in enumerate(tolerance_steps)\n )\n )\n)\n```\n\nThe target is updated each epoch to decay further.\n\n```python\n # update target\n targets = np.where(targets == 1.0, 1.0, (targets - (1.0 / config.n_epochs)).clip(min=0.0))\n```\n\nBy attenuating the target, the range of predicted values narrows, allowing for the detection of finer peaks.\n\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F2133818%2F002ebac2c4da25cdde7a81b27f1ec492%2F2023-12-09%200.15.31.png?generation=1702048562641170&alt=media)\n\n#### Periodicity Filter\n\nAs discussed in [here](https://www.kaggle.com/competitions/child-mind-institute-detect-sleep-states/discussion/456177), there is a daily periodicity in the data when the measuring device is removed. This is leveraged to predict these periods rule-based and used as a filter for inputs and predictions.\n\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F2133818%2Fbd12584427bc9e2e628e5a095419ba2f%2F2023-12-06%2015.11.06.png?generation=1701858205750939&alt=media)\n\n#### Input Features\n\n- categorical features\n - hour\n - minute\n - weekday\n - periodicity flag\n- numerical features\n - anglez / 45\n - enmo.log1p().clip_max(1.0) / 0.1\n - anglez, enmo 12 steps rolling_mean, rolling_std, rolling_max\n - anglez_diff_abs 5 min rolling median\n\n### Change logs\n\n- baseline model (cv: 0.7510) - public: 0.728\n- Add a process to decay the target every epoch (cv: 0.7699, +19pt)\n- Add a periodicity filter to the output (cv: 0.7807, +11pt)\n- Add a periodicity flag to the input as well (cv: 0.7870, +6pt) - public: 0.739\n- batch_size: 16 → 4, hidden_size: 128 → 64, num_layers: 2 → 8 (cv: 0.7985, +11pt) - public: 0.755\n- Normalize the score in the submission file by the daily score sum (cv: 0.8044, +6pt)\n- Remove month and day from the input (cv: 0.8117, +7pt)\n- Trim the edges of the chunk by 30 minutes on both sides (cv: 0.8142, +4pt) - public: 0.765\n- Modify to concatenate the minute features to the final layer (cv: 0.8206, +6pt) - public: 0.768\n\n----\n\n## Post Processing\n\nThis post-processing creates a submission DataFrame to optimize the evaluation metrics. With this post-processing method, we significantly improved our scores (public: 0.768 → **0.790**, private: 0.829 → **0.852** !!!).\n\nThis was a complex procedure, which I will explain step by step.\n\n1. **Data Characteristics**\n\n First, let's discuss the characteristics of the data. As noted in several discussions and notebooks, the second of the target events was always set to zero.\n The competition's evaluation metric doesn't differentiate predictions within a 30-second range from the ground truth event. So, whether the submission timestamp's seconds are 5, 10, 15, 20, ... 25, the same score is returned.\n\n \"\"\n\n2. **Creation of the 2nd Level Model**\n\n The 1st level model's predictions were trained to recognize events within a certain range from the ground truth as positive. However, the 2nd level model transforms these into probabilities of a ground truth event existing for each minute.\n The output of the 1st level model was at seconds 0, 5, 10, ..., but the 2nd level model aggregates these to always be at second 0. Specifically, it inputs aggregated features around hh:mm:00 and learns to predict 1 only at the exact time of an event, otherwise 0. Details of the 2nd level model will be described later.\n\n \"\"\n\n3. **Score Calculation for Each Point**\n\n As explained earlier, submitting any second within the same minute yields the same score. Therefore, we estimate the score at the 15 and 45 second points of each minute, and submit the one with the highest value, effectively submitting the highest score for all points. The method of score estimation is as follows:\n\n For instance, let's estimate the score at 10:00:15.\n\n \"\"\n\n First, we create a window of 12 steps from the point of interest and sum the predictions of the 2nd level model within this window to calculate the `tolerance_12_score`.\n\n \"\"\n\n Similarly, we calculate `tolerance_36_score`, `tolerance_60_score`, ..., for the respective tolerances used in the evaluation, and the sum of these scores is considered the score for the point of interest.\n\n \"\"\n\n We perform this calculation for all points, and for each series, we adopt the point with the highest score and add it to the submission DataFrame.\n\n \"\"\n\n4. **Score Recalculation**\n\n Next, we recalculate the score to determine the next point to be adopted. For example, suppose the point 09:59:15 was chosen.\n\n First, consider updating the `tolerance_12_score`. Events within tolerance 12 of the adopted point cannot match overlappingly with the next point to be submitted.\n\n \"\"\n\n Therefore, when calculating the `tolerance_12_score` for the next point to be adopted, it's necessary to discount the prediction values within 12 steps of the currently adopted point.\n\n \"\"\n\n Likewise, for `tolerance_36_score`, `tolerance_60_score`, ..., we recalculate by discounting the prediction values within 36, 60, ..., steps of the adopted point.\n\n \"\"\n\n With the updated scores calculated, we again adopt the highest scoring point for each series and add it to the submission dataframe.\n\n \"\"\n\n5. **Creating Submissions**\n\n We repeat the above step 4 to extract a sufficient number of submission points, then compile these into a DataFrame to create the submission file.\n\n### Additional Techniques\n\nSeveral other techniques were employed to make the post-processing work effectively:\n\n- Normalize the predictions of the 2nd level model daily.\n- When recalculating the score, calculate the difference from the previous score to reduce the computation.\n- Speed up the above calculations using JIT compilation.\n\n### Details of the 2nd Level Model\n\n- The 2nd level model starts by averaging the 1st level model's predictions on a per-minute basis and then detecting peaks in these averages using `find_peaks` with a height of 0.001 and a distance of 8.\n- Based on the detected peaks, chunks are created from the original time series, capturing 8 minutes before and after each peak. (Recall: 0.9845)\n - This step_size was crucial because the ratio of positive to negative examples changes depending on how many steps are included, affecting the accuracy of subsequent stages. Therefore, we tuned the number of steps for optimal performance.\n - If chunks are connected, they are treated as a single chunk.\n\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F2133818%2F907332d70498f46059a2935461f69941%2F2023-12-06%2019.42.55.png?generation=1701859390836314&alt=media)\n\n- For each chunk, we aggregated features from the 1st model's predictions and other features like anglez and enmo. These aggregated features were then used to train models such as LightGBM and CatBoost.\n- Additionally, we treated each chunk as a sequence for training CNN-RNN, CNN, and Transformer models. As a result, we developed a model that could account for minute biases not fully addressed by the 1st level model.\n\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F2133818%2F4b8ec9d1aecb6172e700a8ce485d226f%2F2023-12-06%2019.43.55.png?generation=1701859451118326&alt=media)\n\n- The predictions of the 2nd level model were sufficiently calibrated, so there was no need for further transformation.\n\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F2133818%2Fad0665bfff9828ebc6516a54e4e20276%2F2024-01-06%2016.46.51.png?generation=1704527223913793&alt=media)", "2550900": "The winning model is from @danielphalen 's notebook! Amazing work!", "2550915": ">As noted in several discussions and notebooks, there was a bias in the minute when ground truth events occurred.\n\nCould you please comment about the bias more?", "2550926": "Please refer to [this discussion](https://www.kaggle.com/competitions/child-mind-institute-detect-sleep-states/discussion/444374).", "2550987": "sakami . Amazing work . Congratulations on winning the comp .", "2551264": "All I can say is wow! I will have to spend more time on this one, incredible solution. A solo model at 9th is honestly just as impressive as the win!", "2551286": "Thank you. I get it.", "2551322": "Great Work! Did you adjust the model to add the skip connections? Also, did you keep the KLDivLoss, or did you use a different loss function?", "2551375": "Having started with the same public notebook, really cool how this progressed! Congratulations", "2551633": "Congrats! Now I'm thinking that step function makes more sense than the Gaussian target because of the metric.", "2551806": "> Did you adjust the model to add the skip connections?\n\nYes. Additionally, I have added several more layers, but the underlying architecture remains unchanged.\n\n> Also, did you keep the KLDivLoss, or did you use a different loss function?\n\nIn my case, using BCEWithLogitsLoss worked better.\n\nI found your notebook to be very helpful. Thank you!!", "2551864": "I am glad it got used! Really I just like the name...\n\nCongratulations again. Your post processing is some nice detailed work, and I think the real difference.", "2551944": "GPU deep learning", "2552191": "Congratulations on top position in this competition. Thanks for sharing details of your solution.", "2552308": "Nice work. Its really interesting", "2553580": "Congratulations on the top position in this competition.", "2554597": "Awesome !!", "2556102": "Very Nice!!", "2556723": "Conngratulations team on your amazing 1st place cash gold finish. I like your technique of reducing the size of target \"gaussian\" during training. That is very ingenious. And your post process is fantastic!", "2556871": "Congratulations on your amazing achievement.", "2560017": "Congratulations !\n\nYour work is very impressive i don't understand everything but i still being impressed.", "2560592": "All I can say is... WOW!!! Excellent work, guys. Simply breathtaking processing and clear explanation behind key processes. The implementation details will definitely be something to see.", "2560852": "Congratulations on your accomplishment!", "2567493": "You can check our code [here](https://github.com/sakami0000/child-mind-institute-detect-sleep-states-1st-place).", "2567792": "sakami \nCongratulation for the 1st place. Very delicate handling of submission. Impressed.\n\nI have some questions.\n\n1. In step 3, why two points of interest are chosen per each minutes (i.e. +-15 of 0 sec points)? Is it better to simply apply step 4-5 on 0 sec points?\n2. What is the target (i.e. loss function and labels) of second level model?", "2567867": "Thank you!!\n\n1. This is determined by the evaluation metrics of this competition. The score is calculated based on whether events submitted fall within a specified tolerance of the Ground Truth events, irrespective of their proximity.\n\n - All Ground Truth events must be at 0 seconds.\n - The smallest tolerance width is in 30-second intervals.\n\n Therefore, regardless of whether the submitted events are at 5, 10, ..., 25 seconds, the same score is returned. Similarly, events at 35, 40, ..., 55 seconds returns the same score. To maximize the score for all points, representative points are extracted at 15-second and 45-second intervals, and the score is computed.\n\n2. The target of 2nd level model is a binary flag indicating whether the ground truth event exists at the point.\n\n > Specifically, it inputs aggregated features around hh:mm:00 and learns to predict 1 only at the exact time of an event, otherwise 0.", "2567917": "> The smallest tolerance width is in 30-second intervals.\n\nAccording to the reference implementation of this competition metric[1], it seems the finest tolerance interval is the open interval (-60sec, +60sec). Isn't that all points [-55, ..., -5, 0, 5, ..., 55] are candidate for matching GTs at 0sec? However, later I come to think we should avoid both 0sec ~~and 30sec~~ because these points are edge of tolerance.\n\nFor example, if we choose 0sec for representative point, it doesn't match to GTs at +-1min points. However, if we choose -15sec or 15 sec, it will mach to GTs at -1min and 1min respectively. Thus, these points has more chance to mach candidate GTs (i.e, twice as likely than 0sec points).\n\n~~And if we choose 30sec for representative point, maybe it is problematic for the step4-5 in your team's solution, because they are edge point from +-30sec points. E.g., it is hard to determine whether to include probabilities at 0sec and 60sec from the candidate point at 30sec.~~ -> It seems the last statement may not correct.\n\n(Feel free to point out if I the above explanation is wrong.)\n\n- [1] https://www.kaggle.com/code/metric/event-detection-ap/notebook", "2567975": "Yes, what you're saying is correct.\n\nThe reason we are only calculating for two points within a minute is to reduce computational complexity. Calculating for all candidate points (5, 10, ..., 55 seconds) or using our method should yield the same score.\n\nFor example, if the tolerance for calculating evaluation metrics is only 1 minute, then submitting any second within the same minute, such as 5, 10, ..., 55 seconds, would result in the same matching ground truth, leading to the same score. Therefore, considering candidate points that represent, for instance, hh:mm:30, would be equivalent to optimizing for all points.\n\nHowever, in this calculation, tolerances like 7.5 minutes or 12.5 minutes, in 30-second intervals, are included. As a result, using hh:mm:15 and hh:mm:45 as representative points will yield different scores. Thus, adopting points at 15 seconds and 45 seconds as representative will be equivalent to optimizing for all points.", "2568101": "> However, in this calculation, tolerances like 7.5 minutes or 12.5 minutes, in 30-second intervals, are included. \n\nYeah, I missed that point. Makes sense. Thanks.", "2570195": "sakami I have one more question.\n\nCan I ask the gain by using step3-5 solely, i.e. not using 2nd level model?\n\nI implemented a slightly modified version of this algorithm, but the gain on my local validation score is not huge (0.8103-> 0.8155; +0.005) as you shared on the solution (private: 0.829 -> 0.852; +0.023).\n\nDoes this technique mainly helps 2nd level model, rather it has score gain by itself?", "2570318": "In our case, performance did not improve with post-processing alone. (w/o daily normalization of the score)\n\nThe 2nd level model and daily normalization of the score were crucial elements in improving performance.", "2570401": "I see. That is aline with my observation. Thanks.", "2629913": "Explained super well", "3042195": "Awesome !!" }, "source": "meta" }