First of all, I would like to thank @tubotubo for sharing your high-quality code, and also thank my teammates @liruiqi577 @brickcoder @xtzhou for their contributions in the competition. Here, I am going to share our team’s “snore like thunder” solution from the following aspects:
\nWe made EDA and readed open discussions found that there are 4 types of data anomalies:
\nTo this end, we have some attempts, such as:
\nBut the above methods didn't completely work. In the end, our preprocessing method was:
\nWe split the dataset group by series into 5 folds. For each fold, we eliminate series with a label missing rate of 100% in the training dataset while without performing any data preprocessing on the validation set. This is done to avoid introducing noise to the training set, and to ensure that the evaluation results of the validation set are more biased towards the real data distribution, which improve our LB score + 0.006.
\nPart of our experiments as below:
\n| Experiment | \nFold0 | \nPublic (single fold) | \nPrivate (5-fold) | \n
|---|---|---|---|
| No preprocess missing data | \n0.751 | \n0.718 | \n0.744 | \n
| Eliminate unlabeled data at the end of train_series & series with missing rate >80% | \n0.739 | \n0.709 | \n0.741 | \n
| Drop train series which don’t have any event labels | \n0.752 | \n0.724 | \n0.749 | \n
train_series[] = train_series[].diff().()\ntrain_series[] = train_series[].rolling(window=, center=, min_periods=).mean()\ntrain_series[] = train_series[].diff().()\ntrain_series[] = train_series[].rolling(window=, center=, min_periods=).mean()\nIn addition, we also made the following features based on open notebooks and our EDA, such as: differential features with different orders, rolling window statistical features, interactive features of enmo and anglez (such as anglez's differential abs * enmo, etc.), anglez_rad_sin/cos, dayofweek/is_weekend (I find that children have different sleeping habits on weekdays and weekends). But strangely enough, too much feature engineering didn’t bring us much benefit.
\n| Experiment | \nFold0 | \nPublic (5-fold) | \nPrivate (5-fold) | \n
|---|---|---|---|
| anglez + enmo + hour_sin + hour_cos | \n0.763 | \n0.731 | \n0.768 | \n
| anglez_abs_diff + enmo_abs_diff + hour_sin + hour_cos | \n0.771 | \n0.741 | \n0.781 | \n
We used 4 models:
\nParameter Tunning: Add more kernel_size 8 for CNNSpectrogram can gain +0.002 online.
\nMulti-Task Learning Objectives: sleep status, onset, wake.
\nLoss Function: For Spec2DCNN and TransformerAutoModel, we use BCE, but with multi-task target weighting, sleep:onset:wake = 0.5:1:1. The purpose of this is to allow the model to focus on learning the last two columns. We tried to train only for the onset and wake columns, but the score was not good. The reason is speculated that the positive samples in these two columns are sparse, and MTL needs to be used to transfer the information from positive samples in the sleep status to the prediction of sleep activity events. Also, I tried KL Loss but it didn't work that well.
\nself.loss_fn = nn.BCEWithLogitsLoss(pos_weight=torch.tensor([,,]))\nAt the same time, we adjusted epoch to 70 and added early stopping with patience=15. The early stopping criterion is the AP of the validation dataset, not the loss of the validation set. batch_size=32.
\n| Experiment | \nFold0 | \nPublic (single fold) | \nPrivate (5-fold) | \n
|---|---|---|---|
| earlystop by val_loss | \n0.750 | \n0.697 | \n0.742 | \n
| earlystop by val_score | \n0.751 | \n0.718 | \n0.744 | \n
| loss_wgt = 1:1:1 | \n0.752 | \n0.724 | \n0.749 | \n
| loss_wgt = 0.5:1:1 | \n0.755 | \n0.723 | \n0.753 | \n
Note: we used the model_weight.pth with the best offline val_score to submit our LB instead of using the best_model.pth with the best offline val_loss。
\nOur post-processing mainly includes:
\n ():\n x1 = dets_arr[:, ]\n x2 = dets_arr[:, ]\n scores = dets_arr[:, ]\n\n areas = x2 - x1\n order = scores.argsort()[::-]\n\n keep = []\n order.size > :\n i = order[]\n keep.append(i)\n xx1 = np.maximum(x1[i], x1[order[:]])\n xx2 = np.minimum(x2[i], x2[order[:]])\n inter = np.maximum(, xx2 - xx1 + )\n ovr = inter / (areas[i] + areas[order[:]] - inter)\n inds = np.where(ovr <= thresh)[]\n order = order[inds + ]\n\n dets_nms_arr = dets_arr[keep,:]\n onset_steps = dets_nms_arr[:, ].tolist()\n wakeup_steps = dets_nms_arr[:, ].tolist()\n nms_save_steps = np.unique(onset_steps + wakeup_steps).tolist()\n nms_save_steps\nIn addition, we set score_th=0.005 (If it is set too low, a large number of events will be detected and cause online scoring errors, so it is fixed at 0.005 here), and use optuna to simultaneously search the parameter distance in find_peaks and the parameter iou_threshold of NMS. Finally, when distance=72 and iou_threshold=0.995, the best performance is achieved.
\n optuna\n\n ():\n score_th = \n distance = trial.suggest_int(, , )\n thresh = trial.suggest_float(, , )\n\n \n val_pred_df = post_process_for_seg(\n keys=keys,\n preds=preds[:, :, [, ]],\n score_th=score_th,\n distance=distance,\n )\n\n \n val_pred_df = val_pred_df.to_pandas()\n nms_pred_dfs = NMS_prediction(val_pred_df, thresh, verbose=)\n score = event_detection_ap(valid_event_df.to_pandas(), nms_pred_dfs)\n -score\n\nstudy = optuna.create_study()\nstudy.optimize(objective, n_trials=)\n(, study.best_params)\n(, study.best_value)\n| Experiment | \nFold0 | \nPubic (5-fold) | \nPrivate (5-fold) | \n
|---|---|---|---|
| find_peak | \n- | \n0.745 | \n0.787 | \n
| find_peak+NMS+optuna | \n- | \n0.746 | \n0.789 | \n
Finally, we average the output probabilities of the following models and then feed into the post processing methods to detect events. By the way, I tried post-processing the detection events for each model and then concating them, but this resulted in too many detections. Even with NMS, I didn't get a better score.
\nThe number of ensemble models: 4 (types of models) * 5 (fold number) = 20.
\n| Experiment | \nFold0 | \nPubic (5-fold) | \nPrivate (5-fold) | \n
|---|---|---|---|
| model1: CNNSpectrogram + Spec2DCNN + UNet1DDecoder | \n0.77209 | \n0.743 | \n0.784 | \n
| model2: PANNsFeatureExtractor + Spec2DCNN + UNet1DDecoder | \n0.777 | \n0.743 | \n0.782 | \n
| model3: PANNsFeatureExtractor + CenterNet + UNet1DDecoder | \n0.75968 | \n0.634 | \n0.68 | \n
| model4: TransformerAutoModel | \n0.74680 | \n- | \n- | \n
| model1 + model2(1:1) | \n- | \n0.746 | \n0.789 | \n
| model1 + model2+model3(1:1:0.4) | \n- | \n0.75 | \n0.786 | \n
| model1 + model2+model3+model4(1:1:0.4:0.2) | \n\n | 0.752 | \n0.787 | \n
Unfortunately, we only considered CenterNet and Transformer to model ensemble with a tentative attitude on the last day, but surprisingly found that a low-CV-scoring model still has a probability of improving final performance as long as it is heterogeneous compared with your previous models. But we didn’t have more opportunities to submit more, which was a profound lesson for me.
\nThoughts not done:
\nData Augmentation: Shift the time within the batch to increase more time diversity and reduce dependence on hour features.
Model: Try more models. Although we try transformer and it didn’t work for us. I am veryyy looking forward to the solutions from top-ranking players.
Thanks again to Kaggle and all Kaggle players. This was a good competition and we learned a lot from it. If you think our solution is useful for you, welcome to upvote and discuss with us.
\nIn addition, this is my first 🥈 silver medal. Thank you everyone for letting me learn a lot. I will continue to work hard. :)
", "messages": [ { "id": "2550342", "postDate": "12/06/2023 01:40:30", "content": "First of all, I would like to thank @tubotubo for sharing your high-quality code, and also thank my teammates @liruiqi577 @brickcoder @xtzhou for their contributions in the competition. Here, I am going to share our team’s “snore like thunder” solution from the following aspects:
\nWe made EDA and readed open discussions found that there are 4 types of data anomalies:
\nTo this end, we have some attempts, such as:
\nBut the above methods didn't completely work. In the end, our preprocessing method was:
\nWe split the dataset group by series into 5 folds. For each fold, we eliminate series with a label missing rate of 100% in the training dataset while without performing any data preprocessing on the validation set. This is done to avoid introducing noise to the training set, and to ensure that the evaluation results of the validation set are more biased towards the real data distribution, which improve our LB score + 0.006.
\nPart of our experiments as below:
\n| Experiment | \nFold0 | \nPublic (single fold) | \nPrivate (5-fold) | \n
|---|---|---|---|
| No preprocess missing data | \n0.751 | \n0.718 | \n0.744 | \n
| Eliminate unlabeled data at the end of train_series & series with missing rate >80% | \n0.739 | \n0.709 | \n0.741 | \n
| Drop train series which don’t have any event labels | \n0.752 | \n0.724 | \n0.749 | \n
train_series[] = train_series[].diff().()\ntrain_series[] = train_series[].rolling(window=, center=, min_periods=).mean()\ntrain_series[] = train_series[].diff().()\ntrain_series[] = train_series[].rolling(window=, center=, min_periods=).mean()\nIn addition, we also made the following features based on open notebooks and our EDA, such as: differential features with different orders, rolling window statistical features, interactive features of enmo and anglez (such as anglez's differential abs * enmo, etc.), anglez_rad_sin/cos, dayofweek/is_weekend (I find that children have different sleeping habits on weekdays and weekends). But strangely enough, too much feature engineering didn’t bring us much benefit.
\n| Experiment | \nFold0 | \nPublic (5-fold) | \nPrivate (5-fold) | \n
|---|---|---|---|
| anglez + enmo + hour_sin + hour_cos | \n0.763 | \n0.731 | \n0.768 | \n
| anglez_abs_diff + enmo_abs_diff + hour_sin + hour_cos | \n0.771 | \n0.741 | \n0.781 | \n
We used 4 models:
\nParameter Tunning: Add more kernel_size 8 for CNNSpectrogram can gain +0.002 online.
\nMulti-Task Learning Objectives: sleep status, onset, wake.
\nLoss Function: For Spec2DCNN and TransformerAutoModel, we use BCE, but with multi-task target weighting, sleep:onset:wake = 0.5:1:1. The purpose of this is to allow the model to focus on learning the last two columns. We tried to train only for the onset and wake columns, but the score was not good. The reason is speculated that the positive samples in these two columns are sparse, and MTL needs to be used to transfer the information from positive samples in the sleep status to the prediction of sleep activity events. Also, I tried KL Loss but it didn't work that well.
\nself.loss_fn = nn.BCEWithLogitsLoss(pos_weight=torch.tensor([,,]))\nAt the same time, we adjusted epoch to 70 and added early stopping with patience=15. The early stopping criterion is the AP of the validation dataset, not the loss of the validation set. batch_size=32.
\n| Experiment | \nFold0 | \nPublic (single fold) | \nPrivate (5-fold) | \n
|---|---|---|---|
| earlystop by val_loss | \n0.750 | \n0.697 | \n0.742 | \n
| earlystop by val_score | \n0.751 | \n0.718 | \n0.744 | \n
| loss_wgt = 1:1:1 | \n0.752 | \n0.724 | \n0.749 | \n
| loss_wgt = 0.5:1:1 | \n0.755 | \n0.723 | \n0.753 | \n
Note: we used the model_weight.pth with the best offline val_score to submit our LB instead of using the best_model.pth with the best offline val_loss。
\nOur post-processing mainly includes:
\n ():\n x1 = dets_arr[:, ]\n x2 = dets_arr[:, ]\n scores = dets_arr[:, ]\n\n areas = x2 - x1\n order = scores.argsort()[::-]\n\n keep = []\n order.size > :\n i = order[]\n keep.append(i)\n xx1 = np.maximum(x1[i], x1[order[:]])\n xx2 = np.minimum(x2[i], x2[order[:]])\n inter = np.maximum(, xx2 - xx1 + )\n ovr = inter / (areas[i] + areas[order[:]] - inter)\n inds = np.where(ovr <= thresh)[]\n order = order[inds + ]\n\n dets_nms_arr = dets_arr[keep,:]\n onset_steps = dets_nms_arr[:, ].tolist()\n wakeup_steps = dets_nms_arr[:, ].tolist()\n nms_save_steps = np.unique(onset_steps + wakeup_steps).tolist()\n nms_save_steps\nIn addition, we set score_th=0.005 (If it is set too low, a large number of events will be detected and cause online scoring errors, so it is fixed at 0.005 here), and use optuna to simultaneously search the parameter distance in find_peaks and the parameter iou_threshold of NMS. Finally, when distance=72 and iou_threshold=0.995, the best performance is achieved.
\n optuna\n\n ():\n score_th = \n distance = trial.suggest_int(, , )\n thresh = trial.suggest_float(, , )\n\n \n val_pred_df = post_process_for_seg(\n keys=keys,\n preds=preds[:, :, [, ]],\n score_th=score_th,\n distance=distance,\n )\n\n \n val_pred_df = val_pred_df.to_pandas()\n nms_pred_dfs = NMS_prediction(val_pred_df, thresh, verbose=)\n score = event_detection_ap(valid_event_df.to_pandas(), nms_pred_dfs)\n -score\n\nstudy = optuna.create_study()\nstudy.optimize(objective, n_trials=)\n(, study.best_params)\n(, study.best_value)\n| Experiment | \nFold0 | \nPubic (5-fold) | \nPrivate (5-fold) | \n
|---|---|---|---|
| find_peak | \n- | \n0.745 | \n0.787 | \n
| find_peak+NMS+optuna | \n- | \n0.746 | \n0.789 | \n
Finally, we average the output probabilities of the following models and then feed into the post processing methods to detect events. By the way, I tried post-processing the detection events for each model and then concating them, but this resulted in too many detections. Even with NMS, I didn't get a better score.
\nThe number of ensemble models: 4 (types of models) * 5 (fold number) = 20.
\n| Experiment | \nFold0 | \nPubic (5-fold) | \nPrivate (5-fold) | \n
|---|---|---|---|
| model1: CNNSpectrogram + Spec2DCNN + UNet1DDecoder | \n0.77209 | \n0.743 | \n0.784 | \n
| model2: PANNsFeatureExtractor + Spec2DCNN + UNet1DDecoder | \n0.777 | \n0.743 | \n0.782 | \n
| model3: PANNsFeatureExtractor + CenterNet + UNet1DDecoder | \n0.75968 | \n0.634 | \n0.68 | \n
| model4: TransformerAutoModel | \n0.74680 | \n- | \n- | \n
| model1 + model2(1:1) | \n- | \n0.746 | \n0.789 | \n
| model1 + model2+model3(1:1:0.4) | \n- | \n0.75 | \n0.786 | \n
| model1 + model2+model3+model4(1:1:0.4:0.2) | \n\n | 0.752 | \n0.787 | \n
Unfortunately, we only considered CenterNet and Transformer to model ensemble with a tentative attitude on the last day, but surprisingly found that a low-CV-scoring model still has a probability of improving final performance as long as it is heterogeneous compared with your previous models. But we didn’t have more opportunities to submit more, which was a profound lesson for me.
\nThoughts not done:
\nData Augmentation: Shift the time within the batch to increase more time diversity and reduce dependence on hour features.
Model: Try more models. Although we try transformer and it didn’t work for us. I am veryyy looking forward to the solutions from top-ranking players.
Thanks again to Kaggle and all Kaggle players. This was a good competition and we learned a lot from it. If you think our solution is useful for you, welcome to upvote and discuss with us.
\nIn addition, this is my first 🥈 silver medal. Thank you everyone for letting me learn a lot. I will continue to work hard. :)
", "rawMarkdown": "First of all, I would like to thank @tubotubo for sharing your high-quality code, and also thank my teammates @liruiqi577 @brickcoder @xtzhou for their contributions in the competition. Here, I am going to share our team’s “**snore like thunder**” solution from the following aspects:\n\n1. Data preprocessing\n2. Feature Engineering\n3. Model\n4. Post Processing\n5. Model Ensemble\n\n## 1. Data preprocessing\nWe made EDA and readed open discussions found that there are 4 types of data anomalies:\n- Some series have a high missing rate and some of them do not even have any event labels;\n- In some series , there are no event annotations in the middle and tail (possibly because the collection activity has stopped);\n- The sleep record is incomplete (a period of sleep is only marked with onset or wakeup).\n- There are outliers in the enmo value.\n\nTo this end, we have some attempts, such as:\n- Eliminate series with high missing rates;\n- Cut the tail of the series without event labels;\n- Upper clip enmo to 1.\n\nBut the above methods didn't completely work. In the end, our preprocessing method was:\n\nWe split the dataset group by series into 5 folds. For each fold, we eliminate series with a label missing rate of 100% in the training dataset while without performing any data preprocessing on the validation set. This is done to avoid introducing noise to the training set, and to ensure that the evaluation results of the validation set are more biased towards the real data distribution, which improve our LB score + 0.006.\n\nPart of our experiments as below:\n| Experiment | Fold0 | Public (single fold) | Private (5-fold) |\n| --- | --- | --- | --- |\n| No preprocess missing data | 0.751 | 0.718 | 0.744 |\n| Eliminate unlabeled data at the end of train_series & series with missing rate >80% | 0.739 | 0.709 | 0.741 |\n| Drop train series which don’t have any event labels | 0.752 | 0.724 | 0.749 |\n\n## 2. Feature Engineering\n- Sensor features: After smoothing the enmo and anglez features, a first-order difference is made to obtain the absolute value. Then replace the original enmo and anglez features with these features, which improve our LB score + 0.01.\n\n```python\ntrain_series['enmo_abs_diff'] = train_series['enmo'].diff().abs()\ntrain_series['enmo'] = train_series['enmo_abs_diff'].rolling(window=5, center=True, min_periods=1).mean()\ntrain_series['anglez_abs_diff'] = train_series['anglez'].diff().abs()\ntrain_series['anglez'] = train_series['anglez_abs_diff'].rolling(window=5, center=True, min_periods=1).mean()\n```\n\n- Time features: sin and cos hour.\n\nIn addition, we also made the following features based on open notebooks and our EDA, such as: differential features with different orders, rolling window statistical features, interactive features of enmo and anglez (such as anglez's differential abs * enmo, etc.), anglez_rad_sin/cos, dayofweek/is_weekend (I find that children have different sleeping habits on weekdays and weekends). But strangely enough, too much feature engineering didn’t bring us much benefit.\n\n| Experiment | Fold0 | Public (5-fold) | Private (5-fold) |\n| --- | --- | --- | --- |\n| anglez + enmo + hour_sin + hour_cos | 0.763 | 0.731 | 0.768 |\n| anglez_abs_diff + enmo_abs_diff + hour_sin + hour_cos | 0.771 | 0.741 | 0.781 |\n\n## 3. Model\nWe used 4 models:\n\n- CNNSpectrogram + Spec2DCNN + UNet1DDecoder;\n- PANNsFeatureExtractor + Spec2DCNN + UNet1DDecoder.\n- PANNsFeatureExtractor + CenterNet + UNet1DDecoder.\n- TransformerAutoModel (xsmall, downsample_rate=8).\n\n**Parameter Tunning**: Add more kernel_size 8 for CNNSpectrogram can gain +0.002 online.\n\n**Multi-Task Learning Objectives**: sleep status, onset, wake.\n\n**Loss Function**: For Spec2DCNN and TransformerAutoModel, we use BCE, but with multi-task target weighting, sleep:onset:wake = 0.5:1:1. The purpose of this is to allow the model to focus on learning the last two columns. We tried to train only for the onset and wake columns, but the score was not good. The reason is speculated that **the positive samples in these two columns are sparse, and MTL needs to be used to transfer the information from positive samples in the sleep status to the prediction of sleep activity events.** Also, I tried KL Loss but it didn't work that well. \n\n```python\nself.loss_fn = nn.BCEWithLogitsLoss(pos_weight=torch.tensor([0.5,1.,1.]))\n```\n\nAt the same time, we adjusted epoch to 70 and added early stopping with patience=15. The early stopping criterion is the AP of the validation dataset, not the loss of the validation set. batch_size=32.\n\n| Experiment | Fold0 | Public (single fold) | Private (5-fold) |\n| --- | --- | --- | --- |\n| earlystop by val_loss | 0.750 | 0.697 | 0.742 |\n| earlystop by val_score | 0.751 | 0.718 | 0.744 |\n| loss_wgt = 1:1:1 | 0.752 | 0.724 | 0.749 |\n| loss_wgt = 0.5:1:1 | 0.755 | 0.723 | 0.753 |\n\nNote: we used the model_weight.pth with the best offline val_score to submit our LB instead of using the best_model.pth with the best offline val_loss。\n\n## 4. Post Processing\n\nOur post-processing mainly includes:\n\n- **find_peaks()**: scipy.signal.find_peaks;\n- **NMS**: This task can be treated as object detection. [onset, wakeup] is regarded as a bounding boxes, and score is the confident of the box. Therefore, I used a time-series NMS. Using NMS can eliminate redundant boxes with high IOU, which increase our AP.\n\n```python\ndef apply_nms(dets_arr, thresh):\n x1 = dets_arr[:, 0]\n x2 = dets_arr[:, 1]\n scores = dets_arr[:, 2]\n \n areas = x2 - x1\n order = scores.argsort()[::-1]\n \n keep = []\n while order.size > 0:\n i = order[0]\n keep.append(i)\n xx1 = np.maximum(x1[i], x1[order[1:]])\n xx2 = np.minimum(x2[i], x2[order[1:]])\n inter = np.maximum(0.0, xx2 - xx1 + 1)\n ovr = inter / (areas[i] + areas[order[1:]] - inter)\n inds = np.where(ovr <= thresh)[0]\n order = order[inds + 1]\n \n dets_nms_arr = dets_arr[keep,:]\n onset_steps = dets_nms_arr[:, 0].tolist()\n wakeup_steps = dets_nms_arr[:, 1].tolist()\n nms_save_steps = np.unique(onset_steps + wakeup_steps).tolist()\n return nms_save_steps\n```\n\nIn addition, we set score_th=0.005 (If it is set too low, a large number of events will be detected and cause online scoring errors, so it is fixed at 0.005 here), and use optuna to simultaneously search the parameter distance in find_peaks and the parameter iou_threshold of NMS. Finally, when distance=72 and iou_threshold=0.995, the best performance is achieved.\n\n```python\nimport optuna\n\ndef objective(trial):\n score_th = 0.005 # trial.suggest_float('score_th', 0.003, 0.006)\n distance = trial.suggest_int('distance', 20, 80)\n thresh = trial.suggest_float('thresh', 0.75, 1.)\n\n # find peak\n val_pred_df = post_process_for_seg(\n keys=keys,\n preds=preds[:, :, [1, 2]],\n score_th=score_th,\n distance=distance,\n )\n\n # nms\n val_pred_df = val_pred_df.to_pandas()\n nms_pred_dfs = NMS_prediction(val_pred_df, thresh, verbose=False)\n score = event_detection_ap(valid_event_df.to_pandas(), nms_pred_dfs)\n return -score\n\nstudy = optuna.create_study()\nstudy.optimize(objective, n_trials=100)\nprint('Best hyperparameters: ', study.best_params)\nprint('Best score: ', study.best_value)\n```\n\n| Experiment | Fold0 | Pubic (5-fold) | Private (5-fold) |\n| --- | --- | --- | --- |\n| find_peak | - | 0.745 | 0.787 |\n| find_peak+NMS+optuna | - | 0.746 | 0.789 |\n\n## 5. Model Ensemble\n\nFinally, we average the output probabilities of the following models and then feed into the post processing methods to detect events. By the way, I tried post-processing the detection events for each model and then concating them, but this resulted in too many detections. Even with NMS, I didn't get a better score.\n\nThe number of ensemble models: 4 (types of models) * 5 (fold number) = 20.\n\n| Experiment | Fold0 | Pubic (5-fold) | Private (5-fold) |\n| --- | --- | --- | --- |\n| model1: CNNSpectrogram + Spec2DCNN + UNet1DDecoder | 0.77209 | 0.743 | 0.784 |\n| model2: PANNsFeatureExtractor + Spec2DCNN + UNet1DDecoder | 0.777 | 0.743 | 0.782 |\n| model3: PANNsFeatureExtractor + CenterNet + UNet1DDecoder | 0.75968 | 0.634 | 0.68 |\n| model4: TransformerAutoModel | 0.74680 | - | - |\n| model1 + model2(1:1) | - | 0.746 | 0.789 |\n| model1 + model2+model3(1:1:0.4) | - | 0.75 | 0.786 |\n| model1 + model2+model3+model4(1:1:0.4:0.2) | | 0.752 | 0.787 |\n\nUnfortunately, we only considered CenterNet and Transformer to model ensemble with a tentative attitude on the last day, but surprisingly found that **a low-CV-scoring model still has a probability of improving final performance as long as it is heterogeneous compared with your previous models**. But we didn’t have more opportunities to submit more, which was a profound lesson for me.\n\nThoughts not done:\n- **Data Augmentation**: Shift the time within the batch to increase more time diversity and reduce dependence on hour features.\n\n- **Model**: Try more models. Although we try transformer and it didn’t work for us. I am veryyy looking forward to the solutions from top-ranking players.\n\nThanks again to Kaggle and all Kaggle players. This was a good competition and we learned a lot from it. If you think our solution is useful for you, welcome to upvote and discuss with us.\n\nIn addition, this is my first 🥈 silver medal. Thank you everyone for letting me learn a lot. I will continue to work hard. :)", "votes": null }, { "id": "2550417", "postDate": "12/06/2023 02:54:25", "content": "Thanks for sharing~
", "rawMarkdown": "Thanks for sharing~", "votes": null }, { "id": "2550422", "postDate": "12/06/2023 03:03:34", "content": "Thanks for sharing~
", "rawMarkdown": "Thanks for sharing~", "votes": null }, { "id": "2550432", "postDate": "12/06/2023 03:11:48", "content": "I also forked @tubotubo 's code, but I mainly focused on Spec2DCNN's model architecture, e.g. replacing smp.UNet with smp.DeepLabV3Plus, etc. Also, you can get a nice boost by mixing feature_extractor and decoder. Thank you very much for sharing!🥳
", "rawMarkdown": "I also forked @tubotubo 's code, but I mainly focused on Spec2DCNN's model architecture, e.g. replacing smp.UNet with smp.DeepLabV3Plus, etc. Also, you can get a nice boost by mixing feature_extractor and decoder. Thank you very much for sharing!🥳", "votes": null }, { "id": "2550460", "postDate": "12/06/2023 03:34:46", "content": "Nice, I haven't try smp.DeepLabV3Plus. Thanks for sharing.
", "rawMarkdown": "Nice, I haven't try smp.DeepLabV3Plus. Thanks for sharing.", "votes": null }, { "id": "2550611", "postDate": "12/06/2023 06:25:12", "content": "Also noticed that feature engineering didnt seem to improve performance, not even sinusoidal time features - was left quite confused!
", "rawMarkdown": "Also noticed that feature engineering didnt seem to improve performance, not even sinusoidal time features - was left quite confused!", "votes": null }, { "id": "2550619", "postDate": "12/06/2023 06:40:23", "content": "Same feeling as your, we have done with many feature works but they don't work at all.
", "rawMarkdown": "Same feeling as your, we have done with many feature works but they don't work at all.", "votes": null } ], "comments": [ { "id": 2550417, "author_name": "liruiqi577", "author_url": "", "post_date": "12/06/2023 02:54:25", "content": "Thanks for sharing~
", "votes": null, "replies": [] }, { "id": 2550422, "author_name": "brickcoder", "author_url": "", "post_date": "12/06/2023 03:03:34", "content": "Thanks for sharing~
", "votes": null, "replies": [] }, { "id": 2550432, "author_name": "jimmyisme1", "author_url": "", "post_date": "12/06/2023 03:11:48", "content": "I also forked @tubotubo 's code, but I mainly focused on Spec2DCNN's model architecture, e.g. replacing smp.UNet with smp.DeepLabV3Plus, etc. Also, you can get a nice boost by mixing feature_extractor and decoder. Thank you very much for sharing!🥳
", "votes": null, "replies": [ { "id": 2550460, "author_name": "alvinai9603", "author_url": "", "post_date": "12/06/2023 03:34:46", "content": "Nice, I haven't try smp.DeepLabV3Plus. Thanks for sharing.
", "votes": null, "replies": [] } ] }, { "id": 2550611, "author_name": "exjustice", "author_url": "", "post_date": "12/06/2023 06:25:12", "content": "Also noticed that feature engineering didnt seem to improve performance, not even sinusoidal time features - was left quite confused!
", "votes": null, "replies": [ { "id": 2550619, "author_name": "alvinai9603", "author_url": "", "post_date": "12/06/2023 06:40:23", "content": "Same feeling as your, we have done with many feature works but they don't work at all.
", "votes": null, "replies": [] } ] } ], "raw_markdown_by_id": { "2550342": "First of all, I would like to thank @tubotubo for sharing your high-quality code, and also thank my teammates @liruiqi577 @brickcoder @xtzhou for their contributions in the competition. Here, I am going to share our team’s “**snore like thunder**” solution from the following aspects:\n\n1. Data preprocessing\n2. Feature Engineering\n3. Model\n4. Post Processing\n5. Model Ensemble\n\n## 1. Data preprocessing\nWe made EDA and readed open discussions found that there are 4 types of data anomalies:\n- Some series have a high missing rate and some of them do not even have any event labels;\n- In some series , there are no event annotations in the middle and tail (possibly because the collection activity has stopped);\n- The sleep record is incomplete (a period of sleep is only marked with onset or wakeup).\n- There are outliers in the enmo value.\n\nTo this end, we have some attempts, such as:\n- Eliminate series with high missing rates;\n- Cut the tail of the series without event labels;\n- Upper clip enmo to 1.\n\nBut the above methods didn't completely work. In the end, our preprocessing method was:\n\nWe split the dataset group by series into 5 folds. For each fold, we eliminate series with a label missing rate of 100% in the training dataset while without performing any data preprocessing on the validation set. This is done to avoid introducing noise to the training set, and to ensure that the evaluation results of the validation set are more biased towards the real data distribution, which improve our LB score + 0.006.\n\nPart of our experiments as below:\n| Experiment | Fold0 | Public (single fold) | Private (5-fold) |\n| --- | --- | --- | --- |\n| No preprocess missing data | 0.751 | 0.718 | 0.744 |\n| Eliminate unlabeled data at the end of train_series & series with missing rate >80% | 0.739 | 0.709 | 0.741 |\n| Drop train series which don’t have any event labels | 0.752 | 0.724 | 0.749 |\n\n## 2. Feature Engineering\n- Sensor features: After smoothing the enmo and anglez features, a first-order difference is made to obtain the absolute value. Then replace the original enmo and anglez features with these features, which improve our LB score + 0.01.\n\n```python\ntrain_series['enmo_abs_diff'] = train_series['enmo'].diff().abs()\ntrain_series['enmo'] = train_series['enmo_abs_diff'].rolling(window=5, center=True, min_periods=1).mean()\ntrain_series['anglez_abs_diff'] = train_series['anglez'].diff().abs()\ntrain_series['anglez'] = train_series['anglez_abs_diff'].rolling(window=5, center=True, min_periods=1).mean()\n```\n\n- Time features: sin and cos hour.\n\nIn addition, we also made the following features based on open notebooks and our EDA, such as: differential features with different orders, rolling window statistical features, interactive features of enmo and anglez (such as anglez's differential abs * enmo, etc.), anglez_rad_sin/cos, dayofweek/is_weekend (I find that children have different sleeping habits on weekdays and weekends). But strangely enough, too much feature engineering didn’t bring us much benefit.\n\n| Experiment | Fold0 | Public (5-fold) | Private (5-fold) |\n| --- | --- | --- | --- |\n| anglez + enmo + hour_sin + hour_cos | 0.763 | 0.731 | 0.768 |\n| anglez_abs_diff + enmo_abs_diff + hour_sin + hour_cos | 0.771 | 0.741 | 0.781 |\n\n## 3. Model\nWe used 4 models:\n\n- CNNSpectrogram + Spec2DCNN + UNet1DDecoder;\n- PANNsFeatureExtractor + Spec2DCNN + UNet1DDecoder.\n- PANNsFeatureExtractor + CenterNet + UNet1DDecoder.\n- TransformerAutoModel (xsmall, downsample_rate=8).\n\n**Parameter Tunning**: Add more kernel_size 8 for CNNSpectrogram can gain +0.002 online.\n\n**Multi-Task Learning Objectives**: sleep status, onset, wake.\n\n**Loss Function**: For Spec2DCNN and TransformerAutoModel, we use BCE, but with multi-task target weighting, sleep:onset:wake = 0.5:1:1. The purpose of this is to allow the model to focus on learning the last two columns. We tried to train only for the onset and wake columns, but the score was not good. The reason is speculated that **the positive samples in these two columns are sparse, and MTL needs to be used to transfer the information from positive samples in the sleep status to the prediction of sleep activity events.** Also, I tried KL Loss but it didn't work that well. \n\n```python\nself.loss_fn = nn.BCEWithLogitsLoss(pos_weight=torch.tensor([0.5,1.,1.]))\n```\n\nAt the same time, we adjusted epoch to 70 and added early stopping with patience=15. The early stopping criterion is the AP of the validation dataset, not the loss of the validation set. batch_size=32.\n\n| Experiment | Fold0 | Public (single fold) | Private (5-fold) |\n| --- | --- | --- | --- |\n| earlystop by val_loss | 0.750 | 0.697 | 0.742 |\n| earlystop by val_score | 0.751 | 0.718 | 0.744 |\n| loss_wgt = 1:1:1 | 0.752 | 0.724 | 0.749 |\n| loss_wgt = 0.5:1:1 | 0.755 | 0.723 | 0.753 |\n\nNote: we used the model_weight.pth with the best offline val_score to submit our LB instead of using the best_model.pth with the best offline val_loss。\n\n## 4. Post Processing\n\nOur post-processing mainly includes:\n\n- **find_peaks()**: scipy.signal.find_peaks;\n- **NMS**: This task can be treated as object detection. [onset, wakeup] is regarded as a bounding boxes, and score is the confident of the box. Therefore, I used a time-series NMS. Using NMS can eliminate redundant boxes with high IOU, which increase our AP.\n\n```python\ndef apply_nms(dets_arr, thresh):\n x1 = dets_arr[:, 0]\n x2 = dets_arr[:, 1]\n scores = dets_arr[:, 2]\n \n areas = x2 - x1\n order = scores.argsort()[::-1]\n \n keep = []\n while order.size > 0:\n i = order[0]\n keep.append(i)\n xx1 = np.maximum(x1[i], x1[order[1:]])\n xx2 = np.minimum(x2[i], x2[order[1:]])\n inter = np.maximum(0.0, xx2 - xx1 + 1)\n ovr = inter / (areas[i] + areas[order[1:]] - inter)\n inds = np.where(ovr <= thresh)[0]\n order = order[inds + 1]\n \n dets_nms_arr = dets_arr[keep,:]\n onset_steps = dets_nms_arr[:, 0].tolist()\n wakeup_steps = dets_nms_arr[:, 1].tolist()\n nms_save_steps = np.unique(onset_steps + wakeup_steps).tolist()\n return nms_save_steps\n```\n\nIn addition, we set score_th=0.005 (If it is set too low, a large number of events will be detected and cause online scoring errors, so it is fixed at 0.005 here), and use optuna to simultaneously search the parameter distance in find_peaks and the parameter iou_threshold of NMS. Finally, when distance=72 and iou_threshold=0.995, the best performance is achieved.\n\n```python\nimport optuna\n\ndef objective(trial):\n score_th = 0.005 # trial.suggest_float('score_th', 0.003, 0.006)\n distance = trial.suggest_int('distance', 20, 80)\n thresh = trial.suggest_float('thresh', 0.75, 1.)\n\n # find peak\n val_pred_df = post_process_for_seg(\n keys=keys,\n preds=preds[:, :, [1, 2]],\n score_th=score_th,\n distance=distance,\n )\n\n # nms\n val_pred_df = val_pred_df.to_pandas()\n nms_pred_dfs = NMS_prediction(val_pred_df, thresh, verbose=False)\n score = event_detection_ap(valid_event_df.to_pandas(), nms_pred_dfs)\n return -score\n\nstudy = optuna.create_study()\nstudy.optimize(objective, n_trials=100)\nprint('Best hyperparameters: ', study.best_params)\nprint('Best score: ', study.best_value)\n```\n\n| Experiment | Fold0 | Pubic (5-fold) | Private (5-fold) |\n| --- | --- | --- | --- |\n| find_peak | - | 0.745 | 0.787 |\n| find_peak+NMS+optuna | - | 0.746 | 0.789 |\n\n## 5. Model Ensemble\n\nFinally, we average the output probabilities of the following models and then feed into the post processing methods to detect events. By the way, I tried post-processing the detection events for each model and then concating them, but this resulted in too many detections. Even with NMS, I didn't get a better score.\n\nThe number of ensemble models: 4 (types of models) * 5 (fold number) = 20.\n\n| Experiment | Fold0 | Pubic (5-fold) | Private (5-fold) |\n| --- | --- | --- | --- |\n| model1: CNNSpectrogram + Spec2DCNN + UNet1DDecoder | 0.77209 | 0.743 | 0.784 |\n| model2: PANNsFeatureExtractor + Spec2DCNN + UNet1DDecoder | 0.777 | 0.743 | 0.782 |\n| model3: PANNsFeatureExtractor + CenterNet + UNet1DDecoder | 0.75968 | 0.634 | 0.68 |\n| model4: TransformerAutoModel | 0.74680 | - | - |\n| model1 + model2(1:1) | - | 0.746 | 0.789 |\n| model1 + model2+model3(1:1:0.4) | - | 0.75 | 0.786 |\n| model1 + model2+model3+model4(1:1:0.4:0.2) | | 0.752 | 0.787 |\n\nUnfortunately, we only considered CenterNet and Transformer to model ensemble with a tentative attitude on the last day, but surprisingly found that **a low-CV-scoring model still has a probability of improving final performance as long as it is heterogeneous compared with your previous models**. But we didn’t have more opportunities to submit more, which was a profound lesson for me.\n\nThoughts not done:\n- **Data Augmentation**: Shift the time within the batch to increase more time diversity and reduce dependence on hour features.\n\n- **Model**: Try more models. Although we try transformer and it didn’t work for us. I am veryyy looking forward to the solutions from top-ranking players.\n\nThanks again to Kaggle and all Kaggle players. This was a good competition and we learned a lot from it. If you think our solution is useful for you, welcome to upvote and discuss with us.\n\nIn addition, this is my first 🥈 silver medal. Thank you everyone for letting me learn a lot. I will continue to work hard. :)", "2550417": "Thanks for sharing~", "2550422": "Thanks for sharing~", "2550432": "I also forked @tubotubo 's code, but I mainly focused on Spec2DCNN's model architecture, e.g. replacing smp.UNet with smp.DeepLabV3Plus, etc. Also, you can get a nice boost by mixing feature_extractor and decoder. Thank you very much for sharing!🥳", "2550460": "Nice, I haven't try smp.DeepLabV3Plus. Thanks for sharing.", "2550611": "Also noticed that feature engineering didnt seem to improve performance, not even sinusoidal time features - was left quite confused!", "2550619": "Same feeling as your, we have done with many feature works but they don't work at all." }, "source": "meta" }