{ "id": 442902, "title": "Ideas for postprocessing?", "url": "/competitions/child-mind-institute-detect-sleep-states/discussion/442902", "author_name": "", "post_date": "2023-09-24T17:12:26.166153100Z", "votes": 5, "comment_count": 3, "views": 0, "content": "

Hello,

\n

I've noticed that there are instances of NAN or unlabeled data in the train_events dataset, and interestingly, my models are making predictions on these unlabeled days. While I've explored a few post-processing techniques to identify these unlabeled days, I've yet to see significant improvements in my CB or LB results.

\n

Here's what I've tried so far:

\n

FIT a model to predict which day is unlabeled: Made a datasets with all the recorded day and unlabeled day. I then trained a LightGBM model with all the time features such as month, weekday, etc, and set the target as whether the day is unlabeled. I decreases the scores column in submission based on the model's predictions. However, this approach resulted in a decrease in my LB score by 0.1.

\n

Weighting Predictions: I experimented with decreasing the prediction scores as the number of recorded days increased. I observed that as time progressed, there were more unlabeled nights, especially towards the end of each series_id. While this approach will dramatically improve predictions for a few series(like from 0.1 to 0.5), but it will decrease most other series's score by around 0.05, ultimately the gain and loss in score just evens up.

\n

I'm keen to hear if anyone has other ideas or techniques to tackle this. Thank you!

", "messages": [ { "id": "2454261", "postDate": "09/24/2023 17:12:26", "content": "

Hello,

\n

I've noticed that there are instances of NAN or unlabeled data in the train_events dataset, and interestingly, my models are making predictions on these unlabeled days. While I've explored a few post-processing techniques to identify these unlabeled days, I've yet to see significant improvements in my CB or LB results.

\n

Here's what I've tried so far:

\n

FIT a model to predict which day is unlabeled: Made a datasets with all the recorded day and unlabeled day. I then trained a LightGBM model with all the time features such as month, weekday, etc, and set the target as whether the day is unlabeled. I decreases the scores column in submission based on the model's predictions. However, this approach resulted in a decrease in my LB score by 0.1.

\n

Weighting Predictions: I experimented with decreasing the prediction scores as the number of recorded days increased. I observed that as time progressed, there were more unlabeled nights, especially towards the end of each series_id. While this approach will dramatically improve predictions for a few series(like from 0.1 to 0.5), but it will decrease most other series's score by around 0.05, ultimately the gain and loss in score just evens up.

\n

I'm keen to hear if anyone has other ideas or techniques to tackle this. Thank you!

", "rawMarkdown": "Hello,\n\nI've noticed that there are instances of NAN or unlabeled data in the train_events dataset, and interestingly, my models are making predictions on these unlabeled days. While I've explored a few post-processing techniques to identify these unlabeled days, I've yet to see significant improvements in my CB or LB results.\n\nHere's what I've tried so far:\n\n**FIT a model to predict which day is unlabeled**: Made a datasets with all the recorded day and unlabeled day. I then trained a LightGBM model with all the time features such as month, weekday, etc, and set the target as whether the day is unlabeled. I decreases the scores column in submission based on the model's predictions. However, this approach resulted in a decrease in my LB score by 0.1.\n\n**Weighting Predictions**: I experimented with decreasing the prediction scores as the number of recorded days increased. I observed that as time progressed, there were more unlabeled nights, especially towards the end of each series_id. While this approach will dramatically improve predictions for a few series(like from 0.1 to 0.5), but it will decrease most other series's score by around 0.05, ultimately the gain and loss in score just evens up.\n\nI'm keen to hear if anyone has other ideas or techniques to tackle this. Thank you!", "votes": null }, { "id": "2454415", "postDate": "09/24/2023 18:52:42", "content": "

@yuchen2066 Considering the number of null lines in the event file at the moment (about 34%), I think this attempt is very worthwhile. However, judging from those results, I was wondering if the trends in the correct labels for the test data are different. What do you think?

", "rawMarkdown": "yuchen2066 Considering the number of null lines in the event file at the moment (about 34%), I think this attempt is very worthwhile. However, judging from those results, I was wondering if the trends in the correct labels for the test data are different. What do you think?", "votes": null }, { "id": "2455579", "postDate": "09/25/2023 15:23:46", "content": "
\n

judging from those results

\n
\n

To be honest, I am just implementing random post-processing ideas. I haven't even applied early stopping or cross-validation to the lgbm model above yet. So, I can't definitively claim that the trends are different in the training and test datasets. Please don't be discouraged by my poor results.

", "rawMarkdown": ">judging from those results\n\nTo be honest, I am just implementing random post-processing ideas. I haven't even applied early stopping or cross-validation to the lgbm model above yet. So, I can't definitively claim that the trends are different in the training and test datasets. Please don't be discouraged by my poor results.", "votes": null }, { "id": "2485105", "postDate": "10/17/2023 00:23:05", "content": "

Hi @yuchen2066 , I in this notebook did this function for postprocesing and my score improve 0.100 points:

\n
def get_events(idx, classifier, file=) :\n\n    test  = pd.read_parquet(f,\n                    filters=[(,,idx)])\n    test = feat_eng(test)\n     = test.drop(columns=drop_cols)\n    test = test[drop_cols]\n\n    preds, probs = classifier.predict(), classifier.predict_proba()[:, ]\n\n    test[] = preds\n    test[] = test[].rolling(+, center=).median()\n    test[] = probs\n\n    test = test[test[]!=]\n\n    test.loc[test[]==, ] = -test.loc[test[]==, ]\n    test[] = test[].rolling(**, center=, min_periods=).mean().bfill().ffill()\n    test[] = test[].diff()\n    test[] = test[].replace({:, :, :np.nan})\n\n    test_wakeup = test[test[]==].groupby(test[].dt.date).agg()\n    test_onset = test[test[]==].groupby(test[].dt.date).agg()\n    test = pd.concat([test_wakeup, test_onset], ignore_index=).sort_values()\n\n    return test\n
", "rawMarkdown": "Hi @yuchen2066 , I in this [notebook](https://www.kaggle.com/code/renatoreggiani/zzzs-ensemble-model-cmi) did this function for postprocesing and my score improve 0.100 points:\n\n\n```\ndef get_events(idx, classifier, file='test_series.parquet') :\n \n test = pd.read_parquet(f'/kaggle/input/child-mind-institute-detect-sleep-states/{file}',\n filters=[('series_id','=',idx)])\n test = feat_eng(test)\n X_test = test.drop(columns=drop_cols)\n test = test[drop_cols]\n\n preds, probs = classifier.predict(X_test), classifier.predict_proba(X_test)[:, 1]\n \n test['prediction'] = preds\n test['prediction'] = test['prediction'].rolling(360+1, center=True).median()\n test['probability'] = probs\n \n test = test[test['prediction']!=2]\n \n test.loc[test['prediction']==0, 'probability'] = 1-test.loc[test['prediction']==0, 'probability']\n test['score'] = test['probability'].rolling(60*12*5, center=True, min_periods=10).mean().bfill().ffill()\n test['pred_diff'] = test['prediction'].diff()\n test['event'] = test['pred_diff'].replace({1:'wakeup', -1:'onset', 0:np.nan})\n \n test_wakeup = test[test['event']=='wakeup'].groupby(test['timestamp'].dt.date).agg('first')\n test_onset = test[test['event']=='onset'].groupby(test['timestamp'].dt.date).agg('last')\n test = pd.concat([test_wakeup, test_onset], ignore_index=True).sort_values('timestamp')\n\n return test\n```", "votes": null } ], "comments": [ { "id": 2454415, "author_name": "mitsuyasuhoshino", "author_url": "", "post_date": "09/24/2023 18:52:42", "content": "

@yuchen2066 Considering the number of null lines in the event file at the moment (about 34%), I think this attempt is very worthwhile. However, judging from those results, I was wondering if the trends in the correct labels for the test data are different. What do you think?

", "votes": null, "replies": [ { "id": 2455579, "author_name": "yuchen2066", "author_url": "", "post_date": "09/25/2023 15:23:46", "content": "
\n

judging from those results

\n
\n

To be honest, I am just implementing random post-processing ideas. I haven't even applied early stopping or cross-validation to the lgbm model above yet. So, I can't definitively claim that the trends are different in the training and test datasets. Please don't be discouraged by my poor results.

", "votes": null, "replies": [] } ] }, { "id": 2485105, "author_name": "renatoreggiani", "author_url": "", "post_date": "10/17/2023 00:23:05", "content": "

Hi @yuchen2066 , I in this notebook did this function for postprocesing and my score improve 0.100 points:

\n
def get_events(idx, classifier, file=) :\n\n    test  = pd.read_parquet(f,\n                    filters=[(,,idx)])\n    test = feat_eng(test)\n     = test.drop(columns=drop_cols)\n    test = test[drop_cols]\n\n    preds, probs = classifier.predict(), classifier.predict_proba()[:, ]\n\n    test[] = preds\n    test[] = test[].rolling(+, center=).median()\n    test[] = probs\n\n    test = test[test[]!=]\n\n    test.loc[test[]==, ] = -test.loc[test[]==, ]\n    test[] = test[].rolling(**, center=, min_periods=).mean().bfill().ffill()\n    test[] = test[].diff()\n    test[] = test[].replace({:, :, :np.nan})\n\n    test_wakeup = test[test[]==].groupby(test[].dt.date).agg()\n    test_onset = test[test[]==].groupby(test[].dt.date).agg()\n    test = pd.concat([test_wakeup, test_onset], ignore_index=).sort_values()\n\n    return test\n
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "2454261": "Hello,\n\nI've noticed that there are instances of NAN or unlabeled data in the train_events dataset, and interestingly, my models are making predictions on these unlabeled days. While I've explored a few post-processing techniques to identify these unlabeled days, I've yet to see significant improvements in my CB or LB results.\n\nHere's what I've tried so far:\n\n**FIT a model to predict which day is unlabeled**: Made a datasets with all the recorded day and unlabeled day. I then trained a LightGBM model with all the time features such as month, weekday, etc, and set the target as whether the day is unlabeled. I decreases the scores column in submission based on the model's predictions. However, this approach resulted in a decrease in my LB score by 0.1.\n\n**Weighting Predictions**: I experimented with decreasing the prediction scores as the number of recorded days increased. I observed that as time progressed, there were more unlabeled nights, especially towards the end of each series_id. While this approach will dramatically improve predictions for a few series(like from 0.1 to 0.5), but it will decrease most other series's score by around 0.05, ultimately the gain and loss in score just evens up.\n\nI'm keen to hear if anyone has other ideas or techniques to tackle this. Thank you!", "2454415": "yuchen2066 Considering the number of null lines in the event file at the moment (about 34%), I think this attempt is very worthwhile. However, judging from those results, I was wondering if the trends in the correct labels for the test data are different. What do you think?", "2455579": ">judging from those results\n\nTo be honest, I am just implementing random post-processing ideas. I haven't even applied early stopping or cross-validation to the lgbm model above yet. So, I can't definitively claim that the trends are different in the training and test datasets. Please don't be discouraged by my poor results.", "2485105": "Hi @yuchen2066 , I in this [notebook](https://www.kaggle.com/code/renatoreggiani/zzzs-ensemble-model-cmi) did this function for postprocesing and my score improve 0.100 points:\n\n\n```\ndef get_events(idx, classifier, file='test_series.parquet') :\n \n test = pd.read_parquet(f'/kaggle/input/child-mind-institute-detect-sleep-states/{file}',\n filters=[('series_id','=',idx)])\n test = feat_eng(test)\n X_test = test.drop(columns=drop_cols)\n test = test[drop_cols]\n\n preds, probs = classifier.predict(X_test), classifier.predict_proba(X_test)[:, 1]\n \n test['prediction'] = preds\n test['prediction'] = test['prediction'].rolling(360+1, center=True).median()\n test['probability'] = probs\n \n test = test[test['prediction']!=2]\n \n test.loc[test['prediction']==0, 'probability'] = 1-test.loc[test['prediction']==0, 'probability']\n test['score'] = test['probability'].rolling(60*12*5, center=True, min_periods=10).mean().bfill().ffill()\n test['pred_diff'] = test['prediction'].diff()\n test['event'] = test['pred_diff'].replace({1:'wakeup', -1:'onset', 0:np.nan})\n \n test_wakeup = test[test['event']=='wakeup'].groupby(test['timestamp'].dt.date).agg('first')\n test_onset = test[test['event']=='onset'].groupby(test['timestamp'].dt.date).agg('last')\n test = pd.concat([test_wakeup, test_onset], ignore_index=True).sort_values('timestamp')\n\n return test\n```" }, "source": "meta" }