{ "id": 447882, "title": "CV vs LB scores", "url": "/competitions/child-mind-institute-detect-sleep-states/discussion/447882", "author_name": "Fritz Cremer", "post_date": "2023-10-17T16:25:05.347000", "votes": 45, "comment_count": 49, "views": 0, "content": "

I consistently get lower LB scores than local CV scores. My cv scores are above 0.7 while LB sits at 0.66. I am not sure whether this is because of the data distribution or differences between the metric function the host provided and the one that is actually used. For example, it seems that a constant change of the score make a difference for LB, but not locally. How does your CV score compare to LB score?

", "messages": [ { "id": 2486021, "postDate": "2023-10-17T16:25:05.347Z", "content": "

I consistently get lower LB scores than local CV scores. My cv scores are above 0.7 while LB sits at 0.66. I am not sure whether this is because of the data distribution or differences between the metric function the host provided and the one that is actually used. For example, it seems that a constant change of the score make a difference for LB, but not locally. How does your CV score compare to LB score?

", "rawMarkdown": "I consistently get lower LB scores than local CV scores. My cv scores are above 0.7 while LB sits at 0.66. I am not sure whether this is because of the data distribution or differences between the metric function the host provided and the one that is actually used. For example, it seems that a constant change of the score make a difference for LB, but not locally. How does your CV score compare to LB score?", "votes": 45 }, { "id": 2491744, "postDate": "2023-10-22T01:58:38.567Z", "content": "

GPU Deep Learning (RNN, CNN, Transformer) with preprocess using GPU RAPIDS cuDF!
\n1 fold out-of-fold. Here are my first few submissions:
\nCV 0.725 LB 0.691
\nCV 0.750 LB 0.714
\nCV 0.770 LB 0.725

", "rawMarkdown": "GPU Deep Learning (RNN, CNN, Transformer) with preprocess using GPU RAPIDS cuDF!\n1 fold out-of-fold. Here are my first few submissions:\nCV 0.725 LB 0.691\nCV 0.750 LB 0.714\nCV 0.770 LB 0.725", "votes": 32, "replies": [ { "id": 2491750, "postDate": "2023-10-22T02:11:00.433Z", "content": "

Wow great work as always Chris! I am excited to see how this works when it comes time to share!

", "rawMarkdown": "Wow great work as always Chris! I am excited to see how this works when it comes time to share! ", "votes": 3 }, { "id": 2492401, "postDate": "2023-10-22T13:35:53.490Z", "content": "

Wow I'm really impressed with how fast you made progress in this competition! I am not sure though, what do you mean by 1 fold, how large is that fold?

", "rawMarkdown": "Wow I'm really impressed with how fast you made progress in this competition! I am not sure though, what do you mean by 1 fold, how large is that fold?", "votes": 1, "replies": [ { "id": 2492706, "postDate": "2023-10-22T19:39:43.210Z", "content": "

Thanks Fritz!

\n

The \"CV score\" is computed using only 1 out of 10 folds from Group 10-fold on series_id. (So it isn't the full 10 folds oof CV score). Then after finding the model parameters, I retrain one model using 100% data with the best hyparameters and submit to LB.

", "rawMarkdown": "Thanks Fritz!\n\nThe \"CV score\" is computed using only 1 out of 10 folds from Group 10-fold on series_id. (So it isn't the full 10 folds oof CV score). Then after finding the model parameters, I retrain one model using 100% data with the best hyparameters and submit to LB.", "votes": 15 }, { "id": 2492816, "postDate": "2023-10-22T23:11:30.610Z", "content": "

Ah I see, thanks for clarification!

", "rawMarkdown": "Ah I see, thanks for clarification!", "votes": 1 }, { "id": 2532429, "postDate": "2023-11-21T04:21:03.330Z", "content": "

@cdeotte thanks for sharing, when retraining using 100% data, how do you know your loss isnt improving and you are not overfitting as you have no validation data

", "rawMarkdown": "@cdeotte thanks for sharing, when retraining using 100% data, how do you know your loss isnt improving and you are not overfitting as you have no validation data" }, { "id": 2542758, "postDate": "2023-11-29T13:41:20.830Z", "content": "

Hi,
\nI just came across the comment, so thought to add some ideas.

\n

we do CV to find best parameters of the model , if some parameters work better than other in each CV fold , then we can say that those parameters are generally good and will give best results with different datasets.

\n

So we get best parameters from CV and then train model again on whole dataset so that we capture all samples of data.

\n

for example. For tree model CV may give us max_depth = 3 , so how the splits will happen will depend on data but on average limiting tree growth to depth of 3 gives best results.
\nSo we are leveraging best parameter guess on average which we obtain from the CV and also utilize the whole dataset so that we capture all samples of data, in this case how the split will happen will be decided based on whole dataset.

\n

This is basic idea but based on business usecase you can set retrain = False in sklearn if you do not want model to retrain on whole data for some reason.

", "rawMarkdown": "Hi,\nI just came across the comment, so thought to add some ideas.\n\nwe do CV to find best parameters of the model , if some parameters work better than other in each CV fold , then we can say that those parameters are generally good and will give best results with different datasets.\n\nSo we get best parameters from CV and then train model again on whole dataset so that we capture all samples of data.\n\nfor example. For tree model CV may give us max_depth = 3 , so how the splits will happen will depend on data but on average limiting tree growth to depth of 3 gives best results.\nSo we are leveraging best parameter guess on average which we obtain from the CV and also utilize the whole dataset so that we capture all samples of data, in this case how the split will happen will be decided based on whole dataset.\n\nThis is basic idea but based on business usecase you can set retrain = False in sklearn if you do not want model to retrain on whole data for some reason.\n" } ] }, { "id": 2504011, "postDate": "2023-10-29T15:33:16.173Z", "content": "

Great CV-LB. I'm trying transformer as well, but I find it's hard to train it, OOM always happens! May I ask how do you solve this problem? @cdeotte

", "rawMarkdown": "Great CV-LB. I'm trying transformer as well, but I find it's hard to train it, OOM always happens! May I ask how do you solve this problem? @cdeotte ", "votes": 1, "replies": [ { "id": 2507348, "postDate": "2023-10-31T23:23:07.953Z", "content": "

@judith007 The best way to diagnose and solve OOM errors is to run your inference notebook interactively in Kaggle notebook (and infer all 277 train series_id) and watch the \"View Session Metrics\" dropdown window. Monitor the memory usage and if any code cell is using too much memory, then find a way to delete previous variables and/or run the current cell with less memory.

\n

Most OOM are GPU VRAM, so you can infer with less batch_size and/or smaller chunks of data at one time. For example write a for-loop that infers small chunks one at a time and save the predictions versus trying to infer all data at once.

", "rawMarkdown": "@judith007 The best way to diagnose and solve OOM errors is to run your inference notebook interactively in Kaggle notebook (and infer all 277 train series_id) and watch the \"View Session Metrics\" dropdown window. Monitor the memory usage and if any code cell is using too much memory, then find a way to delete previous variables and/or run the current cell with less memory. \n\nMost OOM are GPU VRAM, so you can infer with less batch_size and/or smaller chunks of data at one time. For example write a for-loop that infers small chunks one at a time and save the predictions versus trying to infer all data at once. ", "votes": 5 } ] } ] }, { "id": 2512801, "postDate": "2023-11-04T20:47:13.833Z", "content": "

single model
\n5 Folds
\nCV 0.800
\nLB 0.759

", "rawMarkdown": "single model\n5 Folds\nCV 0.800\nLB 0.759", "votes": 18 }, { "id": 2511516, "postDate": "2023-11-03T17:54:33.123Z", "content": "

5 Folds
\nCV(out of fold) 0.811
\nLB(ensemble) 0.770

", "rawMarkdown": "5 Folds\nCV(out of fold) 0.811\nLB(ensemble) 0.770", "votes": 16, "replies": [ { "id": 2519261, "postDate": "2023-11-09T20:50:56.127Z", "content": "

U-Net or Transformer?

", "rawMarkdown": "U-Net or Transformer?" } ] }, { "id": 2500307, "postDate": "2023-10-26T15:20:48.280Z", "content": "

NN 1Fold only(split 5Folds)
\nLocal CV: 0.756 LB: 0.689

", "rawMarkdown": "NN 1Fold only(split 5Folds)\nLocal CV: 0.756 LB: 0.689", "votes": 14 }, { "id": 2529652, "postDate": "2023-11-18T12:43:03.067Z", "content": "

5 folds, GroupKFold
\ncv: 0.822, lb: 0.770
\ncv: 0.818, lb: 0.773
\ncv: 0.814, lb: 0.749
\ncv: 0.794, lb: 0.739
\ncv: 0.785, lb: 0.754

", "rawMarkdown": "5 folds, GroupKFold\ncv: 0.822, lb: 0.770\ncv: 0.818, lb: 0.773\ncv: 0.814, lb: 0.749\ncv: 0.794, lb: 0.739\ncv: 0.785, lb: 0.754", "votes": 11 }, { "id": 2517592, "postDate": "2023-11-08T15:51:37.460Z", "content": "

Transformer GroupKFold (By series id)

\n

5 fold - cv: 0.773 lb-0.748

\n

Update 5 fold: cv- 0.777-lb 0.754

\n

Update 5 fold cv: 0.791 - lb: 0.768

", "rawMarkdown": "Transformer GroupKFold (By series id)\n\n\n5 fold - cv: 0.773 lb-0.748\n\nUpdate 5 fold: cv- 0.777-lb 0.754\n\nUpdate 5 fold cv: 0.791 - lb: 0.768", "votes": 11 }, { "id": 2515422, "postDate": "2023-11-06T23:48:03.503Z", "content": "

single model

\n

4 Folds

\n

CV: 0.7454 ± 0.0199

\n

LB: 0.729

", "rawMarkdown": "single model\n\n4 Folds\n\nCV: 0.7454 ± 0.0199\n\nLB: 0.729", "votes": 10 }, { "id": 2493345, "postDate": "2023-10-23T10:50:26.127Z", "content": "

LSTM model
\n5-fold GroupKFold(by series_id)

\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n
CV scoreLB scorediff
0.6410.6110.030
0.6540.6290.025
0.6670.6450.022
", "rawMarkdown": "LSTM model\n5-fold GroupKFold(by series_id)\n| CV score | LB score | diff |\n| --- | --- | --- |\n| 0.641 | 0.611 | 0.030 |\n| 0.654 | 0.629 | 0.025 |\n| 0.667 | 0.645 | 0.022 |\n\n\n\n", "votes": 9 }, { "id": 2491782, "postDate": "2023-10-22T03:36:21.907Z", "content": "

1 fold xgb
\nCV 654 LB 636
\nBut can't improve anymore, it seems i have to turn to nn models.

", "rawMarkdown": "1 fold xgb\nCV 654 LB 636\nBut can't improve anymore, it seems i have to turn to nn models.", "votes": 10, "replies": [ { "id": 2493410, "postDate": "2023-10-23T11:47:19.013Z", "content": "

Thanks a great XGB model !!

", "rawMarkdown": "Thanks a great XGB model !!" } ] }, { "id": 2512401, "postDate": "2023-11-04T13:46:17.430Z", "content": "

NN 5 folds
\noof CV 0.746
\nLB 0.671

", "rawMarkdown": "NN 5 folds\noof CV 0.746\nLB 0.671", "votes": 8 }, { "id": 2490987, "postDate": "2023-10-21T08:40:35.537Z", "content": "

lightgbm

\n

1fold out-of-fold

\n

cv 0.422 lb 0.429
\ncv 0.533 lb .0.518
\ncv 0.635 lb 0.604
\ncv 0.697 lb 0.658

", "rawMarkdown": "lightgbm\n\n1fold out-of-fold\n\ncv 0.422 lb 0.429\ncv 0.533 lb .0.518\ncv 0.635 lb 0.604\ncv 0.697 lb 0.658", "votes": 8, "replies": [ { "id": 2491740, "postDate": "2023-10-22T01:56:47.197Z", "content": "

Great score with LGBM Adam!

", "rawMarkdown": "Great score with LGBM Adam!", "votes": 1, "replies": [ { "id": 2491759, "postDate": "2023-10-22T02:44:06.103Z", "content": "

Thanks. 😀

", "rawMarkdown": "Thanks. 😀", "votes": 1 } ] } ] }, { "id": 2532778, "postDate": "2023-11-21T10:21:41.197Z", "content": "

GroupKFold: k=5, group=series_id
\nCV:0.518
\nLB:0.749 (ensemble)

\n

My CV score is significantly lower than LB, but looking at other people's posts, it seems like CV is higher.
\nI am calculating the CV score using all 277 series.
\nAre you calculating CV scores for all series?
\nOr are you screening clean series with fewer than 277?

", "rawMarkdown": "GroupKFold: k=5, group=series_id\nCV:0.518\nLB:0.749 (ensemble)\n\nMy CV score is significantly lower than LB, but looking at other people's posts, it seems like CV is higher.\nI am calculating the CV score using all 277 series.\nAre you calculating CV scores for all series?\nOr are you screening clean series with fewer than 277?\n", "votes": 5, "replies": [ { "id": 2533079, "postDate": "2023-11-21T15:17:24.483Z", "content": "

I'm using all 277 series to compute CV.

", "rawMarkdown": "I'm using all 277 series to compute CV.", "votes": 3 }, { "id": 2533293, "postDate": "2023-11-21T18:31:02.740Z", "content": "

Did you drop nulls in train_events when computing cv, that might be a reason.

", "rawMarkdown": "Did you drop nulls in train_events when computing cv, that might be a reason.", "votes": 7, "replies": [ { "id": 2533405, "postDate": "2023-11-21T22:11:21.487Z", "content": "

@ryotayoshinobu @dlukic

\n

thanks a lot!
\nWhen I recalculated the CV, it became 0.785.

", "rawMarkdown": "@ryotayoshinobu @dlukic \n\nthanks a lot!\nWhen I recalculated the CV, it became 0.785.", "votes": 3 }, { "id": 2533408, "postDate": "2023-11-21T22:17:21.103Z", "rawMarkdown": "", "isDeleted": true }, { "id": 2533411, "postDate": "2023-11-21T22:22:52.933Z", "content": "

Yes, the scores of the five models are averaged before post-processing.

", "rawMarkdown": "Yes, the scores of the five models are averaged before post-processing." }, { "id": 2534322, "postDate": "2023-11-22T14:43:02.593Z", "content": "

great idea

", "rawMarkdown": "great idea" } ] } ] }, { "id": 2486471, "postDate": "2023-10-18T00:40:21.740Z", "content": "

What is your CV strategy (e.g 5 fold GroupKFold) and which dataset are you using for CV (if you do not mind sharing) ?

\n

For instance, I'm using 5-fold StratifiedGroupKFold on Zzzs_train_multi.parquet and this LB = 0.536 notebook has CV = 0.468

", "rawMarkdown": "What is your CV strategy (e.g 5 fold GroupKFold) and which dataset are you using for CV (if you do not mind sharing) ?\n\nFor instance, I'm using 5-fold StratifiedGroupKFold on Zzzs_train_multi.parquet and this LB = 0.536 [notebook](https://www.kaggle.com/code/yeoyunsianggeremie/fe-ensemble-added-period-3600) has CV = 0.468", "votes": 6, "replies": [ { "id": 2486994, "postDate": "2023-10-18T09:40:39.203Z", "content": "

I am using 4-fold GroupKFold. I filter some samples out for training but validate on the entire dataset.

", "rawMarkdown": "I am using 4-fold GroupKFold. I filter some samples out for training but validate on the entire dataset.", "votes": 2 } ] }, { "id": 2486938, "postDate": "2023-10-18T08:51:00.593Z", "content": "

Same here
\nCV 0.553 - 0.487
\nCV 0.561 - LB 0.501
\nCV 0.571 - LB 0.516

", "rawMarkdown": "Same here\nCV 0.553 - 0.487\nCV 0.561 - LB 0.501\nCV 0.571 - LB 0.516", "votes": 3 }, { "id": 2491777, "postDate": "2023-10-22T03:26:15.857Z", "content": "

NN:1fold out-of-fold cv 599 lb 632

", "rawMarkdown": "NN:1fold out-of-fold cv 599 lb 632", "votes": 4 }, { "id": 2545039, "postDate": "2023-12-01T08:53:34.687Z", "content": "

Since the beginning of the competition, I've tried several models, and I always have a huge difference between CV and LB score. I still haven't figured out why.
\nMethodology : I'm using a light dataset made of only complete series, that I split for validation (no cross validation, just plain simple train/test split). I train my model on the training set, test accuracy on the validation set, then do some post-processing (smoothing). When I make a submission, my CV score ranges from 0.6 to 0.75, but my LB score has never been higher than 0.3 (my last try had 0.68 CV, but only 0.09 LB).
\nDoes anyone think of a reason why this could happen ?

", "rawMarkdown": "Since the beginning of the competition, I've tried several models, and I always have a huge difference between CV and LB score. I still haven't figured out why. \nMethodology : I'm using a light dataset made of only complete series, that I split for validation (no cross validation, just plain simple train/test split). I train my model on the training set, test accuracy on the validation set, then do some post-processing (smoothing). When I make a submission, my CV score ranges from 0.6 to 0.75, but my LB score has never been higher than 0.3 (my last try had 0.68 CV, but only 0.09 LB).\nDoes anyone think of a reason why this could happen ?", "votes": 1 }, { "id": 2527447, "postDate": "2023-11-16T14:29:39.780Z", "content": "

Can you give an example of cross validation code for your model. Very blogadarin in advance

", "rawMarkdown": "Can you give an example of cross validation code for your model. Very blogadarin in advance", "votes": 1 }, { "id": 2486979, "postDate": "2023-10-18T09:18:45.853Z", "content": "

The CV is lower then LB by 0.1 with lightGBM, and the CV is higher than LB by 0.4 for RNN.

", "rawMarkdown": "The CV is lower then LB by 0.1 with lightGBM, and the CV is higher than LB by 0.4 for RNN.", "votes": 1 }, { "id": 2486322, "postDate": "2023-10-17T20:27:28.477Z", "content": "

I think many have not yet got CV working, so until someone posts a notebook that gives people a starting point on computing CV I don't know if this will get answered sadly

", "rawMarkdown": "I think many have not yet got CV working, so until someone posts a notebook that gives people a starting point on computing CV I don't know if this will get answered sadly", "votes": 1, "replies": [ { "id": 2489337, "postDate": "2023-10-19T23:23:59.103Z", "content": "

I think the issue is the size of the data, so doing CV seems to take a long time and also may take a lot of memory.

", "rawMarkdown": "I think the issue is the size of the data, so doing CV seems to take a long time and also may take a lot of memory.", "votes": 1, "replies": [ { "id": 2489360, "postDate": "2023-10-19T23:52:44.290Z", "content": "

Yep absolutely. I think the way to do it would be to limit the data, maybe separated by series_id and to get a framework working in kaggle with an unreasonably small dataset just to get it to work. After that I think it could be scaled up outside of kaggle resources for training and moved back to kaggle for inference only. Thats at least first thought though

", "rawMarkdown": "Yep absolutely. I think the way to do it would be to limit the data, maybe separated by series_id and to get a framework working in kaggle with an unreasonably small dataset just to get it to work. After that I think it could be scaled up outside of kaggle resources for training and moved back to kaggle for inference only. Thats at least first thought though", "votes": 1 } ] } ] }, { "id": 2497618, "postDate": "2023-10-24T18:26:00.817Z", "content": "

RNN + CNN model
\n1 fold out of 4 folds (K-Fold)
\nCV: 0.662 LB: 0.623

", "rawMarkdown": "RNN + CNN model\n1 fold out of 4 folds (K-Fold)\nCV: 0.662 LB: 0.623", "votes": 2 }, { "id": 2498445, "postDate": "2023-10-25T10:08:47.273Z", "content": "
\n

For example, it seems that a constant change of the score make a difference for LB, but not locally.

\n
\n

What do you mean? What did you try exactly?

", "rawMarkdown": ">For example, it seems that a constant change of the score make a difference for LB, but not locally. \n\nWhat do you mean? What did you try exactly?", "votes": 1, "replies": [ { "id": 2499318, "postDate": "2023-10-26T01:21:25.117Z", "content": "

Ahmet, great job jumping high on the LB with your first submissions!

", "rawMarkdown": "Ahmet, great job jumping high on the LB with your first submissions!", "votes": 1, "replies": [ { "id": 2499615, "postDate": "2023-10-26T06:37:02.497Z", "content": "

Thanks Chris, happy to see you here!

", "rawMarkdown": "Thanks Chris, happy to see you here!", "votes": 1 } ] }, { "id": 2500365, "postDate": "2023-10-26T15:56:27.307Z", "content": "

With my local metric, when setting the score column to various constants (e.g. setting the score to a unique constant like 1 or 0.5 for all predictions), the metric didn't change. However, when trying the same during a submissions, I actually had somewhat different LB score. I only tested this several weeks ago but didn't investigate further.

", "rawMarkdown": "With my local metric, when setting the score column to various constants (e.g. setting the score to a unique constant like 1 or 0.5 for all predictions), the metric didn't change. However, when trying the same during a submissions, I actually had somewhat different LB score. I only tested this several weeks ago but didn't investigate further.", "votes": 1, "replies": [ { "id": 2500747, "postDate": "2023-10-27T01:49:43.933Z", "content": "

It's like AUC. Only rank matters.

", "rawMarkdown": "It's like AUC. Only rank matters.", "votes": 1 }, { "id": 2501496, "postDate": "2023-10-27T13:48:44.427Z", "content": "

Yes that's what I thought, but it seems when I do the same thing for a submission, it does matter what constant I use. I am not sure why though

", "rawMarkdown": "Yes that's what I thought, but it seems when I do the same thing for a submission, it does matter what constant I use. I am not sure why though", "votes": 2 }, { "id": 2502429, "postDate": "2023-10-28T07:27:57.540Z", "content": "

Thanks for sharing. How much does it matter?

", "rawMarkdown": "Thanks for sharing. How much does it matter?" }, { "id": 2519724, "postDate": "2023-11-10T09:29:32.703Z", "content": "

Not a lot. Maybe it is just because the competition host uses some kind of metric that sorts unstable.

", "rawMarkdown": "Not a lot. Maybe it is just because the competition host uses some kind of metric that sorts unstable." } ] } ] } ], "comments": [ { "id": 2491744, "author_name": "Chris Deotte", "author_url": "", "post_date": "2023-10-22T01:58:38.567000", "content": "

GPU Deep Learning (RNN, CNN, Transformer) with preprocess using GPU RAPIDS cuDF!
\n1 fold out-of-fold. Here are my first few submissions:
\nCV 0.725 LB 0.691
\nCV 0.750 LB 0.714
\nCV 0.770 LB 0.725

", "votes": 32, "replies": [ { "id": 2491750, "author_name": "Cody_Null", "author_url": "", "post_date": "2023-10-22T02:11:00.433000", "content": "

Wow great work as always Chris! I am excited to see how this works when it comes time to share!

", "votes": 3, "replies": [] }, { "id": 2492401, "author_name": "Fritz Cremer", "author_url": "", "post_date": "2023-10-22T13:35:53.490000", "content": "

Wow I'm really impressed with how fast you made progress in this competition! I am not sure though, what do you mean by 1 fold, how large is that fold?

", "votes": 1, "replies": [ { "id": 2492706, "author_name": "Chris Deotte", "author_url": "", "post_date": "2023-10-22T19:39:43.210000", "content": "

Thanks Fritz!

\n

The \"CV score\" is computed using only 1 out of 10 folds from Group 10-fold on series_id. (So it isn't the full 10 folds oof CV score). Then after finding the model parameters, I retrain one model using 100% data with the best hyparameters and submit to LB.

", "votes": 15, "replies": [] }, { "id": 2492816, "author_name": "Fritz Cremer", "author_url": "", "post_date": "2023-10-22T23:11:30.610000", "content": "

Ah I see, thanks for clarification!

", "votes": 1, "replies": [] }, { "id": 2532429, "author_name": "Brainiac", "author_url": "", "post_date": "2023-11-21T04:21:03.330000", "content": "

@cdeotte thanks for sharing, when retraining using 100% data, how do you know your loss isnt improving and you are not overfitting as you have no validation data

", "votes": 0, "replies": [] }, { "id": 2542758, "author_name": "Rohit Chaudhari", "author_url": "", "post_date": "2023-11-29T13:41:20.830000", "content": "

Hi,
\nI just came across the comment, so thought to add some ideas.

\n

we do CV to find best parameters of the model , if some parameters work better than other in each CV fold , then we can say that those parameters are generally good and will give best results with different datasets.

\n

So we get best parameters from CV and then train model again on whole dataset so that we capture all samples of data.

\n

for example. For tree model CV may give us max_depth = 3 , so how the splits will happen will depend on data but on average limiting tree growth to depth of 3 gives best results.
\nSo we are leveraging best parameter guess on average which we obtain from the CV and also utilize the whole dataset so that we capture all samples of data, in this case how the split will happen will be decided based on whole dataset.

\n

This is basic idea but based on business usecase you can set retrain = False in sklearn if you do not want model to retrain on whole data for some reason.

", "votes": 0, "replies": [] } ] }, { "id": 2504011, "author_name": "Shiner", "author_url": "", "post_date": "2023-10-29T15:33:16.173000", "content": "

Great CV-LB. I'm trying transformer as well, but I find it's hard to train it, OOM always happens! May I ask how do you solve this problem? @cdeotte

", "votes": 1, "replies": [ { "id": 2507348, "author_name": "Chris Deotte", "author_url": "", "post_date": "2023-10-31T23:23:07.953000", "content": "

@judith007 The best way to diagnose and solve OOM errors is to run your inference notebook interactively in Kaggle notebook (and infer all 277 train series_id) and watch the \"View Session Metrics\" dropdown window. Monitor the memory usage and if any code cell is using too much memory, then find a way to delete previous variables and/or run the current cell with less memory.

\n

Most OOM are GPU VRAM, so you can infer with less batch_size and/or smaller chunks of data at one time. For example write a for-loop that infers small chunks one at a time and save the predictions versus trying to infer all data at once.

", "votes": 5, "replies": [] } ] } ] }, { "id": 2512801, "author_name": "Takoi", "author_url": "", "post_date": "2023-11-04T20:47:13.833000", "content": "

single model
\n5 Folds
\nCV 0.800
\nLB 0.759

", "votes": 18, "replies": [] }, { "id": 2511516, "author_name": "K_mat", "author_url": "", "post_date": "2023-11-03T17:54:33.123000", "content": "

5 Folds
\nCV(out of fold) 0.811
\nLB(ensemble) 0.770

", "votes": 16, "replies": [ { "id": 2519261, "author_name": "Anand Tirkey", "author_url": "", "post_date": "2023-11-09T20:50:56.127000", "content": "

U-Net or Transformer?

", "votes": 0, "replies": [] } ] }, { "id": 2500307, "author_name": "tereka", "author_url": "", "post_date": "2023-10-26T15:20:48.280000", "content": "

NN 1Fold only(split 5Folds)
\nLocal CV: 0.756 LB: 0.689

", "votes": 14, "replies": [] }, { "id": 2529652, "author_name": "penguin46", "author_url": "", "post_date": "2023-11-18T12:43:03.067000", "content": "

5 folds, GroupKFold
\ncv: 0.822, lb: 0.770
\ncv: 0.818, lb: 0.773
\ncv: 0.814, lb: 0.749
\ncv: 0.794, lb: 0.739
\ncv: 0.785, lb: 0.754

", "votes": 11, "replies": [] }, { "id": 2517592, "author_name": "NikhilMishra", "author_url": "", "post_date": "2023-11-08T15:51:37.460000", "content": "

Transformer GroupKFold (By series id)

\n

5 fold - cv: 0.773 lb-0.748

\n

Update 5 fold: cv- 0.777-lb 0.754

\n

Update 5 fold cv: 0.791 - lb: 0.768

", "votes": 11, "replies": [] }, { "id": 2515422, "author_name": "shu421", "author_url": "", "post_date": "2023-11-06T23:48:03.503000", "content": "

single model

\n

4 Folds

\n

CV: 0.7454 ± 0.0199

\n

LB: 0.729

", "votes": 10, "replies": [] }, { "id": 2493345, "author_name": "kuto", "author_url": "", "post_date": "2023-10-23T10:50:26.127000", "content": "

LSTM model
\n5-fold GroupKFold(by series_id)

\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n
CV scoreLB scorediff
0.6410.6110.030
0.6540.6290.025
0.6670.6450.022
", "votes": 9, "replies": [] }, { "id": 2491782, "author_name": "嘴爷", "author_url": "", "post_date": "2023-10-22T03:36:21.907000", "content": "

1 fold xgb
\nCV 654 LB 636
\nBut can't improve anymore, it seems i have to turn to nn models.

", "votes": 10, "replies": [ { "id": 2493410, "author_name": "Chris Deotte", "author_url": "", "post_date": "2023-10-23T11:47:19.013000", "content": "

Thanks a great XGB model !!

", "votes": 0, "replies": [] } ] }, { "id": 2512401, "author_name": "Taro_pan", "author_url": "", "post_date": "2023-11-04T13:46:17.430000", "content": "

NN 5 folds
\noof CV 0.746
\nLB 0.671

", "votes": 8, "replies": [] }, { "id": 2490987, "author_name": "ADAM.", "author_url": "", "post_date": "2023-10-21T08:40:35.537000", "content": "

lightgbm

\n

1fold out-of-fold

\n

cv 0.422 lb 0.429
\ncv 0.533 lb .0.518
\ncv 0.635 lb 0.604
\ncv 0.697 lb 0.658

", "votes": 8, "replies": [ { "id": 2491740, "author_name": "Chris Deotte", "author_url": "", "post_date": "2023-10-22T01:56:47.197000", "content": "

Great score with LGBM Adam!

", "votes": 1, "replies": [ { "id": 2491759, "author_name": "ADAM.", "author_url": "", "post_date": "2023-10-22T02:44:06.103000", "content": "

Thanks. 😀

", "votes": 1, "replies": [] } ] } ] }, { "id": 2532778, "author_name": "ohkawa3", "author_url": "", "post_date": "2023-11-21T10:21:41.197000", "content": "

GroupKFold: k=5, group=series_id
\nCV:0.518
\nLB:0.749 (ensemble)

\n

My CV score is significantly lower than LB, but looking at other people's posts, it seems like CV is higher.
\nI am calculating the CV score using all 277 series.
\nAre you calculating CV scores for all series?
\nOr are you screening clean series with fewer than 277?

", "votes": 5, "replies": [ { "id": 2533079, "author_name": "penguin46", "author_url": "", "post_date": "2023-11-21T15:17:24.483000", "content": "

I'm using all 277 series to compute CV.

", "votes": 3, "replies": [] }, { "id": 2533293, "author_name": "denis", "author_url": "", "post_date": "2023-11-21T18:31:02.740000", "content": "

Did you drop nulls in train_events when computing cv, that might be a reason.

", "votes": 7, "replies": [ { "id": 2533405, "author_name": "ohkawa3", "author_url": "", "post_date": "2023-11-21T22:11:21.487000", "content": "

@ryotayoshinobu @dlukic

\n

thanks a lot!
\nWhen I recalculated the CV, it became 0.785.

", "votes": 3, "replies": [] }, { "id": 2533408, "author_name": "", "author_url": "", "post_date": "2023-11-21T22:17:21.103000", "content": "", "votes": 0, "replies": [] }, { "id": 2533411, "author_name": "ohkawa3", "author_url": "", "post_date": "2023-11-21T22:22:52.933000", "content": "

Yes, the scores of the five models are averaged before post-processing.

", "votes": 0, "replies": [] }, { "id": 2534322, "author_name": "Lamal", "author_url": "", "post_date": "2023-11-22T14:43:02.593000", "content": "

great idea

", "votes": 0, "replies": [] } ] } ] }, { "id": 2486471, "author_name": "Geremie Yeo", "author_url": "", "post_date": "2023-10-18T00:40:21.740000", "content": "

What is your CV strategy (e.g 5 fold GroupKFold) and which dataset are you using for CV (if you do not mind sharing) ?

\n

For instance, I'm using 5-fold StratifiedGroupKFold on Zzzs_train_multi.parquet and this LB = 0.536 notebook has CV = 0.468

", "votes": 6, "replies": [ { "id": 2486994, "author_name": "Fritz Cremer", "author_url": "", "post_date": "2023-10-18T09:40:39.203000", "content": "

I am using 4-fold GroupKFold. I filter some samples out for training but validate on the entire dataset.

", "votes": 2, "replies": [] } ] }, { "id": 2486938, "author_name": "Vadim Timakin", "author_url": "", "post_date": "2023-10-18T08:51:00.593000", "content": "

Same here
\nCV 0.553 - 0.487
\nCV 0.561 - LB 0.501
\nCV 0.571 - LB 0.516

", "votes": 3, "replies": [] }, { "id": 2491777, "author_name": "sayoulala", "author_url": "", "post_date": "2023-10-22T03:26:15.857000", "content": "

NN:1fold out-of-fold cv 599 lb 632

", "votes": 4, "replies": [] }, { "id": 2545039, "author_name": "Matthias J", "author_url": "", "post_date": "2023-12-01T08:53:34.687000", "content": "

Since the beginning of the competition, I've tried several models, and I always have a huge difference between CV and LB score. I still haven't figured out why.
\nMethodology : I'm using a light dataset made of only complete series, that I split for validation (no cross validation, just plain simple train/test split). I train my model on the training set, test accuracy on the validation set, then do some post-processing (smoothing). When I make a submission, my CV score ranges from 0.6 to 0.75, but my LB score has never been higher than 0.3 (my last try had 0.68 CV, but only 0.09 LB).
\nDoes anyone think of a reason why this could happen ?

", "votes": 1, "replies": [] }, { "id": 2527447, "author_name": "Mark", "author_url": "", "post_date": "2023-11-16T14:29:39.780000", "content": "

Can you give an example of cross validation code for your model. Very blogadarin in advance

", "votes": 1, "replies": [] }, { "id": 2486979, "author_name": "Yuchen20", "author_url": "", "post_date": "2023-10-18T09:18:45.853000", "content": "

The CV is lower then LB by 0.1 with lightGBM, and the CV is higher than LB by 0.4 for RNN.

", "votes": 1, "replies": [] }, { "id": 2486322, "author_name": "Cody_Null", "author_url": "", "post_date": "2023-10-17T20:27:28.477000", "content": "

I think many have not yet got CV working, so until someone posts a notebook that gives people a starting point on computing CV I don't know if this will get answered sadly

", "votes": 1, "replies": [ { "id": 2489337, "author_name": "Marcus Chan", "author_url": "", "post_date": "2023-10-19T23:23:59.103000", "content": "

I think the issue is the size of the data, so doing CV seems to take a long time and also may take a lot of memory.

", "votes": 1, "replies": [ { "id": 2489360, "author_name": "Cody_Null", "author_url": "", "post_date": "2023-10-19T23:52:44.290000", "content": "

Yep absolutely. I think the way to do it would be to limit the data, maybe separated by series_id and to get a framework working in kaggle with an unreasonably small dataset just to get it to work. After that I think it could be scaled up outside of kaggle resources for training and moved back to kaggle for inference only. Thats at least first thought though

", "votes": 1, "replies": [] } ] } ] }, { "id": 2497618, "author_name": "Nghi Huynh", "author_url": "", "post_date": "2023-10-24T18:26:00.817000", "content": "

RNN + CNN model
\n1 fold out of 4 folds (K-Fold)
\nCV: 0.662 LB: 0.623

", "votes": 2, "replies": [] }, { "id": 2498445, "author_name": "Ahmet Erdem", "author_url": "", "post_date": "2023-10-25T10:08:47.273000", "content": "
\n

For example, it seems that a constant change of the score make a difference for LB, but not locally.

\n
\n

What do you mean? What did you try exactly?

", "votes": 1, "replies": [ { "id": 2499318, "author_name": "Chris Deotte", "author_url": "", "post_date": "2023-10-26T01:21:25.117000", "content": "

Ahmet, great job jumping high on the LB with your first submissions!

", "votes": 1, "replies": [ { "id": 2499615, "author_name": "Ahmet Erdem", "author_url": "", "post_date": "2023-10-26T06:37:02.497000", "content": "

Thanks Chris, happy to see you here!

", "votes": 1, "replies": [] } ] }, { "id": 2500365, "author_name": "Fritz Cremer", "author_url": "", "post_date": "2023-10-26T15:56:27.307000", "content": "

With my local metric, when setting the score column to various constants (e.g. setting the score to a unique constant like 1 or 0.5 for all predictions), the metric didn't change. However, when trying the same during a submissions, I actually had somewhat different LB score. I only tested this several weeks ago but didn't investigate further.

", "votes": 1, "replies": [ { "id": 2500747, "author_name": "ADAM.", "author_url": "", "post_date": "2023-10-27T01:49:43.933000", "content": "

It's like AUC. Only rank matters.

", "votes": 1, "replies": [] }, { "id": 2501496, "author_name": "Fritz Cremer", "author_url": "", "post_date": "2023-10-27T13:48:44.427000", "content": "

Yes that's what I thought, but it seems when I do the same thing for a submission, it does matter what constant I use. I am not sure why though

", "votes": 2, "replies": [] }, { "id": 2502429, "author_name": "Ahmet Erdem", "author_url": "", "post_date": "2023-10-28T07:27:57.540000", "content": "

Thanks for sharing. How much does it matter?

", "votes": 0, "replies": [] }, { "id": 2519724, "author_name": "Fritz Cremer", "author_url": "", "post_date": "2023-11-10T09:29:32.703000", "content": "

Not a lot. Maybe it is just because the competition host uses some kind of metric that sorts unstable.

", "votes": 0, "replies": [] } ] } ] } ], "raw_markdown_by_id": { "2486021": "I consistently get lower LB scores than local CV scores. My cv scores are above 0.7 while LB sits at 0.66. I am not sure whether this is because of the data distribution or differences between the metric function the host provided and the one that is actually used. For example, it seems that a constant change of the score make a difference for LB, but not locally. How does your CV score compare to LB score?", "2491744": "GPU Deep Learning (RNN, CNN, Transformer) with preprocess using GPU RAPIDS cuDF!\n1 fold out-of-fold. Here are my first few submissions:\nCV 0.725 LB 0.691\nCV 0.750 LB 0.714\nCV 0.770 LB 0.725", "2512801": "single model\n5 Folds\nCV 0.800\nLB 0.759", "2511516": "5 Folds\nCV(out of fold) 0.811\nLB(ensemble) 0.770", "2500307": "NN 1Fold only(split 5Folds)\nLocal CV: 0.756 LB: 0.689", "2529652": "5 folds, GroupKFold\ncv: 0.822, lb: 0.770\ncv: 0.818, lb: 0.773\ncv: 0.814, lb: 0.749\ncv: 0.794, lb: 0.739\ncv: 0.785, lb: 0.754", "2517592": "Transformer GroupKFold (By series id)\n\n\n5 fold - cv: 0.773 lb-0.748\n\nUpdate 5 fold: cv- 0.777-lb 0.754\n\nUpdate 5 fold cv: 0.791 - lb: 0.768", "2515422": "single model\n\n4 Folds\n\nCV: 0.7454 ± 0.0199\n\nLB: 0.729", "2493345": "LSTM model\n5-fold GroupKFold(by series_id)\n| CV score | LB score | diff |\n| --- | --- | --- |\n| 0.641 | 0.611 | 0.030 |\n| 0.654 | 0.629 | 0.025 |\n| 0.667 | 0.645 | 0.022 |\n\n\n\n", "2491782": "1 fold xgb\nCV 654 LB 636\nBut can't improve anymore, it seems i have to turn to nn models.", "2512401": "NN 5 folds\noof CV 0.746\nLB 0.671", "2490987": "lightgbm\n\n1fold out-of-fold\n\ncv 0.422 lb 0.429\ncv 0.533 lb .0.518\ncv 0.635 lb 0.604\ncv 0.697 lb 0.658", "2532778": "GroupKFold: k=5, group=series_id\nCV:0.518\nLB:0.749 (ensemble)\n\nMy CV score is significantly lower than LB, but looking at other people's posts, it seems like CV is higher.\nI am calculating the CV score using all 277 series.\nAre you calculating CV scores for all series?\nOr are you screening clean series with fewer than 277?\n", "2486471": "What is your CV strategy (e.g 5 fold GroupKFold) and which dataset are you using for CV (if you do not mind sharing) ?\n\nFor instance, I'm using 5-fold StratifiedGroupKFold on Zzzs_train_multi.parquet and this LB = 0.536 [notebook](https://www.kaggle.com/code/yeoyunsianggeremie/fe-ensemble-added-period-3600) has CV = 0.468", "2486938": "Same here\nCV 0.553 - 0.487\nCV 0.561 - LB 0.501\nCV 0.571 - LB 0.516", "2491777": "NN:1fold out-of-fold cv 599 lb 632", "2545039": "Since the beginning of the competition, I've tried several models, and I always have a huge difference between CV and LB score. I still haven't figured out why. \nMethodology : I'm using a light dataset made of only complete series, that I split for validation (no cross validation, just plain simple train/test split). I train my model on the training set, test accuracy on the validation set, then do some post-processing (smoothing). When I make a submission, my CV score ranges from 0.6 to 0.75, but my LB score has never been higher than 0.3 (my last try had 0.68 CV, but only 0.09 LB).\nDoes anyone think of a reason why this could happen ?", "2527447": "Can you give an example of cross validation code for your model. Very blogadarin in advance", "2486979": "The CV is lower then LB by 0.1 with lightGBM, and the CV is higher than LB by 0.4 for RNN.", "2486322": "I think many have not yet got CV working, so until someone posts a notebook that gives people a starting point on computing CV I don't know if this will get answered sadly", "2497618": "RNN + CNN model\n1 fold out of 4 folds (K-Fold)\nCV: 0.662 LB: 0.623", "2498445": ">For example, it seems that a constant change of the score make a difference for LB, but not locally. \n\nWhat do you mean? What did you try exactly?" } }