![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F7279405%2Fc250e84961e3c29af8c8b4bf1e709fda%2FScreenshot%202025-08-10%20193847.png?generation=1754844635197396&alt=media\")
I will address an issue some of the top public kernel have, nothing new for top teams here, sorry ! 🙂
\nIf you’ve trained a few models, you’ve probably noticed:
\nThis is expected. The F1 metric is inherently unstable, the dataset is small, and the task is not trivial.
\nThis also means we need to be very careful when optimizing models !
\nThe goal of k-fold CV is to evaluate your model across the entire dataset in a consistent way by training k models with the same hyperparameters. This leads to a more robust selection process.
\nFor instance, the k models have to be trained with :
\nWhat you should not do:
\nUse 100 epochs for fold 0 and 60 for fold 1 just because those are the epochs where each fold individually peaked.
When doing so, you are reducing robustness of you validation setup, and artificially boosting your CV score. Instead, if 80 epochs works reasonably well across folds, stick with it.
\nThe number of epochs that maximizes the average CV score is a better estimator of (private) test performance than the per-fold optimal values.
\nKeep this in mind when considering early-stopping at epoch 60:
\n\n\nWhat if your fold 0 validation score stopped increasing at 60 epoch, but your test score did not ?
\n
(Only in the case of unstable CV scores, do whatever you want with your big tabular datasets)
\nWhen using checkpointing, you are using a different number of epochs for each fold.
\nAND you are choosing the checkpoint that maximizes the fold score.
\nIt is the best way to have an overestimated CV and to lose robustness.
Instead, train all folds for the same number of epochs, and always use the final checkpoint(s).
\nSame issue: you end up with a different number of epochs per fold, guided by a weak estimator—individual fold validation scores.
\nThis strategy relies on per-fold validation scores to adapt learning rate schedules, once again breaking consistency.
\nInstead, I recommend using a scheduler that progressively decreases your learning rate. E.g. Linear or Cosine :
\n torch.optim.lr_scheduler import CosineAnnealingLR, LinearLR\ncosine = CosineAnnealingLR(optimizer, num_epochs, =1e-6)\nlinear = LinearLR(optimizer, =1.0, =0.0001, =num_epochs)\nThis will also bring more stability to your training, removing the need for 1. and 2.
\n--
\nThanks for reading and happy parameter optimization !
\nAnd remember, trust your CV !
\n(you can, now that you've removed fold-score-dependent components)
I will address an issue some of the top public kernel have, nothing new for top teams here, sorry ! 🙂
\nIf you’ve trained a few models, you’ve probably noticed:
\nThis is expected. The F1 metric is inherently unstable, the dataset is small, and the task is not trivial.
\nThis also means we need to be very careful when optimizing models !
\nThe goal of k-fold CV is to evaluate your model across the entire dataset in a consistent way by training k models with the same hyperparameters. This leads to a more robust selection process.
\nFor instance, the k models have to be trained with :
\nWhat you should not do:
\nUse 100 epochs for fold 0 and 60 for fold 1 just because those are the epochs where each fold individually peaked.
When doing so, you are reducing robustness of you validation setup, and artificially boosting your CV score. Instead, if 80 epochs works reasonably well across folds, stick with it.
\nThe number of epochs that maximizes the average CV score is a better estimator of (private) test performance than the per-fold optimal values.
\nKeep this in mind when considering early-stopping at epoch 60:
\n\n\nWhat if your fold 0 validation score stopped increasing at 60 epoch, but your test score did not ?
\n
(Only in the case of unstable CV scores, do whatever you want with your big tabular datasets)
\nWhen using checkpointing, you are using a different number of epochs for each fold.
\nAND you are choosing the checkpoint that maximizes the fold score.
\nIt is the best way to have an overestimated CV and to lose robustness.
Instead, train all folds for the same number of epochs, and always use the final checkpoint(s).
\nSame issue: you end up with a different number of epochs per fold, guided by a weak estimator—individual fold validation scores.
\nThis strategy relies on per-fold validation scores to adapt learning rate schedules, once again breaking consistency.
\nInstead, I recommend using a scheduler that progressively decreases your learning rate. E.g. Linear or Cosine :
\n torch.optim.lr_scheduler import CosineAnnealingLR, LinearLR\ncosine = CosineAnnealingLR(optimizer, num_epochs, =1e-6)\nlinear = LinearLR(optimizer, =1.0, =0.0001, =num_epochs)\nThis will also bring more stability to your training, removing the need for 1. and 2.
\n--
\nThanks for reading and happy parameter optimization !
\nAnd remember, trust your CV !
\n(you can, now that you've removed fold-score-dependent components)
All of them are enabled, I'm an aware sinner. 😌
", "rawMarkdown": "All of them are enabled, I'm an aware sinner. 😌", "votes": 9 }, { "id": 3265188, "postDate": "2025-08-07T06:41:57.723Z", "content": "One competition-specific note: Since around 50% of the private test set will have missing thermo and tof values (a pattern not present in the training data); if your models rely on these features, you should try to simulate this missingness during validation…
", "rawMarkdown": "One competition-specific note: Since around 50% of the private test set will have missing thermo and tof values (a pattern not present in the training data); if your models rely on these features, you should try to simulate this missingness during validation...", "votes": 5, "replies": [ { "id": 3265276, "postDate": "2025-08-07T09:01:42.143Z", "content": "100% True
", "rawMarkdown": "100% True\n\n", "votes": 1 } ] }, { "id": 3263638, "postDate": "2025-08-05T13:45:37.410Z", "content": "Thanks for the tips! One question, how would you recommend comparing different hyperparameters/FE if training the full k-fold takes too long? I'm currently tuning on 1 fold but I feel like that would lead to overfitting on that fold?
", "rawMarkdown": "Thanks for the tips! One question, how would you recommend comparing different hyperparameters/FE if training the full k-fold takes too long? I'm currently tuning on 1 fold but I feel like that would lead to overfitting on that fold?", "votes": 1, "replies": [ { "id": 3263661, "postDate": "2025-08-05T14:19:22.893Z", "content": "You can experiment on 1 or 2 folds and once you're satisfied with what you have, ensure the result translates to a k-fold improvement. That's what I usually do. Single fold can already be enough to reject a lot of hypothesis.
", "rawMarkdown": "You can experiment on 1 or 2 folds and once you're satisfied with what you have, ensure the result translates to a k-fold improvement. That's what I usually do. Single fold can already be enough to reject a lot of hypothesis.", "votes": 1, "replies": [ { "id": 3263940, "postDate": "2025-08-05T22:57:31.287Z", "content": "\n\nSingle fold can already be enough to reject a lot of hypothesis.
\n
It will work, to a degree.
\nChance to beat people that perform full k-fold --> more robust experiments?
\nLow.
\nUnless you are very good and/or have great luck.
One more question: when tuning the model/FE, should I keep the same number of epochs for each experiment? (each experiment here is 1 fold only)
", "rawMarkdown": "One more question: when tuning the model/FE, should I keep the same number of epochs for each experiment? (each experiment here is 1 fold only)" }, { "id": 3264343, "postDate": "2025-08-06T14:00:12.610Z", "content": "If you change the model architecture by a lot you might need to adapt the number of epochs. Otherwise I like to keep it fixed.
", "rawMarkdown": "If you change the model architecture by a lot you might need to adapt the number of epochs. Otherwise I like to keep it fixed." } ] } ] }, { "id": 3263545, "postDate": "2025-08-05T11:10:13.797Z", "content": "Thanks for sharing. I have a few questions, though. You suggest avoiding early stopping, but many textbooks recommend it. So, when do you think early stopping is actually useful? You also said, \"If 80 epochs works reasonably well across folds, stick with it.\" But how do we know that 80 epochs is enough? Are we still using the validation set to decide on that number? Also, you recommend using the same number of epochs for all folds, but different folds might not converge at the same time. Isn’t that a problem?
", "rawMarkdown": "Thanks for sharing. I have a few questions, though. You suggest avoiding early stopping, but many textbooks recommend it. So, when do you think early stopping is actually useful? You also said, \"If 80 epochs works reasonably well across folds, stick with it.\" But how do we know that 80 epochs is enough? Are we still using the validation set to decide on that number? Also, you recommend using the same number of epochs for all folds, but different folds might not converge at the same time. Isn’t that a problem?", "votes": 1, "replies": [ { "id": 3263556, "postDate": "2025-08-05T11:20:59.497Z", "content": "\n\nBut how do we know that 80 epochs is enough
\n
Train your k folds with 60, 80, 100 epochs - compute your CV score and use whichever scores best.
\n\n\nDifferent folds might not converge at the same time. Isn’t that a problem?
\n
If your training strategy is good (good lr scheduling, good augmentations), folds should converge at the same time. Keep in mind that single-fold validation f1-score is often not a good indicator of convergence.
", "rawMarkdown": "> But how do we know that 80 epochs is enough\n\nTrain your *k* folds with 60, 80, 100 epochs - compute your CV score and use whichever scores best.\n\n> Different folds might not converge at the same time. Isn’t that a problem?\n\nIf your training strategy is good (good lr scheduling, good augmentations), folds should converge at the same time. Keep in mind that single-fold validation f1-score is often not a good indicator of convergence." } ] }, { "id": 3263824, "postDate": "2025-08-05T18:54:29.277Z", "content": "Great insights. Sometimes Kagglers ignore these because it may perform better on kfold bagging on submission. Ensemble of overfit models can perform better (same idea as random forest). So in most cases Kagglers (including me) don't exactly validate their test case (single models on validation vs kfold bagging on test)
", "rawMarkdown": "Great insights. Sometimes Kagglers ignore these because it may perform better on kfold bagging on submission. Ensemble of overfit models can perform better (same idea as random forest). So in most cases Kagglers (including me) don't exactly validate their test case (single models on validation vs kfold bagging on test)", "votes": 2 }, { "id": 3267190, "postDate": "2025-08-10T16:54:11.283Z", "content": "Theo, thanks for the interesting advice! I have a question then, can these graphs help us in this case or not?
\n
Yes, you can use them to build an understanding on how your models behave. E.g. regarding overfitting, instability. They're useful to tweak epochs and lr
", "rawMarkdown": "Yes, you can use them to build an understanding on how your models behave. E.g. regarding overfitting, instability. They're useful to tweak epochs and lr", "votes": 1 } ] }, { "id": 3264561, "postDate": "2025-08-06T18:06:15.150Z", "content": "\n\n\n
\n- Early stopping
\nSame issue: you end up with a different number of epochs per fold, guided by a weak estimator—individual fold validation scores.
\n
Sometimes, it could be helpful for faster training if you have ideas you want to iterate fast.
", "rawMarkdown": "> 2. Early stopping\n\n> Same issue: you end up with a different number of epochs per fold, guided by a weak estimator—individual fold validation scores.\n\nSometimes, it could be helpful for faster training if you have ideas you want to iterate fast. ", "replies": [ { "id": 3265277, "postDate": "2025-08-07T09:02:42.413Z", "content": "If your validation scores are stable yes :)
", "rawMarkdown": "If your validation scores are stable yes :)", "replies": [ { "id": 3265867, "postDate": "2025-08-08T03:46:11.337Z", "content": "For this competition i didn't do anything fancy for validation scheme.
\nI did ablate using early stopping (apprx 5 subs), and i observe small correlation between improvement in cv and public lb.
\nI think the nature of small dataset in this competition generally has more noise than that of from usage of early-stopping.
\nAlso, submission to public is average prediction for 5 folds, so i think it helps to reduce \"overfitting\" from early stopping.
", "rawMarkdown": "For this competition i didn't do anything fancy for validation scheme.\n\nI did ablate using early stopping (apprx 5 subs), and i observe small correlation between improvement in cv and public lb.\n\nI think the nature of small dataset in this competition generally has more noise than that of from usage of early-stopping. \n\nAlso, submission to public is average prediction for 5 folds, so i think it helps to reduce \"overfitting\" from early stopping. " } ] } ] }, { "id": 3263663, "postDate": "2025-08-05T14:23:42.270Z", "content": "![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F15039814%2F9f684a6622a8ac1a48b6441cead2ef91%2FEkran%20Resmi%202025-08-05%2017.11.11.png?generation=1754403114078820&alt=media\")
Hi, Thanks for the tips on robust validation. It's a very important point for me as a beginner.
\nFollowing your advice, I've been training with a consistent setup. Specifically, I'm using StratifiedGroupKFold, and the model architecture and all hyperparameters are identical for each fold. However, as you can see in the attached graph, one fold (fold_5) is a clear underperformer compared to the rest.
\nGiven your post, would you still consider it a bad practice to either:
Also, is it generally considered normal for one fold to perform worse than the others like this?
", "rawMarkdown": "\n\n\nHi, Thanks for the tips on robust validation. It's a very important point for me as a beginner.\nFollowing your advice, I've been training with a consistent setup. Specifically, I'm using StratifiedGroupKFold, and the model architecture and all hyperparameters are identical for each fold. However, as you can see in the attached graph, one fold (fold_5) is a clear underperformer compared to the rest.\nGiven your post, would you still consider it a bad practice to either:\n1. Use the best checkpoint for this specific fold, even if it breaks the \"fixed epoch\" rule?\n2. Exclude this poorly performing fold from the final ensemble?\n\nAlso, is it generally considered normal for one fold to perform worse than the others like this?", "replies": [ { "id": 3263803, "postDate": "2025-08-05T18:18:41.290Z", "content": "\n\nGiven your post, would you still consider it a bad practice to either:
\n
\n Use the best checkpoint for this specific fold, even if it breaks the \"fixed epoch\" rule?
Yes
\n\n\nExclude this poorly performing fold from the final ensemble?
\n
If you think the checkpoint will do poorly on test data, you can exclude it.
\n--
\nOverall, you should try to figure out why fold 5 performs poorly, 0.55 looks low, you can probably improve that.
Very interesting post/discussion, did you personally noticed a increase in your LB score by using the epoch with the best CV mean instead of the best epoch with the peak score for each different fold?
", "rawMarkdown": "Very interesting post/discussion, did you personally noticed a increase in your LB score by using the epoch with the best CV mean instead of the best epoch with the peak score for each different fold?", "replies": [ { "id": 3263557, "postDate": "2025-08-05T11:22:19.377Z", "content": "Not really since LB is a bit unstable - but it pays in the long run since you select your final models using a more reliable score
", "rawMarkdown": "Not really since LB is a bit unstable - but it pays in the long run since you select your final models using a more reliable score", "votes": 1, "replies": [ { "id": 3267859, "postDate": "2025-08-12T03:28:27.700Z", "content": "My LB score decreased 0.005, CV decreased 0.002 after doing your suggestions. Do you think this is normal and is just because of the unstable LB? I just really don’t know what to trust anymore…
", "rawMarkdown": "My LB score decreased 0.005, CV decreased 0.002 after doing your suggestions. Do you think this is normal and is just because of the unstable LB? I just really don’t know what to trust anymore…" }, { "id": 3267936, "postDate": "2025-08-12T06:40:16.957Z", "content": "\n\nDo you think this is normal and is just because of the unstable LB?
\n
Could be yes, I've seen significant CV improvements lead to LB drops.
\nIt could also be that the early-stopping approach blends better when averaging k-folds.
", "rawMarkdown": "> Do you think this is normal and is just because of the unstable LB?\n\nCould be yes, I've seen significant CV improvements lead to LB drops.\n\nIt could also be that the early-stopping approach blends better when averaging k-folds." }, { "id": 3267946, "postDate": "2025-08-12T06:57:19.793Z", "content": "My worst fold score = 0.87, my best fold score = 0.91. And I was stuck for a long time, unable to improve CV and LB. Maybe ensembling a number of models will boost the LB significantly. Gradient boosting might be useful 🤤
", "rawMarkdown": "My worst fold score = 0.87, my best fold score = 0.91. And I was stuck for a long time, unable to improve CV and LB. Maybe ensembling a number of models will boost the LB significantly. Gradient boosting might be useful 🤤" }, { "id": 3267954, "postDate": "2025-08-12T07:01:58.423Z", "content": "I have tried comparing the last epoch and the best epoch, and the differences in both CV and LB are not significant, around 0.001-0.002.
", "rawMarkdown": "I have tried comparing the last epoch and the best epoch, and the differences in both CV and LB are not significant, around 0.001-0.002." } ] } ] }, { "id": 3265511, "postDate": "2025-08-07T15:44:34.900Z", "content": "I absolutely agree with your fact regarding F1 score being highly unstable. It works/ is great for further actions if the given dataset is skewed and large dataset. Because, it wont be optimal to perform hyperparameter tuning on large dataset given that constrains of your hardware specifications and at such cases better to stick on to F1 score. But if your dataset is less skewed and small, its always better to hyperparameter tuning and cross-validation method
", "rawMarkdown": "I absolutely agree with your fact regarding F1 score being highly unstable. It works/ is great for further actions if the given dataset is skewed and large dataset. Because, it wont be optimal to perform hyperparameter tuning on large dataset given that constrains of your hardware specifications and at such cases better to stick on to F1 score. But if your dataset is less skewed and small, its always better to hyperparameter tuning and cross-validation method", "isDeleted": true }, { "id": 3268205, "postDate": "2025-08-12T15:14:50.800Z", "content": "Thanks for the tips…!
", "rawMarkdown": "Thanks for the tips...!" } ], "comments": [ { "id": 3263492, "author_name": "Anil Ozturk", "author_url": "", "post_date": "2025-08-05T09:24:50.687000", "content": "All of them are enabled, I'm an aware sinner. 😌
", "votes": 9, "replies": [] }, { "id": 3265188, "author_name": "Ertuğrul Demir", "author_url": "", "post_date": "2025-08-07T06:41:57.723000", "content": "One competition-specific note: Since around 50% of the private test set will have missing thermo and tof values (a pattern not present in the training data); if your models rely on these features, you should try to simulate this missingness during validation…
", "votes": 5, "replies": [ { "id": 3265276, "author_name": "Theo Viel", "author_url": "", "post_date": "2025-08-07T09:01:42.143000", "content": "100% True
", "votes": 1, "replies": [] } ] }, { "id": 3263638, "author_name": "Mahog", "author_url": "", "post_date": "2025-08-05T13:45:37.410000", "content": "Thanks for the tips! One question, how would you recommend comparing different hyperparameters/FE if training the full k-fold takes too long? I'm currently tuning on 1 fold but I feel like that would lead to overfitting on that fold?
", "votes": 1, "replies": [ { "id": 3263661, "author_name": "Theo Viel", "author_url": "", "post_date": "2025-08-05T14:19:22.893000", "content": "You can experiment on 1 or 2 folds and once you're satisfied with what you have, ensure the result translates to a k-fold improvement. That's what I usually do. Single fold can already be enough to reject a lot of hypothesis.
", "votes": 1, "replies": [ { "id": 3263940, "author_name": "greySnow", "author_url": "", "post_date": "2025-08-05T22:57:31.287000", "content": "\n\nSingle fold can already be enough to reject a lot of hypothesis.
\n
It will work, to a degree.
\nChance to beat people that perform full k-fold --> more robust experiments?
\nLow.
\nUnless you are very good and/or have great luck.
One more question: when tuning the model/FE, should I keep the same number of epochs for each experiment? (each experiment here is 1 fold only)
", "votes": 0, "replies": [] }, { "id": 3264343, "author_name": "Theo Viel", "author_url": "", "post_date": "2025-08-06T14:00:12.610000", "content": "If you change the model architecture by a lot you might need to adapt the number of epochs. Otherwise I like to keep it fixed.
", "votes": 0, "replies": [] } ] } ] }, { "id": 3263545, "author_name": "atl132", "author_url": "", "post_date": "2025-08-05T11:10:13.797000", "content": "Thanks for sharing. I have a few questions, though. You suggest avoiding early stopping, but many textbooks recommend it. So, when do you think early stopping is actually useful? You also said, \"If 80 epochs works reasonably well across folds, stick with it.\" But how do we know that 80 epochs is enough? Are we still using the validation set to decide on that number? Also, you recommend using the same number of epochs for all folds, but different folds might not converge at the same time. Isn’t that a problem?
", "votes": 1, "replies": [ { "id": 3263556, "author_name": "Theo Viel", "author_url": "", "post_date": "2025-08-05T11:20:59.497000", "content": "\n\nBut how do we know that 80 epochs is enough
\n
Train your k folds with 60, 80, 100 epochs - compute your CV score and use whichever scores best.
\n\n\nDifferent folds might not converge at the same time. Isn’t that a problem?
\n
If your training strategy is good (good lr scheduling, good augmentations), folds should converge at the same time. Keep in mind that single-fold validation f1-score is often not a good indicator of convergence.
", "votes": 0, "replies": [] } ] }, { "id": 3263824, "author_name": "Ahmet Erdem", "author_url": "", "post_date": "2025-08-05T18:54:29.277000", "content": "Great insights. Sometimes Kagglers ignore these because it may perform better on kfold bagging on submission. Ensemble of overfit models can perform better (same idea as random forest). So in most cases Kagglers (including me) don't exactly validate their test case (single models on validation vs kfold bagging on test)
", "votes": 2, "replies": [] }, { "id": 3267190, "author_name": "AI-Cat", "author_url": "", "post_date": "2025-08-10T16:54:11.283000", "content": "Theo, thanks for the interesting advice! I have a question then, can these graphs help us in this case or not?
\n
Yes, you can use them to build an understanding on how your models behave. E.g. regarding overfitting, instability. They're useful to tweak epochs and lr
", "votes": 1, "replies": [] } ] }, { "id": 3264561, "author_name": "Gerwyn", "author_url": "", "post_date": "2025-08-06T18:06:15.150000", "content": "\n\n\n
\n- Early stopping
\nSame issue: you end up with a different number of epochs per fold, guided by a weak estimator—individual fold validation scores.
\n
Sometimes, it could be helpful for faster training if you have ideas you want to iterate fast.
", "votes": 0, "replies": [ { "id": 3265277, "author_name": "Theo Viel", "author_url": "", "post_date": "2025-08-07T09:02:42.413000", "content": "If your validation scores are stable yes :)
", "votes": 0, "replies": [ { "id": 3265867, "author_name": "Gerwyn", "author_url": "", "post_date": "2025-08-08T03:46:11.337000", "content": "For this competition i didn't do anything fancy for validation scheme.
\nI did ablate using early stopping (apprx 5 subs), and i observe small correlation between improvement in cv and public lb.
\nI think the nature of small dataset in this competition generally has more noise than that of from usage of early-stopping.
\nAlso, submission to public is average prediction for 5 folds, so i think it helps to reduce \"overfitting\" from early stopping.
", "votes": 0, "replies": [] } ] } ] }, { "id": 3263663, "author_name": "Hüseyin Bayoğlu", "author_url": "", "post_date": "2025-08-05T14:23:42.270000", "content": "
Hi, Thanks for the tips on robust validation. It's a very important point for me as a beginner.
\nFollowing your advice, I've been training with a consistent setup. Specifically, I'm using StratifiedGroupKFold, and the model architecture and all hyperparameters are identical for each fold. However, as you can see in the attached graph, one fold (fold_5) is a clear underperformer compared to the rest.
\nGiven your post, would you still consider it a bad practice to either:
Also, is it generally considered normal for one fold to perform worse than the others like this?
", "votes": 0, "replies": [ { "id": 3263803, "author_name": "Theo Viel", "author_url": "", "post_date": "2025-08-05T18:18:41.290000", "content": "\n\nGiven your post, would you still consider it a bad practice to either:
\n
\n Use the best checkpoint for this specific fold, even if it breaks the \"fixed epoch\" rule?
Yes
\n\n\nExclude this poorly performing fold from the final ensemble?
\n
If you think the checkpoint will do poorly on test data, you can exclude it.
\n--
\nOverall, you should try to figure out why fold 5 performs poorly, 0.55 looks low, you can probably improve that.
Very interesting post/discussion, did you personally noticed a increase in your LB score by using the epoch with the best CV mean instead of the best epoch with the peak score for each different fold?
", "votes": 0, "replies": [ { "id": 3263557, "author_name": "Theo Viel", "author_url": "", "post_date": "2025-08-05T11:22:19.377000", "content": "Not really since LB is a bit unstable - but it pays in the long run since you select your final models using a more reliable score
", "votes": 1, "replies": [ { "id": 3267859, "author_name": "Ravon", "author_url": "", "post_date": "2025-08-12T03:28:27.700000", "content": "My LB score decreased 0.005, CV decreased 0.002 after doing your suggestions. Do you think this is normal and is just because of the unstable LB? I just really don’t know what to trust anymore…
", "votes": 0, "replies": [] }, { "id": 3267936, "author_name": "Theo Viel", "author_url": "", "post_date": "2025-08-12T06:40:16.957000", "content": "\n\nDo you think this is normal and is just because of the unstable LB?
\n
Could be yes, I've seen significant CV improvements lead to LB drops.
\nIt could also be that the early-stopping approach blends better when averaging k-folds.
", "votes": 0, "replies": [] }, { "id": 3267946, "author_name": "Quan Vu", "author_url": "", "post_date": "2025-08-12T06:57:19.793000", "content": "My worst fold score = 0.87, my best fold score = 0.91. And I was stuck for a long time, unable to improve CV and LB. Maybe ensembling a number of models will boost the LB significantly. Gradient boosting might be useful 🤤
", "votes": 0, "replies": [] }, { "id": 3267954, "author_name": "Quan Vu", "author_url": "", "post_date": "2025-08-12T07:01:58.423000", "content": "I have tried comparing the last epoch and the best epoch, and the differences in both CV and LB are not significant, around 0.001-0.002.
", "votes": 0, "replies": [] } ] } ] }, { "id": 3265511, "author_name": "", "author_url": "", "post_date": "2025-08-07T15:44:34.900000", "content": "I absolutely agree with your fact regarding F1 score being highly unstable. It works/ is great for further actions if the given dataset is skewed and large dataset. Because, it wont be optimal to perform hyperparameter tuning on large dataset given that constrains of your hardware specifications and at such cases better to stick on to F1 score. But if your dataset is less skewed and small, its always better to hyperparameter tuning and cross-validation method
", "votes": 0, "replies": [] }, { "id": 3268205, "author_name": "Sarah Arshad", "author_url": "", "post_date": "2025-08-12T15:14:50.800000", "content": "Thanks for the tips…!
", "votes": 0, "replies": [] } ], "raw_markdown_by_id": { "3263464": "I will address an issue some of the top public kernel have, nothing new for top teams here, sorry ! 🙂\n\nIf you’ve trained a few models, you’ve probably noticed:\n- Small hyperparameter changes can lead to significant score differences (e.g., > ±0.005 F1)\n- Scores can vary widely between different folds (> ± 0.01 F1)\n\nThis is expected. The F1 metric is inherently unstable, the dataset is small, and the task is not trivial.\n\nThis also means we need to be very careful when optimizing models !\n\n## *k*-fold Cross-Validation\n\nThe goal of *k*-fold CV is to evaluate your model across the entire dataset in a consistent way by training *k* models with the **same hyperparameters**. This leads to a more robust selection process.\n\nFor instance, the *k* models have to be trained with :\n- The same learning rate and associated scheduling strategy\n- The same number of epochs\nThis consistency allows you to properly evaluate which (learning rate, epoch count) pair works best for your overall CV setup.\n\nWhat you should **not** do:\nUse 100 epochs for fold 0 and 60 for fold 1 just because those are the epochs where each fold individually peaked.\n\nWhen doing so, you are reducing robustness of you validation setup, and artificially boosting your CV score. Instead, if 80 epochs works reasonably well across folds, stick with it.\n\n**The number of epochs that maximizes the average CV score is a better estimator of (private) test performance than the per-fold optimal values.**\n\nKeep this in mind when considering early-stopping at epoch 60:\n> *What if your fold 0 validation score stopped increasing at 60 epoch, but your test score did not ?*\n\n## Practices to Avoid\n\n(Only in the case of unstable CV scores, do whatever you want with your big tabular datasets)\n\n#### 1. Checkpointing \n\nWhen using checkpointing, you are using a different number of epochs for each fold.\nAND you are choosing the checkpoint that maximizes the fold score.\nIt is the best way to have an overestimated CV and to lose robustness. \n\n**Instead,** train all folds for the same number of epochs, and **always** use the final checkpoint(s).\n\n#### 2. Early stopping\n\nSame issue: you end up with a different number of epochs per fold, guided by a weak estimator—individual fold validation scores.\n\n#### 3. Reduce LR on Plateau\n\nThis strategy relies on per-fold validation scores to adapt learning rate schedules, once again breaking consistency.\n\n**Instead,** I recommend using a scheduler that progressively decreases your learning rate. E.g. Linear or Cosine :\n```\nfrom torch.optim.lr_scheduler import CosineAnnealingLR, LinearLR\ncosine = CosineAnnealingLR(optimizer, num_epochs, eta_min=1e-6)\nlinear = LinearLR(optimizer, start_factor=1.0, end_factor=0.0001, total_iters=num_epochs)\n```\nThis will also bring more stability to your training, removing the need for 1. and 2.\n\n-- \n\nThanks for reading and happy parameter optimization !\n\nAnd remember, trust your CV !\n(you can, now that you've removed fold-score-dependent components)", "3263492": "All of them are enabled, I'm an aware sinner. 😌", "3265188": "One competition-specific note: Since around 50% of the private test set will have missing thermo and tof values (a pattern not present in the training data); if your models rely on these features, you should try to simulate this missingness during validation...", "3263638": "Thanks for the tips! One question, how would you recommend comparing different hyperparameters/FE if training the full k-fold takes too long? I'm currently tuning on 1 fold but I feel like that would lead to overfitting on that fold?", "3263545": "Thanks for sharing. I have a few questions, though. You suggest avoiding early stopping, but many textbooks recommend it. So, when do you think early stopping is actually useful? You also said, \"If 80 epochs works reasonably well across folds, stick with it.\" But how do we know that 80 epochs is enough? Are we still using the validation set to decide on that number? Also, you recommend using the same number of epochs for all folds, but different folds might not converge at the same time. Isn’t that a problem?", "3263824": "Great insights. Sometimes Kagglers ignore these because it may perform better on kfold bagging on submission. Ensemble of overfit models can perform better (same idea as random forest). So in most cases Kagglers (including me) don't exactly validate their test case (single models on validation vs kfold bagging on test)", "3267190": "Theo, thanks for the interesting advice! I have a question then, can these graphs help us in this case or not?\n\n", "3264561": "> 2. Early stopping\n\n> Same issue: you end up with a different number of epochs per fold, guided by a weak estimator—individual fold validation scores.\n\nSometimes, it could be helpful for faster training if you have ideas you want to iterate fast. ", "3263663": "\n\n\nHi, Thanks for the tips on robust validation. It's a very important point for me as a beginner.\nFollowing your advice, I've been training with a consistent setup. Specifically, I'm using StratifiedGroupKFold, and the model architecture and all hyperparameters are identical for each fold. However, as you can see in the attached graph, one fold (fold_5) is a clear underperformer compared to the rest.\nGiven your post, would you still consider it a bad practice to either:\n1. Use the best checkpoint for this specific fold, even if it breaks the \"fixed epoch\" rule?\n2. Exclude this poorly performing fold from the final ensemble?\n\nAlso, is it generally considered normal for one fold to perform worse than the others like this?", "3263553": "Very interesting post/discussion, did you personally noticed a increase in your LB score by using the epoch with the best CV mean instead of the best epoch with the peak score for each different fold?", "3265511": "I absolutely agree with your fact regarding F1 score being highly unstable. It works/ is great for further actions if the given dataset is skewed and large dataset. Because, it wont be optimal to perform hyperparameter tuning on large dataset given that constrains of your hardware specifications and at such cases better to stick on to F1 score. But if your dataset is less skewed and small, its always better to hyperparameter tuning and cross-validation method", "3268205": "Thanks for the tips...!" } }