Do you think people are overfitting to the public leaderboard at this point?\nDo you expect a big private-LB shake-up?\nAny tips to avoid chasing the public LB too much?
\nShare your idea
", "messages": [ { "id": 3394820, "postDate": "2026-01-21T17:59:55.943Z", "content": "Looking at the difference between folds on my CV I lean towards the expectation of a pretty big shakeup.
\nA huge source of some past shakeups have been hundreds/thousands of folks using a high scoring shared notebook as their submission(s). So I am hoping that the current 0.73 shared notebook is hugely over-fitting and my lower scoring model is a winner :)
", "rawMarkdown": "Looking at the difference between folds on my CV I lean towards the expectation of a pretty big shakeup.\n\nA huge source of some past shakeups have been hundreds/thousands of folks using a high scoring shared notebook as their submission(s). So I am hoping that the current 0.73 shared notebook is hugely over-fitting and my lower scoring model is a winner :)\n\n\n\n\n\n", "votes": 1, "replies": [ { "id": 3394832, "postDate": "2026-01-21T18:08:40.133Z", "content": "Lol, I have had the same thought - \"I wish everyone above us is overfitting and everybody below is underfitting.\" :) But I don't think there would be a big shake.
", "rawMarkdown": "Lol, I have had the same thought - \"I wish everyone above us is overfitting and everybody below is underfitting.\" :) But I don't think there would be a big shake.", "replies": [ { "id": 3394842, "postDate": "2026-01-21T18:29:07.763Z", "content": "OK - if you drop more than 10 places than the next time you come to Pittsburgh PA You will buy me one of our famous Big Ben sandwiches. If you don't drop by more than 10 I will buy you lunch the next time I come to Palani !
", "rawMarkdown": "OK - if you drop more than 10 places than the next time you come to Pittsburgh PA You will buy me one of our famous Big Ben sandwiches. If you don't drop by more than 10 I will buy you lunch the next time I come to Palani !", "votes": 1 }, { "id": 3394861, "postDate": "2026-01-21T19:03:47.297Z", "content": "Okay. Deal šš¤
", "rawMarkdown": "Okay. Deal šš¤", "votes": 3 }, { "id": 3398356, "postDate": "2026-01-29T02:52:59.720Z", "content": "I owe you a sandwich. The final score was exactly what I thought it would be, I picked a 0.66 and 0.68 for my submissions but sadly my models did not achieve that score. Congrats.
", "rawMarkdown": "I owe you a sandwich. The final score was exactly what I thought it would be, I picked a 0.66 and 0.68 for my submissions but sadly my models did not achieve that score. Congrats." } ] }, { "id": 3394917, "postDate": "2026-01-21T21:16:56.083Z", "rawMarkdown": "", "isDeleted": true }, { "id": 3398546, "postDate": "2026-01-29T09:56:56.307Z", "content": "\n\nLooking at the difference between folds on my CV I lean towards the expectation of a pretty big shakeup.
\nA huge source of some past shakeups have been hundreds/thousands of folks using a high scoring shared notebook as their submission(s). So I am hoping that the current 0.73 shared notebook is hugely over-fitting and my lower scoring model is a winner :)
\n
Would like to extend my respect to you. I was here in the competition to revisit my skills. Was not even interested in competing and I was closely following what you were commenting about. You were 100 right! on the shakeup!
", "rawMarkdown": "> Looking at the difference between folds on my CV I lean towards the expectation of a pretty big shakeup.\n> \n> A huge source of some past shakeups have been hundreds/thousands of folks using a high scoring shared notebook as their submission(s). So I am hoping that the current 0.73 shared notebook is hugely over-fitting and my lower scoring model is a winner :)\n\nWould like to extend my respect to you. I was here in the competition to revisit my skills. Was not even interested in competing and I was closely following what you were commenting about. You were 100 right! on the shakeup!\n" } ] }, { "id": 3394684, "postDate": "2026-01-21T13:49:52.533Z", "content": "I think CV and lb are n depends on number of head, you trained on, as i trained on 3 head, and derive 2 from that 3, then my cv and lb are similla, but when i increase number of head to 4 or 5 cv and lb are different.
", "rawMarkdown": "I think CV and lb are n depends on number of head, you trained on, as i trained on 3 head, and derive 2 from that 3, then my cv and lb are similla, but when i increase number of head to 4 or 5 cv and lb are different.", "votes": 1, "replies": [ { "id": 3394693, "postDate": "2026-01-21T14:17:02.770Z", "content": "That's strange. I encourage you to make sure of it again as 4 heads worked much better for me than 3 or 5. And I also mentioned it in detail along with a comparison of each approach in my last discussion. Do check that out here.
", "rawMarkdown": "That's strange. I encourage you to make sure of it again as 4 heads worked much better for me than 3 or 5. And I also mentioned it in detail along with a comparison of each approach in my last discussion. Do check that out [here](https://www.kaggle.com/competitions/csiro-biomass/discussion/669012).", "votes": 1, "replies": [ { "id": 3394960, "postDate": "2026-01-22T01:06:36.997Z", "content": "Yes I have gone through your explanation, but as i found this, when in 4 head cv is 0.51 and lb is 0.71[in all case using/not-using tta, post-processing etc same results] but when 3 heads cv and lb are aligned, get almost same cv and lb.
", "rawMarkdown": "Yes I have gone through your explanation, but as i found this, when in 4 head cv is 0.51 and lb is 0.71[in all case using/not-using tta, post-processing etc same results] but when 3 heads cv and lb are aligned, get almost same cv and lb. " } ] } ] }, { "id": 3394646, "postDate": "2026-01-21T12:27:19.860Z", "content": "Above 0.73, I don't believe the shakeup is going to be of more than 0.02.
\nWhich means, A person getting 0.73 let's say, cannot get a rank above someone getting 0.75. Unless they have used simple trial and error and their scores are very unreliable.
\nFor me, my CV gets me 0.72 Train R2, 0.72 Val R2 and 0.72 LB, all at the same time. \nIt only starts deviating after 0.72. \nI hence don't expect my Private LB to be higher than someone with Public LB 0.77 at best and lower than someone with Public LB 0.73 at worst.
", "rawMarkdown": "Above 0.73, I don't believe the shakeup is going to be of more than 0.02. \n\nWhich means, A person getting 0.73 let's say, cannot get a rank above someone getting 0.75. Unless they have used simple trial and error and their scores are very unreliable. \n\nFor me, my CV gets me 0.72 Train R2, 0.72 Val R2 and 0.72 LB, all at the same time. \nIt only starts deviating after 0.72. \nI hence don't expect my Private LB to be higher than someone with Public LB 0.77 at best and lower than someone with Public LB 0.73 at worst. ", "votes": 2, "replies": [ { "id": 3394666, "postDate": "2026-01-21T13:01:47.883Z", "content": "When the host split the test set into public and private parts, is it done randomly, or do they intentionally put some harder samples into the private test set? If the public and private splits are very different, it becomes really hard to estimate what the final score will be.
", "rawMarkdown": "When the host split the test set into public and private parts, is it done randomly, or do they intentionally put some harder samples into the private test set? If the public and private splits are very different, it becomes really hard to estimate what the final score will be.", "replies": [ { "id": 3394695, "postDate": "2026-01-21T14:22:37.193Z", "content": "As far as I remember, the train and test set have the same states, species, etc. and the distinction comes from time. So there are two possibilities here
\nx&y) as Public LB, 50% samples from each bin (let's say date) were taken for Public LB and the other were taken for Private LB.x, Public LB is from year y and Private LB is from year z. This one would be tough for everyone but at the same time would be really a lot of fun.I remember the host mentioning somewhere that the splits are not going to be entirely random. I hope there's not much distribution shift, otherwise this will meet the same fate as NeurIPS Polymer Prediction Competition.
", "rawMarkdown": "I remember the host mentioning somewhere that the splits are not going to be entirely random. I hope there's not much distribution shift, otherwise this will meet the same fate as NeurIPS Polymer Prediction Competition." }, { "id": 3394743, "postDate": "2026-01-21T15:56:49.840Z", "content": "That one was once in a decade experience. At no cost are we going to see something like that soon.
", "rawMarkdown": "That one was once in a decade experience. At no cost are we going to see something like that soon." } ] } ] }, { "id": 3394623, "postDate": "2026-01-21T11:30:56.997Z", "content": "Do you think people are overfitting to the public leaderboard at this point?\nDo you expect a big private-LB shake-up?\nAny tips to avoid chasing the public LB too much?
\nShare your idea
", "rawMarkdown": "Do you think people are overfitting to the public leaderboard at this point?\nDo you expect a big private-LB shake-up?\nAny tips to avoid chasing the public LB too much?\n\nShare your idea" }, { "id": 3394745, "postDate": "2026-01-21T16:01:37.323Z", "content": "From a data perspective, I feel like the only data that really tests generalization in this competition is the 400 images on the public LB and the 300 images in the training set. So if a solution does well on the private LB, it basically means it has already āpassedā the generalization check on those 400 public-LB samples. And if your CV correlates strongly with the public LB, then you can kind of say the model generalizes well on this 700-sample universe.
\nThere are two possibilities.
\nFirst, if the public and private sets are just randomly split, then because the sample size is small, Iād still expect some variance. Even with a random split, youāre going to see some shake-up.
\nSecond, if thereās distribution shift between public and private, then the shake-up could be huge. You can often tell this when CV and public LB correlation is low. Low correlation usually means your validation target doesnāt match the public LB target which is just like the distribution shift between public and private. And you can see it in practice: with many different CV splits, you get cases where CV goes up but public LB goes down, or CV goes down but public LB goes up. If you imagine any of those splits being the private LB, the variance would be massive.
", "rawMarkdown": "From a data perspective, I feel like the only data that really tests generalization in this competition is the 400 images on the public LB and the 300 images in the training set. So if a solution does well on the private LB, it basically means it has already āpassedā the generalization check on those 400 public-LB samples. And if your CV correlates strongly with the public LB, then you can kind of say the model generalizes well on this 700-sample universe.\n\nThere are two possibilities.\n\nFirst, if the public and private sets are just randomly split, then because the sample size is small, Iād still expect some variance. Even with a random split, youāre going to see some shake-up.\n\nSecond, if thereās distribution shift between public and private, then the shake-up could be huge. You can often tell this when CV and public LB correlation is low. Low correlation usually means your validation target doesnāt match the public LB target which is just like the distribution shift between public and private. And you can see it in practice: with many different CV splits, you get cases where CV goes up but public LB goes down, or CV goes down but public LB goes up. If you imagine any of those splits being the private LB, the variance would be massive.", "replies": [ { "id": 3395049, "postDate": "2026-01-22T06:45:14.617Z", "content": "I think the scores that rank high are relatively reliable, at least they have good generalization ability on 700 images. Of course, there must be models with relatively low LB that will turn the tables in the final test set.
", "rawMarkdown": "I think the scores that rank high are relatively reliable, at least they have good generalization ability on 700 images. Of course, there must be models with relatively low LB that will turn the tables in the final test set.", "votes": -1 } ] } ], "comments": [ { "id": 3394820, "author_name": "PC Jimmmy", "author_url": "", "post_date": "2026-01-21T17:59:55.943000", "content": "Looking at the difference between folds on my CV I lean towards the expectation of a pretty big shakeup.
\nA huge source of some past shakeups have been hundreds/thousands of folks using a high scoring shared notebook as their submission(s). So I am hoping that the current 0.73 shared notebook is hugely over-fitting and my lower scoring model is a winner :)
", "votes": 1, "replies": [ { "id": 3394832, "author_name": "Paritosh Kumar Tripathi", "author_url": "", "post_date": "2026-01-21T18:08:40.133000", "content": "Lol, I have had the same thought - \"I wish everyone above us is overfitting and everybody below is underfitting.\" :) But I don't think there would be a big shake.
", "votes": 0, "replies": [ { "id": 3394842, "author_name": "PC Jimmmy", "author_url": "", "post_date": "2026-01-21T18:29:07.763000", "content": "OK - if you drop more than 10 places than the next time you come to Pittsburgh PA You will buy me one of our famous Big Ben sandwiches. If you don't drop by more than 10 I will buy you lunch the next time I come to Palani !
", "votes": 1, "replies": [] }, { "id": 3394861, "author_name": "Paritosh Kumar Tripathi", "author_url": "", "post_date": "2026-01-21T19:03:47.297000", "content": "Okay. Deal šš¤
", "votes": 3, "replies": [] }, { "id": 3398356, "author_name": "PC Jimmmy", "author_url": "", "post_date": "2026-01-29T02:52:59.720000", "content": "I owe you a sandwich. The final score was exactly what I thought it would be, I picked a 0.66 and 0.68 for my submissions but sadly my models did not achieve that score. Congrats.
", "votes": 0, "replies": [] } ] }, { "id": 3394917, "author_name": "", "author_url": "", "post_date": "2026-01-21T21:16:56.083000", "content": "", "votes": 0, "replies": [] }, { "id": 3398546, "author_name": "WajihullahBaig", "author_url": "", "post_date": "2026-01-29T09:56:56.307000", "content": "\n\nLooking at the difference between folds on my CV I lean towards the expectation of a pretty big shakeup.
\nA huge source of some past shakeups have been hundreds/thousands of folks using a high scoring shared notebook as their submission(s). So I am hoping that the current 0.73 shared notebook is hugely over-fitting and my lower scoring model is a winner :)
\n
Would like to extend my respect to you. I was here in the competition to revisit my skills. Was not even interested in competing and I was closely following what you were commenting about. You were 100 right! on the shakeup!
", "votes": 0, "replies": [] } ] }, { "id": 3394684, "author_name": "Rocker Ritesh", "author_url": "", "post_date": "2026-01-21T13:49:52.533000", "content": "I think CV and lb are n depends on number of head, you trained on, as i trained on 3 head, and derive 2 from that 3, then my cv and lb are similla, but when i increase number of head to 4 or 5 cv and lb are different.
", "votes": 1, "replies": [ { "id": 3394693, "author_name": "Komil Parmar", "author_url": "", "post_date": "2026-01-21T14:17:02.770000", "content": "That's strange. I encourage you to make sure of it again as 4 heads worked much better for me than 3 or 5. And I also mentioned it in detail along with a comparison of each approach in my last discussion. Do check that out here.
", "votes": 1, "replies": [ { "id": 3394960, "author_name": "Rocker Ritesh", "author_url": "", "post_date": "2026-01-22T01:06:36.997000", "content": "Yes I have gone through your explanation, but as i found this, when in 4 head cv is 0.51 and lb is 0.71[in all case using/not-using tta, post-processing etc same results] but when 3 heads cv and lb are aligned, get almost same cv and lb.
", "votes": 0, "replies": [] } ] } ] }, { "id": 3394646, "author_name": "Komil Parmar", "author_url": "", "post_date": "2026-01-21T12:27:19.860000", "content": "Above 0.73, I don't believe the shakeup is going to be of more than 0.02.
\nWhich means, A person getting 0.73 let's say, cannot get a rank above someone getting 0.75. Unless they have used simple trial and error and their scores are very unreliable.
\nFor me, my CV gets me 0.72 Train R2, 0.72 Val R2 and 0.72 LB, all at the same time. \nIt only starts deviating after 0.72. \nI hence don't expect my Private LB to be higher than someone with Public LB 0.77 at best and lower than someone with Public LB 0.73 at worst.
", "votes": 2, "replies": [ { "id": 3394666, "author_name": "Forrest_xlz", "author_url": "", "post_date": "2026-01-21T13:01:47.883000", "content": "When the host split the test set into public and private parts, is it done randomly, or do they intentionally put some harder samples into the private test set? If the public and private splits are very different, it becomes really hard to estimate what the final score will be.
", "votes": 0, "replies": [ { "id": 3394695, "author_name": "Komil Parmar", "author_url": "", "post_date": "2026-01-21T14:22:37.193000", "content": "As far as I remember, the train and test set have the same states, species, etc. and the distinction comes from time. So there are two possibilities here
\nx&y) as Public LB, 50% samples from each bin (let's say date) were taken for Public LB and the other were taken for Private LB.x, Public LB is from year y and Private LB is from year z. This one would be tough for everyone but at the same time would be really a lot of fun.I remember the host mentioning somewhere that the splits are not going to be entirely random. I hope there's not much distribution shift, otherwise this will meet the same fate as NeurIPS Polymer Prediction Competition.
", "votes": 0, "replies": [] }, { "id": 3394743, "author_name": "Komil Parmar", "author_url": "", "post_date": "2026-01-21T15:56:49.840000", "content": "That one was once in a decade experience. At no cost are we going to see something like that soon.
", "votes": 0, "replies": [] } ] } ] }, { "id": 3394745, "author_name": "Forrest_xlz", "author_url": "", "post_date": "2026-01-21T16:01:37.323000", "content": "From a data perspective, I feel like the only data that really tests generalization in this competition is the 400 images on the public LB and the 300 images in the training set. So if a solution does well on the private LB, it basically means it has already āpassedā the generalization check on those 400 public-LB samples. And if your CV correlates strongly with the public LB, then you can kind of say the model generalizes well on this 700-sample universe.
\nThere are two possibilities.
\nFirst, if the public and private sets are just randomly split, then because the sample size is small, Iād still expect some variance. Even with a random split, youāre going to see some shake-up.
\nSecond, if thereās distribution shift between public and private, then the shake-up could be huge. You can often tell this when CV and public LB correlation is low. Low correlation usually means your validation target doesnāt match the public LB target which is just like the distribution shift between public and private. And you can see it in practice: with many different CV splits, you get cases where CV goes up but public LB goes down, or CV goes down but public LB goes up. If you imagine any of those splits being the private LB, the variance would be massive.
", "votes": 0, "replies": [ { "id": 3395049, "author_name": "YYH", "author_url": "", "post_date": "2026-01-22T06:45:14.617000", "content": "I think the scores that rank high are relatively reliable, at least they have good generalization ability on 700 images. Of course, there must be models with relatively low LB that will turn the tables in the final test set.
", "votes": -1, "replies": [] } ] } ], "raw_markdown_by_id": { "3394820": "Looking at the difference between folds on my CV I lean towards the expectation of a pretty big shakeup.\n\nA huge source of some past shakeups have been hundreds/thousands of folks using a high scoring shared notebook as their submission(s). So I am hoping that the current 0.73 shared notebook is hugely over-fitting and my lower scoring model is a winner :)\n\n\n\n\n\n", "3394684": "I think CV and lb are n depends on number of head, you trained on, as i trained on 3 head, and derive 2 from that 3, then my cv and lb are similla, but when i increase number of head to 4 or 5 cv and lb are different.", "3394646": "Above 0.73, I don't believe the shakeup is going to be of more than 0.02. \n\nWhich means, A person getting 0.73 let's say, cannot get a rank above someone getting 0.75. Unless they have used simple trial and error and their scores are very unreliable. \n\nFor me, my CV gets me 0.72 Train R2, 0.72 Val R2 and 0.72 LB, all at the same time. \nIt only starts deviating after 0.72. \nI hence don't expect my Private LB to be higher than someone with Public LB 0.77 at best and lower than someone with Public LB 0.73 at worst. ", "3394623": "Do you think people are overfitting to the public leaderboard at this point?\nDo you expect a big private-LB shake-up?\nAny tips to avoid chasing the public LB too much?\n\nShare your idea", "3394745": "From a data perspective, I feel like the only data that really tests generalization in this competition is the 400 images on the public LB and the 300 images in the training set. So if a solution does well on the private LB, it basically means it has already āpassedā the generalization check on those 400 public-LB samples. And if your CV correlates strongly with the public LB, then you can kind of say the model generalizes well on this 700-sample universe.\n\nThere are two possibilities.\n\nFirst, if the public and private sets are just randomly split, then because the sample size is small, Iād still expect some variance. Even with a random split, youāre going to see some shake-up.\n\nSecond, if thereās distribution shift between public and private, then the shake-up could be huge. You can often tell this when CV and public LB correlation is low. Low correlation usually means your validation target doesnāt match the public LB target which is just like the distribution shift between public and private. And you can see it in practice: with many different CV splits, you get cases where CV goes up but public LB goes down, or CV goes down but public LB goes up. If you imagine any of those splits being the private LB, the variance would be massive." } }