The main discussion about \"new ideas to improve\" is now too long
The current discussion page should let us share more ideas, focusing on the meaning of current leaderboard scores.
----- Start -----
\nMy best guess:
\nParticipants with scores around 0.66 have a good chance of actually winning the competition. This score is extremely hard to achieve without any additional tricks.
Scores like 0.76, 0.77 probably use postprocessing which works well with the public leaderboard. An example I can think of is shifting the mean of predictions.
Now the question is whether we should trust our 0.66 scores for private test scoring or try leaderboard probing? Your own ideas go here…
The main discussion about \"new ideas to improve\" is now too long
The current discussion page should let us share more ideas, focusing on the meaning of current leaderboard scores.
----- Start -----
\nMy best guess:
\nParticipants with scores around 0.66 have a good chance of actually winning the competition. This score is extremely hard to achieve without any additional tricks.
Scores like 0.76, 0.77 probably use postprocessing which works well with the public leaderboard. An example I can think of is shifting the mean of predictions.
Now the question is whether we should trust our 0.66 scores for private test scoring or try leaderboard probing? Your own ideas go here…
Actually a truly simple implementation can achieve LB 0.74+. It's hard to say is 0.03 diff critical, but I believe 0.10+ diff is. In my case, a LB 0.75+ model's performance is always better than LB 0.65-. And by the way, trying to achieve higher public LB score is also an interesting part in contests.
", "rawMarkdown": "Actually a truly simple implementation can achieve LB 0.74+. It's hard to say is 0.03 diff critical, but I believe 0.10+ diff is. In my case, a LB 0.75+ model's performance is always better than LB 0.65-. And by the way, trying to achieve higher public LB score is also an interesting part in contests.", "votes": 8, "replies": [ { "id": 3379876, "postDate": "2025-12-21T04:03:14.633Z", "content": "I think the truly simple implementation is the \"seed\"
", "rawMarkdown": "I think the truly simple implementation is the \"seed\"", "votes": 2, "replies": [ { "id": 3379881, "postDate": "2025-12-21T04:28:15.430Z", "content": "Seed can make at most ±0.02 LB change in my case.
", "rawMarkdown": "Seed can make at most ±0.02 LB change in my case.", "votes": 1 }, { "id": 3379937, "postDate": "2025-12-21T08:27:40.970Z", "rawMarkdown": "", "votes": -1, "isDeleted": true }, { "id": 3379962, "postDate": "2025-12-21T09:23:48.060Z", "content": "I agree. Seed is something that might feel like a luck factory now, but not in Private Leaderboard. The strategy used in CV matters much more than the seed.
", "rawMarkdown": "I agree. Seed is something that might feel like a luck factory now, but not in Private Leaderboard. The strategy used in CV matters much more than the seed. ", "votes": 1 }, { "id": 3382981, "postDate": "2025-12-29T03:09:38.823Z", "content": "indeed , the strategy used in cv is important,
", "rawMarkdown": "indeed , the strategy used in cv is important," }, { "id": 3382991, "postDate": "2025-12-29T03:58:10.060Z", "content": "Do you mean that changing the seed will change the composition of the training data for each fold, and by doing so, it will improve the score?
", "rawMarkdown": "Do you mean that changing the seed will change the composition of the training data for each fold, and by doing so, it will improve the score?" } ] }, { "id": 3380055, "postDate": "2025-12-21T14:26:07.703Z", "content": "Thanks what has a been your best single model of we may ask @linkinpony
", "rawMarkdown": "Thanks what has a been your best single model of we may ask @linkinpony ", "replies": [ { "id": 3380063, "postDate": "2025-12-21T15:08:53.407Z", "content": "0.76 @gauravbrills
", "rawMarkdown": "0.76 @gauravbrills ", "votes": 4 } ] }, { "id": 3381528, "postDate": "2025-12-24T20:37:31.570Z", "content": "How long does it take for your code to execute, if it's not a secret?
", "rawMarkdown": "How long does it take for your code to execute, if it's not a secret?", "replies": [ { "id": 3381786, "postDate": "2025-12-25T14:29:20.433Z", "content": "FWIW, I have models that score in the 0.65-0.68 range that have a wide range of scoring times.
\nEverything from 1 hour to 5 hours
\nAdmittedly, I have done almost no work to optimize the times on these submissions, since the data set is so small and there appears to be ample time for most of my submissions.
", "rawMarkdown": "FWIW, I have models that score in the 0.65-0.68 range that have a wide range of scoring times.\n\nEverything from 1 hour to 5 hours\n\nAdmittedly, I have done almost no work to optimize the times on these submissions, since the data set is so small and there appears to be ample time for most of my submissions." }, { "id": 3381790, "postDate": "2025-12-25T14:36:09.003Z", "content": "From minutes to hours. It can vary in a very large range.
", "rawMarkdown": "From minutes to hours. It can vary in a very large range.", "votes": 1 } ] }, { "id": 3389099, "postDate": "2026-01-10T09:28:52.920Z", "content": "So impressive! I can only make it to ~0.72. May I ask how long does such models (LB 0.74+) take for inference? Mine is very fast, < 20 mins.
", "rawMarkdown": "So impressive! I can only make it to ~0.72. May I ask how long does such models (LB 0.74+) take for inference? Mine is very fast, < 20 mins." } ] }, { "id": 3379773, "postDate": "2025-12-20T17:36:11.257Z", "content": "It's difficult to tell
", "rawMarkdown": "It's difficult to tell ", "votes": 1 }, { "id": 3379763, "postDate": "2025-12-20T16:53:55.373Z", "content": "I agree，haha.😄
", "rawMarkdown": "I agree，haha.😄", "votes": 1, "replies": [ { "id": 3379795, "postDate": "2025-12-20T18:14:44.087Z", "content": "I think that going from 0.66 to 0.71 was due to good use of the train data. For me, my local CV and my lb score are statistically equivalent. I am struggling to get above 0.71, but I think there is room above 0.7 that is not leaderboard probing yet.
", "rawMarkdown": "I think that going from 0.66 to 0.71 was due to good use of the train data. For me, my local CV and my lb score are statistically equivalent. I am struggling to get above 0.71, but I think there is room above 0.7 that is not leaderboard probing yet.", "votes": 2, "replies": [ { "id": 3379961, "postDate": "2025-12-21T09:20:42.407Z", "content": "For me this exact same movement from 0.66 to 0.71 was due to change in training strategy alone which basically includes the techniques associated with making the training or loss curve better, like loss itself, optimiser, scheduling, regularisation etc. I didn't change my data nor the CV strategy for this.
\nBut the \"good use of train data\" actually got me thinking @timkrige. What if for my model, I learn to estimate the number of epochs or steps it takes to converge on both train and val set the best. I then multiply it with 5/4 and train my model on entire data with that epochs or steps. Same thing was mentioned by Chris Deotte in one of the solution discussion, but that was playground series and hence tabular data. Don't know if that would work here, but it's worth the try, especially if that's exactly what you meant by \"good use of train data\".
", "rawMarkdown": "For me this exact same movement from 0.66 to 0.71 was due to change in training strategy alone which basically includes the techniques associated with making the training or loss curve better, like loss itself, optimiser, scheduling, regularisation etc. I didn't change my data nor the CV strategy for this. \n\nBut the \"good use of train data\" actually got me thinking @timkrige. What if for my model, I learn to estimate the number of epochs or steps it takes to converge on both train and val set the best. I then multiply it with 5/4 and train my model on entire data with that epochs or steps. Same thing was mentioned by Chris Deotte in one of the solution discussion, but that was playground series and hence tabular data. Don't know if that would work here, but it's worth the try, especially if that's exactly what you meant by \"good use of train data\". ", "votes": 3 }, { "id": 3381871, "postDate": "2025-12-25T19:26:11.583Z", "content": "This was not what I meant, but it sounded cool so I tried it. my models are far too out of control to know when to stop - one epoch wrong and you are 0.3 different on the LB, and even the best was only 0.6.
\nI meant looking at the train data and using it in a way that makes sense, a lot of it is already described in the discussions.
", "rawMarkdown": "This was not what I meant, but it sounded cool so I tried it. my models are far too out of control to know when to stop - one epoch wrong and you are 0.3 different on the LB, and even the best was only 0.6.\n\nI meant looking at the train data and using it in a way that makes sense, a lot of it is already described in the discussions.\n\n", "votes": 1 }, { "id": 3388600, "postDate": "2026-01-09T07:46:33.123Z", "content": "The way to stop it is to decide based on CV. \nI trained the same Dino model on CV, got the average train loss when I hit early stopping, and then trained the same model on full dataset stopping when the training loss goes below that mean train loss. Let this epoch be E.\nI got a score of 0.7 which is the same score I got on majority of the folds. \nI then tried submitting the checkpoint from that E-1 and that E+1 and found that both got 0.69 on E+1 while getting 0.7 on E-1. I then chose E-2 and got 0.69 again.
\nFrom what I know, Full dataset model should always be better if stopped correctly. Especially, in such low data regime. To me this only indicates that we should leave the idea of trying more and more models, training strategies, and instead search for better datasets, or try pseudo labelling and finetuning on test data.
", "rawMarkdown": "The way to stop it is to decide based on CV. \nI trained the same Dino model on CV, got the average train loss when I hit early stopping, and then trained the same model on full dataset stopping when the training loss goes below that mean train loss. Let this epoch be E.\nI got a score of 0.7 which is the same score I got on majority of the folds. \nI then tried submitting the checkpoint from that E-1 and that E+1 and found that both got 0.69 on E+1 while getting 0.7 on E-1. I then chose E-2 and got 0.69 again.\n\nFrom what I know, Full dataset model should always be better if stopped correctly. Especially, in such low data regime. To me this only indicates that we should leave the idea of trying more and more models, training strategies, and instead search for better datasets, or try pseudo labelling and finetuning on test data. ", "votes": 1 }, { "id": 3388601, "postDate": "2026-01-09T07:53:12.593Z", "content": "One more very interesting trajectory I saw is that, when I over complicated things, by adding much more augmentations, much more regularization, minimal unfreezing etc to the same model that I earlier trained with minimal regularization and augmentation, full model training, Both got an identical training trajectory as well as an identical train-val loss gap. Early stopping in more regularized version only meant chopping off that train-val loss trajectory instead of meaning a better trajectory.
\nInfact, the regularized version was instead performing poorly on LB comparatively.
\nMy next idea is to try train the model entirely, with same weight decay and lr on the entire model, while trying to make the minimum number of gradient steps, as opposed to just unfreezing last few layers, discriminative lr and weight decays, and training for much more gradient steps confidently.
", "rawMarkdown": "One more very interesting trajectory I saw is that, when I over complicated things, by adding much more augmentations, much more regularization, minimal unfreezing etc to the same model that I earlier trained with minimal regularization and augmentation, full model training, Both got an identical training trajectory as well as an identical train-val loss gap. Early stopping in more regularized version only meant chopping off that train-val loss trajectory instead of meaning a better trajectory.\n\nInfact, the regularized version was instead performing poorly on LB comparatively.\n\nMy next idea is to try train the model entirely, with same weight decay and lr on the entire model, while trying to make the minimum number of gradient steps, as opposed to just unfreezing last few layers, discriminative lr and weight decays, and training for much more gradient steps confidently. ", "votes": 1 } ] } ] }, { "id": 3379946, "postDate": "2025-12-21T08:44:10.957Z", "rawMarkdown": "", "votes": -1, "isDeleted": true }, { "id": 3379943, "postDate": "2025-12-21T08:41:57.487Z", "rawMarkdown": "", "isDeleted": true } ], "comments": [ { "id": 3379855, "author_name": "Sayaka Miki", "author_url": "", "post_date": "2025-12-21T01:51:05.193000", "content": "Actually a truly simple implementation can achieve LB 0.74+. It's hard to say is 0.03 diff critical, but I believe 0.10+ diff is. In my case, a LB 0.75+ model's performance is always better than LB 0.65-. And by the way, trying to achieve higher public LB score is also an interesting part in contests.
", "votes": 8, "replies": [ { "id": 3379876, "author_name": "Paritosh Kumar Tripathi", "author_url": "", "post_date": "2025-12-21T04:03:14.633000", "content": "I think the truly simple implementation is the \"seed\"
", "votes": 2, "replies": [ { "id": 3379881, "author_name": "Sayaka Miki", "author_url": "", "post_date": "2025-12-21T04:28:15.430000", "content": "Seed can make at most ±0.02 LB change in my case.
", "votes": 1, "replies": [] }, { "id": 3379937, "author_name": "", "author_url": "", "post_date": "2025-12-21T08:27:40.970000", "content": "", "votes": -1, "replies": [] }, { "id": 3379962, "author_name": "Komil Parmar", "author_url": "", "post_date": "2025-12-21T09:23:48.060000", "content": "I agree. Seed is something that might feel like a luck factory now, but not in Private Leaderboard. The strategy used in CV matters much more than the seed.
", "votes": 1, "replies": [] }, { "id": 3382981, "author_name": "", "author_url": "", "post_date": "2025-12-29T03:09:38.823000", "content": "indeed , the strategy used in cv is important,
", "votes": 0, "replies": [] }, { "id": 3382991, "author_name": "", "author_url": "", "post_date": "2025-12-29T03:58:10.060000", "content": "Do you mean that changing the seed will change the composition of the training data for each fold, and by doing so, it will improve the score?
", "votes": 0, "replies": [] } ] }, { "id": 3380055, "author_name": "Gaurav Rawat", "author_url": "", "post_date": "2025-12-21T14:26:07.703000", "content": "Thanks what has a been your best single model of we may ask @linkinpony
", "votes": 0, "replies": [ { "id": 3380063, "author_name": "Sayaka Miki", "author_url": "", "post_date": "2025-12-21T15:08:53.407000", "content": "0.76 @gauravbrills
", "votes": 4, "replies": [] } ] }, { "id": 3381528, "author_name": "Mark", "author_url": "", "post_date": "2025-12-24T20:37:31.570000", "content": "How long does it take for your code to execute, if it's not a secret?
", "votes": 0, "replies": [ { "id": 3381786, "author_name": "chemdatafarmer", "author_url": "", "post_date": "2025-12-25T14:29:20.433000", "content": "FWIW, I have models that score in the 0.65-0.68 range that have a wide range of scoring times.
\nEverything from 1 hour to 5 hours
\nAdmittedly, I have done almost no work to optimize the times on these submissions, since the data set is so small and there appears to be ample time for most of my submissions.
", "votes": 0, "replies": [] }, { "id": 3381790, "author_name": "Sayaka Miki", "author_url": "", "post_date": "2025-12-25T14:36:09.003000", "content": "From minutes to hours. It can vary in a very large range.
", "votes": 1, "replies": [] } ] }, { "id": 3389099, "author_name": "DiamondH", "author_url": "", "post_date": "2026-01-10T09:28:52.920000", "content": "So impressive! I can only make it to ~0.72. May I ask how long does such models (LB 0.74+) take for inference? Mine is very fast, < 20 mins.
", "votes": 0, "replies": [] } ] }, { "id": 3379773, "author_name": "Med Ali Bouchhioua", "author_url": "", "post_date": "2025-12-20T17:36:11.257000", "content": "It's difficult to tell
", "votes": 1, "replies": [] }, { "id": 3379763, "author_name": "Boredom", "author_url": "", "post_date": "2025-12-20T16:53:55.373000", "content": "I agree，haha.😄
", "votes": 1, "replies": [ { "id": 3379795, "author_name": "Tim Krige", "author_url": "", "post_date": "2025-12-20T18:14:44.087000", "content": "I think that going from 0.66 to 0.71 was due to good use of the train data. For me, my local CV and my lb score are statistically equivalent. I am struggling to get above 0.71, but I think there is room above 0.7 that is not leaderboard probing yet.
", "votes": 2, "replies": [ { "id": 3379961, "author_name": "Komil Parmar", "author_url": "", "post_date": "2025-12-21T09:20:42.407000", "content": "For me this exact same movement from 0.66 to 0.71 was due to change in training strategy alone which basically includes the techniques associated with making the training or loss curve better, like loss itself, optimiser, scheduling, regularisation etc. I didn't change my data nor the CV strategy for this.
\nBut the \"good use of train data\" actually got me thinking @timkrige. What if for my model, I learn to estimate the number of epochs or steps it takes to converge on both train and val set the best. I then multiply it with 5/4 and train my model on entire data with that epochs or steps. Same thing was mentioned by Chris Deotte in one of the solution discussion, but that was playground series and hence tabular data. Don't know if that would work here, but it's worth the try, especially if that's exactly what you meant by \"good use of train data\".
", "votes": 3, "replies": [] }, { "id": 3381871, "author_name": "Tim Krige", "author_url": "", "post_date": "2025-12-25T19:26:11.583000", "content": "This was not what I meant, but it sounded cool so I tried it. my models are far too out of control to know when to stop - one epoch wrong and you are 0.3 different on the LB, and even the best was only 0.6.
\nI meant looking at the train data and using it in a way that makes sense, a lot of it is already described in the discussions.
", "votes": 1, "replies": [] }, { "id": 3388600, "author_name": "Komil Parmar", "author_url": "", "post_date": "2026-01-09T07:46:33.123000", "content": "The way to stop it is to decide based on CV. \nI trained the same Dino model on CV, got the average train loss when I hit early stopping, and then trained the same model on full dataset stopping when the training loss goes below that mean train loss. Let this epoch be E.\nI got a score of 0.7 which is the same score I got on majority of the folds. \nI then tried submitting the checkpoint from that E-1 and that E+1 and found that both got 0.69 on E+1 while getting 0.7 on E-1. I then chose E-2 and got 0.69 again.
\nFrom what I know, Full dataset model should always be better if stopped correctly. Especially, in such low data regime. To me this only indicates that we should leave the idea of trying more and more models, training strategies, and instead search for better datasets, or try pseudo labelling and finetuning on test data.
", "votes": 1, "replies": [] }, { "id": 3388601, "author_name": "Komil Parmar", "author_url": "", "post_date": "2026-01-09T07:53:12.593000", "content": "One more very interesting trajectory I saw is that, when I over complicated things, by adding much more augmentations, much more regularization, minimal unfreezing etc to the same model that I earlier trained with minimal regularization and augmentation, full model training, Both got an identical training trajectory as well as an identical train-val loss gap. Early stopping in more regularized version only meant chopping off that train-val loss trajectory instead of meaning a better trajectory.
\nInfact, the regularized version was instead performing poorly on LB comparatively.
\nMy next idea is to try train the model entirely, with same weight decay and lr on the entire model, while trying to make the minimum number of gradient steps, as opposed to just unfreezing last few layers, discriminative lr and weight decays, and training for much more gradient steps confidently.
", "votes": 1, "replies": [] } ] } ] }, { "id": 3379946, "author_name": "", "author_url": "", "post_date": "2025-12-21T08:44:10.957000", "content": "", "votes": -1, "replies": [] }, { "id": 3379943, "author_name": "", "author_url": "", "post_date": "2025-12-21T08:41:57.487000", "content": "", "votes": 0, "replies": [] } ], "raw_markdown_by_id": { "3379744": "- The main discussion about \"new ideas to improve\" is now too long\n\n- The current discussion page should let us share more ideas, focusing on the meaning of current leaderboard scores.\n\n----- Start -----\n\nMy best guess:\n\n- Participants with scores around 0.66 have a good chance of actually winning the competition. This score is extremely hard to achieve without any additional tricks. \n\n- Scores like 0.76, 0.77 probably use postprocessing which works well with the public leaderboard. An example I can think of is shifting the mean of predictions.\n\n- Now the question is whether we should trust our 0.66 scores for private test scoring or try leaderboard probing? Your own ideas go here... ", "3379855": "Actually a truly simple implementation can achieve LB 0.74+. It's hard to say is 0.03 diff critical, but I believe 0.10+ diff is. In my case, a LB 0.75+ model's performance is always better than LB 0.65-. And by the way, trying to achieve higher public LB score is also an interesting part in contests.", "3379773": "It's difficult to tell ", "3379763": "I agree，haha.😄", "3379946": "", "3379943": "" } }