Since the competition only has 1 file in the dummy test files, it is hard to know how much is the model predicting for each test files. However, I noticed that for that one particular dummy file, the models ive been training produces very similar results no matter how many architecture I tried. No matter how many models I tried, I can't seem to also break the 0.70 wall. One thing that came to my mind is \"Could it be that the domain is very different in test compared to train\". So I switched into a simpler model using dino and ridge regression and initially I got 0.58 on LB. By using the hypothesis, I multiplied each result with a factor of 1.05 and got and LB of 0.55. When I changed the factor into 0.95 and 0.90, I got an LB of 0.61 and 0.62. Maybe the target in test domain is much different than train and it is the reason why some models are struggling to get a high score in the LB despite having high score on CV.
", "messages": [ { "id": "3384852", "postDate": "01/02/2026 05:57:19", "content": "Since the competition only has 1 file in the dummy test files, it is hard to know how much is the model predicting for each test files. However, I noticed that for that one particular dummy file, the models ive been training produces very similar results no matter how many architecture I tried. No matter how many models I tried, I can't seem to also break the 0.70 wall. One thing that came to my mind is \"Could it be that the domain is very different in test compared to train\". So I switched into a simpler model using dino and ridge regression and initially I got 0.58 on LB. By using the hypothesis, I multiplied each result with a factor of 1.05 and got and LB of 0.55. When I changed the factor into 0.95 and 0.90, I got an LB of 0.61 and 0.62. Maybe the target in test domain is much different than train and it is the reason why some models are struggling to get a high score in the LB despite having high score on CV.
", "rawMarkdown": "Since the competition only has 1 file in the dummy test files, it is hard to know how much is the model predicting for each test files. However, I noticed that for that one particular dummy file, the models ive been training produces very similar results no matter how many architecture I tried. No matter how many models I tried, I can't seem to also break the 0.70 wall. One thing that came to my mind is \"Could it be that the domain is very different in test compared to train\". So I switched into a simpler model using dino and ridge regression and initially I got 0.58 on LB. By using the hypothesis, I multiplied each result with a factor of 1.05 and got and LB of 0.55. When I changed the factor into 0.95 and 0.90, I got an LB of 0.61 and 0.62. Maybe the target in test domain is much different than train and it is the reason why some models are struggling to get a high score in the LB despite having high score on CV.", "votes": null }, { "id": "3384873", "postDate": "01/02/2026 07:07:14", "content": "Let me try it. 😂 I’ve tried a ton of methods but still can’t improve the LB, and the CV is unstable.
", "rawMarkdown": "Let me try it. 😂 I’ve tried a ton of methods but still can’t improve the LB, and the CV is unstable.", "votes": null }, { "id": "3384888", "postDate": "01/02/2026 07:47:55", "content": "I think your post‑processing approach also makes sense, because when you multiply by a factor, larger values will have their distances reduced more. For example: 1* (1 - 0.95) = 0.05 but 100 * (1 - 0.95) = 5.
\nNote that RMSE measures the distance from the predicted value to the ground truth. So if multiplying your model’s outputs by 0.95 leads to an improvement, it suggests that the predicted values are higher than the ground truth.
\nThe means of Clover, Dead, and Green are quite small (6, 12, 26), and models often tend to predict values close to the mean, so it’s very likely that samples with large true values are being pulled down to values smaller than their actual values.
", "rawMarkdown": "I think your post‑processing approach also makes sense, because when you multiply by a factor, larger values will have their distances reduced more. For example: 1* (1 - 0.95) = 0.05 but 100 * (1 - 0.95) = 5.\n\nNote that RMSE measures the distance from the predicted value to the ground truth. So if multiplying your model’s outputs by 0.95 leads to an improvement, it suggests that the predicted values are higher than the ground truth. \n\nThe means of Clover, Dead, and Green are quite small (6, 12, 26), and models often tend to predict values close to the mean, so it’s very likely that samples with large true values are being pulled down to values smaller than their actual values.", "votes": null }, { "id": "3385011", "postDate": "01/02/2026 12:24:55", "content": "Yes, that's what I thought. How did it go for you ?
", "rawMarkdown": "Yes, that's what I thought. How did it go for you ?", "votes": null }, { "id": "3385046", "postDate": "01/02/2026 13:35:13", "content": "Magic multiplier like the Walmart comp all over again 🤣
\nBut this doesn't make sense in the context of this dataset 🤔
", "rawMarkdown": "Magic multiplier like the [Walmart comp](https://www.kaggle.com/competitions/m5-forecasting-uncertainty/overview) all over again 🤣 \n\nBut this doesn't make sense in the context of this dataset 🤔", "votes": null }, { "id": "3385054", "postDate": "01/02/2026 13:54:55", "content": "I still haven't checked thoroughly but here's my hunch. It is said that the data is taken on 2014-2017 on the paper. The training data is taken on 2015. Maybe the one on test is taken on the year where the biomass is much more or much less than the ones in 2015. Neural network are kinda lazy and tend to predict the mean right. If the NN is trained on the 2015 data, it would tend to predict the mean of 2015. If the mean of the test is much higher than the 2015, it would make sense to multiply the prediction by a certain factor
", "rawMarkdown": "I still haven't checked thoroughly but here's my hunch. It is said that the data is taken on 2014-2017 on the paper. The training data is taken on 2015. Maybe the one on test is taken on the year where the biomass is much more or much less than the ones in 2015. Neural network are kinda lazy and tend to predict the mean right. If the NN is trained on the 2015 data, it would tend to predict the mean of 2015. If the mean of the test is much higher than the 2015, it would make sense to multiply the prediction by a certain factor", "votes": null }, { "id": "3385058", "postDate": "01/02/2026 14:00:10", "content": "The organizers have said the split is not random and while the biomass distribution and states are all represented in the test set, there are some holdout dates.
\nIf you look carefully, you will notice all the months of the year have train data, except for two. You will also notice that for one season, we only have train data for one state. You will also notice that clover is almost completely 0 for that unique season. This may explain your factor being less than 1. It also may explain why stratified group fold or group fold CVs tend to moderately correlate with lb scores, to a point at least.
\nI've been trying to get a model over 0.7 without exploiting anything about the test distribution. I'm not super interested in trying to specifically do better in the months I suspect are held out, but it's probably helpful for maximum LB score.
", "rawMarkdown": "The organizers have said the split is not random and while the biomass distribution and states are all represented in the test set, there are some holdout dates. \n\nIf you look carefully, you will notice all the months of the year have train data, except for two. You will also notice that for one season, we only have train data for one state. You will also notice that clover is almost completely 0 for that unique season. This may explain your factor being less than 1. It also may explain why stratified group fold or group fold CVs tend to moderately correlate with lb scores, to a point at least.\n\nI've been trying to get a model over 0.7 without exploiting anything about the test distribution. I'm not super interested in trying to specifically do better in the months I suspect are held out, but it's probably helpful for maximum LB score.", "votes": null }, { "id": "3385060", "postDate": "01/02/2026 14:02:18", "content": "If you look carefully, you will notice all the months of the year have train data, except for two.\nalso entire year 2014, 2016, 2017. Not in train at all.
", "rawMarkdown": "```\nIf you look carefully, you will notice all the months of the year have train data, except for two.\n```\n\nalso entire year 2014, 2016, 2017. Not in train at all.", "votes": null }, { "id": "3385061", "postDate": "01/02/2026 14:04:37", "content": "If the NN is trained on the 2015 data, it would tend to predict the mean of 2015\nagreed. well done to our models then, it memorized the training data 😂
", "rawMarkdown": "```\nIf the NN is trained on the 2015 data, it would tend to predict the mean of 2015\n```\nagreed. well done to our models then, it memorized the training data 😂", "votes": null }, { "id": "3385077", "postDate": "01/02/2026 14:16:27", "content": "This is genuinely a hard problem.
\nTBH with the limited data set, non-uniformity of pictures (human shadows, note cards, etc), and apparent time splitting I'm a little surprised our models are as good as they are! Pre-trained embeddings are pretty stellar these days and the data must have fairly reasonable signal.
", "rawMarkdown": "This is genuinely a hard problem.\n\nTBH with the limited data set, non-uniformity of pictures (human shadows, note cards, etc), and apparent time splitting I'm a little surprised our models are as good as they are! Pre-trained embeddings are pretty stellar these days and the data must have fairly reasonable signal.", "votes": null }, { "id": "3385187", "postDate": "01/02/2026 17:20:27", "content": "I multiplied the model’s predicted values by 0.95, and the LB decreased slightly from 0.75 to 0.75 😅 🤣 🤤
", "rawMarkdown": "I multiplied the model’s predicted values by 0.95, and the LB decreased slightly from 0.75 to 0.75 😅 🤣 🤤", "votes": null }, { "id": "3385304", "postDate": "01/03/2026 00:40:20", "content": "Same for me - 0.73 to a lower 0.73 🤣
", "rawMarkdown": "Same for me - 0.73 to a lower 0.73 🤣", "votes": null }, { "id": "3385308", "postDate": "01/03/2026 00:55:18", "content": "You very likely need a different calibration curve for each of the five labels. Trying to build that now - if it works I will not tell you about it :)
", "rawMarkdown": "You very likely need a different calibration curve for each of the five labels. Trying to build that now - if it works I will not tell you about it :)", "votes": null }, { "id": "3385331", "postDate": "01/03/2026 03:40:29", "content": "This sounds like some kind of leaderboard probing. You can use some strategy like submit a mean value predict with factor and binary search the factor to probe public test dataset's distribution, and get higher public LB score by these infomation. But I don't think this method will work well in private LB, especially you can only get 2 digits info on LB.
", "rawMarkdown": "This sounds like some kind of leaderboard probing. You can use some strategy like submit a mean value predict with factor and binary search the factor to probe public test dataset's distribution, and get higher public LB score by these infomation. But I don't think this method will work well in private LB, especially you can only get 2 digits info on LB.", "votes": null }, { "id": "3385335", "postDate": "01/03/2026 04:00:44", "content": "Yes exactly like that. I've been looking some of your comments on using a very simple trick that can get an LB of 0.74+. I thought that this kind of magic multiplier might be one of them. Any other hints maybe ?
", "rawMarkdown": "Yes exactly like that. I've been looking some of your comments on using a very simple trick that can get an LB of 0.74+. I thought that this kind of magic multiplier might be one of them. Any other hints maybe ?", "votes": null }, { "id": "3385349", "postDate": "01/03/2026 05:08:16", "content": "Nope. You can reach 0.74 without any probing.
", "rawMarkdown": "Nope. You can reach 0.74 without any probing.", "votes": null }, { "id": "3385350", "postDate": "01/03/2026 05:14:47", "content": "That's very impressive actually. I've tried so many models starting from simple MLP, patch level attention MLP, SVR, linewr regression, and many others but I still can't get through 0.7 haha. Makes me wonder what approach the top scorers including you are using
", "rawMarkdown": "That's very impressive actually. I've tried so many models starting from simple MLP, patch level attention MLP, SVR, linewr regression, and many others but I still can't get through 0.7 haha. Makes me wonder what approach the top scorers including you are using", "votes": null }, { "id": "3385375", "postDate": "01/03/2026 06:35:25", "content": "I multiplied the model’s predicted values by 1.03, and the LB decreased from 0.75 to 0.75
", "rawMarkdown": "I multiplied the model’s predicted values by 1.03, and the LB decreased from 0.75 to 0.75", "votes": null }, { "id": "3385682", "postDate": "01/03/2026 17:37:22", "content": "Two things likely at play.
\nMost of my models under-estimate the larger biomass images. An adjustment like you did might off set a model that either over or under predicts the larger biomass images. I think as a rule the poorer the model the more likely your factor change will help. A 0.70 model not get as big a band for the buck as a 0.55 model would get.
Since my current model is under predicting the larger biomass images I am very sensitive to the number of images in test that have big biomass - this means sensitive to the month of the year where the grass is at it's tallest and sensitive to the number of states in test since grass grows differently in terms of biomass by state. (which is why getting a good fold strategy is a pain)
All this would lead me to predict a big shakeup in final LB dependent on how kaggle split the two private test groups, and the risk or benefit of using a factor adjustment as you suggest. I sure hope kaggle did not slip in different years - rain and temperature data do suggest very likely a year to year difference in some states. I don't even want to think about that adjustment :)
", "rawMarkdown": "Two things likely at play.\n\n1. Most of my models under-estimate the larger biomass images. An adjustment like you did might off set a model that either over or under predicts the larger biomass images. I think as a rule the poorer the model the more likely your factor change will help. A 0.70 model not get as big a band for the buck as a 0.55 model would get.\n\n2. Since my current model is under predicting the larger biomass images I am very sensitive to the number of images in test that have big biomass - this means sensitive to the month of the year where the grass is at it's tallest and sensitive to the number of states in test since grass grows differently in terms of biomass by state. (which is why getting a good fold strategy is a pain)\n\nAll this would lead me to predict a big shakeup in final LB dependent on how kaggle split the two private test groups, and the risk or benefit of using a factor adjustment as you suggest. I sure hope kaggle did not slip in different years - rain and temperature data do suggest very likely a year to year difference in some states. I don't even want to think about that adjustment :)", "votes": null }, { "id": "3385689", "postDate": "01/03/2026 17:48:57", "content": "If you do want to go to a year to year adjustment - I just made three notebooks public that show how to get rain and temperature data. I am sharing them since I could not see any way to benefit from their use :)
", "rawMarkdown": "If you do want to go to a year to year adjustment - I just made three notebooks public that show how to get rain and temperature data. I am sharing them since I could not see any way to benefit from their use :)", "votes": null }, { "id": "3385698", "postDate": "01/03/2026 18:11:42", "content": "I built a linear calibration for each label. At the time it turned out the model I used it on was a 0.50 LB. The adjustment did raise it to 0.54. But when I tried it on my best 0.68 model it did pretty much nothing. Even revised the calibration to a quadratic and it did not help my best. Did not even try it on the 0.50 model. Probably will keep the quadratic in my prediction code but its not the magic I hoped for.
", "rawMarkdown": "I built a linear calibration for each label. At the time it turned out the model I used it on was a 0.50 LB. The adjustment did raise it to 0.54. But when I tried it on my best 0.68 model it did pretty much nothing. Even revised the calibration to a quadratic and it did not help my best. Did not even try it on the 0.50 model. Probably will keep the quadratic in my prediction code but its not the magic I hoped for.", "votes": null }, { "id": "3387740", "postDate": "01/07/2026 15:15:40", "content": "you are right.the models over 0.7 on lb get lower benefit ,but the models under 0.7 would get 0.1-0.2 benefit
", "rawMarkdown": "you are right.the models over 0.7 on lb get lower benefit ,but the models under 0.7 would get 0.1-0.2 benefit", "votes": null } ], "comments": [ { "id": 3384873, "author_name": "quan0095", "author_url": "", "post_date": "01/02/2026 07:07:14", "content": "Let me try it. 😂 I’ve tried a ton of methods but still can’t improve the LB, and the CV is unstable.
", "votes": null, "replies": [ { "id": 3384888, "author_name": "quan0095", "author_url": "", "post_date": "01/02/2026 07:47:55", "content": "I think your post‑processing approach also makes sense, because when you multiply by a factor, larger values will have their distances reduced more. For example: 1* (1 - 0.95) = 0.05 but 100 * (1 - 0.95) = 5.
\nNote that RMSE measures the distance from the predicted value to the ground truth. So if multiplying your model’s outputs by 0.95 leads to an improvement, it suggests that the predicted values are higher than the ground truth.
\nThe means of Clover, Dead, and Green are quite small (6, 12, 26), and models often tend to predict values close to the mean, so it’s very likely that samples with large true values are being pulled down to values smaller than their actual values.
", "votes": null, "replies": [ { "id": 3385011, "author_name": "darrenamadeusmartin", "author_url": "", "post_date": "01/02/2026 12:24:55", "content": "Yes, that's what I thought. How did it go for you ?
", "votes": null, "replies": [ { "id": 3385187, "author_name": "quan0095", "author_url": "", "post_date": "01/02/2026 17:20:27", "content": "I multiplied the model’s predicted values by 0.95, and the LB decreased slightly from 0.75 to 0.75 😅 🤣 🤤
", "votes": null, "replies": [ { "id": 3385304, "author_name": "yeoyunsianggeremie", "author_url": "", "post_date": "01/03/2026 00:40:20", "content": "Same for me - 0.73 to a lower 0.73 🤣
", "votes": null, "replies": [] } ] } ] }, { "id": 3385308, "author_name": "pcjimmmy", "author_url": "", "post_date": "01/03/2026 00:55:18", "content": "You very likely need a different calibration curve for each of the five labels. Trying to build that now - if it works I will not tell you about it :)
", "votes": null, "replies": [] }, { "id": 3385375, "author_name": "quan0095", "author_url": "", "post_date": "01/03/2026 06:35:25", "content": "I multiplied the model’s predicted values by 1.03, and the LB decreased from 0.75 to 0.75
", "votes": null, "replies": [] }, { "id": 3385698, "author_name": "pcjimmmy", "author_url": "", "post_date": "01/03/2026 18:11:42", "content": "I built a linear calibration for each label. At the time it turned out the model I used it on was a 0.50 LB. The adjustment did raise it to 0.54. But when I tried it on my best 0.68 model it did pretty much nothing. Even revised the calibration to a quadratic and it did not help my best. Did not even try it on the 0.50 model. Probably will keep the quadratic in my prediction code but its not the magic I hoped for.
", "votes": null, "replies": [] } ] } ] }, { "id": 3385046, "author_name": "yeoyunsianggeremie", "author_url": "", "post_date": "01/02/2026 13:35:13", "content": "Magic multiplier like the Walmart comp all over again 🤣
\nBut this doesn't make sense in the context of this dataset 🤔
", "votes": null, "replies": [ { "id": 3385054, "author_name": "darrenamadeusmartin", "author_url": "", "post_date": "01/02/2026 13:54:55", "content": "I still haven't checked thoroughly but here's my hunch. It is said that the data is taken on 2014-2017 on the paper. The training data is taken on 2015. Maybe the one on test is taken on the year where the biomass is much more or much less than the ones in 2015. Neural network are kinda lazy and tend to predict the mean right. If the NN is trained on the 2015 data, it would tend to predict the mean of 2015. If the mean of the test is much higher than the 2015, it would make sense to multiply the prediction by a certain factor
", "votes": null, "replies": [ { "id": 3385061, "author_name": "yeoyunsianggeremie", "author_url": "", "post_date": "01/02/2026 14:04:37", "content": "If the NN is trained on the 2015 data, it would tend to predict the mean of 2015\nagreed. well done to our models then, it memorized the training data 😂
", "votes": null, "replies": [] } ] } ] }, { "id": 3385058, "author_name": "chemdatafarmer", "author_url": "", "post_date": "01/02/2026 14:00:10", "content": "The organizers have said the split is not random and while the biomass distribution and states are all represented in the test set, there are some holdout dates.
\nIf you look carefully, you will notice all the months of the year have train data, except for two. You will also notice that for one season, we only have train data for one state. You will also notice that clover is almost completely 0 for that unique season. This may explain your factor being less than 1. It also may explain why stratified group fold or group fold CVs tend to moderately correlate with lb scores, to a point at least.
\nI've been trying to get a model over 0.7 without exploiting anything about the test distribution. I'm not super interested in trying to specifically do better in the months I suspect are held out, but it's probably helpful for maximum LB score.
", "votes": null, "replies": [ { "id": 3385060, "author_name": "yeoyunsianggeremie", "author_url": "", "post_date": "01/02/2026 14:02:18", "content": "If you look carefully, you will notice all the months of the year have train data, except for two.\nalso entire year 2014, 2016, 2017. Not in train at all.
", "votes": null, "replies": [ { "id": 3385077, "author_name": "chemdatafarmer", "author_url": "", "post_date": "01/02/2026 14:16:27", "content": "This is genuinely a hard problem.
\nTBH with the limited data set, non-uniformity of pictures (human shadows, note cards, etc), and apparent time splitting I'm a little surprised our models are as good as they are! Pre-trained embeddings are pretty stellar these days and the data must have fairly reasonable signal.
", "votes": null, "replies": [] } ] } ] }, { "id": 3385331, "author_name": "linkinpony", "author_url": "", "post_date": "01/03/2026 03:40:29", "content": "This sounds like some kind of leaderboard probing. You can use some strategy like submit a mean value predict with factor and binary search the factor to probe public test dataset's distribution, and get higher public LB score by these infomation. But I don't think this method will work well in private LB, especially you can only get 2 digits info on LB.
", "votes": null, "replies": [ { "id": 3385335, "author_name": "darrenamadeusmartin", "author_url": "", "post_date": "01/03/2026 04:00:44", "content": "Yes exactly like that. I've been looking some of your comments on using a very simple trick that can get an LB of 0.74+. I thought that this kind of magic multiplier might be one of them. Any other hints maybe ?
", "votes": null, "replies": [ { "id": 3385349, "author_name": "linkinpony", "author_url": "", "post_date": "01/03/2026 05:08:16", "content": "Nope. You can reach 0.74 without any probing.
", "votes": null, "replies": [ { "id": 3385350, "author_name": "darrenamadeusmartin", "author_url": "", "post_date": "01/03/2026 05:14:47", "content": "That's very impressive actually. I've tried so many models starting from simple MLP, patch level attention MLP, SVR, linewr regression, and many others but I still can't get through 0.7 haha. Makes me wonder what approach the top scorers including you are using
", "votes": null, "replies": [] } ] } ] } ] }, { "id": 3385682, "author_name": "pcjimmmy", "author_url": "", "post_date": "01/03/2026 17:37:22", "content": "Two things likely at play.
\nMost of my models under-estimate the larger biomass images. An adjustment like you did might off set a model that either over or under predicts the larger biomass images. I think as a rule the poorer the model the more likely your factor change will help. A 0.70 model not get as big a band for the buck as a 0.55 model would get.
Since my current model is under predicting the larger biomass images I am very sensitive to the number of images in test that have big biomass - this means sensitive to the month of the year where the grass is at it's tallest and sensitive to the number of states in test since grass grows differently in terms of biomass by state. (which is why getting a good fold strategy is a pain)
All this would lead me to predict a big shakeup in final LB dependent on how kaggle split the two private test groups, and the risk or benefit of using a factor adjustment as you suggest. I sure hope kaggle did not slip in different years - rain and temperature data do suggest very likely a year to year difference in some states. I don't even want to think about that adjustment :)
", "votes": null, "replies": [ { "id": 3385689, "author_name": "pcjimmmy", "author_url": "", "post_date": "01/03/2026 17:48:57", "content": "If you do want to go to a year to year adjustment - I just made three notebooks public that show how to get rain and temperature data. I am sharing them since I could not see any way to benefit from their use :)
", "votes": null, "replies": [] }, { "id": 3387740, "author_name": "", "author_url": "", "post_date": "01/07/2026 15:15:40", "content": "you are right.the models over 0.7 on lb get lower benefit ,but the models under 0.7 would get 0.1-0.2 benefit
", "votes": null, "replies": [] } ] } ], "raw_markdown_by_id": { "3384852": "Since the competition only has 1 file in the dummy test files, it is hard to know how much is the model predicting for each test files. However, I noticed that for that one particular dummy file, the models ive been training produces very similar results no matter how many architecture I tried. No matter how many models I tried, I can't seem to also break the 0.70 wall. One thing that came to my mind is \"Could it be that the domain is very different in test compared to train\". So I switched into a simpler model using dino and ridge regression and initially I got 0.58 on LB. By using the hypothesis, I multiplied each result with a factor of 1.05 and got and LB of 0.55. When I changed the factor into 0.95 and 0.90, I got an LB of 0.61 and 0.62. Maybe the target in test domain is much different than train and it is the reason why some models are struggling to get a high score in the LB despite having high score on CV.", "3384873": "Let me try it. 😂 I’ve tried a ton of methods but still can’t improve the LB, and the CV is unstable.", "3384888": "I think your post‑processing approach also makes sense, because when you multiply by a factor, larger values will have their distances reduced more. For example: 1* (1 - 0.95) = 0.05 but 100 * (1 - 0.95) = 5.\n\nNote that RMSE measures the distance from the predicted value to the ground truth. So if multiplying your model’s outputs by 0.95 leads to an improvement, it suggests that the predicted values are higher than the ground truth. \n\nThe means of Clover, Dead, and Green are quite small (6, 12, 26), and models often tend to predict values close to the mean, so it’s very likely that samples with large true values are being pulled down to values smaller than their actual values.", "3385011": "Yes, that's what I thought. How did it go for you ?", "3385046": "Magic multiplier like the [Walmart comp](https://www.kaggle.com/competitions/m5-forecasting-uncertainty/overview) all over again 🤣 \n\nBut this doesn't make sense in the context of this dataset 🤔", "3385054": "I still haven't checked thoroughly but here's my hunch. It is said that the data is taken on 2014-2017 on the paper. The training data is taken on 2015. Maybe the one on test is taken on the year where the biomass is much more or much less than the ones in 2015. Neural network are kinda lazy and tend to predict the mean right. If the NN is trained on the 2015 data, it would tend to predict the mean of 2015. If the mean of the test is much higher than the 2015, it would make sense to multiply the prediction by a certain factor", "3385058": "The organizers have said the split is not random and while the biomass distribution and states are all represented in the test set, there are some holdout dates. \n\nIf you look carefully, you will notice all the months of the year have train data, except for two. You will also notice that for one season, we only have train data for one state. You will also notice that clover is almost completely 0 for that unique season. This may explain your factor being less than 1. It also may explain why stratified group fold or group fold CVs tend to moderately correlate with lb scores, to a point at least.\n\nI've been trying to get a model over 0.7 without exploiting anything about the test distribution. I'm not super interested in trying to specifically do better in the months I suspect are held out, but it's probably helpful for maximum LB score.", "3385060": "```\nIf you look carefully, you will notice all the months of the year have train data, except for two.\n```\n\nalso entire year 2014, 2016, 2017. Not in train at all.", "3385061": "```\nIf the NN is trained on the 2015 data, it would tend to predict the mean of 2015\n```\nagreed. well done to our models then, it memorized the training data 😂", "3385077": "This is genuinely a hard problem.\n\nTBH with the limited data set, non-uniformity of pictures (human shadows, note cards, etc), and apparent time splitting I'm a little surprised our models are as good as they are! Pre-trained embeddings are pretty stellar these days and the data must have fairly reasonable signal.", "3385187": "I multiplied the model’s predicted values by 0.95, and the LB decreased slightly from 0.75 to 0.75 😅 🤣 🤤", "3385304": "Same for me - 0.73 to a lower 0.73 🤣", "3385308": "You very likely need a different calibration curve for each of the five labels. Trying to build that now - if it works I will not tell you about it :)", "3385331": "This sounds like some kind of leaderboard probing. You can use some strategy like submit a mean value predict with factor and binary search the factor to probe public test dataset's distribution, and get higher public LB score by these infomation. But I don't think this method will work well in private LB, especially you can only get 2 digits info on LB.", "3385335": "Yes exactly like that. I've been looking some of your comments on using a very simple trick that can get an LB of 0.74+. I thought that this kind of magic multiplier might be one of them. Any other hints maybe ?", "3385349": "Nope. You can reach 0.74 without any probing.", "3385350": "That's very impressive actually. I've tried so many models starting from simple MLP, patch level attention MLP, SVR, linewr regression, and many others but I still can't get through 0.7 haha. Makes me wonder what approach the top scorers including you are using", "3385375": "I multiplied the model’s predicted values by 1.03, and the LB decreased from 0.75 to 0.75", "3385682": "Two things likely at play.\n\n1. Most of my models under-estimate the larger biomass images. An adjustment like you did might off set a model that either over or under predicts the larger biomass images. I think as a rule the poorer the model the more likely your factor change will help. A 0.70 model not get as big a band for the buck as a 0.55 model would get.\n\n2. Since my current model is under predicting the larger biomass images I am very sensitive to the number of images in test that have big biomass - this means sensitive to the month of the year where the grass is at it's tallest and sensitive to the number of states in test since grass grows differently in terms of biomass by state. (which is why getting a good fold strategy is a pain)\n\nAll this would lead me to predict a big shakeup in final LB dependent on how kaggle split the two private test groups, and the risk or benefit of using a factor adjustment as you suggest. I sure hope kaggle did not slip in different years - rain and temperature data do suggest very likely a year to year difference in some states. I don't even want to think about that adjustment :)", "3385689": "If you do want to go to a year to year adjustment - I just made three notebooks public that show how to get rain and temperature data. I am sharing them since I could not see any way to benefit from their use :)", "3385698": "I built a linear calibration for each label. At the time it turned out the model I used it on was a 0.50 LB. The adjustment did raise it to 0.54. But when I tried it on my best 0.68 model it did pretty much nothing. Even revised the calibration to a quadratic and it did not help my best. Did not even try it on the 0.50 model. Probably will keep the quadratic in my prediction code but its not the magic I hoped for.", "3387740": "you are right.the models over 0.7 on lb get lower benefit ,but the models under 0.7 would get 0.1-0.2 benefit" }, "source": "meta" }