{
  "id": 640451,
  "title": "Good CV (0.69) but leaderboard drops to –1000+ — need help",
  "url": "/competitions/csiro-biomass/discussion/640451",
  "author_name": "",
  "post_date": "2025-11-25T18:37:16.360668200Z",
  "votes": -1,
  "comment_count": 4,
  "views": 0,
  "content": "<p>My model trains well and gets stable validation scores around 0.70–0.72 on a single fold.\nExample:</p>\n<p>Best Val Score: 0.7208\nPer-target R²:\n  Dry_Total_g    : 0.6432\n  Dry_Green_g    : 0.7862\n  Dry_Dead_g     : 0.4109\n  Dry_Clover_g   : 0.7744\n  GDM_g          : 0.7433</p>\n<p>But when I submit the notebook, the public leaderboard drops to values like –1000.25, –1.45, etc.</p>\n<p>The big warning sign is the test predictions</p>\n<p>Here’s a sample from my submission:</p>\n<p>Dry_Clover_g    → 1000.0\nDry_Dead_g      → 4.22\nDry_Green_g     → 0.0\nDry_Total_g     → 1000.0\nGDM_g           → 1000.0</p>\n<p>Most values collapse to 0 or 1000, even though validation predictions look normal.</p>\n<p>My setup</p>\n<p>ResNet50 backbone (local pretrained weights)</p>\n<p>Custom head</p>\n<p>Training with LogMSELoss (model predicts original-scale grams, but loss is in log-space)</p>\n<p>Validation also uses log-space R² (matching competition metric)</p>\n<p>Inference applies:</p>\n<p>log_pred = model(x)\npred = np.expm1(log_pred)\npred = np.clip(pred, 0, 800)</p>\n<p>What could cause good CV but catastrophic submission scores?</p>\n<p>I’m considering:</p>\n<p>Incorrect model loading in the notebook (PyTorch weights_only issue?)</p>\n<p>Wrong log → exp pipeline during inference</p>\n<p>Train/test transform mismatch</p>\n<p>Output → target mapping error</p>\n<p>Something off in how predictions are clipped</p>\n<p>If anyone has seen this pattern or knows common pitfalls in this competition, I’d appreciate the help.</p>",
  "messages": [
    {
      "id": "3348329",
      "postDate": "11/25/2025 18:37:16",
      "content": "<p>My model trains well and gets stable validation scores around 0.70–0.72 on a single fold.\nExample:</p>\n<p>Best Val Score: 0.7208\nPer-target R²:\n  Dry_Total_g    : 0.6432\n  Dry_Green_g    : 0.7862\n  Dry_Dead_g     : 0.4109\n  Dry_Clover_g   : 0.7744\n  GDM_g          : 0.7433</p>\n<p>But when I submit the notebook, the public leaderboard drops to values like –1000.25, –1.45, etc.</p>\n<p>The big warning sign is the test predictions</p>\n<p>Here’s a sample from my submission:</p>\n<p>Dry_Clover_g    → 1000.0\nDry_Dead_g      → 4.22\nDry_Green_g     → 0.0\nDry_Total_g     → 1000.0\nGDM_g           → 1000.0</p>\n<p>Most values collapse to 0 or 1000, even though validation predictions look normal.</p>\n<p>My setup</p>\n<p>ResNet50 backbone (local pretrained weights)</p>\n<p>Custom head</p>\n<p>Training with LogMSELoss (model predicts original-scale grams, but loss is in log-space)</p>\n<p>Validation also uses log-space R² (matching competition metric)</p>\n<p>Inference applies:</p>\n<p>log_pred = model(x)\npred = np.expm1(log_pred)\npred = np.clip(pred, 0, 800)</p>\n<p>What could cause good CV but catastrophic submission scores?</p>\n<p>I’m considering:</p>\n<p>Incorrect model loading in the notebook (PyTorch weights_only issue?)</p>\n<p>Wrong log → exp pipeline during inference</p>\n<p>Train/test transform mismatch</p>\n<p>Output → target mapping error</p>\n<p>Something off in how predictions are clipped</p>\n<p>If anyone has seen this pattern or knows common pitfalls in this competition, I’d appreciate the help.</p>",
      "rawMarkdown": "My model trains well and gets stable validation scores around 0.70–0.72 on a single fold.\nExample:\n\nBest Val Score: 0.7208\nPer-target R²:\n  Dry_Total_g    : 0.6432\n  Dry_Green_g    : 0.7862\n  Dry_Dead_g     : 0.4109\n  Dry_Clover_g   : 0.7744\n  GDM_g          : 0.7433\n\n\nBut when I submit the notebook, the public leaderboard drops to values like –1000.25, –1.45, etc.\n\nThe big warning sign is the test predictions\n\nHere’s a sample from my submission:\n\nDry_Clover_g    → 1000.0\nDry_Dead_g      → 4.22\nDry_Green_g     → 0.0\nDry_Total_g     → 1000.0\nGDM_g           → 1000.0\n\n\nMost values collapse to 0 or 1000, even though validation predictions look normal.\n\nMy setup\n\nResNet50 backbone (local pretrained weights)\n\nCustom head\n\nTraining with LogMSELoss (model predicts original-scale grams, but loss is in log-space)\n\nValidation also uses log-space R² (matching competition metric)\n\nInference applies:\n\nlog_pred = model(x)\npred = np.expm1(log_pred)\npred = np.clip(pred, 0, 800)\n\nWhat could cause good CV but catastrophic submission scores?\n\nI’m considering:\n\nIncorrect model loading in the notebook (PyTorch weights_only issue?)\n\nWrong log → exp pipeline during inference\n\nTrain/test transform mismatch\n\nOutput → target mapping error\n\nSomething off in how predictions are clipped\n\nIf anyone has seen this pattern or knows common pitfalls in this competition, I’d appreciate the help.",
      "votes": null
    },
    {
      "id": "3348368",
      "postDate": "11/25/2025 19:16:46",
      "content": "<p>This statement seems wrong - Validation also uses log-space R² (matching competition metric)</p>\n<p>**The competition metric does not use log space.  **</p>\n<p>Just guessing that your code has some log confusion.  Unless you think you have a magic winner - make your notebook public and share it here.  Pretty hard to debug with the info you have shared.</p>",
      "rawMarkdown": "This statement seems wrong - Validation also uses log-space R² (matching competition metric)\n\n**The competition metric does not use log space.  **\n\nJust guessing that your code has some log confusion.  Unless you think you have a magic winner - make your notebook public and share it here.  Pretty hard to debug with the info you have shared.",
      "votes": null
    },
    {
      "id": "3348703",
      "postDate": "11/26/2025 06:13:12",
      "content": "<p>this is the notebook \n<a href=\"https://www.kaggle.com/code/bingos/csiro-image2biomass\" target=\"_blank\">https://www.kaggle.com/code/bingos/csiro-image2biomass</a></p>",
      "rawMarkdown": "this is the notebook \nhttps://www.kaggle.com/code/bingos/csiro-image2biomass",
      "votes": null
    },
    {
      "id": "3348932",
      "postDate": "11/26/2025 10:11:08",
      "content": "<p>When I ran it did not get great R2 values -0.18 on fold 1 and 0.51  on fold 2. I don't see anything close to 0.70, so would not be surprised if the LB score is less than 0.</p>\n<p>Will submit but heading to bed now so won't see the score for 8 hours.  </p>",
      "rawMarkdown": "When I ran it did not get great R2 values -0.18 on fold 1 and 0.51  on fold 2. I don't see anything close to 0.70, so would not be surprised if the LB score is less than 0.\n\nWill submit but heading to bed now so won't see the score for 8 hours.",
      "votes": null
    },
    {
      "id": "3349482",
      "postDate": "11/26/2025 16:37:26",
      "content": "<p>LB 0.35 - got r2 of 0.18 and 0.52 on the submission run.   That's pretty much what to expect.  Model works, just not a big enough lad to handle the job.</p>",
      "rawMarkdown": "LB 0.35 - got r2 of 0.18 and 0.52 on the submission run.   That's pretty much what to expect.  Model works, just not a big enough lad to handle the job.",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 3348368,
      "author_name": "pcjimmmy",
      "author_url": "",
      "post_date": "11/25/2025 19:16:46",
      "content": "<p>This statement seems wrong - Validation also uses log-space R² (matching competition metric)</p>\n<p>**The competition metric does not use log space.  **</p>\n<p>Just guessing that your code has some log confusion.  Unless you think you have a magic winner - make your notebook public and share it here.  Pretty hard to debug with the info you have shared.</p>",
      "votes": null,
      "replies": [
        {
          "id": 3348703,
          "author_name": "bingos",
          "author_url": "",
          "post_date": "11/26/2025 06:13:12",
          "content": "<p>this is the notebook \n<a href=\"https://www.kaggle.com/code/bingos/csiro-image2biomass\" target=\"_blank\">https://www.kaggle.com/code/bingos/csiro-image2biomass</a></p>",
          "votes": null,
          "replies": [
            {
              "id": 3348932,
              "author_name": "pcjimmmy",
              "author_url": "",
              "post_date": "11/26/2025 10:11:08",
              "content": "<p>When I ran it did not get great R2 values -0.18 on fold 1 and 0.51  on fold 2. I don't see anything close to 0.70, so would not be surprised if the LB score is less than 0.</p>\n<p>Will submit but heading to bed now so won't see the score for 8 hours.  </p>",
              "votes": null,
              "replies": [
                {
                  "id": 3349482,
                  "author_name": "pcjimmmy",
                  "author_url": "",
                  "post_date": "11/26/2025 16:37:26",
                  "content": "<p>LB 0.35 - got r2 of 0.18 and 0.52 on the submission run.   That's pretty much what to expect.  Model works, just not a big enough lad to handle the job.</p>",
                  "votes": null,
                  "replies": []
                }
              ]
            }
          ]
        }
      ]
    }
  ],
  "raw_markdown_by_id": {
    "3348329": "My model trains well and gets stable validation scores around 0.70–0.72 on a single fold.\nExample:\n\nBest Val Score: 0.7208\nPer-target R²:\n  Dry_Total_g    : 0.6432\n  Dry_Green_g    : 0.7862\n  Dry_Dead_g     : 0.4109\n  Dry_Clover_g   : 0.7744\n  GDM_g          : 0.7433\n\n\nBut when I submit the notebook, the public leaderboard drops to values like –1000.25, –1.45, etc.\n\nThe big warning sign is the test predictions\n\nHere’s a sample from my submission:\n\nDry_Clover_g    → 1000.0\nDry_Dead_g      → 4.22\nDry_Green_g     → 0.0\nDry_Total_g     → 1000.0\nGDM_g           → 1000.0\n\n\nMost values collapse to 0 or 1000, even though validation predictions look normal.\n\nMy setup\n\nResNet50 backbone (local pretrained weights)\n\nCustom head\n\nTraining with LogMSELoss (model predicts original-scale grams, but loss is in log-space)\n\nValidation also uses log-space R² (matching competition metric)\n\nInference applies:\n\nlog_pred = model(x)\npred = np.expm1(log_pred)\npred = np.clip(pred, 0, 800)\n\nWhat could cause good CV but catastrophic submission scores?\n\nI’m considering:\n\nIncorrect model loading in the notebook (PyTorch weights_only issue?)\n\nWrong log → exp pipeline during inference\n\nTrain/test transform mismatch\n\nOutput → target mapping error\n\nSomething off in how predictions are clipped\n\nIf anyone has seen this pattern or knows common pitfalls in this competition, I’d appreciate the help.",
    "3348368": "This statement seems wrong - Validation also uses log-space R² (matching competition metric)\n\n**The competition metric does not use log space.  **\n\nJust guessing that your code has some log confusion.  Unless you think you have a magic winner - make your notebook public and share it here.  Pretty hard to debug with the info you have shared.",
    "3348703": "this is the notebook \nhttps://www.kaggle.com/code/bingos/csiro-image2biomass",
    "3348932": "When I ran it did not get great R2 values -0.18 on fold 1 and 0.51  on fold 2. I don't see anything close to 0.70, so would not be surprised if the LB score is less than 0.\n\nWill submit but heading to bed now so won't see the score for 8 hours.",
    "3349482": "LB 0.35 - got r2 of 0.18 and 0.52 on the submission run.   That's pretty much what to expect.  Model works, just not a big enough lad to handle the job."
  },
  "source": "meta"
}