{
  "id": 641829,
  "title": "Persistent Submission Scoring Error in CSIRO Image2Biomass – Potential Backend Issue",
  "url": "/competitions/csiro-biomass/discussion/641829",
  "author_name": "",
  "post_date": "2025-11-27T10:58:43.021649900Z",
  "votes": -1,
  "comment_count": 2,
  "views": 0,
  "content": "<p>We've encountered recurring \"Submission Scoring Error\" failures in the CSIRO Image2Biomass Prediction competition, despite passing all format validations. Our diagnostic tests on a baseline notebook (\"biomass-dynamic-baseline-v1\") confirm: notebook executes successfully (19.5s on GPU P100), generates a valid long-format CSV with 5 rows for the dummy test image (ID1001187975), adheres to hierarchical constraints (sum error: 0.0), and includes variance (411.4033) to avoid constant prediction issues. Logs show clean runs with no NaNs, infs, or negatives. Timeline: Submissions succeeded ~10 days ago (e.g., values 18.0, 12.2, 2.3, 22.2, 37.7), but all fail since ~3-7 days ago, even varied baselines like (25, 8, 4, 18, 55). This rules out user-side errors in predictions, format, or code scaling (simulated for 800+ images). Evidence points to a Kaggle backend glitch in the R² scoring phase on the hidden test set. Similar reports in threads #628987, #639169, etc., suggest a platform-wide issue starting mid-November 2025.</p>",
  "messages": [
    {
      "id": "3350137",
      "postDate": "11/27/2025 10:58:43",
      "content": "<p>We've encountered recurring \"Submission Scoring Error\" failures in the CSIRO Image2Biomass Prediction competition, despite passing all format validations. Our diagnostic tests on a baseline notebook (\"biomass-dynamic-baseline-v1\") confirm: notebook executes successfully (19.5s on GPU P100), generates a valid long-format CSV with 5 rows for the dummy test image (ID1001187975), adheres to hierarchical constraints (sum error: 0.0), and includes variance (411.4033) to avoid constant prediction issues. Logs show clean runs with no NaNs, infs, or negatives. Timeline: Submissions succeeded ~10 days ago (e.g., values 18.0, 12.2, 2.3, 22.2, 37.7), but all fail since ~3-7 days ago, even varied baselines like (25, 8, 4, 18, 55). This rules out user-side errors in predictions, format, or code scaling (simulated for 800+ images). Evidence points to a Kaggle backend glitch in the R² scoring phase on the hidden test set. Similar reports in threads #628987, #639169, etc., suggest a platform-wide issue starting mid-November 2025.</p>",
      "rawMarkdown": "We've encountered recurring \"Submission Scoring Error\" failures in the CSIRO Image2Biomass Prediction competition, despite passing all format validations. Our diagnostic tests on a baseline notebook (\"biomass-dynamic-baseline-v1\") confirm: notebook executes successfully (19.5s on GPU P100), generates a valid long-format CSV with 5 rows for the dummy test image (ID1001187975), adheres to hierarchical constraints (sum error: 0.0), and includes variance (411.4033) to avoid constant prediction issues. Logs show clean runs with no NaNs, infs, or negatives. Timeline: Submissions succeeded ~10 days ago (e.g., values 18.0, 12.2, 2.3, 22.2, 37.7), but all fail since ~3-7 days ago, even varied baselines like (25, 8, 4, 18, 55). This rules out user-side errors in predictions, format, or code scaling (simulated for 800+ images). Evidence points to a Kaggle backend glitch in the R² scoring phase on the hidden test set. Similar reports in threads #628987, #639169, etc., suggest a platform-wide issue starting mid-November 2025.",
      "votes": null
    },
    {
      "id": "3350577",
      "postDate": "11/27/2025 16:47:38",
      "content": "<p>For clarification - you are running the exact same baseline notebook with zero changes that had previous successful LB score?</p>\n<p>With that notebook if you replace test.csv with train.csv it successfully predicts all 357 images?</p>\n<p>Since it's a baseline can you make it public and share a link here?  There may be something unique in you notebook, as based on the raw leader board data around 219 of the 1514 folks in this competition have had a submission today that appears to be successful.</p>",
      "rawMarkdown": "For clarification - you are running the exact same baseline notebook with zero changes that had previous successful LB score?\n\nWith that notebook if you replace test.csv with train.csv it successfully predicts all 357 images?\n\nSince it's a baseline can you make it public and share a link here?  There may be something unique in you notebook, as based on the raw leader board data around 219 of the 1514 folks in this competition have had a submission today that appears to be successful.",
      "votes": null
    },
    {
      "id": "3352127",
      "postDate": "11/29/2025 00:34:10",
      "content": "<p>solved based on my past successful submissions!: Code Execution Requirements</p>\n<ol>\n<li>Never Load Training Data in Submission Code Your working submission avoided train.csv loading. Loading large datasets during inference causes \"Notebook Threw Exception\" errors. Use training data only during model development, not in submission notebooks. Pre-compute statistics offline.</li>\n<li>Keep Code Minimal and Fast Simplicity prevents failures. Avoid unnecessary imports (numpy.random), complex operations, or heavy computations. The working submission had ~15 lines of code versus previous 100+ line versions. Less code = fewer failure points.</li>\n<li>Dynamic Image ID Extraction Always extract image IDs from test.csv at runtime: test_df['sample_id'].str.split('__').str[0].unique(). Never hardcode image IDs. The hidden test set has 800+ images; your code must scale automatically.</li>\n<li>Correct Hierarchical Constraint Critical: Dry_Total = Dry_Clover + Dry_Dead + Dry_Green (GDM is separate, NOT included in Total). This was verified from training data. Violating this causes scoring errors.</li>\n<li>Exact Alphabetical Order Targets must be: Clover, Dead, Green, Total, GDM. Match sample_submission.csv order exactly.</li>\n<li>No Random Operations Avoid np.random with seeds in submission code. Use deterministic predictions or pre-computed values.</li>\n<li>Error Handling Add try-except blocks for file operations. Check if files exist before reading.</li>\n<li>Memory Efficiency With 800+ images × 5 targets = 4000+ rows, use efficient data structures. Avoid loading unnecessary data into memory.\nFor your next real model submission: Train offline, save model weights, load only weights in submission code, generate predictions efficiently, enforce hierarchy in post-processing.</li>\n</ol>",
      "rawMarkdown": "solved based on my past successful submissions!: Code Execution Requirements\n1. Never Load Training Data in Submission Code Your working submission avoided train.csv loading. Loading large datasets during inference causes \"Notebook Threw Exception\" errors. Use training data only during model development, not in submission notebooks. Pre-compute statistics offline.\n2. Keep Code Minimal and Fast Simplicity prevents failures. Avoid unnecessary imports (numpy.random), complex operations, or heavy computations. The working submission had ~15 lines of code versus previous 100+ line versions. Less code = fewer failure points.\n3. Dynamic Image ID Extraction Always extract image IDs from test.csv at runtime: test_df['sample_id'].str.split('__').str[0].unique(). Never hardcode image IDs. The hidden test set has 800+ images; your code must scale automatically.\n4. Correct Hierarchical Constraint Critical: Dry_Total = Dry_Clover + Dry_Dead + Dry_Green (GDM is separate, NOT included in Total). This was verified from training data. Violating this causes scoring errors.\n5. Exact Alphabetical Order Targets must be: Clover, Dead, Green, Total, GDM. Match sample_submission.csv order exactly.\n6. No Random Operations Avoid np.random with seeds in submission code. Use deterministic predictions or pre-computed values.\n7. Error Handling Add try-except blocks for file operations. Check if files exist before reading.\n8. Memory Efficiency With 800+ images × 5 targets = 4000+ rows, use efficient data structures. Avoid loading unnecessary data into memory.\nFor your next real model submission: Train offline, save model weights, load only weights in submission code, generate predictions efficiently, enforce hierarchy in post-processing.",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 3350577,
      "author_name": "pcjimmmy",
      "author_url": "",
      "post_date": "11/27/2025 16:47:38",
      "content": "<p>For clarification - you are running the exact same baseline notebook with zero changes that had previous successful LB score?</p>\n<p>With that notebook if you replace test.csv with train.csv it successfully predicts all 357 images?</p>\n<p>Since it's a baseline can you make it public and share a link here?  There may be something unique in you notebook, as based on the raw leader board data around 219 of the 1514 folks in this competition have had a submission today that appears to be successful.</p>",
      "votes": null,
      "replies": [
        {
          "id": 3352127,
          "author_name": "mohammadjafarzamani",
          "author_url": "",
          "post_date": "11/29/2025 00:34:10",
          "content": "<p>solved based on my past successful submissions!: Code Execution Requirements</p>\n<ol>\n<li>Never Load Training Data in Submission Code Your working submission avoided train.csv loading. Loading large datasets during inference causes \"Notebook Threw Exception\" errors. Use training data only during model development, not in submission notebooks. Pre-compute statistics offline.</li>\n<li>Keep Code Minimal and Fast Simplicity prevents failures. Avoid unnecessary imports (numpy.random), complex operations, or heavy computations. The working submission had ~15 lines of code versus previous 100+ line versions. Less code = fewer failure points.</li>\n<li>Dynamic Image ID Extraction Always extract image IDs from test.csv at runtime: test_df['sample_id'].str.split('__').str[0].unique(). Never hardcode image IDs. The hidden test set has 800+ images; your code must scale automatically.</li>\n<li>Correct Hierarchical Constraint Critical: Dry_Total = Dry_Clover + Dry_Dead + Dry_Green (GDM is separate, NOT included in Total). This was verified from training data. Violating this causes scoring errors.</li>\n<li>Exact Alphabetical Order Targets must be: Clover, Dead, Green, Total, GDM. Match sample_submission.csv order exactly.</li>\n<li>No Random Operations Avoid np.random with seeds in submission code. Use deterministic predictions or pre-computed values.</li>\n<li>Error Handling Add try-except blocks for file operations. Check if files exist before reading.</li>\n<li>Memory Efficiency With 800+ images × 5 targets = 4000+ rows, use efficient data structures. Avoid loading unnecessary data into memory.\nFor your next real model submission: Train offline, save model weights, load only weights in submission code, generate predictions efficiently, enforce hierarchy in post-processing.</li>\n</ol>",
          "votes": null,
          "replies": []
        }
      ]
    }
  ],
  "raw_markdown_by_id": {
    "3350137": "We've encountered recurring \"Submission Scoring Error\" failures in the CSIRO Image2Biomass Prediction competition, despite passing all format validations. Our diagnostic tests on a baseline notebook (\"biomass-dynamic-baseline-v1\") confirm: notebook executes successfully (19.5s on GPU P100), generates a valid long-format CSV with 5 rows for the dummy test image (ID1001187975), adheres to hierarchical constraints (sum error: 0.0), and includes variance (411.4033) to avoid constant prediction issues. Logs show clean runs with no NaNs, infs, or negatives. Timeline: Submissions succeeded ~10 days ago (e.g., values 18.0, 12.2, 2.3, 22.2, 37.7), but all fail since ~3-7 days ago, even varied baselines like (25, 8, 4, 18, 55). This rules out user-side errors in predictions, format, or code scaling (simulated for 800+ images). Evidence points to a Kaggle backend glitch in the R² scoring phase on the hidden test set. Similar reports in threads #628987, #639169, etc., suggest a platform-wide issue starting mid-November 2025.",
    "3350577": "For clarification - you are running the exact same baseline notebook with zero changes that had previous successful LB score?\n\nWith that notebook if you replace test.csv with train.csv it successfully predicts all 357 images?\n\nSince it's a baseline can you make it public and share a link here?  There may be something unique in you notebook, as based on the raw leader board data around 219 of the 1514 folks in this competition have had a submission today that appears to be successful.",
    "3352127": "solved based on my past successful submissions!: Code Execution Requirements\n1. Never Load Training Data in Submission Code Your working submission avoided train.csv loading. Loading large datasets during inference causes \"Notebook Threw Exception\" errors. Use training data only during model development, not in submission notebooks. Pre-compute statistics offline.\n2. Keep Code Minimal and Fast Simplicity prevents failures. Avoid unnecessary imports (numpy.random), complex operations, or heavy computations. The working submission had ~15 lines of code versus previous 100+ line versions. Less code = fewer failure points.\n3. Dynamic Image ID Extraction Always extract image IDs from test.csv at runtime: test_df['sample_id'].str.split('__').str[0].unique(). Never hardcode image IDs. The hidden test set has 800+ images; your code must scale automatically.\n4. Correct Hierarchical Constraint Critical: Dry_Total = Dry_Clover + Dry_Dead + Dry_Green (GDM is separate, NOT included in Total). This was verified from training data. Violating this causes scoring errors.\n5. Exact Alphabetical Order Targets must be: Clover, Dead, Green, Total, GDM. Match sample_submission.csv order exactly.\n6. No Random Operations Avoid np.random with seeds in submission code. Use deterministic predictions or pre-computed values.\n7. Error Handling Add try-except blocks for file operations. Check if files exist before reading.\n8. Memory Efficiency With 800+ images × 5 targets = 4000+ rows, use efficient data structures. Avoid loading unnecessary data into memory.\nFor your next real model submission: Train offline, save model weights, load only weights in submission code, generate predictions efficiently, enforce hierarchy in post-processing."
  },
  "source": "meta"
}