I've been stuck on this problem forever. My submissions all threw an exception every time. One of the suggestions was to substitute the test data with the train data - which I did in the notebook I submitted. I've also marked down the block of cells towards the end that worked (successful) and so ran my model back against the 'new' test images. It gave an r2 of 0.87 so that model must be doing something good. But why is the marked down code failing - throwing an exception every time
", "messages": [ { "id": "3383795", "postDate": "12/30/2025 21:19:24", "content": "I've been stuck on this problem forever. My submissions all threw an exception every time. One of the suggestions was to substitute the test data with the train data - which I did in the notebook I submitted. I've also marked down the block of cells towards the end that worked (successful) and so ran my model back against the 'new' test images. It gave an r2 of 0.87 so that model must be doing something good. But why is the marked down code failing - throwing an exception every time
", "rawMarkdown": "I've been stuck on this problem forever. My submissions all threw an exception every time. One of the suggestions was to substitute the test data with the train data - which I did in the notebook I submitted. I've also marked down the block of cells towards the end that worked (successful) and so ran my model back against the 'new' test images. It gave an r2 of 0.87 so that model must be doing something good. But why is the marked down code failing - throwing an exception every time", "votes": null }, { "id": "3383799", "postDate": "12/30/2025 21:46:27", "content": "Sounds like you did it wrong.
\nSubstitute the test with train.
\nRun the notebook - you can run it in the editor - Run All or Save Version and than look at it when done.
\nLook at the errors generated - almost certain there will be some. If there are none and all 357 images are predicted than you probably have the format wrong.
\nYou DON'T submit this notebook - you just run it.
\nMake it public and put a link here - pretty sure we can figure out the error.
", "rawMarkdown": "Sounds like you did it wrong.\n\nSubstitute the test with train.\n\nRun the notebook - you can run it in the editor - Run All or Save Version and than look at it when done.\n\nLook at the errors generated - almost certain there will be some. If there are none and all 357 images are predicted than you probably have the format wrong. \n\nYou DON'T submit this notebook - you just run it.\n\nMake it public and put a link here - pretty sure we can figure out the error.", "votes": null }, { "id": "3383800", "postDate": "12/30/2025 21:48:28", "content": "OK - I see your notebook - it helps if you put a link in the discussion topic.
", "rawMarkdown": "OK - I see your notebook - it helps if you put a link in the discussion topic.", "votes": null }, { "id": "3383803", "postDate": "12/30/2025 21:53:39", "content": "This competition a bit unique - the sample_submission.csv file very common in the years I have been here - for some reason in this competition the file is NOT available when kaggle runs the code in the hidden folder. This is the killer !!!! You need to change your code to avoid use of the sample_submission.csv file.
\ndf_sub = pd.read_csv('/kaggle/input/csiro-biomass/sample_submission.csv')
Thanks - I did not use df_sub. That's just the way I always start. I marked down the block of code I was running for submission with the failures. It shows this code\nsubmission = pd.merge(\n template[['sample_id']], \n pd.DataFrame(results), \n on='sample_id', \n how='left'\n).fillna({'target': 0.0}) \nI ran the train image test on the last blocs along with the r2 score. If you can see something wrong with the marked down code (the actual submission code which failed )- lmk. Sorry that the format for my code does not format in this block. Cheers!
", "rawMarkdown": "Thanks - I did not use df_sub. That's just the way I always start. I marked down the block of code I was running for submission with the failures. It shows this code\nsubmission = pd.merge(\n template[['sample_id']], \n pd.DataFrame(results), \n on='sample_id', \n how='left'\n).fillna({'target': 0.0}) \nI ran the train image test on the last blocs along with the r2 score. If you can see something wrong with the marked down code (the actual submission code which failed )- lmk. Sorry that the format for my code does not format in this block. Cheers!", "votes": null }, { "id": "3383850", "postDate": "12/30/2025 23:33:34", "content": "PC_Jimmmy - the code is public. It is named CSIRO_Biomass_4. The code I use in a submit run is the marked down part near the end. If you can provide guidance on the exception failures I'd be very thankful. I won't try again until I make some changes to that markdown block. Did you mean anything else about providing a link?
", "rawMarkdown": "PC_Jimmmy - the code is public. It is named CSIRO_Biomass_4. The code I use in a submit run is the marked down part near the end. If you can provide guidance on the exception failures I'd be very thankful. I won't try again until I make some changes to that markdown block. Did you mean anything else about providing a link?", "votes": null }, { "id": "3383852", "postDate": "12/30/2025 23:57:46", "content": "If you don't use it than just comment it out.
\nFor some reason I can't save your code. Going to take it to my local machine. Will hollar if I find something.
\nYou need to make the model also public for anyone to run the code completly.
\nRegarding link - when looking for help its best if you put a link to the notebook here in the discussion.
\nRegarding appearance of the code you put in the text - seems strange that one of the world's largest forces on the world wide web would have such a crappy text message app. I am often crussing when attempting to put code in one of these messages - it almost always looks bad.
", "rawMarkdown": "If you don't use it than just comment it out. \n\nFor some reason I can't save your code. Going to take it to my local machine. Will hollar if I find something.\n\nYou need to make the model also public for anyone to run the code completly.\n\nRegarding link - when looking for help its best if you put a link to the [notebook](https://www.kaggle.com/code/arosielle/csiro-biomass-4) here in the discussion.\n\nRegarding appearance of the code you put in the text - seems strange that one of the world's largest forces on the world wide web would have such a crappy text message app. I am often crussing when attempting to put code in one of these messages - it almost always looks bad.", "votes": null }, { "id": "3383856", "postDate": "12/31/2025 00:20:34", "content": "You also read the sample_submission.csv in your prediction code - you need to remove that one also.
", "rawMarkdown": "You also read the sample_submission.csv in your prediction code - you need to remove that one also.", "votes": null }, { "id": "3383860", "postDate": "12/31/2025 00:48:28", "content": "I could not figure out the error in your prediction code - it was easier to just re-write it with Gemini Pro. You need to change the folder - pretty sure you can't reach my Desktop. You also did some excessive changes to the working folder - I can't find where on my machine the submission.csv file ended up :) Do not go crazy on kaggle setting working folders, etc - they will almost always work with the defaults. Let us know if you can get it working. I had a bunch of different image files in my test folder so I needed the check for jpg and png.
\nI removed the model from your notebook and I can now save it. I made the changes below as well as killing the early read of the sample_submission.csv file. Will see if I can submit :)
\nimport glob\nimport os\nimport numpy as np\nimport pandas as pd\nimport torch\nfrom torch.utils.data import DataLoader\nfrom tqdm import tqdm\n\n# 1. SETUP PATHS & FILES\n# Check the directory from your previous notebook\nTEST_IMG_DIR = '/home/james/Desktop/csiro-biomass/test'\n\n# Gather files (Case insensitive check for both jpg and png)\ntest_files = glob.glob(os.path.join(TEST_IMG_DIR, '*.jpg')) + \\\n glob.glob(os.path.join(TEST_IMG_DIR, '*.JPG')) + \\\n glob.glob(os.path.join(TEST_IMG_DIR, '*.png')) + \\\n glob.glob(os.path.join(TEST_IMG_DIR, '*.PNG'))\n\ntest_files = sorted(list(set(test_files))) # Remove duplicates if any\n\nprint(f\"Found {len(test_files)} test images in {TEST_IMG_DIR}\")\n\nif len(test_files) == 0:\n raise ValueError(f\"CRITICAL ERROR: No images found in {TEST_IMG_DIR}. Check your path or file extensions!\")\n\n# 2. PREPARE DATASET\n# Re-initialize the dataset to be sure it uses the files we just found\ntest_ds = CSIROInferenceDataset(test_files, transform=valid_transform)\ntest_loader = DataLoader(test_ds, batch_size=32, shuffle=False, num_workers=2)\n\n# 3. RUN INFERENCE\n# model.eval()\nresults = [] # Initialize strictly before the loop\n\nprint(\"Starting inference loop...\")\n\nwith torch.no_grad():\n for imgs, img_ids in tqdm(test_loader, desc=\"Predicting\"):\n imgs = imgs.to(device)\n outputs = model(imgs)\n\n # Handle different output types (Dict vs Tensor)\n if isinstance(outputs, dict) and 'biomass' in outputs:\n preds = outputs['biomass'].cpu().numpy()\n else:\n preds = outputs.cpu().numpy()\n\n # Unscale\n # Create dummy array to match scaler's expected shape (usually 9 columns based on previous errors)\n dummy = np.zeros((preds.shape[0], 9))\n dummy[:, :5] = preds\n unscaled = scaler.inverse_transform(dummy)[:, :5]\n\n # Save results\n for i, img_id in enumerate(img_ids):\n for j, col in enumerate(TARGET_COLS):\n results.append({\n 'sample_id': f\"{img_id}__{col}\",\n 'target': max(0.0, float(unscaled[i, j]))\n })\n\n# 4. SAVE SUBMISSION\nif len(results) > 0:\n submission = pd.DataFrame(results)\n submission.to_csv('submission.csv', index=False)\n print(\"\\nSUCCESS: Inference complete.\")\n print(f\"Saved submission.csv with {len(submission)} rows.\")\n print(submission.head())\nelse:\n print(\"\\nFAILURE: Loop finished but 'results' is empty. Check your model output or dataloader.\")\nSubmission success 0.13 LB Run time is not too bad - it saved a best score at epoch 28 so you could easily ramp up the epochs. On my local it had the best at epoch 16 so I did not explore higher epoch numbers. Let me know if you can't get it going and would like me to make my code public.
", "rawMarkdown": "Submission success 0.13 LB Run time is not too bad - it saved a best score at epoch 28 so you could easily ramp up the epochs. On my local it had the best at epoch 16 so I did not explore higher epoch numbers. Let me know if you can't get it going and would like me to make my code public.", "votes": null } ], "comments": [ { "id": 3383799, "author_name": "pcjimmmy", "author_url": "", "post_date": "12/30/2025 21:46:27", "content": "Sounds like you did it wrong.
\nSubstitute the test with train.
\nRun the notebook - you can run it in the editor - Run All or Save Version and than look at it when done.
\nLook at the errors generated - almost certain there will be some. If there are none and all 357 images are predicted than you probably have the format wrong.
\nYou DON'T submit this notebook - you just run it.
\nMake it public and put a link here - pretty sure we can figure out the error.
", "votes": null, "replies": [ { "id": 3383800, "author_name": "pcjimmmy", "author_url": "", "post_date": "12/30/2025 21:48:28", "content": "OK - I see your notebook - it helps if you put a link in the discussion topic.
", "votes": null, "replies": [] }, { "id": 3383803, "author_name": "pcjimmmy", "author_url": "", "post_date": "12/30/2025 21:53:39", "content": "This competition a bit unique - the sample_submission.csv file very common in the years I have been here - for some reason in this competition the file is NOT available when kaggle runs the code in the hidden folder. This is the killer !!!! You need to change your code to avoid use of the sample_submission.csv file.
\ndf_sub = pd.read_csv('/kaggle/input/csiro-biomass/sample_submission.csv')
Thanks - I did not use df_sub. That's just the way I always start. I marked down the block of code I was running for submission with the failures. It shows this code\nsubmission = pd.merge(\n template[['sample_id']], \n pd.DataFrame(results), \n on='sample_id', \n how='left'\n).fillna({'target': 0.0}) \nI ran the train image test on the last blocs along with the r2 score. If you can see something wrong with the marked down code (the actual submission code which failed )- lmk. Sorry that the format for my code does not format in this block. Cheers!
", "votes": null, "replies": [ { "id": 3383850, "author_name": "arosielle", "author_url": "", "post_date": "12/30/2025 23:33:34", "content": "PC_Jimmmy - the code is public. It is named CSIRO_Biomass_4. The code I use in a submit run is the marked down part near the end. If you can provide guidance on the exception failures I'd be very thankful. I won't try again until I make some changes to that markdown block. Did you mean anything else about providing a link?
", "votes": null, "replies": [ { "id": 3383852, "author_name": "pcjimmmy", "author_url": "", "post_date": "12/30/2025 23:57:46", "content": "If you don't use it than just comment it out.
\nFor some reason I can't save your code. Going to take it to my local machine. Will hollar if I find something.
\nYou need to make the model also public for anyone to run the code completly.
\nRegarding link - when looking for help its best if you put a link to the notebook here in the discussion.
\nRegarding appearance of the code you put in the text - seems strange that one of the world's largest forces on the world wide web would have such a crappy text message app. I am often crussing when attempting to put code in one of these messages - it almost always looks bad.
", "votes": null, "replies": [ { "id": 3383856, "author_name": "pcjimmmy", "author_url": "", "post_date": "12/31/2025 00:20:34", "content": "You also read the sample_submission.csv in your prediction code - you need to remove that one also.
", "votes": null, "replies": [ { "id": 3383860, "author_name": "pcjimmmy", "author_url": "", "post_date": "12/31/2025 00:48:28", "content": "I could not figure out the error in your prediction code - it was easier to just re-write it with Gemini Pro. You need to change the folder - pretty sure you can't reach my Desktop. You also did some excessive changes to the working folder - I can't find where on my machine the submission.csv file ended up :) Do not go crazy on kaggle setting working folders, etc - they will almost always work with the defaults. Let us know if you can get it working. I had a bunch of different image files in my test folder so I needed the check for jpg and png.
\nI removed the model from your notebook and I can now save it. I made the changes below as well as killing the early read of the sample_submission.csv file. Will see if I can submit :)
\nimport glob\nimport os\nimport numpy as np\nimport pandas as pd\nimport torch\nfrom torch.utils.data import DataLoader\nfrom tqdm import tqdm\n\n# 1. SETUP PATHS & FILES\n# Check the directory from your previous notebook\nTEST_IMG_DIR = '/home/james/Desktop/csiro-biomass/test'\n\n# Gather files (Case insensitive check for both jpg and png)\ntest_files = glob.glob(os.path.join(TEST_IMG_DIR, '*.jpg')) + \\\n glob.glob(os.path.join(TEST_IMG_DIR, '*.JPG')) + \\\n glob.glob(os.path.join(TEST_IMG_DIR, '*.png')) + \\\n glob.glob(os.path.join(TEST_IMG_DIR, '*.PNG'))\n\ntest_files = sorted(list(set(test_files))) # Remove duplicates if any\n\nprint(f\"Found {len(test_files)} test images in {TEST_IMG_DIR}\")\n\nif len(test_files) == 0:\n raise ValueError(f\"CRITICAL ERROR: No images found in {TEST_IMG_DIR}. Check your path or file extensions!\")\n\n# 2. PREPARE DATASET\n# Re-initialize the dataset to be sure it uses the files we just found\ntest_ds = CSIROInferenceDataset(test_files, transform=valid_transform)\ntest_loader = DataLoader(test_ds, batch_size=32, shuffle=False, num_workers=2)\n\n# 3. RUN INFERENCE\n# model.eval()\nresults = [] # Initialize strictly before the loop\n\nprint(\"Starting inference loop...\")\n\nwith torch.no_grad():\n for imgs, img_ids in tqdm(test_loader, desc=\"Predicting\"):\n imgs = imgs.to(device)\n outputs = model(imgs)\n\n # Handle different output types (Dict vs Tensor)\n if isinstance(outputs, dict) and 'biomass' in outputs:\n preds = outputs['biomass'].cpu().numpy()\n else:\n preds = outputs.cpu().numpy()\n\n # Unscale\n # Create dummy array to match scaler's expected shape (usually 9 columns based on previous errors)\n dummy = np.zeros((preds.shape[0], 9))\n dummy[:, :5] = preds\n unscaled = scaler.inverse_transform(dummy)[:, :5]\n\n # Save results\n for i, img_id in enumerate(img_ids):\n for j, col in enumerate(TARGET_COLS):\n results.append({\n 'sample_id': f\"{img_id}__{col}\",\n 'target': max(0.0, float(unscaled[i, j]))\n })\n\n# 4. SAVE SUBMISSION\nif len(results) > 0:\n submission = pd.DataFrame(results)\n submission.to_csv('submission.csv', index=False)\n print(\"\\nSUCCESS: Inference complete.\")\n print(f\"Saved submission.csv with {len(submission)} rows.\")\n print(submission.head())\nelse:\n print(\"\\nFAILURE: Loop finished but 'results' is empty. Check your model output or dataloader.\")\nSubmission success 0.13 LB Run time is not too bad - it saved a best score at epoch 28 so you could easily ramp up the epochs. On my local it had the best at epoch 16 so I did not explore higher epoch numbers. Let me know if you can't get it going and would like me to make my code public.
", "votes": null, "replies": [] } ] } ] } ] } ] } ] } ] } ], "raw_markdown_by_id": { "3383795": "I've been stuck on this problem forever. My submissions all threw an exception every time. One of the suggestions was to substitute the test data with the train data - which I did in the notebook I submitted. I've also marked down the block of cells towards the end that worked (successful) and so ran my model back against the 'new' test images. It gave an r2 of 0.87 so that model must be doing something good. But why is the marked down code failing - throwing an exception every time", "3383799": "Sounds like you did it wrong.\n\nSubstitute the test with train.\n\nRun the notebook - you can run it in the editor - Run All or Save Version and than look at it when done.\n\nLook at the errors generated - almost certain there will be some. If there are none and all 357 images are predicted than you probably have the format wrong. \n\nYou DON'T submit this notebook - you just run it.\n\nMake it public and put a link here - pretty sure we can figure out the error.", "3383800": "OK - I see your notebook - it helps if you put a link in the discussion topic.", "3383803": "This competition a bit unique - the sample_submission.csv file very common in the years I have been here - for some reason in this competition the file is NOT available when kaggle runs the code in the hidden folder. This is the killer !!!! You need to change your code to avoid use of the sample_submission.csv file. \n\n`df_sub = pd.read_csv('/kaggle/input/csiro-biomass/sample_submission.csv')`", "3383835": "Thanks - I did not use df_sub. That's just the way I always start. I marked down the block of code I was running for submission with the failures. It shows this code\nsubmission = pd.merge(\n template[['sample_id']], \n pd.DataFrame(results), \n on='sample_id', \n how='left'\n).fillna({'target': 0.0}) \nI ran the train image test on the last blocs along with the r2 score. If you can see something wrong with the marked down code (the actual submission code which failed )- lmk. Sorry that the format for my code does not format in this block. Cheers!", "3383850": "PC_Jimmmy - the code is public. It is named CSIRO_Biomass_4. The code I use in a submit run is the marked down part near the end. If you can provide guidance on the exception failures I'd be very thankful. I won't try again until I make some changes to that markdown block. Did you mean anything else about providing a link?", "3383852": "If you don't use it than just comment it out. \n\nFor some reason I can't save your code. Going to take it to my local machine. Will hollar if I find something.\n\nYou need to make the model also public for anyone to run the code completly.\n\nRegarding link - when looking for help its best if you put a link to the [notebook](https://www.kaggle.com/code/arosielle/csiro-biomass-4) here in the discussion.\n\nRegarding appearance of the code you put in the text - seems strange that one of the world's largest forces on the world wide web would have such a crappy text message app. I am often crussing when attempting to put code in one of these messages - it almost always looks bad.", "3383856": "You also read the sample_submission.csv in your prediction code - you need to remove that one also.", "3383860": "I could not figure out the error in your prediction code - it was easier to just re-write it with Gemini Pro. You need to change the folder - pretty sure you can't reach my Desktop. You also did some excessive changes to the working folder - I can't find where on my machine the submission.csv file ended up :) Do not go crazy on kaggle setting working folders, etc - they will almost always work with the defaults. Let us know if you can get it working. I had a bunch of different image files in my test folder so I needed the check for jpg and png.\n\nI removed the model from your notebook and I can now save it. I made the changes below as well as killing the early read of the sample_submission.csv file. Will see if I can submit :)\n\n```python\n\nimport glob\nimport os\nimport numpy as np\nimport pandas as pd\nimport torch\nfrom torch.utils.data import DataLoader\nfrom tqdm import tqdm\n\n# 1. SETUP PATHS & FILES\n# Check the directory from your previous notebook\nTEST_IMG_DIR = '/home/james/Desktop/csiro-biomass/test'\n\n# Gather files (Case insensitive check for both jpg and png)\ntest_files = glob.glob(os.path.join(TEST_IMG_DIR, '*.jpg')) + \\\n glob.glob(os.path.join(TEST_IMG_DIR, '*.JPG')) + \\\n glob.glob(os.path.join(TEST_IMG_DIR, '*.png')) + \\\n glob.glob(os.path.join(TEST_IMG_DIR, '*.PNG'))\n\ntest_files = sorted(list(set(test_files))) # Remove duplicates if any\n\nprint(f\"Found {len(test_files)} test images in {TEST_IMG_DIR}\")\n\nif len(test_files) == 0:\n raise ValueError(f\"CRITICAL ERROR: No images found in {TEST_IMG_DIR}. Check your path or file extensions!\")\n\n# 2. PREPARE DATASET\n# Re-initialize the dataset to be sure it uses the files we just found\ntest_ds = CSIROInferenceDataset(test_files, transform=valid_transform)\ntest_loader = DataLoader(test_ds, batch_size=32, shuffle=False, num_workers=2)\n\n# 3. RUN INFERENCE\n# model.eval()\nresults = [] # Initialize strictly before the loop\n\nprint(\"Starting inference loop...\")\n\nwith torch.no_grad():\n for imgs, img_ids in tqdm(test_loader, desc=\"Predicting\"):\n imgs = imgs.to(device)\n outputs = model(imgs)\n \n # Handle different output types (Dict vs Tensor)\n if isinstance(outputs, dict) and 'biomass' in outputs:\n preds = outputs['biomass'].cpu().numpy()\n else:\n preds = outputs.cpu().numpy()\n \n # Unscale\n # Create dummy array to match scaler's expected shape (usually 9 columns based on previous errors)\n dummy = np.zeros((preds.shape[0], 9))\n dummy[:, :5] = preds\n unscaled = scaler.inverse_transform(dummy)[:, :5]\n \n # Save results\n for i, img_id in enumerate(img_ids):\n for j, col in enumerate(TARGET_COLS):\n results.append({\n 'sample_id': f\"{img_id}__{col}\",\n 'target': max(0.0, float(unscaled[i, j]))\n })\n\n# 4. SAVE SUBMISSION\nif len(results) > 0:\n submission = pd.DataFrame(results)\n submission.to_csv('submission.csv', index=False)\n print(\"\\nSUCCESS: Inference complete.\")\n print(f\"Saved submission.csv with {len(submission)} rows.\")\n print(submission.head())\nelse:\n print(\"\\nFAILURE: Loop finished but 'results' is empty. Check your model output or dataloader.\")\n```", "3383924": "Submission success 0.13 LB Run time is not too bad - it saved a best score at epoch 28 so you could easily ramp up the epochs. On my local it had the best at epoch 16 so I did not explore higher epoch numbers. Let me know if you can't get it going and would like me to make my code public." }, "source": "meta" }