{
  "id": 667179,
  "title": "Does Kaggle provide test.csv or test/images during inference?",
  "url": "/competitions/csiro-biomass/discussion/667179",
  "author_name": "",
  "post_date": "2026-01-11T12:54:16.489994Z",
  "votes": -2,
  "comment_count": 9,
  "views": 0,
  "content": "<p>Hello folks, </p>\n<p>I need some help from any of you working on this competetion. I'm new to kaggle platform and not understanding what actually happens behind the scene when I press the submit button.</p>\n<p>For this competetion especially, I'm in a intuition that kaggle will run the notebook with hidden data, like test.csv and test/images, provided the paths are correct. </p>\n<p>When i'm running the notebook on single image provided for validating the code it's working, but when I submit the code, it's throwing exception.</p>\n<p>I couldn't even trace where am I going wrong, where to check for code output etc.\n<a href=\"https://www.kaggle.com/code/jordan1980/dinov2-vitl14-features-cv5/edit/run/291259997\" target=\"_blank\">notebook is here</a> </p>\n<p>I appreciate your inputs. \nMany thanks in advance for your time.</p>",
  "messages": [
    {
      "id": "3389567",
      "postDate": "01/11/2026 12:54:16",
      "content": "<p>Hello folks, </p>\n<p>I need some help from any of you working on this competetion. I'm new to kaggle platform and not understanding what actually happens behind the scene when I press the submit button.</p>\n<p>For this competetion especially, I'm in a intuition that kaggle will run the notebook with hidden data, like test.csv and test/images, provided the paths are correct. </p>\n<p>When i'm running the notebook on single image provided for validating the code it's working, but when I submit the code, it's throwing exception.</p>\n<p>I couldn't even trace where am I going wrong, where to check for code output etc.\n<a href=\"https://www.kaggle.com/code/jordan1980/dinov2-vitl14-features-cv5/edit/run/291259997\" target=\"_blank\">notebook is here</a> </p>\n<p>I appreciate your inputs. \nMany thanks in advance for your time.</p>",
      "rawMarkdown": "Hello folks, \n\nI need some help from any of you working on this competetion. I'm new to kaggle platform and not understanding what actually happens behind the scene when I press the submit button.\n\nFor this competetion especially, I'm in a intuition that kaggle will run the notebook with hidden data, like test.csv and test/images, provided the paths are correct. \n\nWhen i'm running the notebook on single image provided for validating the code it's working, but when I submit the code, it's throwing exception.\n\nI couldn't even trace where am I going wrong, where to check for code output etc.\n[notebook is here](https://www.kaggle.com/code/jordan1980/dinov2-vitl14-features-cv5/edit/run/291259997) \n\nI appreciate your inputs. \nMany thanks in advance for your time.",
      "votes": null
    },
    {
      "id": "3389635",
      "postDate": "01/11/2026 16:01:20",
      "content": "<p>Your link is broken.  I also can't seem to find a shared notebook.</p>",
      "rawMarkdown": "Your link is broken.  I also can't seem to find a shared notebook.",
      "votes": null
    },
    {
      "id": "3389748",
      "postDate": "01/11/2026 20:16:01",
      "content": "<p>Those are provided during inference. Probably you have some other problem, e.g. handling more than 1 image, or wrong submission format.</p>",
      "rawMarkdown": "Those are provided during inference. Probably you have some other problem, e.g. handling more than 1 image, or wrong submission format.",
      "votes": null
    },
    {
      "id": "3389841",
      "postDate": "01/12/2026 04:08:27",
      "content": "<p><a href=\"https://www.kaggle.com/Jimmmy\" target=\"_blank\">@Jimmmy</a> and <a href=\"https://www.kaggle.com/tennogh\" target=\"_blank\">@tennogh</a> ,</p>\n<p>many thanks for your time, </p>\n<p>Can you please check the following code and tell me where is the problem.</p>\n<h1>===============================</h1>\n<h1>1. Imports &amp; Paths</h1>\n<h1>===============================</h1>\n<p>import os\nimport numpy as np\nimport pandas as pd\nimport torch\nimport tensorflow as tf</p>\n<p>from PIL import Image\nfrom torchvision import transforms\nfrom tqdm import tqdm\nfrom tensorflow.keras import layers\nfrom keras import ops</p>\n<h1>Kaggle paths</h1>\n<p>DATA_ROOT = \"/kaggle/input/csiro-biomass\"\nTEST_CSV  = f\"{DATA_ROOT}/test.csv\"\nSAMPLE_SUB = f\"{DATA_ROOT}/sample_submission.csv\"</p>\n<p>TEST_IMAGE_ROOT = DATA_ROOT\nMODEL_DIR = \"/kaggle/input/cv-5-fold-models\"<br>\nDINO_DIR  = \"/kaggle/input/dinovit/dinov2\"          # offline dinov2 repo + weights</p>\n<p>WORK_DIR = \"/kaggle/working\"\nFEATURE_DIR = f\"{WORK_DIR}/features_test\"\nos.makedirs(FEATURE_DIR, exist_ok=True)</p>\n<p>DEVICE = \"cuda\" if torch.cuda.is_available() else \"cpu\"</p>\n<h1>===============================</h1>\n<h1>2. Load test.csv (SOURCE OF TRUTH)</h1>\n<h1>===============================</h1>\n<p>test_df = pd.read_csv(TEST_CSV)</p>\n<p>test_image_paths = test_df[\"image_path\"].unique()</p>\n<p>print(\"Number of test images:\", len(test_image_paths))</p>\n<p>print(\"Test image paths:\", test_image_paths)</p>\n<h1>===============================</h1>\n<h1>3. Load DINOv2 ViT-L/14 (offline)</h1>\n<h1>===============================</h1>\n<p>model_dino = torch.hub.load(\n    DINO_DIR,\n    \"dinov2_vitl14\",\n    source=\"local\",\n    pretrained=False,\n    weights=os.path.join(DINO_DIR, \"dinov2_vitl14.pth\")\n)</p>\n<p>model_dino.eval().to(DEVICE)</p>\n<p><a href=\"https://www.kaggle.com/torch.no\" target=\"_blank\">@torch.no</a>_grad()\ndef extract_features(img_path):\n    img = Image.open(img_path).convert(\"RGB\")\n    x = transform(img).unsqueeze(0).to(DEVICE)\n    out = model_dino.forward_features(x)\n    return out[\"x_norm_patchtokens\"].cpu().numpy()[0]  # (256, 1024)</p>\n<h1>===============================</h1>\n<h1>4. Extract DINO features for test images</h1>\n<h1>===============================</h1>\n<p>for rel_path in tqdm(test_image_paths, desc=\"Extracting DINO features\"):\n    abs_path = os.path.join(TEST_IMAGE_ROOT, rel_path)\n    image_id = os.path.splitext(os.path.basename(rel_path))[0]</p>\n<pre><code>feats = extract_features(abs_path)\nnp.save(os.path.join(FEATURE_DIR, f\"{image_id}.npy\"), feats)\n</code></pre>\n<p>print(\"Saved test feature files:\", os.listdir(FEATURE_DIR))\nExtracting DINO features: 100%|██████████| 1/1 [00:00&lt;00:00,  1.28it/s]\nSaved test feature files: ['ID1001187975.npy']</p>\n<h1>===============================</h1>\n<h1>5. TF Dataset</h1>\n<h1>===============================</h1>\n<p>def make_test_dataset(feature_dir, batch_size=16):\n    files = sorted(<a target=\"_blank\">\n        os.path.join(feature_dir, f)\n        for f in os.listdir(feature_dir)\n        if f.endswith(\".npy\")\n    </a>)</p>\n<pre><code>def gen():\n    for f in files:\n        yield np.load(f).astype(\"float32\")\n\nds = tf.data.Dataset.from_generator(\n    gen,\n    output_signature=tf.TensorSpec((256, 1024), tf.float32)\n)\n\nids = [os.path.basename(f).replace(\".npy\", \"\") for f in files]\nreturn ds.batch(batch_size), ids\n</code></pre>\n<p>test_ds, test_ids = make_test_dataset(FEATURE_DIR)</p>\n<h1>===============================</h1>\n<h1>6. Custom AttentionPooling layer</h1>\n<h1>===============================</h1>\n<p><a href=\"https://www.kaggle.com/tf.keras.utils.register\" target=\"_blank\">@tf.keras.utils.register</a>_keras_serializable(package=\"Custom\")\nclass AttentionPooling(layers.Layer):\n    def <strong>init</strong>(self, <strong>kwargs):\n        super().<strong>init</strong>(</strong>kwargs)\n        self.score = layers.Dense(1)</p>\n<pre><code>def build(self, input_shape):\n    # input_shape = (batch, num_patches, feature_dim)\n    self.score.build(input_shape)\n    super().build(input_shape)\n\ndef call(self, x):\n    w = ops.softmax(self.score(x), axis=1)\n    return ops.sum(w * x, axis=1)\n\ndef get_config(self):\n    return super().get_config()\n</code></pre>\n<h1>===============================</h1>\n<h1>7. Load 5-fold models &amp; ensemble</h1>\n<h1>===============================</h1>\n<p>fold_models = []\nfor i in range(5):\n    m = tf.keras.models.load_model(\n        os.path.join(MODEL_DIR, f\"image_only_fold{i}.keras\"),\n        compile=False\n    )\n    fold_models.append(m)</p>\n<p>all_preds = [m.predict(test_ds, verbose=0) for m in fold_models]\nensemble_preds = np.mean(all_preds, axis=0)</p>\n<p>print(\"Ensemble preds shape:\", ensemble_preds.shape)</p>\n<p>calibration_params = np.array([\n    [0.9525, 1.3930],\n    [0.7887, 3.1789],\n    [0.8678, 1.7714],\n    [0.7900, 10.1653],\n    [0.8243, 5.3555]], \n    dtype=np.float32\n)</p>\n<p>calibrated_preds = np.zeros_like(ensemble_preds)</p>\n<p>for t in range(5):\n    a, b = calibration_params[t]\n    calibrated_preds[:, t] = a * ensemble_preds[:, t] + b</p>\n<p>calibrated_preds = np.clip(calibrated_preds, 0, None)\nprint(\"Min per target:\", calibrated_preds.min(axis=0))\nMin per target: [17.061354 21.137712 65.494606 98.899605 79.84999 ]</p>\n<h1>===============================</h1>\n<h1>9. Build submission (STRICT mapping)</h1>\n<h1>===============================</h1>\n<p>TRAIN_TARGET_ORDER = [\n    \"Dry_Clover_g\",\n    \"Dry_Dead_g\",\n    \"Dry_Green_g\",\n    \"Dry_Total_g\",\n    \"GDM_g\",\n]</p>\n<p>pred_map = {\n    img_id: {t: float(v) for t, v in zip(TRAIN_TARGET_ORDER, vec)}\n    for img_id, vec in zip(test_ids, calibrated_preds)\n}</p>\n<p>sample_sub = pd.read_csv(SAMPLE_SUB)</p>\n<p>out = []\nfor sid in sample_sub[\"sample_id\"]:\n    img_id, target_name = sid.split(\"__\")\n    out.append(pred_map[img_id][target_name])</p>\n<p>sample_sub[\"target\"] = out\nsample_sub.to_csv(\"submission.csv\", index=False)</p>\n<p>print(\"submission.csv written\")\nsubmission.csv written</p>",
      "rawMarkdown": "Jimmmy and @tennogh ,\n\nmany thanks for your time, \n\nCan you please check the following code and tell me where is the problem.\n# ===============================\n# 1. Imports & Paths\n# ===============================\nimport os\nimport numpy as np\nimport pandas as pd\nimport torch\nimport tensorflow as tf\n\nfrom PIL import Image\nfrom torchvision import transforms\nfrom tqdm import tqdm\nfrom tensorflow.keras import layers\nfrom keras import ops\n# Kaggle paths\nDATA_ROOT = \"/kaggle/input/csiro-biomass\"\nTEST_CSV  = f\"{DATA_ROOT}/test.csv\"\nSAMPLE_SUB = f\"{DATA_ROOT}/sample_submission.csv\"\n\nTEST_IMAGE_ROOT = DATA_ROOT\nMODEL_DIR = \"/kaggle/input/cv-5-fold-models\"       \nDINO_DIR  = \"/kaggle/input/dinovit/dinov2\"          # offline dinov2 repo + weights\n\nWORK_DIR = \"/kaggle/working\"\nFEATURE_DIR = f\"{WORK_DIR}/features_test\"\nos.makedirs(FEATURE_DIR, exist_ok=True)\n\nDEVICE = \"cuda\" if torch.cuda.is_available() else \"cpu\"\n# ===============================\n# 2. Load test.csv (SOURCE OF TRUTH)\n# ===============================\ntest_df = pd.read_csv(TEST_CSV)\n\ntest_image_paths = test_df[\"image_path\"].unique()\n\nprint(\"Number of test images:\", len(test_image_paths))\n\nprint(\"Test image paths:\", test_image_paths)\n\n\n# ===============================\n# 3. Load DINOv2 ViT-L/14 (offline)\n# ===============================\nmodel_dino = torch.hub.load(\n    DINO_DIR,\n    \"dinov2_vitl14\",\n    source=\"local\",\n    pretrained=False,\n    weights=os.path.join(DINO_DIR, \"dinov2_vitl14.pth\")\n)\n\nmodel_dino.eval().to(DEVICE)\n\n@torch.no_grad()\ndef extract_features(img_path):\n    img = Image.open(img_path).convert(\"RGB\")\n    x = transform(img).unsqueeze(0).to(DEVICE)\n    out = model_dino.forward_features(x)\n    return out[\"x_norm_patchtokens\"].cpu().numpy()[0]  # (256, 1024)\n\n# ===============================\n# 4. Extract DINO features for test images\n# ===============================\nfor rel_path in tqdm(test_image_paths, desc=\"Extracting DINO features\"):\n    abs_path = os.path.join(TEST_IMAGE_ROOT, rel_path)\n    image_id = os.path.splitext(os.path.basename(rel_path))[0]\n\n    feats = extract_features(abs_path)\n    np.save(os.path.join(FEATURE_DIR, f\"{image_id}.npy\"), feats)\n\nprint(\"Saved test feature files:\", os.listdir(FEATURE_DIR))\nExtracting DINO features: 100%|██████████| 1/1 [00:00<00:00,  1.28it/s]\nSaved test feature files: ['ID1001187975.npy']\n# ===============================\n# 5. TF Dataset\n# ===============================\ndef make_test_dataset(feature_dir, batch_size=16):\n    files = sorted([\n        os.path.join(feature_dir, f)\n        for f in os.listdir(feature_dir)\n        if f.endswith(\".npy\")\n    ])\n\n    def gen():\n        for f in files:\n            yield np.load(f).astype(\"float32\")\n\n    ds = tf.data.Dataset.from_generator(\n        gen,\n        output_signature=tf.TensorSpec((256, 1024), tf.float32)\n    )\n\n    ids = [os.path.basename(f).replace(\".npy\", \"\") for f in files]\n    return ds.batch(batch_size), ids\ntest_ds, test_ids = make_test_dataset(FEATURE_DIR)\n\n# ===============================\n# 6. Custom AttentionPooling layer\n# ===============================\n@tf.keras.utils.register_keras_serializable(package=\"Custom\")\nclass AttentionPooling(layers.Layer):\n    def __init__(self, **kwargs):\n        super().__init__(**kwargs)\n        self.score = layers.Dense(1)\n\n    def build(self, input_shape):\n        # input_shape = (batch, num_patches, feature_dim)\n        self.score.build(input_shape)\n        super().build(input_shape)\n\n    def call(self, x):\n        w = ops.softmax(self.score(x), axis=1)\n        return ops.sum(w * x, axis=1)\n\n    def get_config(self):\n        return super().get_config()\n\n# ===============================\n# 7. Load 5-fold models & ensemble\n# ===============================\nfold_models = []\nfor i in range(5):\n    m = tf.keras.models.load_model(\n        os.path.join(MODEL_DIR, f\"image_only_fold{i}.keras\"),\n        compile=False\n    )\n    fold_models.append(m)\n\nall_preds = [m.predict(test_ds, verbose=0) for m in fold_models]\nensemble_preds = np.mean(all_preds, axis=0)\n\nprint(\"Ensemble preds shape:\", ensemble_preds.shape)\n\ncalibration_params = np.array([\n    [0.9525, 1.3930],\n    [0.7887, 3.1789],\n    [0.8678, 1.7714],\n    [0.7900, 10.1653],\n    [0.8243, 5.3555]], \n    dtype=np.float32\n)\n\ncalibrated_preds = np.zeros_like(ensemble_preds)\n\nfor t in range(5):\n    a, b = calibration_params[t]\n    calibrated_preds[:, t] = a * ensemble_preds[:, t] + b\n\ncalibrated_preds = np.clip(calibrated_preds, 0, None)\nprint(\"Min per target:\", calibrated_preds.min(axis=0))\nMin per target: [17.061354 21.137712 65.494606 98.899605 79.84999 ]\n# ===============================\n# 9. Build submission (STRICT mapping)\n# ===============================\nTRAIN_TARGET_ORDER = [\n    \"Dry_Clover_g\",\n    \"Dry_Dead_g\",\n    \"Dry_Green_g\",\n    \"Dry_Total_g\",\n    \"GDM_g\",\n]\n\npred_map = {\n    img_id: {t: float(v) for t, v in zip(TRAIN_TARGET_ORDER, vec)}\n    for img_id, vec in zip(test_ids, calibrated_preds)\n}\n\nsample_sub = pd.read_csv(SAMPLE_SUB)\n\nout = []\nfor sid in sample_sub[\"sample_id\"]:\n    img_id, target_name = sid.split(\"__\")\n    out.append(pred_map[img_id][target_name])\n\nsample_sub[\"target\"] = out\nsample_sub.to_csv(\"submission.csv\", index=False)\n\nprint(\"submission.csv written\")\nsubmission.csv written",
      "votes": null
    },
    {
      "id": "3389850",
      "postDate": "01/12/2026 04:38:32",
      "content": "<p>Its best if you make your notebook public and than share a link here.  Just reading code is sometimes not an easy task.</p>\n<p>As noted in a number of discussion posts you cannot read the sample_submission.csv file.  It exists for the folder with the single test image that we see but for some reason it's not in the folder that kaggle switches to for running a submission for score.  Most of the notebooks I have helped fix with this error work ok by just replacing it with test.csv.</p>",
      "rawMarkdown": "Its best if you make your notebook public and than share a link here.  Just reading code is sometimes not an easy task.\n\nAs noted in a number of discussion posts you cannot read the sample_submission.csv file.  It exists for the folder with the single test image that we see but for some reason it's not in the folder that kaggle switches to for running a submission for score.  Most of the notebooks I have helped fix with this error work ok by just replacing it with test.csv.",
      "votes": null
    },
    {
      "id": "3389880",
      "postDate": "01/12/2026 06:05:19",
      "content": "<p>Hi,\njust made the notebook public.</p>\n<p><a href=\"https://www.kaggle.com/code/jordan1980/dinov2-vitl14-features-cv5\" target=\"_blank\">Notebook link</a>\nThank you</p>",
      "rawMarkdown": "Hi,\njust made the notebook public.\n\n[Notebook link](https://www.kaggle.com/code/jordan1980/dinov2-vitl14-features-cv5)\nThank you",
      "votes": null
    },
    {
      "id": "3390123",
      "postDate": "01/12/2026 16:06:10",
      "content": "<p>This version has you reading the sample_submission.csv file - that will not work.</p>\n<p>Any datasets also need to be made public when your seeking help.  We cannot run the code and find issues with private data sets.</p>",
      "rawMarkdown": "This version has you reading the sample_submission.csv file - that will not work.\n\nAny datasets also need to be made public when your seeking help.  We cannot run the code and find issues with private data sets.",
      "votes": null
    },
    {
      "id": "3390168",
      "postDate": "01/12/2026 17:25:56",
      "content": "<p>I'm sorry, </p>\n<p>Here is the link for cross validation<a href=\"https://www.kaggle.com/code/jordan1980/dinov2-vitl14-features-cv5\" target=\"_blank\">CV models</a>\nLink for feature extraction<a href=\"https://www.kaggle.com/code/jordan1980/dinov2-vitl14-features-cv5\" target=\"_blank\">dinovitl4</a>\n<a href=\"https://www.kaggle.com/code/jordan1980/dinov2-vitl14-features-cv5\" target=\"_blank\">Notebook</a></p>\n<p>Nothing else uploaded from local machine.\nThank you</p>",
      "rawMarkdown": "I'm sorry, \n\nHere is the link for cross validation[CV models](https://www.kaggle.com/code/jordan1980/dinov2-vitl14-features-cv5)\nLink for feature extraction[dinovitl4](https://www.kaggle.com/code/jordan1980/dinov2-vitl14-features-cv5)\n[Notebook](https://www.kaggle.com/code/jordan1980/dinov2-vitl14-features-cv5)\n\nNothing else uploaded from local machine.\nThank you",
      "votes": null
    },
    {
      "id": "3390169",
      "postDate": "01/12/2026 17:29:26",
      "content": "<p>When I go to the notebook I still see two private datasets.  I assume one of them is the model (Dino) and the other are your trained weights for the model.  </p>\n<p>Change the sample_submission.csv to test.csv and see if it will run before you bother to make the two private datasets public.</p>\n<p>NFG</p>\n<p><code>SAMPLE_SUB = f\"{DATA_ROOT}/sample_submission.csv\"</code></p>\n<p>GOOD - maybe :)</p>\n<p><code>SAMPLE_SUB = f\"{DATA_ROOT}/test.csv\"</code></p>",
      "rawMarkdown": "When I go to the notebook I still see two private datasets.  I assume one of them is the model (Dino) and the other are your trained weights for the model.  \n\nChange the sample_submission.csv to test.csv and see if it will run before you bother to make the two private datasets public.\n\nNFG\n\n`SAMPLE_SUB = f\"{DATA_ROOT}/sample_submission.csv\"`\n\nGOOD - maybe :)\n\n`SAMPLE_SUB = f\"{DATA_ROOT}/test.csv\"`",
      "votes": null
    },
    {
      "id": "3390489",
      "postDate": "01/13/2026 09:01:54",
      "content": "<p>Thanks a lot, got it worked after this change :)</p>",
      "rawMarkdown": "Thanks a lot, got it worked after this change :)",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 3389635,
      "author_name": "pcjimmmy",
      "author_url": "",
      "post_date": "01/11/2026 16:01:20",
      "content": "<p>Your link is broken.  I also can't seem to find a shared notebook.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 3389748,
      "author_name": "tennogh",
      "author_url": "",
      "post_date": "01/11/2026 20:16:01",
      "content": "<p>Those are provided during inference. Probably you have some other problem, e.g. handling more than 1 image, or wrong submission format.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 3389841,
      "author_name": "jordan1980",
      "author_url": "",
      "post_date": "01/12/2026 04:08:27",
      "content": "<p><a href=\"https://www.kaggle.com/Jimmmy\" target=\"_blank\">@Jimmmy</a> and <a href=\"https://www.kaggle.com/tennogh\" target=\"_blank\">@tennogh</a> ,</p>\n<p>many thanks for your time, </p>\n<p>Can you please check the following code and tell me where is the problem.</p>\n<h1>===============================</h1>\n<h1>1. Imports &amp; Paths</h1>\n<h1>===============================</h1>\n<p>import os\nimport numpy as np\nimport pandas as pd\nimport torch\nimport tensorflow as tf</p>\n<p>from PIL import Image\nfrom torchvision import transforms\nfrom tqdm import tqdm\nfrom tensorflow.keras import layers\nfrom keras import ops</p>\n<h1>Kaggle paths</h1>\n<p>DATA_ROOT = \"/kaggle/input/csiro-biomass\"\nTEST_CSV  = f\"{DATA_ROOT}/test.csv\"\nSAMPLE_SUB = f\"{DATA_ROOT}/sample_submission.csv\"</p>\n<p>TEST_IMAGE_ROOT = DATA_ROOT\nMODEL_DIR = \"/kaggle/input/cv-5-fold-models\"<br>\nDINO_DIR  = \"/kaggle/input/dinovit/dinov2\"          # offline dinov2 repo + weights</p>\n<p>WORK_DIR = \"/kaggle/working\"\nFEATURE_DIR = f\"{WORK_DIR}/features_test\"\nos.makedirs(FEATURE_DIR, exist_ok=True)</p>\n<p>DEVICE = \"cuda\" if torch.cuda.is_available() else \"cpu\"</p>\n<h1>===============================</h1>\n<h1>2. Load test.csv (SOURCE OF TRUTH)</h1>\n<h1>===============================</h1>\n<p>test_df = pd.read_csv(TEST_CSV)</p>\n<p>test_image_paths = test_df[\"image_path\"].unique()</p>\n<p>print(\"Number of test images:\", len(test_image_paths))</p>\n<p>print(\"Test image paths:\", test_image_paths)</p>\n<h1>===============================</h1>\n<h1>3. Load DINOv2 ViT-L/14 (offline)</h1>\n<h1>===============================</h1>\n<p>model_dino = torch.hub.load(\n    DINO_DIR,\n    \"dinov2_vitl14\",\n    source=\"local\",\n    pretrained=False,\n    weights=os.path.join(DINO_DIR, \"dinov2_vitl14.pth\")\n)</p>\n<p>model_dino.eval().to(DEVICE)</p>\n<p><a href=\"https://www.kaggle.com/torch.no\" target=\"_blank\">@torch.no</a>_grad()\ndef extract_features(img_path):\n    img = Image.open(img_path).convert(\"RGB\")\n    x = transform(img).unsqueeze(0).to(DEVICE)\n    out = model_dino.forward_features(x)\n    return out[\"x_norm_patchtokens\"].cpu().numpy()[0]  # (256, 1024)</p>\n<h1>===============================</h1>\n<h1>4. Extract DINO features for test images</h1>\n<h1>===============================</h1>\n<p>for rel_path in tqdm(test_image_paths, desc=\"Extracting DINO features\"):\n    abs_path = os.path.join(TEST_IMAGE_ROOT, rel_path)\n    image_id = os.path.splitext(os.path.basename(rel_path))[0]</p>\n<pre><code>feats = extract_features(abs_path)\nnp.save(os.path.join(FEATURE_DIR, f\"{image_id}.npy\"), feats)\n</code></pre>\n<p>print(\"Saved test feature files:\", os.listdir(FEATURE_DIR))\nExtracting DINO features: 100%|██████████| 1/1 [00:00&lt;00:00,  1.28it/s]\nSaved test feature files: ['ID1001187975.npy']</p>\n<h1>===============================</h1>\n<h1>5. TF Dataset</h1>\n<h1>===============================</h1>\n<p>def make_test_dataset(feature_dir, batch_size=16):\n    files = sorted(<a target=\"_blank\">\n        os.path.join(feature_dir, f)\n        for f in os.listdir(feature_dir)\n        if f.endswith(\".npy\")\n    </a>)</p>\n<pre><code>def gen():\n    for f in files:\n        yield np.load(f).astype(\"float32\")\n\nds = tf.data.Dataset.from_generator(\n    gen,\n    output_signature=tf.TensorSpec((256, 1024), tf.float32)\n)\n\nids = [os.path.basename(f).replace(\".npy\", \"\") for f in files]\nreturn ds.batch(batch_size), ids\n</code></pre>\n<p>test_ds, test_ids = make_test_dataset(FEATURE_DIR)</p>\n<h1>===============================</h1>\n<h1>6. Custom AttentionPooling layer</h1>\n<h1>===============================</h1>\n<p><a href=\"https://www.kaggle.com/tf.keras.utils.register\" target=\"_blank\">@tf.keras.utils.register</a>_keras_serializable(package=\"Custom\")\nclass AttentionPooling(layers.Layer):\n    def <strong>init</strong>(self, <strong>kwargs):\n        super().<strong>init</strong>(</strong>kwargs)\n        self.score = layers.Dense(1)</p>\n<pre><code>def build(self, input_shape):\n    # input_shape = (batch, num_patches, feature_dim)\n    self.score.build(input_shape)\n    super().build(input_shape)\n\ndef call(self, x):\n    w = ops.softmax(self.score(x), axis=1)\n    return ops.sum(w * x, axis=1)\n\ndef get_config(self):\n    return super().get_config()\n</code></pre>\n<h1>===============================</h1>\n<h1>7. Load 5-fold models &amp; ensemble</h1>\n<h1>===============================</h1>\n<p>fold_models = []\nfor i in range(5):\n    m = tf.keras.models.load_model(\n        os.path.join(MODEL_DIR, f\"image_only_fold{i}.keras\"),\n        compile=False\n    )\n    fold_models.append(m)</p>\n<p>all_preds = [m.predict(test_ds, verbose=0) for m in fold_models]\nensemble_preds = np.mean(all_preds, axis=0)</p>\n<p>print(\"Ensemble preds shape:\", ensemble_preds.shape)</p>\n<p>calibration_params = np.array([\n    [0.9525, 1.3930],\n    [0.7887, 3.1789],\n    [0.8678, 1.7714],\n    [0.7900, 10.1653],\n    [0.8243, 5.3555]], \n    dtype=np.float32\n)</p>\n<p>calibrated_preds = np.zeros_like(ensemble_preds)</p>\n<p>for t in range(5):\n    a, b = calibration_params[t]\n    calibrated_preds[:, t] = a * ensemble_preds[:, t] + b</p>\n<p>calibrated_preds = np.clip(calibrated_preds, 0, None)\nprint(\"Min per target:\", calibrated_preds.min(axis=0))\nMin per target: [17.061354 21.137712 65.494606 98.899605 79.84999 ]</p>\n<h1>===============================</h1>\n<h1>9. Build submission (STRICT mapping)</h1>\n<h1>===============================</h1>\n<p>TRAIN_TARGET_ORDER = [\n    \"Dry_Clover_g\",\n    \"Dry_Dead_g\",\n    \"Dry_Green_g\",\n    \"Dry_Total_g\",\n    \"GDM_g\",\n]</p>\n<p>pred_map = {\n    img_id: {t: float(v) for t, v in zip(TRAIN_TARGET_ORDER, vec)}\n    for img_id, vec in zip(test_ids, calibrated_preds)\n}</p>\n<p>sample_sub = pd.read_csv(SAMPLE_SUB)</p>\n<p>out = []\nfor sid in sample_sub[\"sample_id\"]:\n    img_id, target_name = sid.split(\"__\")\n    out.append(pred_map[img_id][target_name])</p>\n<p>sample_sub[\"target\"] = out\nsample_sub.to_csv(\"submission.csv\", index=False)</p>\n<p>print(\"submission.csv written\")\nsubmission.csv written</p>",
      "votes": null,
      "replies": [
        {
          "id": 3389850,
          "author_name": "pcjimmmy",
          "author_url": "",
          "post_date": "01/12/2026 04:38:32",
          "content": "<p>Its best if you make your notebook public and than share a link here.  Just reading code is sometimes not an easy task.</p>\n<p>As noted in a number of discussion posts you cannot read the sample_submission.csv file.  It exists for the folder with the single test image that we see but for some reason it's not in the folder that kaggle switches to for running a submission for score.  Most of the notebooks I have helped fix with this error work ok by just replacing it with test.csv.</p>",
          "votes": null,
          "replies": [
            {
              "id": 3389880,
              "author_name": "jordan1980",
              "author_url": "",
              "post_date": "01/12/2026 06:05:19",
              "content": "<p>Hi,\njust made the notebook public.</p>\n<p><a href=\"https://www.kaggle.com/code/jordan1980/dinov2-vitl14-features-cv5\" target=\"_blank\">Notebook link</a>\nThank you</p>",
              "votes": null,
              "replies": [
                {
                  "id": 3390123,
                  "author_name": "pcjimmmy",
                  "author_url": "",
                  "post_date": "01/12/2026 16:06:10",
                  "content": "<p>This version has you reading the sample_submission.csv file - that will not work.</p>\n<p>Any datasets also need to be made public when your seeking help.  We cannot run the code and find issues with private data sets.</p>",
                  "votes": null,
                  "replies": [
                    {
                      "id": 3390168,
                      "author_name": "jordan1980",
                      "author_url": "",
                      "post_date": "01/12/2026 17:25:56",
                      "content": "<p>I'm sorry, </p>\n<p>Here is the link for cross validation<a href=\"https://www.kaggle.com/code/jordan1980/dinov2-vitl14-features-cv5\" target=\"_blank\">CV models</a>\nLink for feature extraction<a href=\"https://www.kaggle.com/code/jordan1980/dinov2-vitl14-features-cv5\" target=\"_blank\">dinovitl4</a>\n<a href=\"https://www.kaggle.com/code/jordan1980/dinov2-vitl14-features-cv5\" target=\"_blank\">Notebook</a></p>\n<p>Nothing else uploaded from local machine.\nThank you</p>",
                      "votes": null,
                      "replies": [
                        {
                          "id": 3390169,
                          "author_name": "pcjimmmy",
                          "author_url": "",
                          "post_date": "01/12/2026 17:29:26",
                          "content": "<p>When I go to the notebook I still see two private datasets.  I assume one of them is the model (Dino) and the other are your trained weights for the model.  </p>\n<p>Change the sample_submission.csv to test.csv and see if it will run before you bother to make the two private datasets public.</p>\n<p>NFG</p>\n<p><code>SAMPLE_SUB = f\"{DATA_ROOT}/sample_submission.csv\"</code></p>\n<p>GOOD - maybe :)</p>\n<p><code>SAMPLE_SUB = f\"{DATA_ROOT}/test.csv\"</code></p>",
                          "votes": null,
                          "replies": [
                            {
                              "id": 3390489,
                              "author_name": "jordan1980",
                              "author_url": "",
                              "post_date": "01/13/2026 09:01:54",
                              "content": "<p>Thanks a lot, got it worked after this change :)</p>",
                              "votes": null,
                              "replies": []
                            }
                          ]
                        }
                      ]
                    }
                  ]
                }
              ]
            }
          ]
        }
      ]
    }
  ],
  "raw_markdown_by_id": {
    "3389567": "Hello folks, \n\nI need some help from any of you working on this competetion. I'm new to kaggle platform and not understanding what actually happens behind the scene when I press the submit button.\n\nFor this competetion especially, I'm in a intuition that kaggle will run the notebook with hidden data, like test.csv and test/images, provided the paths are correct. \n\nWhen i'm running the notebook on single image provided for validating the code it's working, but when I submit the code, it's throwing exception.\n\nI couldn't even trace where am I going wrong, where to check for code output etc.\n[notebook is here](https://www.kaggle.com/code/jordan1980/dinov2-vitl14-features-cv5/edit/run/291259997) \n\nI appreciate your inputs. \nMany thanks in advance for your time.",
    "3389635": "Your link is broken.  I also can't seem to find a shared notebook.",
    "3389748": "Those are provided during inference. Probably you have some other problem, e.g. handling more than 1 image, or wrong submission format.",
    "3389841": "Jimmmy and @tennogh ,\n\nmany thanks for your time, \n\nCan you please check the following code and tell me where is the problem.\n# ===============================\n# 1. Imports & Paths\n# ===============================\nimport os\nimport numpy as np\nimport pandas as pd\nimport torch\nimport tensorflow as tf\n\nfrom PIL import Image\nfrom torchvision import transforms\nfrom tqdm import tqdm\nfrom tensorflow.keras import layers\nfrom keras import ops\n# Kaggle paths\nDATA_ROOT = \"/kaggle/input/csiro-biomass\"\nTEST_CSV  = f\"{DATA_ROOT}/test.csv\"\nSAMPLE_SUB = f\"{DATA_ROOT}/sample_submission.csv\"\n\nTEST_IMAGE_ROOT = DATA_ROOT\nMODEL_DIR = \"/kaggle/input/cv-5-fold-models\"       \nDINO_DIR  = \"/kaggle/input/dinovit/dinov2\"          # offline dinov2 repo + weights\n\nWORK_DIR = \"/kaggle/working\"\nFEATURE_DIR = f\"{WORK_DIR}/features_test\"\nos.makedirs(FEATURE_DIR, exist_ok=True)\n\nDEVICE = \"cuda\" if torch.cuda.is_available() else \"cpu\"\n# ===============================\n# 2. Load test.csv (SOURCE OF TRUTH)\n# ===============================\ntest_df = pd.read_csv(TEST_CSV)\n\ntest_image_paths = test_df[\"image_path\"].unique()\n\nprint(\"Number of test images:\", len(test_image_paths))\n\nprint(\"Test image paths:\", test_image_paths)\n\n\n# ===============================\n# 3. Load DINOv2 ViT-L/14 (offline)\n# ===============================\nmodel_dino = torch.hub.load(\n    DINO_DIR,\n    \"dinov2_vitl14\",\n    source=\"local\",\n    pretrained=False,\n    weights=os.path.join(DINO_DIR, \"dinov2_vitl14.pth\")\n)\n\nmodel_dino.eval().to(DEVICE)\n\n@torch.no_grad()\ndef extract_features(img_path):\n    img = Image.open(img_path).convert(\"RGB\")\n    x = transform(img).unsqueeze(0).to(DEVICE)\n    out = model_dino.forward_features(x)\n    return out[\"x_norm_patchtokens\"].cpu().numpy()[0]  # (256, 1024)\n\n# ===============================\n# 4. Extract DINO features for test images\n# ===============================\nfor rel_path in tqdm(test_image_paths, desc=\"Extracting DINO features\"):\n    abs_path = os.path.join(TEST_IMAGE_ROOT, rel_path)\n    image_id = os.path.splitext(os.path.basename(rel_path))[0]\n\n    feats = extract_features(abs_path)\n    np.save(os.path.join(FEATURE_DIR, f\"{image_id}.npy\"), feats)\n\nprint(\"Saved test feature files:\", os.listdir(FEATURE_DIR))\nExtracting DINO features: 100%|██████████| 1/1 [00:00<00:00,  1.28it/s]\nSaved test feature files: ['ID1001187975.npy']\n# ===============================\n# 5. TF Dataset\n# ===============================\ndef make_test_dataset(feature_dir, batch_size=16):\n    files = sorted([\n        os.path.join(feature_dir, f)\n        for f in os.listdir(feature_dir)\n        if f.endswith(\".npy\")\n    ])\n\n    def gen():\n        for f in files:\n            yield np.load(f).astype(\"float32\")\n\n    ds = tf.data.Dataset.from_generator(\n        gen,\n        output_signature=tf.TensorSpec((256, 1024), tf.float32)\n    )\n\n    ids = [os.path.basename(f).replace(\".npy\", \"\") for f in files]\n    return ds.batch(batch_size), ids\ntest_ds, test_ids = make_test_dataset(FEATURE_DIR)\n\n# ===============================\n# 6. Custom AttentionPooling layer\n# ===============================\n@tf.keras.utils.register_keras_serializable(package=\"Custom\")\nclass AttentionPooling(layers.Layer):\n    def __init__(self, **kwargs):\n        super().__init__(**kwargs)\n        self.score = layers.Dense(1)\n\n    def build(self, input_shape):\n        # input_shape = (batch, num_patches, feature_dim)\n        self.score.build(input_shape)\n        super().build(input_shape)\n\n    def call(self, x):\n        w = ops.softmax(self.score(x), axis=1)\n        return ops.sum(w * x, axis=1)\n\n    def get_config(self):\n        return super().get_config()\n\n# ===============================\n# 7. Load 5-fold models & ensemble\n# ===============================\nfold_models = []\nfor i in range(5):\n    m = tf.keras.models.load_model(\n        os.path.join(MODEL_DIR, f\"image_only_fold{i}.keras\"),\n        compile=False\n    )\n    fold_models.append(m)\n\nall_preds = [m.predict(test_ds, verbose=0) for m in fold_models]\nensemble_preds = np.mean(all_preds, axis=0)\n\nprint(\"Ensemble preds shape:\", ensemble_preds.shape)\n\ncalibration_params = np.array([\n    [0.9525, 1.3930],\n    [0.7887, 3.1789],\n    [0.8678, 1.7714],\n    [0.7900, 10.1653],\n    [0.8243, 5.3555]], \n    dtype=np.float32\n)\n\ncalibrated_preds = np.zeros_like(ensemble_preds)\n\nfor t in range(5):\n    a, b = calibration_params[t]\n    calibrated_preds[:, t] = a * ensemble_preds[:, t] + b\n\ncalibrated_preds = np.clip(calibrated_preds, 0, None)\nprint(\"Min per target:\", calibrated_preds.min(axis=0))\nMin per target: [17.061354 21.137712 65.494606 98.899605 79.84999 ]\n# ===============================\n# 9. Build submission (STRICT mapping)\n# ===============================\nTRAIN_TARGET_ORDER = [\n    \"Dry_Clover_g\",\n    \"Dry_Dead_g\",\n    \"Dry_Green_g\",\n    \"Dry_Total_g\",\n    \"GDM_g\",\n]\n\npred_map = {\n    img_id: {t: float(v) for t, v in zip(TRAIN_TARGET_ORDER, vec)}\n    for img_id, vec in zip(test_ids, calibrated_preds)\n}\n\nsample_sub = pd.read_csv(SAMPLE_SUB)\n\nout = []\nfor sid in sample_sub[\"sample_id\"]:\n    img_id, target_name = sid.split(\"__\")\n    out.append(pred_map[img_id][target_name])\n\nsample_sub[\"target\"] = out\nsample_sub.to_csv(\"submission.csv\", index=False)\n\nprint(\"submission.csv written\")\nsubmission.csv written",
    "3389850": "Its best if you make your notebook public and than share a link here.  Just reading code is sometimes not an easy task.\n\nAs noted in a number of discussion posts you cannot read the sample_submission.csv file.  It exists for the folder with the single test image that we see but for some reason it's not in the folder that kaggle switches to for running a submission for score.  Most of the notebooks I have helped fix with this error work ok by just replacing it with test.csv.",
    "3389880": "Hi,\njust made the notebook public.\n\n[Notebook link](https://www.kaggle.com/code/jordan1980/dinov2-vitl14-features-cv5)\nThank you",
    "3390123": "This version has you reading the sample_submission.csv file - that will not work.\n\nAny datasets also need to be made public when your seeking help.  We cannot run the code and find issues with private data sets.",
    "3390168": "I'm sorry, \n\nHere is the link for cross validation[CV models](https://www.kaggle.com/code/jordan1980/dinov2-vitl14-features-cv5)\nLink for feature extraction[dinovitl4](https://www.kaggle.com/code/jordan1980/dinov2-vitl14-features-cv5)\n[Notebook](https://www.kaggle.com/code/jordan1980/dinov2-vitl14-features-cv5)\n\nNothing else uploaded from local machine.\nThank you",
    "3390169": "When I go to the notebook I still see two private datasets.  I assume one of them is the model (Dino) and the other are your trained weights for the model.  \n\nChange the sample_submission.csv to test.csv and see if it will run before you bother to make the two private datasets public.\n\nNFG\n\n`SAMPLE_SUB = f\"{DATA_ROOT}/sample_submission.csv\"`\n\nGOOD - maybe :)\n\n`SAMPLE_SUB = f\"{DATA_ROOT}/test.csv\"`",
    "3390489": "Thanks a lot, got it worked after this change :)"
  },
  "source": "meta"
}