{
  "id": 614431,
  "title": "Test image",
  "url": "/competitions/csiro-biomass/discussion/614431",
  "author_name": "",
  "post_date": "2025-11-04T00:07:08.849148300Z",
  "votes": 3,
  "comment_count": 4,
  "views": 0,
  "content": "<p>Why there is only one image for testing?</p>",
  "messages": [
    {
      "id": "3310938",
      "postDate": "11/04/2025 00:07:08",
      "content": "<p>Why there is only one image for testing?</p>",
      "rawMarkdown": "Why there is only one image for testing?",
      "votes": null
    },
    {
      "id": "3310955",
      "postDate": "11/04/2025 00:54:03",
      "content": "<p>One image is provided only to help you configure/write your notebook. Your notebook should assume that there will be multiple images in that directory and hence make records in submission.csv for each one of them.\nThen, when you actually submit the notebook, at that time, there will be more images there, hence your notebook can access them during submission/evaluation only.</p>",
      "rawMarkdown": "One image is provided only to help you configure/write your notebook. Your notebook should assume that there will be multiple images in that directory and hence make records in submission.csv for each one of them.\nThen, when you actually submit the notebook, at that time, there will be more images there, hence your notebook can access them during submission/evaluation only.",
      "votes": null
    },
    {
      "id": "3311051",
      "postDate": "11/04/2025 05:37:15",
      "content": "<p>That's because the public test set only contains one image; the rest are in the hidden test set. When you submit your code, the test set will be swapped for the hidden one to run inference and generate the final predictions.</p>",
      "rawMarkdown": "That's because the public test set only contains one image; the rest are in the hidden test set. When you submit your code, the test set will be swapped for the hidden one to run inference and generate the final predictions.",
      "votes": null
    },
    {
      "id": "3312279",
      "postDate": "11/06/2025 21:23:17",
      "content": "<p>Many thanks for clarification</p>",
      "rawMarkdown": "Many thanks for clarification",
      "votes": null
    },
    {
      "id": "3384504",
      "postDate": "01/01/2026 11:46:33",
      "content": "<p>Hi Komil, \nThank you for your advice.\nWhen I submit the notebook, it looks that the notebook processes only one test image. Does your notebook process multiple test image and could the notebook count the processes of the multiple images? Or is it right that the log shows only one process though it actually processes multiple images in the hidden manner? </p>\n<p>Relevant part of my output log (snipped for brevity):</p>\n<p>[train] images=357 in /kaggle/input/csiro-biomass/train\n… (many similar lines omitted)\n[train] image 352 / 357  id=ID972274220\n[train] image 353 / 357  id=ID975115267\n[train] image 354 / 357  id=ID978026131\n[train] image 355 / 357  id=ID980538882\n[train] image 356 / 357  id=ID980878870\n[train] image 357 / 357  id=ID983582017\nprocessed images: 357\n[train] saved ids -&gt; /kaggle/working/dinov2_cache/train_ids.csv\n[train] saved emb -&gt; /kaggle/working/dinov2_cache/train_emb__kaggle_input_dinov2-base-offline.npy\n[train] embedding shape = (357, 768) (should be N x 768 for dinov2-base)\n[test] device=cuda, num_gpus=2\n[test] images=1 in /kaggle/input/csiro-biomass/test\nprocessed images: 1\n[test] saved ids -&gt; /kaggle/working/dinov2_cache/test_ids.csv\n[test] saved emb -&gt; /kaggle/working/dinov2_cache/test_emb__kaggle_input_dinov2-base-offline.npy\n[test] embedding shape = (1, 768) (should be N x 768 for dinov2-base)</p>\n<p>when running below script.</p>\n<p>def extract_embeddings(img_dir: Path, out_prefix: str):\n    device = \"cuda\" if torch.cuda.is_available() else \"cpu\"\n    n_gpu = torch.cuda.device_count()\n    print(f\"[{out_prefix}] device={device}, num_gpus={n_gpu}\")</p>\n<pre><code>img_paths = sorted(img_dir.glob(\"*.jpg\"))\nprint(f\"[{out_prefix}] images={len(img_paths)} in {img_dir}\")\n\nprocessor = AutoImageProcessor.from_pretrained(MODEL_DIR)\nmodel = AutoModel.from_pretrained(MODEL_DIR).eval()\n\nif device == \"cuda\":\n    model = model.to(\"cuda:0\")\n\n    if n_gpu &gt;= 2:\n        model = torch.nn.DataParallel(model, device_ids=list(range(n_gpu)))\nelse:\n    model = model.to(\"cpu\")\n\nds = ImagePathDataset(img_paths)\n\ndl = DataLoader(\n    ds,\n    batch_size=BATCH_SIZE,\n    shuffle=False,\n    num_workers=NUM_WORKERS,\n    pin_memory=(device == \"cuda\"),\n    collate_fn=lambda b: my_collate(b, processor),\n)\n\nall_ids = []\nall_embs = []\n\npbar = tqdm(dl, desc=f\"Extract {out_prefix}\")\nfor ids, inputs in pbar:\n    # this shows the all id of the images in both training and test data.\n    for j, id_ in enumerate(ids):\n        print(f\"[{out_prefix}] image {len(all_ids)+j+1} / {len(ds)}  id={id_}\")\n\n    if device == \"cuda\":\n        inputs = {k: v.to(\"cuda:0\", non_blocking=True) for k, v in inputs.items()}\n\n        with torch.autocast(device_type=\"cuda\", dtype=torch.float16):\n            out = model(**inputs)\n    else:\n        out = model(**inputs)\n\n    # CLS token embedding: (B, hidden)\n    emb = out.last_hidden_state[:, 0, :].detach().cpu().numpy()\n    all_ids.extend(ids)\n    print(\"processed images:\", len(all_ids))\n    all_embs.append(emb)\n\nE = np.concatenate(all_embs, axis=0)\n\n# save\npd.Series(all_ids, name=\"id\").to_csv(OUT_DIR / f\"{out_prefix}_ids.csv\", index=False)\nnp.save(OUT_DIR / f\"{out_prefix}_emb_{MODEL_DIR.replace('/','_')}.npy\", E)\n\nprint(f\"[{out_prefix}] saved ids -&gt; {OUT_DIR / f'{out_prefix}_ids.csv'}\")\nprint(f\"[{out_prefix}] saved emb -&gt; {OUT_DIR / f'{out_prefix}_emb_{MODEL_DIR.replace('/','_')}.npy'}\")\nprint(f\"[{out_prefix}] embedding shape = {E.shape} (should be N x 768 for dinov2-base)\")\n</code></pre>\n<h1>execution</h1>\n<p>extract_embeddings(TRAIN_DIR, \"train\")\nextract_embeddings(TEST_DIR, \"test\")</p>",
      "rawMarkdown": "Hi Komil, \nThank you for your advice.\nWhen I submit the notebook, it looks that the notebook processes only one test image. Does your notebook process multiple test image and could the notebook count the processes of the multiple images? Or is it right that the log shows only one process though it actually processes multiple images in the hidden manner? \n\nRelevant part of my output log (snipped for brevity):\n\n[train] images=357 in /kaggle/input/csiro-biomass/train\n... (many similar lines omitted)\n[train] image 352 / 357  id=ID972274220\n[train] image 353 / 357  id=ID975115267\n[train] image 354 / 357  id=ID978026131\n[train] image 355 / 357  id=ID980538882\n[train] image 356 / 357  id=ID980878870\n[train] image 357 / 357  id=ID983582017\nprocessed images: 357\n[train] saved ids -> /kaggle/working/dinov2_cache/train_ids.csv\n[train] saved emb -> /kaggle/working/dinov2_cache/train_emb__kaggle_input_dinov2-base-offline.npy\n[train] embedding shape = (357, 768) (should be N x 768 for dinov2-base)\n[test] device=cuda, num_gpus=2\n[test] images=1 in /kaggle/input/csiro-biomass/test\nprocessed images: 1\n[test] saved ids -> /kaggle/working/dinov2_cache/test_ids.csv\n[test] saved emb -> /kaggle/working/dinov2_cache/test_emb__kaggle_input_dinov2-base-offline.npy\n[test] embedding shape = (1, 768) (should be N x 768 for dinov2-base)\n\nwhen running below script.\n\ndef extract_embeddings(img_dir: Path, out_prefix: str):\n    device = \"cuda\" if torch.cuda.is_available() else \"cpu\"\n    n_gpu = torch.cuda.device_count()\n    print(f\"[{out_prefix}] device={device}, num_gpus={n_gpu}\")\n\n\n    img_paths = sorted(img_dir.glob(\"*.jpg\"))\n    print(f\"[{out_prefix}] images={len(img_paths)} in {img_dir}\")\n\n    processor = AutoImageProcessor.from_pretrained(MODEL_DIR)\n    model = AutoModel.from_pretrained(MODEL_DIR).eval()\n\n    if device == \"cuda\":\n        model = model.to(\"cuda:0\")\n        \n        if n_gpu >= 2:\n            model = torch.nn.DataParallel(model, device_ids=list(range(n_gpu)))\n    else:\n        model = model.to(\"cpu\")\n\n    ds = ImagePathDataset(img_paths)\n \n    dl = DataLoader(\n        ds,\n        batch_size=BATCH_SIZE,\n        shuffle=False,\n        num_workers=NUM_WORKERS,\n        pin_memory=(device == \"cuda\"),\n        collate_fn=lambda b: my_collate(b, processor),\n    )\n\n    all_ids = []\n    all_embs = []\n\n    pbar = tqdm(dl, desc=f\"Extract {out_prefix}\")\n    for ids, inputs in pbar:\n        # this shows the all id of the images in both training and test data.\n        for j, id_ in enumerate(ids):\n            print(f\"[{out_prefix}] image {len(all_ids)+j+1} / {len(ds)}  id={id_}\")\n            \n        if device == \"cuda\":\n            inputs = {k: v.to(\"cuda:0\", non_blocking=True) for k, v in inputs.items()}\n        \n            with torch.autocast(device_type=\"cuda\", dtype=torch.float16):\n                out = model(**inputs)\n        else:\n            out = model(**inputs)\n\n        # CLS token embedding: (B, hidden)\n        emb = out.last_hidden_state[:, 0, :].detach().cpu().numpy()\n        all_ids.extend(ids)\n        print(\"processed images:\", len(all_ids))\n        all_embs.append(emb)\n\n    E = np.concatenate(all_embs, axis=0)\n\n    # save\n    pd.Series(all_ids, name=\"id\").to_csv(OUT_DIR / f\"{out_prefix}_ids.csv\", index=False)\n    np.save(OUT_DIR / f\"{out_prefix}_emb_{MODEL_DIR.replace('/','_')}.npy\", E)\n\n    print(f\"[{out_prefix}] saved ids -> {OUT_DIR / f'{out_prefix}_ids.csv'}\")\n    print(f\"[{out_prefix}] saved emb -> {OUT_DIR / f'{out_prefix}_emb_{MODEL_DIR.replace('/','_')}.npy'}\")\n    print(f\"[{out_prefix}] embedding shape = {E.shape} (should be N x 768 for dinov2-base)\")\n\n\n#execution\nextract_embeddings(TRAIN_DIR, \"train\")\nextract_embeddings(TEST_DIR, \"test\")",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 3310955,
      "author_name": "komilparmar",
      "author_url": "",
      "post_date": "11/04/2025 00:54:03",
      "content": "<p>One image is provided only to help you configure/write your notebook. Your notebook should assume that there will be multiple images in that directory and hence make records in submission.csv for each one of them.\nThen, when you actually submit the notebook, at that time, there will be more images there, hence your notebook can access them during submission/evaluation only.</p>",
      "votes": null,
      "replies": [
        {
          "id": 3312279,
          "author_name": "lding2016",
          "author_url": "",
          "post_date": "11/06/2025 21:23:17",
          "content": "<p>Many thanks for clarification</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 3384504,
          "author_name": "yuzurumizuno",
          "author_url": "",
          "post_date": "01/01/2026 11:46:33",
          "content": "<p>Hi Komil, \nThank you for your advice.\nWhen I submit the notebook, it looks that the notebook processes only one test image. Does your notebook process multiple test image and could the notebook count the processes of the multiple images? Or is it right that the log shows only one process though it actually processes multiple images in the hidden manner? </p>\n<p>Relevant part of my output log (snipped for brevity):</p>\n<p>[train] images=357 in /kaggle/input/csiro-biomass/train\n… (many similar lines omitted)\n[train] image 352 / 357  id=ID972274220\n[train] image 353 / 357  id=ID975115267\n[train] image 354 / 357  id=ID978026131\n[train] image 355 / 357  id=ID980538882\n[train] image 356 / 357  id=ID980878870\n[train] image 357 / 357  id=ID983582017\nprocessed images: 357\n[train] saved ids -&gt; /kaggle/working/dinov2_cache/train_ids.csv\n[train] saved emb -&gt; /kaggle/working/dinov2_cache/train_emb__kaggle_input_dinov2-base-offline.npy\n[train] embedding shape = (357, 768) (should be N x 768 for dinov2-base)\n[test] device=cuda, num_gpus=2\n[test] images=1 in /kaggle/input/csiro-biomass/test\nprocessed images: 1\n[test] saved ids -&gt; /kaggle/working/dinov2_cache/test_ids.csv\n[test] saved emb -&gt; /kaggle/working/dinov2_cache/test_emb__kaggle_input_dinov2-base-offline.npy\n[test] embedding shape = (1, 768) (should be N x 768 for dinov2-base)</p>\n<p>when running below script.</p>\n<p>def extract_embeddings(img_dir: Path, out_prefix: str):\n    device = \"cuda\" if torch.cuda.is_available() else \"cpu\"\n    n_gpu = torch.cuda.device_count()\n    print(f\"[{out_prefix}] device={device}, num_gpus={n_gpu}\")</p>\n<pre><code>img_paths = sorted(img_dir.glob(\"*.jpg\"))\nprint(f\"[{out_prefix}] images={len(img_paths)} in {img_dir}\")\n\nprocessor = AutoImageProcessor.from_pretrained(MODEL_DIR)\nmodel = AutoModel.from_pretrained(MODEL_DIR).eval()\n\nif device == \"cuda\":\n    model = model.to(\"cuda:0\")\n\n    if n_gpu &gt;= 2:\n        model = torch.nn.DataParallel(model, device_ids=list(range(n_gpu)))\nelse:\n    model = model.to(\"cpu\")\n\nds = ImagePathDataset(img_paths)\n\ndl = DataLoader(\n    ds,\n    batch_size=BATCH_SIZE,\n    shuffle=False,\n    num_workers=NUM_WORKERS,\n    pin_memory=(device == \"cuda\"),\n    collate_fn=lambda b: my_collate(b, processor),\n)\n\nall_ids = []\nall_embs = []\n\npbar = tqdm(dl, desc=f\"Extract {out_prefix}\")\nfor ids, inputs in pbar:\n    # this shows the all id of the images in both training and test data.\n    for j, id_ in enumerate(ids):\n        print(f\"[{out_prefix}] image {len(all_ids)+j+1} / {len(ds)}  id={id_}\")\n\n    if device == \"cuda\":\n        inputs = {k: v.to(\"cuda:0\", non_blocking=True) for k, v in inputs.items()}\n\n        with torch.autocast(device_type=\"cuda\", dtype=torch.float16):\n            out = model(**inputs)\n    else:\n        out = model(**inputs)\n\n    # CLS token embedding: (B, hidden)\n    emb = out.last_hidden_state[:, 0, :].detach().cpu().numpy()\n    all_ids.extend(ids)\n    print(\"processed images:\", len(all_ids))\n    all_embs.append(emb)\n\nE = np.concatenate(all_embs, axis=0)\n\n# save\npd.Series(all_ids, name=\"id\").to_csv(OUT_DIR / f\"{out_prefix}_ids.csv\", index=False)\nnp.save(OUT_DIR / f\"{out_prefix}_emb_{MODEL_DIR.replace('/','_')}.npy\", E)\n\nprint(f\"[{out_prefix}] saved ids -&gt; {OUT_DIR / f'{out_prefix}_ids.csv'}\")\nprint(f\"[{out_prefix}] saved emb -&gt; {OUT_DIR / f'{out_prefix}_emb_{MODEL_DIR.replace('/','_')}.npy'}\")\nprint(f\"[{out_prefix}] embedding shape = {E.shape} (should be N x 768 for dinov2-base)\")\n</code></pre>\n<h1>execution</h1>\n<p>extract_embeddings(TRAIN_DIR, \"train\")\nextract_embeddings(TEST_DIR, \"test\")</p>",
          "votes": null,
          "replies": []
        }
      ]
    },
    {
      "id": 3311051,
      "author_name": "gunianxue",
      "author_url": "",
      "post_date": "11/04/2025 05:37:15",
      "content": "<p>That's because the public test set only contains one image; the rest are in the hidden test set. When you submit your code, the test set will be swapped for the hidden one to run inference and generate the final predictions.</p>",
      "votes": null,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "3310938": "Why there is only one image for testing?",
    "3310955": "One image is provided only to help you configure/write your notebook. Your notebook should assume that there will be multiple images in that directory and hence make records in submission.csv for each one of them.\nThen, when you actually submit the notebook, at that time, there will be more images there, hence your notebook can access them during submission/evaluation only.",
    "3311051": "That's because the public test set only contains one image; the rest are in the hidden test set. When you submit your code, the test set will be swapped for the hidden one to run inference and generate the final predictions.",
    "3312279": "Many thanks for clarification",
    "3384504": "Hi Komil, \nThank you for your advice.\nWhen I submit the notebook, it looks that the notebook processes only one test image. Does your notebook process multiple test image and could the notebook count the processes of the multiple images? Or is it right that the log shows only one process though it actually processes multiple images in the hidden manner? \n\nRelevant part of my output log (snipped for brevity):\n\n[train] images=357 in /kaggle/input/csiro-biomass/train\n... (many similar lines omitted)\n[train] image 352 / 357  id=ID972274220\n[train] image 353 / 357  id=ID975115267\n[train] image 354 / 357  id=ID978026131\n[train] image 355 / 357  id=ID980538882\n[train] image 356 / 357  id=ID980878870\n[train] image 357 / 357  id=ID983582017\nprocessed images: 357\n[train] saved ids -> /kaggle/working/dinov2_cache/train_ids.csv\n[train] saved emb -> /kaggle/working/dinov2_cache/train_emb__kaggle_input_dinov2-base-offline.npy\n[train] embedding shape = (357, 768) (should be N x 768 for dinov2-base)\n[test] device=cuda, num_gpus=2\n[test] images=1 in /kaggle/input/csiro-biomass/test\nprocessed images: 1\n[test] saved ids -> /kaggle/working/dinov2_cache/test_ids.csv\n[test] saved emb -> /kaggle/working/dinov2_cache/test_emb__kaggle_input_dinov2-base-offline.npy\n[test] embedding shape = (1, 768) (should be N x 768 for dinov2-base)\n\nwhen running below script.\n\ndef extract_embeddings(img_dir: Path, out_prefix: str):\n    device = \"cuda\" if torch.cuda.is_available() else \"cpu\"\n    n_gpu = torch.cuda.device_count()\n    print(f\"[{out_prefix}] device={device}, num_gpus={n_gpu}\")\n\n\n    img_paths = sorted(img_dir.glob(\"*.jpg\"))\n    print(f\"[{out_prefix}] images={len(img_paths)} in {img_dir}\")\n\n    processor = AutoImageProcessor.from_pretrained(MODEL_DIR)\n    model = AutoModel.from_pretrained(MODEL_DIR).eval()\n\n    if device == \"cuda\":\n        model = model.to(\"cuda:0\")\n        \n        if n_gpu >= 2:\n            model = torch.nn.DataParallel(model, device_ids=list(range(n_gpu)))\n    else:\n        model = model.to(\"cpu\")\n\n    ds = ImagePathDataset(img_paths)\n \n    dl = DataLoader(\n        ds,\n        batch_size=BATCH_SIZE,\n        shuffle=False,\n        num_workers=NUM_WORKERS,\n        pin_memory=(device == \"cuda\"),\n        collate_fn=lambda b: my_collate(b, processor),\n    )\n\n    all_ids = []\n    all_embs = []\n\n    pbar = tqdm(dl, desc=f\"Extract {out_prefix}\")\n    for ids, inputs in pbar:\n        # this shows the all id of the images in both training and test data.\n        for j, id_ in enumerate(ids):\n            print(f\"[{out_prefix}] image {len(all_ids)+j+1} / {len(ds)}  id={id_}\")\n            \n        if device == \"cuda\":\n            inputs = {k: v.to(\"cuda:0\", non_blocking=True) for k, v in inputs.items()}\n        \n            with torch.autocast(device_type=\"cuda\", dtype=torch.float16):\n                out = model(**inputs)\n        else:\n            out = model(**inputs)\n\n        # CLS token embedding: (B, hidden)\n        emb = out.last_hidden_state[:, 0, :].detach().cpu().numpy()\n        all_ids.extend(ids)\n        print(\"processed images:\", len(all_ids))\n        all_embs.append(emb)\n\n    E = np.concatenate(all_embs, axis=0)\n\n    # save\n    pd.Series(all_ids, name=\"id\").to_csv(OUT_DIR / f\"{out_prefix}_ids.csv\", index=False)\n    np.save(OUT_DIR / f\"{out_prefix}_emb_{MODEL_DIR.replace('/','_')}.npy\", E)\n\n    print(f\"[{out_prefix}] saved ids -> {OUT_DIR / f'{out_prefix}_ids.csv'}\")\n    print(f\"[{out_prefix}] saved emb -> {OUT_DIR / f'{out_prefix}_emb_{MODEL_DIR.replace('/','_')}.npy'}\")\n    print(f\"[{out_prefix}] embedding shape = {E.shape} (should be N x 768 for dinov2-base)\")\n\n\n#execution\nextract_embeddings(TRAIN_DIR, \"train\")\nextract_embeddings(TEST_DIR, \"test\")"
  },
  "source": "meta"
}