{
  "id": 570792,
  "title": "Split Datasets WildlifeDatasets",
  "url": "/competitions/animal-clef-2025/discussion/570792",
  "author_name": "",
  "post_date": "2025-03-30T19:31:17.631533800Z",
  "votes": 3,
  "comment_count": 3,
  "views": 0,
  "content": "<p>I'm trying to split the set, so I can run the model faster. I've tried a few ways but when I get to extractor() I get the error below.</p>\n<pre><code>ds_train_wild = AnimalCLEF2025(INPUT_DIR, transform=transform_display, load_label=)\nds_valid_wild = AnimalCLEF2025(INPUT_DIR, transform=transform_display, load_label=)\n\n wildlife_datasets  splits\n\nsplitter = splits.ClosedSetSplit()\nidx_train, idx_valid = splitter.split(ds_train_wild.df)[]\n\n\n\nds_train_wild.df = ds_train_wild.df.loc[idx_train]\nds_train_wild\n</code></pre>\n<pre><code>TypeError                                 Traceback (most recent call last)\n&lt;ipython-input-30-1f1f27d22870&gt; in &lt;cell line: 1&gt;()\n----&gt; 1 features_database = extractor(train_dataset_database)\n      2 features_database\n\n/usr/local/lib/python3.10/dist-packages/wildlife_tools/features/deep.py in __call__(self, dataset)\n     54         )\n     55         outputs = []\n---&gt; 56         for image, _ in tqdm(loader, mininterval=1, ncols=100):\n     57             with torch.no_grad():\n     58                 output = self.model(image.to(self.device))\n\n/usr/local/lib/python3.10/dist-packages/tqdm/std.py in __iter__(self)\n   1179 \n   1180         try:\n-&gt; 1181             for obj in iterable:\n   1182                 yield obj\n   1183                 # Update and possibly print the progressbar.\n\n/usr/local/lib/python3.10/dist-packages/torch/utils/data/dataloader.py in __next__(self)\n    699                 # TODO(https://github.com/pytorch/pytorch/issues/76750)\n    700                 self._reset()  # type: ignore[call-arg]\n--&gt; 701             data = self._next_data()\n    702             self._num_yielded += 1\n    703             if (\n\n/usr/local/lib/python3.10/dist-packages/torch/utils/data/dataloader.py in _next_data(self)\n    755     def _next_data(self):\n    756         index = self._next_index()  # may raise StopIteration\n--&gt; 757         data = self._dataset_fetcher.fetch(index)  # may raise StopIteration\n    758         if self._pin_memory:\n    759             data = _utils.pin_memory.pin_memory(data, self._pin_memory_device)\n\n/usr/local/lib/python3.10/dist-packages/torch/utils/data/_utils/fetch.py in fetch(self, possibly_batched_index)\n     53         else:\n     54             data = self.dataset[possibly_batched_index]\n---&gt; 55         return self.collate_fn(data)\n\n/usr/local/lib/python3.10/dist-packages/torch/utils/data/_utils/collate.py in default_collate(batch)\n    396         &gt;&gt;&gt; default_collate(batch)  # Handle `CustomType` automatically\n    397     \"\"\"\n--&gt; 398     return collate(batch, collate_fn_map=default_collate_fn_map)\n\n/usr/local/lib/python3.10/dist-packages/torch/utils/data/_utils/collate.py in collate(batch, collate_fn_map)\n    209 \n    210         if isinstance(elem, tuple):\n--&gt; 211             return [\n    212                 collate(samples, collate_fn_map=collate_fn_map)\n    213                 for samples in transposed\n\n/usr/local/lib/python3.10/dist-packages/torch/utils/data/_utils/collate.py in &lt;listcomp&gt;(.0)\n    210         if isinstance(elem, tuple):\n    211             return [\n--&gt; 212                 collate(samples, collate_fn_map=collate_fn_map)\n    213                 for samples in transposed\n    214             ]  # Backwards compatibility.\n\n/usr/local/lib/python3.10/dist-packages/torch/utils/data/_utils/collate.py in collate(batch, collate_fn_map)\n    238                 ]\n    239 \n--&gt; 240     raise TypeError(default_collate_err_msg_format.format(elem_type))\n    241 \n    242 \n\nTypeError: default_collate: batch must contain tensors, numpy arrays, numbers, dicts or lists; found &lt;class 'PIL.Image.Image'&gt;\n</code></pre>",
  "messages": [
    {
      "id": "3163441",
      "postDate": "03/30/2025 19:31:17",
      "content": "<p>I'm trying to split the set, so I can run the model faster. I've tried a few ways but when I get to extractor() I get the error below.</p>\n<pre><code>ds_train_wild = AnimalCLEF2025(INPUT_DIR, transform=transform_display, load_label=)\nds_valid_wild = AnimalCLEF2025(INPUT_DIR, transform=transform_display, load_label=)\n\n wildlife_datasets  splits\n\nsplitter = splits.ClosedSetSplit()\nidx_train, idx_valid = splitter.split(ds_train_wild.df)[]\n\n\n\nds_train_wild.df = ds_train_wild.df.loc[idx_train]\nds_train_wild\n</code></pre>\n<pre><code>TypeError                                 Traceback (most recent call last)\n&lt;ipython-input-30-1f1f27d22870&gt; in &lt;cell line: 1&gt;()\n----&gt; 1 features_database = extractor(train_dataset_database)\n      2 features_database\n\n/usr/local/lib/python3.10/dist-packages/wildlife_tools/features/deep.py in __call__(self, dataset)\n     54         )\n     55         outputs = []\n---&gt; 56         for image, _ in tqdm(loader, mininterval=1, ncols=100):\n     57             with torch.no_grad():\n     58                 output = self.model(image.to(self.device))\n\n/usr/local/lib/python3.10/dist-packages/tqdm/std.py in __iter__(self)\n   1179 \n   1180         try:\n-&gt; 1181             for obj in iterable:\n   1182                 yield obj\n   1183                 # Update and possibly print the progressbar.\n\n/usr/local/lib/python3.10/dist-packages/torch/utils/data/dataloader.py in __next__(self)\n    699                 # TODO(https://github.com/pytorch/pytorch/issues/76750)\n    700                 self._reset()  # type: ignore[call-arg]\n--&gt; 701             data = self._next_data()\n    702             self._num_yielded += 1\n    703             if (\n\n/usr/local/lib/python3.10/dist-packages/torch/utils/data/dataloader.py in _next_data(self)\n    755     def _next_data(self):\n    756         index = self._next_index()  # may raise StopIteration\n--&gt; 757         data = self._dataset_fetcher.fetch(index)  # may raise StopIteration\n    758         if self._pin_memory:\n    759             data = _utils.pin_memory.pin_memory(data, self._pin_memory_device)\n\n/usr/local/lib/python3.10/dist-packages/torch/utils/data/_utils/fetch.py in fetch(self, possibly_batched_index)\n     53         else:\n     54             data = self.dataset[possibly_batched_index]\n---&gt; 55         return self.collate_fn(data)\n\n/usr/local/lib/python3.10/dist-packages/torch/utils/data/_utils/collate.py in default_collate(batch)\n    396         &gt;&gt;&gt; default_collate(batch)  # Handle `CustomType` automatically\n    397     \"\"\"\n--&gt; 398     return collate(batch, collate_fn_map=default_collate_fn_map)\n\n/usr/local/lib/python3.10/dist-packages/torch/utils/data/_utils/collate.py in collate(batch, collate_fn_map)\n    209 \n    210         if isinstance(elem, tuple):\n--&gt; 211             return [\n    212                 collate(samples, collate_fn_map=collate_fn_map)\n    213                 for samples in transposed\n\n/usr/local/lib/python3.10/dist-packages/torch/utils/data/_utils/collate.py in &lt;listcomp&gt;(.0)\n    210         if isinstance(elem, tuple):\n    211             return [\n--&gt; 212                 collate(samples, collate_fn_map=collate_fn_map)\n    213                 for samples in transposed\n    214             ]  # Backwards compatibility.\n\n/usr/local/lib/python3.10/dist-packages/torch/utils/data/_utils/collate.py in collate(batch, collate_fn_map)\n    238                 ]\n    239 \n--&gt; 240     raise TypeError(default_collate_err_msg_format.format(elem_type))\n    241 \n    242 \n\nTypeError: default_collate: batch must contain tensors, numpy arrays, numbers, dicts or lists; found &lt;class 'PIL.Image.Image'&gt;\n</code></pre>",
      "rawMarkdown": "I'm trying to split the set, so I can run the model faster. I've tried a few ways but when I get to extractor() I get the error below.\n\n\n\n```python\nds_train_wild = AnimalCLEF2025(INPUT_DIR, transform=transform_display, load_label=True)\nds_valid_wild = AnimalCLEF2025(INPUT_DIR, transform=transform_display, load_label=True)\n\nfrom wildlife_datasets import splits\n\nsplitter = splits.ClosedSetSplit(0.2)\nidx_train, idx_valid = splitter.split(ds_train_wild.df)[0]\n\n# df_train = ds_train_wild.df.loc[idx_train]\n\nds_train_wild.df = ds_train_wild.df.loc[idx_train]\nds_train_wild\n\n```\n\n\n```txt\n\nTypeError                                 Traceback (most recent call last)\n<ipython-input-30-1f1f27d22870> in <cell line: 1>()\n----> 1 features_database = extractor(train_dataset_database)\n      2 features_database\n\n/usr/local/lib/python3.10/dist-packages/wildlife_tools/features/deep.py in __call__(self, dataset)\n     54         )\n     55         outputs = []\n---> 56         for image, _ in tqdm(loader, mininterval=1, ncols=100):\n     57             with torch.no_grad():\n     58                 output = self.model(image.to(self.device))\n\n/usr/local/lib/python3.10/dist-packages/tqdm/std.py in __iter__(self)\n   1179 \n   1180         try:\n-> 1181             for obj in iterable:\n   1182                 yield obj\n   1183                 # Update and possibly print the progressbar.\n\n/usr/local/lib/python3.10/dist-packages/torch/utils/data/dataloader.py in __next__(self)\n    699                 # TODO(https://github.com/pytorch/pytorch/issues/76750)\n    700                 self._reset()  # type: ignore[call-arg]\n--> 701             data = self._next_data()\n    702             self._num_yielded += 1\n    703             if (\n\n/usr/local/lib/python3.10/dist-packages/torch/utils/data/dataloader.py in _next_data(self)\n    755     def _next_data(self):\n    756         index = self._next_index()  # may raise StopIteration\n--> 757         data = self._dataset_fetcher.fetch(index)  # may raise StopIteration\n    758         if self._pin_memory:\n    759             data = _utils.pin_memory.pin_memory(data, self._pin_memory_device)\n\n/usr/local/lib/python3.10/dist-packages/torch/utils/data/_utils/fetch.py in fetch(self, possibly_batched_index)\n     53         else:\n     54             data = self.dataset[possibly_batched_index]\n---> 55         return self.collate_fn(data)\n\n/usr/local/lib/python3.10/dist-packages/torch/utils/data/_utils/collate.py in default_collate(batch)\n    396         >>> default_collate(batch)  # Handle `CustomType` automatically\n    397     \"\"\"\n--> 398     return collate(batch, collate_fn_map=default_collate_fn_map)\n\n/usr/local/lib/python3.10/dist-packages/torch/utils/data/_utils/collate.py in collate(batch, collate_fn_map)\n    209 \n    210         if isinstance(elem, tuple):\n--> 211             return [\n    212                 collate(samples, collate_fn_map=collate_fn_map)\n    213                 for samples in transposed\n\n/usr/local/lib/python3.10/dist-packages/torch/utils/data/_utils/collate.py in <listcomp>(.0)\n    210         if isinstance(elem, tuple):\n    211             return [\n--> 212                 collate(samples, collate_fn_map=collate_fn_map)\n    213                 for samples in transposed\n    214             ]  # Backwards compatibility.\n\n/usr/local/lib/python3.10/dist-packages/torch/utils/data/_utils/collate.py in collate(batch, collate_fn_map)\n    238                 ]\n    239 \n--> 240     raise TypeError(default_collate_err_msg_format.format(elem_type))\n    241 \n    242 \n\nTypeError: default_collate: batch must contain tensors, numpy arrays, numbers, dicts or lists; found <class 'PIL.Image.Image'>\n```",
      "votes": null
    },
    {
      "id": "3163825",
      "postDate": "03/31/2025 07:55:56",
      "content": "<p>Hi, thanks for the question. My fast guess is that you overwrite an inner attribute of the <code>WildlifeDataset</code> class (we will try to raise warning when something like this happens in the next version of the package). If you change the next-to-last line in your code by</p>\n<pre><code> = ds_train_wild.get_subset(idx_train)\n</code></pre>\n<p>will your code work? If not, I will take a deeper look into it. I would also like to ask how is it to work with wildlife-datasets? Is there anything which was difficult to understand or which you would like to be improved?</p>\n<p>Best,</p>\n<p>Lukas</p>",
      "rawMarkdown": "Hi, thanks for the question. My fast guess is that you overwrite an inner attribute of the `WildlifeDataset` class (we will try to raise warning when something like this happens in the next version of the package). If you change the next-to-last line in your code by\n\n```\nds_train_wild = ds_train_wild.get_subset(idx_train)\n```\n\nwill your code work? If not, I will take a deeper look into it. I would also like to ask how is it to work with wildlife-datasets? Is there anything which was difficult to understand or which you would like to be improved?\n\nBest,\n\nLukas",
      "votes": null
    },
    {
      "id": "3165721",
      "postDate": "04/01/2025 00:58:45",
      "content": "<p><a href=\"https://www.kaggle.com/wildlifedatasets\" target=\"_blank\">@wildlifedatasets</a>  Thank you very much for your help, I managed to do it here. It's been a really good challenge, it caught my attention when I saw it here on Kaggle. I usually do challenges with tabular data. This one with images is being fun and I'm able to learn a lot, I'm reading your papers calmly to be able to better understand the libraries that are being used in the examples. I'm dividing the notebook of the example to understand each step. The architecture that you set up for loading images, I found a difference, very cool. Congratulations on your work.<br>\nThe difficulty I'm having is more in the model, as I have the practice of creating the architecture, using something more ready-made than huggingface and improving the result ends up being a challenge. Thank you very much for this opportunity, it's great to always be learning more.</p>\n<p><img src=\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F13034312%2Fdae88b4434d1dcb9e5ecbad0208e99df%2FScreenshot%20from%202025-03-31%2021-52-22.png?generation=1743468778686286&amp;alt=media\" alt=\"\"></p>",
      "rawMarkdown": "wildlifedatasets  Thank you very much for your help, I managed to do it here. It's been a really good challenge, it caught my attention when I saw it here on Kaggle. I usually do challenges with tabular data. This one with images is being fun and I'm able to learn a lot, I'm reading your papers calmly to be able to better understand the libraries that are being used in the examples. I'm dividing the notebook of the example to understand each step. The architecture that you set up for loading images, I found a difference, very cool. Congratulations on your work.\nThe difficulty I'm having is more in the model, as I have the practice of creating the architecture, using something more ready-made than huggingface and improving the result ends up being a challenge. Thank you very much for this opportunity, it's great to always be learning more.\n\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F13034312%2Fdae88b4434d1dcb9e5ecbad0208e99df%2FScreenshot%20from%202025-03-31%2021-52-22.png?generation=1743468778686286&alt=media)",
      "votes": null
    },
    {
      "id": "3166915",
      "postDate": "04/01/2025 06:24:08",
      "content": "<p><a href=\"https://www.kaggle.com/sc0v1n0\" target=\"_blank\">@sc0v1n0</a> Thanks a lot for the feedback, it is nice to know that people appreciate our work :) </p>",
      "rawMarkdown": "sc0v1n0 Thanks a lot for the feedback, it is nice to know that people appreciate our work :)",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 3163825,
      "author_name": "wildlifedatasets",
      "author_url": "",
      "post_date": "03/31/2025 07:55:56",
      "content": "<p>Hi, thanks for the question. My fast guess is that you overwrite an inner attribute of the <code>WildlifeDataset</code> class (we will try to raise warning when something like this happens in the next version of the package). If you change the next-to-last line in your code by</p>\n<pre><code> = ds_train_wild.get_subset(idx_train)\n</code></pre>\n<p>will your code work? If not, I will take a deeper look into it. I would also like to ask how is it to work with wildlife-datasets? Is there anything which was difficult to understand or which you would like to be improved?</p>\n<p>Best,</p>\n<p>Lukas</p>",
      "votes": null,
      "replies": [
        {
          "id": 3165721,
          "author_name": "sc0v1n0",
          "author_url": "",
          "post_date": "04/01/2025 00:58:45",
          "content": "<p><a href=\"https://www.kaggle.com/wildlifedatasets\" target=\"_blank\">@wildlifedatasets</a>  Thank you very much for your help, I managed to do it here. It's been a really good challenge, it caught my attention when I saw it here on Kaggle. I usually do challenges with tabular data. This one with images is being fun and I'm able to learn a lot, I'm reading your papers calmly to be able to better understand the libraries that are being used in the examples. I'm dividing the notebook of the example to understand each step. The architecture that you set up for loading images, I found a difference, very cool. Congratulations on your work.<br>\nThe difficulty I'm having is more in the model, as I have the practice of creating the architecture, using something more ready-made than huggingface and improving the result ends up being a challenge. Thank you very much for this opportunity, it's great to always be learning more.</p>\n<p><img src=\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F13034312%2Fdae88b4434d1dcb9e5ecbad0208e99df%2FScreenshot%20from%202025-03-31%2021-52-22.png?generation=1743468778686286&amp;alt=media\" alt=\"\"></p>",
          "votes": null,
          "replies": [
            {
              "id": 3166915,
              "author_name": "wildlifedatasets",
              "author_url": "",
              "post_date": "04/01/2025 06:24:08",
              "content": "<p><a href=\"https://www.kaggle.com/sc0v1n0\" target=\"_blank\">@sc0v1n0</a> Thanks a lot for the feedback, it is nice to know that people appreciate our work :) </p>",
              "votes": null,
              "replies": []
            }
          ]
        }
      ]
    }
  ],
  "raw_markdown_by_id": {
    "3163441": "I'm trying to split the set, so I can run the model faster. I've tried a few ways but when I get to extractor() I get the error below.\n\n\n\n```python\nds_train_wild = AnimalCLEF2025(INPUT_DIR, transform=transform_display, load_label=True)\nds_valid_wild = AnimalCLEF2025(INPUT_DIR, transform=transform_display, load_label=True)\n\nfrom wildlife_datasets import splits\n\nsplitter = splits.ClosedSetSplit(0.2)\nidx_train, idx_valid = splitter.split(ds_train_wild.df)[0]\n\n# df_train = ds_train_wild.df.loc[idx_train]\n\nds_train_wild.df = ds_train_wild.df.loc[idx_train]\nds_train_wild\n\n```\n\n\n```txt\n\nTypeError                                 Traceback (most recent call last)\n<ipython-input-30-1f1f27d22870> in <cell line: 1>()\n----> 1 features_database = extractor(train_dataset_database)\n      2 features_database\n\n/usr/local/lib/python3.10/dist-packages/wildlife_tools/features/deep.py in __call__(self, dataset)\n     54         )\n     55         outputs = []\n---> 56         for image, _ in tqdm(loader, mininterval=1, ncols=100):\n     57             with torch.no_grad():\n     58                 output = self.model(image.to(self.device))\n\n/usr/local/lib/python3.10/dist-packages/tqdm/std.py in __iter__(self)\n   1179 \n   1180         try:\n-> 1181             for obj in iterable:\n   1182                 yield obj\n   1183                 # Update and possibly print the progressbar.\n\n/usr/local/lib/python3.10/dist-packages/torch/utils/data/dataloader.py in __next__(self)\n    699                 # TODO(https://github.com/pytorch/pytorch/issues/76750)\n    700                 self._reset()  # type: ignore[call-arg]\n--> 701             data = self._next_data()\n    702             self._num_yielded += 1\n    703             if (\n\n/usr/local/lib/python3.10/dist-packages/torch/utils/data/dataloader.py in _next_data(self)\n    755     def _next_data(self):\n    756         index = self._next_index()  # may raise StopIteration\n--> 757         data = self._dataset_fetcher.fetch(index)  # may raise StopIteration\n    758         if self._pin_memory:\n    759             data = _utils.pin_memory.pin_memory(data, self._pin_memory_device)\n\n/usr/local/lib/python3.10/dist-packages/torch/utils/data/_utils/fetch.py in fetch(self, possibly_batched_index)\n     53         else:\n     54             data = self.dataset[possibly_batched_index]\n---> 55         return self.collate_fn(data)\n\n/usr/local/lib/python3.10/dist-packages/torch/utils/data/_utils/collate.py in default_collate(batch)\n    396         >>> default_collate(batch)  # Handle `CustomType` automatically\n    397     \"\"\"\n--> 398     return collate(batch, collate_fn_map=default_collate_fn_map)\n\n/usr/local/lib/python3.10/dist-packages/torch/utils/data/_utils/collate.py in collate(batch, collate_fn_map)\n    209 \n    210         if isinstance(elem, tuple):\n--> 211             return [\n    212                 collate(samples, collate_fn_map=collate_fn_map)\n    213                 for samples in transposed\n\n/usr/local/lib/python3.10/dist-packages/torch/utils/data/_utils/collate.py in <listcomp>(.0)\n    210         if isinstance(elem, tuple):\n    211             return [\n--> 212                 collate(samples, collate_fn_map=collate_fn_map)\n    213                 for samples in transposed\n    214             ]  # Backwards compatibility.\n\n/usr/local/lib/python3.10/dist-packages/torch/utils/data/_utils/collate.py in collate(batch, collate_fn_map)\n    238                 ]\n    239 \n--> 240     raise TypeError(default_collate_err_msg_format.format(elem_type))\n    241 \n    242 \n\nTypeError: default_collate: batch must contain tensors, numpy arrays, numbers, dicts or lists; found <class 'PIL.Image.Image'>\n```",
    "3163825": "Hi, thanks for the question. My fast guess is that you overwrite an inner attribute of the `WildlifeDataset` class (we will try to raise warning when something like this happens in the next version of the package). If you change the next-to-last line in your code by\n\n```\nds_train_wild = ds_train_wild.get_subset(idx_train)\n```\n\nwill your code work? If not, I will take a deeper look into it. I would also like to ask how is it to work with wildlife-datasets? Is there anything which was difficult to understand or which you would like to be improved?\n\nBest,\n\nLukas",
    "3165721": "wildlifedatasets  Thank you very much for your help, I managed to do it here. It's been a really good challenge, it caught my attention when I saw it here on Kaggle. I usually do challenges with tabular data. This one with images is being fun and I'm able to learn a lot, I'm reading your papers calmly to be able to better understand the libraries that are being used in the examples. I'm dividing the notebook of the example to understand each step. The architecture that you set up for loading images, I found a difference, very cool. Congratulations on your work.\nThe difficulty I'm having is more in the model, as I have the practice of creating the architecture, using something more ready-made than huggingface and improving the result ends up being a challenge. Thank you very much for this opportunity, it's great to always be learning more.\n\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F13034312%2Fdae88b4434d1dcb9e5ecbad0208e99df%2FScreenshot%20from%202025-03-31%2021-52-22.png?generation=1743468778686286&alt=media)",
    "3166915": "sc0v1n0 Thanks a lot for the feedback, it is nice to know that people appreciate our work :)"
  },
  "source": "meta"
}