{
  "id": 572569,
  "title": "data download instructions bug",
  "url": "/competitions/dishcovery-vlm-mtf-cvpr-2025/discussion/572569",
  "author_name": "",
  "post_date": "2025-04-10T07:05:46.980125Z",
  "votes": null,
  "comment_count": 2,
  "views": 0,
  "content": "<p>The data tab includes this command:</p>\n<pre><code>img2dataset \\\n   mtf2025_web_metadata.parquet \\\n    \\\n    \\\n    \\\n   webdataset \\\n    \\\n   16 \\\n   64 \\\n   512 \\\n   keep_ratio \\\n   '[]' \\\n   False\n</code></pre>\n<p>but the flag save_additional_columns results in an error since the column caption already exists</p>\n<p>The command should be:</p>\n<pre><code>img2dataset \\\n   mtf2025_web_metadata.parquet \\\n    \\\n    \\\n    \\\n   webdataset \\\n    \\\n   16 \\\n   64 \\\n   512 \\\n   keep_ratio \\\n   False\n</code></pre>",
  "messages": [
    {
      "id": "3175465",
      "postDate": "04/10/2025 07:05:46",
      "content": "<p>The data tab includes this command:</p>\n<pre><code>img2dataset \\\n   mtf2025_web_metadata.parquet \\\n    \\\n    \\\n    \\\n   webdataset \\\n    \\\n   16 \\\n   64 \\\n   512 \\\n   keep_ratio \\\n   '[]' \\\n   False\n</code></pre>\n<p>but the flag save_additional_columns results in an error since the column caption already exists</p>\n<p>The command should be:</p>\n<pre><code>img2dataset \\\n   mtf2025_web_metadata.parquet \\\n    \\\n    \\\n    \\\n   webdataset \\\n    \\\n   16 \\\n   64 \\\n   512 \\\n   keep_ratio \\\n   False\n</code></pre>",
      "rawMarkdown": "The data tab includes this command:\n```\nimg2dataset \\\n  --url_list mtf2025_web_metadata.parquet \\\n  --input_format \"parquet\" \\\n  --url_col \"url\" \\\n  --caption_col \"caption\" \\\n  --output_format webdataset \\\n  --output_folder ./mtf2025_web_images \\\n  --processes_count 16 \\\n  --thread_count 64 \\\n  --image_size 512 \\\n  --resize_mode keep_ratio \\\n  --save_additional_columns '[\"caption\"]' \\\n  --enable_wandb False\n```\nbut the flag save_additional_columns results in an error since the column caption already exists\n\nThe command should be:\n```\nimg2dataset \\\n  --url_list mtf2025_web_metadata.parquet \\\n  --input_format \"parquet\" \\\n  --url_col \"url\" \\\n  --caption_col \"caption\" \\\n  --output_format webdataset \\\n  --output_folder ./mtf2025_web_images \\\n  --processes_count 16 \\\n  --thread_count 64 \\\n  --image_size 512 \\\n  --resize_mode keep_ratio \\\n  --enable_wandb False\n```",
      "votes": null
    },
    {
      "id": "3178761",
      "postDate": "04/14/2025 15:17:41",
      "content": "<p>Hi Anthony! Thanks for detection the issue and letting us know. </p>\n<p>We have already changed the instructions in the hugging face page.</p>",
      "rawMarkdown": "Hi Anthony! Thanks for detection the issue and letting us know. \n\nWe have already changed the instructions in the hugging face page.",
      "votes": null
    },
    {
      "id": "3180379",
      "postDate": "04/16/2025 14:31:16",
      "content": "<p>I encountered this problem when using img2dataset to download data. I can't download any images. Do I need to modify the configuration of the image_size?</p>\n<pre><code>  - success: . - failed to download: . - failed to resize: . - images per sec:  - count: \n   - success: . - failed to download: . - failed to resize: . - images per sec:  - count: \n</code></pre>",
      "rawMarkdown": "I encountered this problem when using img2dataset to download data. I can't download any images. Do I need to modify the configuration of the image_size?\n```\nworker  - success: 0.000 - failed to download: 0.000 - failed to resize: 1.000 - images per sec: 3 - count: 10\ntotal   - success: 0.000 - failed to download: 0.000 - failed to resize: 1.000 - images per sec: 3 - count: 10\n```",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 3178761,
      "author_name": "metafood25vlms",
      "author_url": "",
      "post_date": "04/14/2025 15:17:41",
      "content": "<p>Hi Anthony! Thanks for detection the issue and letting us know. </p>\n<p>We have already changed the instructions in the hugging face page.</p>",
      "votes": null,
      "replies": [
        {
          "id": 3180379,
          "author_name": "sunnylx",
          "author_url": "",
          "post_date": "04/16/2025 14:31:16",
          "content": "<p>I encountered this problem when using img2dataset to download data. I can't download any images. Do I need to modify the configuration of the image_size?</p>\n<pre><code>  - success: . - failed to download: . - failed to resize: . - images per sec:  - count: \n   - success: . - failed to download: . - failed to resize: . - images per sec:  - count: \n</code></pre>",
          "votes": null,
          "replies": []
        }
      ]
    }
  ],
  "raw_markdown_by_id": {
    "3175465": "The data tab includes this command:\n```\nimg2dataset \\\n  --url_list mtf2025_web_metadata.parquet \\\n  --input_format \"parquet\" \\\n  --url_col \"url\" \\\n  --caption_col \"caption\" \\\n  --output_format webdataset \\\n  --output_folder ./mtf2025_web_images \\\n  --processes_count 16 \\\n  --thread_count 64 \\\n  --image_size 512 \\\n  --resize_mode keep_ratio \\\n  --save_additional_columns '[\"caption\"]' \\\n  --enable_wandb False\n```\nbut the flag save_additional_columns results in an error since the column caption already exists\n\nThe command should be:\n```\nimg2dataset \\\n  --url_list mtf2025_web_metadata.parquet \\\n  --input_format \"parquet\" \\\n  --url_col \"url\" \\\n  --caption_col \"caption\" \\\n  --output_format webdataset \\\n  --output_folder ./mtf2025_web_images \\\n  --processes_count 16 \\\n  --thread_count 64 \\\n  --image_size 512 \\\n  --resize_mode keep_ratio \\\n  --enable_wandb False\n```",
    "3178761": "Hi Anthony! Thanks for detection the issue and letting us know. \n\nWe have already changed the instructions in the hugging face page.",
    "3180379": "I encountered this problem when using img2dataset to download data. I can't download any images. Do I need to modify the configuration of the image_size?\n```\nworker  - success: 0.000 - failed to download: 0.000 - failed to resize: 1.000 - images per sec: 3 - count: 10\ntotal   - success: 0.000 - failed to download: 0.000 - failed to resize: 1.000 - images per sec: 3 - count: 10\n```"
  },
  "source": "meta"
}