{
  "id": 48908,
  "title": "Sorting training data into sub-directories",
  "url": "/competitions/dog-breed-identification/discussion/48908",
  "author_name": "KaranJaisingh",
  "post_date": "2018-02-04T08:12:48.532000",
  "votes": 9,
  "comment_count": 7,
  "views": 0,
  "content": "<p>I'm unable to find a method to sort the 10222 training images into 120 different sub-folders - could someone please advise me on how to do so?\nI've made a list of length 10222 with a number from 1 to 120 (a different number for each breed) in accordance with each image entry, if this makes things any easier.\nThanks in advance!</p>",
  "messages": [
    {
      "id": 277775,
      "postDate": "2018-02-04T08:12:48.533Z",
      "content": "<p>I'm unable to find a method to sort the 10222 training images into 120 different sub-folders - could someone please advise me on how to do so?\nI've made a list of length 10222 with a number from 1 to 120 (a different number for each breed) in accordance with each image entry, if this makes things any easier.\nThanks in advance!</p>",
      "rawMarkdown": "I'm unable to find a method to sort the 10222 training images into 120 different sub-folders - could someone please advise me on how to do so?\nI've made a list of length 10222 with a number from 1 to 120 (a different number for each breed) in accordance with each image entry, if this makes things any easier.\nThanks in advance!",
      "votes": 9
    },
    {
      "id": 537632,
      "postDate": "2019-05-27T11:18:55.963Z",
      "content": "<p>The code above gives IOError, hence he we are back to square one.</p>",
      "rawMarkdown": "The code above gives IOError, hence he we are back to square one.",
      "votes": 1
    },
    {
      "id": 277817,
      "postDate": "2018-02-04T12:38:44.143Z",
      "content": "<p>I just did this a few minutes ago, haven't  checked whether there are already some solutions to this in the forum/kernel section (but I suppose there might be some more elegant ones). </p>\n\n<p>PATH is the path to your project root folder (ie where the train folder is)</p>\n\n<p>labels.csv is the list you already created (two columns, id for image name and breed for class); you might have to make some modifications there.</p>\n\n<p>'''\nimport pandas as pd\nimport os</p>\n\n<p>labels = pd.read_csv(f'{PATH}labels.csv')</p>\n\n<h2>create validation directory</h2>\n\n<p>os.mkdir(f'{PATH}valid')</p>\n\n<h2>.. and class directories</h2>\n\n<p>for f in labels.breed.unique():\n    os.mkdir(f'{PATH}train/{f}')\n    os.mkdir(f'{PATH}valid/{f}')</p>\n\n<h2>share of images for validation set</h2>\n\n<p>n = 0.2</p>\n\n<h2>move all pics in class folders</h2>\n\n<p>for p in labels.itertuples():\n    file_path = f'{PATH}train/{p.id}.jpg'\n    train_path = f'{PATH}train/{p.breed}/{p.id}.jpg'</p>\n\n<pre><code>os.rename(f'{file_path}', f'{train_path}')\n</code></pre>\n\n<h2>move valid pics to valid folder</h2>\n\n<p>for f in os.listdir(f'{PATH}train'):\n    pics = os.listdir(f'{PATH}train/{f}')\n    num_pics = len(pics)\n    num_val_pics = round(n * num_pics)</p>\n\n<pre><code>val_pics = random.sample(pics, num_val_pics)\n\nfor p in val_pics:\n    file_path = f'{PATH}train/{f}/{p}'\n    val_path = f'{PATH}valid/{f}/{p}'\n\n    os.rename(f'{file_path}', f'{val_path}')\n</code></pre>\n\n<p>'''</p>",
      "rawMarkdown": "I just did this a few minutes ago, haven't  checked whether there are already some solutions to this in the forum/kernel section (but I suppose there might be some more elegant ones). \n\nPATH is the path to your project root folder (ie where the train folder is)\n\nlabels.csv is the list you already created (two columns, id for image name and breed for class); you might have to make some modifications there.\n\n'''\nimport pandas as pd\nimport os\n\nlabels = pd.read_csv(f'{PATH}labels.csv')\n\n## create validation directory\nos.mkdir(f'{PATH}valid')\n## .. and class directories\nfor f in labels.breed.unique():\n    os.mkdir(f'{PATH}train/{f}')\n    os.mkdir(f'{PATH}valid/{f}')\n\n## share of images for validation set\nn = 0.2\n\n## move all pics in class folders\nfor p in labels.itertuples():\n    file_path = f'{PATH}train/{p.id}.jpg'\n    train_path = f'{PATH}train/{p.breed}/{p.id}.jpg'\n    \n    os.rename(f'{file_path}', f'{train_path}')\n\n## move valid pics to valid folder\nfor f in os.listdir(f'{PATH}train'):\n    pics = os.listdir(f'{PATH}train/{f}')\n    num_pics = len(pics)\n    num_val_pics = round(n * num_pics)\n    \n    val_pics = random.sample(pics, num_val_pics)\n    \n    for p in val_pics:\n        file_path = f'{PATH}train/{f}/{p}'\n        val_path = f'{PATH}valid/{f}/{p}'\n    \n        os.rename(f'{file_path}', f'{val_path}')\n'''",
      "votes": 3
    },
    {
      "id": 279549,
      "postDate": "2018-02-08T08:27:02.910Z",
      "content": "<p>I wrote the following code to separate the images into sub-folders, which may meet the need.</p>\n\n<pre><code>from __future__ import print_function\nimport pandas as pd\nimport shutil\nimport os\nimport sys\n\nlabels = pd.read_csv('labels.csv')\n\n# Create `train_sep` directory\ntrain_dir = 'train/'\ntrain_sep_dir = 'train_sep/'\nif not os.path.exists(train_sep_dir):\n    os.mkdir(train_sep_dir)\n\nfor filename, class_name in labels.values:\n    # Create subdirectory with `class_name`\n    if not os.path.exists(train_sep_dir + class_name):\n        os.mkdir(train_sep_dir + class_name)\n    src_path = train_dir + filename + '.jpg'\n    dst_path = train_sep_dir + class_name + '/' + filename + '.jpg'\n    try:\n        shutil.copy(src_path, dst_path)\n    except IOError as e:\n        print('Unable to copy file {} to {}'\n              .format(src_path, dst_path))\n    except:\n        print('When try copy file {} to {}, unexpected error: {}'\n              .format(src_path, dst_path, sys.exc_info()))\n</code></pre>",
      "rawMarkdown": "I wrote the following code to separate the images into sub-folders, which may meet the need.\n\n    from __future__ import print_function\n    import pandas as pd\n    import shutil\n    import os\n    import sys\n\n    labels = pd.read_csv('labels.csv')\n\n    # Create `train_sep` directory\n    train_dir = 'train/'\n    train_sep_dir = 'train_sep/'\n    if not os.path.exists(train_sep_dir):\n        os.mkdir(train_sep_dir)\n\n    for filename, class_name in labels.values:\n        # Create subdirectory with `class_name`\n        if not os.path.exists(train_sep_dir + class_name):\n            os.mkdir(train_sep_dir + class_name)\n        src_path = train_dir + filename + '.jpg'\n        dst_path = train_sep_dir + class_name + '/' + filename + '.jpg'\n        try:\n            shutil.copy(src_path, dst_path)\n        except IOError as e:\n            print('Unable to copy file {} to {}'\n                  .format(src_path, dst_path))\n        except:\n            print('When try copy file {} to {}, unexpected error: {}'\n                  .format(src_path, dst_path, sys.exc_info()))",
      "votes": 4,
      "replies": [
        {
          "id": 623712,
          "postDate": "2019-09-11T08:28:40.923Z",
          "content": "<p>hai sir while using your code I'm getting below error</p>\n\n<p>File \"\", line 20\n  src_path = train_dir + image_id + '.jpg'\n           ^\nSyntaxError: invalid syntax</p>",
          "rawMarkdown": "hai sir while using your code I'm getting below error\n\n File \""
        }
      ]
    },
    {
      "id": 296177,
      "postDate": "2018-03-14T21:04:21.527Z",
      "content": "<p>Thank you. This thread makes it so much easier </p>",
      "rawMarkdown": "Thank you. This thread makes it so much easier "
    },
    {
      "id": 565930,
      "postDate": "2019-07-01T15:10:41.383Z",
      "rawMarkdown": "",
      "isDeleted": true,
      "replies": [
        {
          "id": 1073530,
          "postDate": "2020-11-09T15:41:03.053Z",
          "content": "<p>What if i wanted to put them into their respective folders, like 120 folders each for a category of dog, and i want to put the images into them</p>",
          "rawMarkdown": "What if i wanted to put them into their respective folders, like 120 folders each for a category of dog, and i want to put the images into them"
        }
      ]
    }
  ],
  "comments": [
    {
      "id": 537632,
      "author_name": "Pranjalya Tiwari",
      "author_url": "",
      "post_date": "2019-05-27T11:18:55.963000",
      "content": "<p>The code above gives IOError, hence he we are back to square one.</p>",
      "votes": 1,
      "replies": []
    },
    {
      "id": 277817,
      "author_name": "Bln_data_team",
      "author_url": "",
      "post_date": "2018-02-04T12:38:44.143000",
      "content": "<p>I just did this a few minutes ago, haven't  checked whether there are already some solutions to this in the forum/kernel section (but I suppose there might be some more elegant ones). </p>\n\n<p>PATH is the path to your project root folder (ie where the train folder is)</p>\n\n<p>labels.csv is the list you already created (two columns, id for image name and breed for class); you might have to make some modifications there.</p>\n\n<p>'''\nimport pandas as pd\nimport os</p>\n\n<p>labels = pd.read_csv(f'{PATH}labels.csv')</p>\n\n<h2>create validation directory</h2>\n\n<p>os.mkdir(f'{PATH}valid')</p>\n\n<h2>.. and class directories</h2>\n\n<p>for f in labels.breed.unique():\n    os.mkdir(f'{PATH}train/{f}')\n    os.mkdir(f'{PATH}valid/{f}')</p>\n\n<h2>share of images for validation set</h2>\n\n<p>n = 0.2</p>\n\n<h2>move all pics in class folders</h2>\n\n<p>for p in labels.itertuples():\n    file_path = f'{PATH}train/{p.id}.jpg'\n    train_path = f'{PATH}train/{p.breed}/{p.id}.jpg'</p>\n\n<pre><code>os.rename(f'{file_path}', f'{train_path}')\n</code></pre>\n\n<h2>move valid pics to valid folder</h2>\n\n<p>for f in os.listdir(f'{PATH}train'):\n    pics = os.listdir(f'{PATH}train/{f}')\n    num_pics = len(pics)\n    num_val_pics = round(n * num_pics)</p>\n\n<pre><code>val_pics = random.sample(pics, num_val_pics)\n\nfor p in val_pics:\n    file_path = f'{PATH}train/{f}/{p}'\n    val_path = f'{PATH}valid/{f}/{p}'\n\n    os.rename(f'{file_path}', f'{val_path}')\n</code></pre>\n\n<p>'''</p>",
      "votes": 3,
      "replies": []
    },
    {
      "id": 279549,
      "author_name": "Du Ang",
      "author_url": "",
      "post_date": "2018-02-08T08:27:02.910000",
      "content": "<p>I wrote the following code to separate the images into sub-folders, which may meet the need.</p>\n\n<pre><code>from __future__ import print_function\nimport pandas as pd\nimport shutil\nimport os\nimport sys\n\nlabels = pd.read_csv('labels.csv')\n\n# Create `train_sep` directory\ntrain_dir = 'train/'\ntrain_sep_dir = 'train_sep/'\nif not os.path.exists(train_sep_dir):\n    os.mkdir(train_sep_dir)\n\nfor filename, class_name in labels.values:\n    # Create subdirectory with `class_name`\n    if not os.path.exists(train_sep_dir + class_name):\n        os.mkdir(train_sep_dir + class_name)\n    src_path = train_dir + filename + '.jpg'\n    dst_path = train_sep_dir + class_name + '/' + filename + '.jpg'\n    try:\n        shutil.copy(src_path, dst_path)\n    except IOError as e:\n        print('Unable to copy file {} to {}'\n              .format(src_path, dst_path))\n    except:\n        print('When try copy file {} to {}, unexpected error: {}'\n              .format(src_path, dst_path, sys.exc_info()))\n</code></pre>",
      "votes": 4,
      "replies": [
        {
          "id": 623712,
          "author_name": "hemanthdatascientist",
          "author_url": "",
          "post_date": "2019-09-11T08:28:40.923000",
          "content": "<p>hai sir while using your code I'm getting below error</p>\n\n<p>File \"\", line 20\n  src_path = train_dir + image_id + '.jpg'\n           ^\nSyntaxError: invalid syntax</p>",
          "votes": 0,
          "replies": []
        }
      ]
    },
    {
      "id": 296177,
      "author_name": "",
      "author_url": "",
      "post_date": "2018-03-14T21:04:21.527000",
      "content": "<p>Thank you. This thread makes it so much easier </p>",
      "votes": 0,
      "replies": []
    },
    {
      "id": 565930,
      "author_name": "",
      "author_url": "",
      "post_date": "2019-07-01T15:10:41.383000",
      "content": "",
      "votes": 0,
      "replies": [
        {
          "id": 1073530,
          "author_name": "Jo Amps",
          "author_url": "",
          "post_date": "2020-11-09T15:41:03.053000",
          "content": "<p>What if i wanted to put them into their respective folders, like 120 folders each for a category of dog, and i want to put the images into them</p>",
          "votes": 0,
          "replies": []
        }
      ]
    }
  ],
  "raw_markdown_by_id": {
    "277775": "I'm unable to find a method to sort the 10222 training images into 120 different sub-folders - could someone please advise me on how to do so?\nI've made a list of length 10222 with a number from 1 to 120 (a different number for each breed) in accordance with each image entry, if this makes things any easier.\nThanks in advance!",
    "537632": "The code above gives IOError, hence he we are back to square one.",
    "277817": "I just did this a few minutes ago, haven't  checked whether there are already some solutions to this in the forum/kernel section (but I suppose there might be some more elegant ones). \n\nPATH is the path to your project root folder (ie where the train folder is)\n\nlabels.csv is the list you already created (two columns, id for image name and breed for class); you might have to make some modifications there.\n\n'''\nimport pandas as pd\nimport os\n\nlabels = pd.read_csv(f'{PATH}labels.csv')\n\n## create validation directory\nos.mkdir(f'{PATH}valid')\n## .. and class directories\nfor f in labels.breed.unique():\n    os.mkdir(f'{PATH}train/{f}')\n    os.mkdir(f'{PATH}valid/{f}')\n\n## share of images for validation set\nn = 0.2\n\n## move all pics in class folders\nfor p in labels.itertuples():\n    file_path = f'{PATH}train/{p.id}.jpg'\n    train_path = f'{PATH}train/{p.breed}/{p.id}.jpg'\n    \n    os.rename(f'{file_path}', f'{train_path}')\n\n## move valid pics to valid folder\nfor f in os.listdir(f'{PATH}train'):\n    pics = os.listdir(f'{PATH}train/{f}')\n    num_pics = len(pics)\n    num_val_pics = round(n * num_pics)\n    \n    val_pics = random.sample(pics, num_val_pics)\n    \n    for p in val_pics:\n        file_path = f'{PATH}train/{f}/{p}'\n        val_path = f'{PATH}valid/{f}/{p}'\n    \n        os.rename(f'{file_path}', f'{val_path}')\n'''",
    "279549": "I wrote the following code to separate the images into sub-folders, which may meet the need.\n\n    from __future__ import print_function\n    import pandas as pd\n    import shutil\n    import os\n    import sys\n\n    labels = pd.read_csv('labels.csv')\n\n    # Create `train_sep` directory\n    train_dir = 'train/'\n    train_sep_dir = 'train_sep/'\n    if not os.path.exists(train_sep_dir):\n        os.mkdir(train_sep_dir)\n\n    for filename, class_name in labels.values:\n        # Create subdirectory with `class_name`\n        if not os.path.exists(train_sep_dir + class_name):\n            os.mkdir(train_sep_dir + class_name)\n        src_path = train_dir + filename + '.jpg'\n        dst_path = train_sep_dir + class_name + '/' + filename + '.jpg'\n        try:\n            shutil.copy(src_path, dst_path)\n        except IOError as e:\n            print('Unable to copy file {} to {}'\n                  .format(src_path, dst_path))\n        except:\n            print('When try copy file {} to {}, unexpected error: {}'\n                  .format(src_path, dst_path, sys.exc_info()))",
    "296177": "Thank you. This thread makes it so much easier ",
    "565930": ""
  }
}