{
  "id": 54676,
  "title": "Fast.ai code",
  "url": "/competitions/dogs-vs-cats/discussion/54676",
  "author_name": "",
  "post_date": "2018-04-16T17:30:06.790462100Z",
  "votes": 4,
  "comment_count": 3,
  "views": 0,
  "content": "<p>I wanted to try fast.ai with this dataset, but the implementation of fast.ai needs the data to be stored in two different folders, each one of them containing either the cats or the dogs. However, I cannot try to make this because I am not allowed to create directories or move files from one directory to another. Any thoughts on how can I do it without having to change the fast.ai code?</p>",
  "messages": [
    {
      "id": "315051",
      "postDate": "04/16/2018 17:30:06",
      "content": "<p>I wanted to try fast.ai with this dataset, but the implementation of fast.ai needs the data to be stored in two different folders, each one of them containing either the cats or the dogs. However, I cannot try to make this because I am not allowed to create directories or move files from one directory to another. Any thoughts on how can I do it without having to change the fast.ai code?</p>",
      "rawMarkdown": "I wanted to try fast.ai with this dataset, but the implementation of fast.ai needs the data to be stored in two different folders, each one of them containing either the cats or the dogs. However, I cannot try to make this because I am not allowed to create directories or move files from one directory to another. Any thoughts on how can I do it without having to change the fast.ai code?",
      "votes": null
    },
    {
      "id": "315101",
      "postDate": "04/16/2018 18:39:13",
      "content": "<p>create your own dataset with the required structure and upload it on kaggle, but you still will not be able to use fastai in kernels, since it is not installed</p>",
      "rawMarkdown": "create your own dataset with the required structure and upload it on kaggle, but you still will not be able to use fastai in kernels, since it is not installed",
      "votes": null
    },
    {
      "id": "315184",
      "postDate": "04/16/2018 20:46:14",
      "content": "<p>You can reorganize your data by creating symbolic links.  </p>\n\n<pre><code>from fastai.conv_learner import *\nfrom sklearn.model_selection import train_test_split \nPATH = \"../input/dogs-vs-cats-redux-kernels-edition/\"\nsz=224\nroot_prefix = PATH\ntrain_filenames = os.listdir('%s/train/' % (root_prefix))\ntest_filenames  = os.listdir('%s/test/'  % (root_prefix))\nmy_train = train_filenames\nmy_train, my_cv = train_test_split(train_filenames, test_size=0.1, random_state=0)\n\nfrom pathlib import Path\n!cp -as \"$(pwd)/../input/\" \"$(pwd)/COPY\"\nroot_prefix = 'COPY'\n\ndef remove_and_create_class(dirname):\n    if os.path.exists(dirname):\n        shutil.rmtree(dirname)\n    os.mkdir(dirname)\n    os.mkdir(dirname+'/cat')\n    os.mkdir(dirname+'/dog')\n\nremove_and_create_class('%s/train' % (root_prefix))\nremove_and_create_class('%s/valid' % (root_prefix))\n\nfor filename in filter(lambda x: x.split(\".\")[0] == \"cat\", my_train):\n    os.symlink('%s/train/' % (root_prefix)+filename, '%s/train/cat/' % (root_prefix)+filename)\nfor filename in filter(lambda x: x.split(\".\")[0] == \"dog\", my_train):\n    os.symlink('%s/train/' % (root_prefix)+filename, '%s/train/dog/' % (root_prefix)+filename)\nfor filename in filter(lambda x: x.split(\".\")[0] == \"cat\", my_cv):\n    os.symlink('%s/train/' % (root_prefix)+filename, '%s/valid/cat/' % (root_prefix)+filename)\nfor filename in filter(lambda x: x.split(\".\")[0] == \"dog\", my_cv):\n    os.symlink('%s/train/' % (root_prefix)+filename, '%s/valid/dog/' % (root_prefix)+filename)\n</code></pre>",
      "rawMarkdown": "You can reorganize your data by creating symbolic links.  \n\n    from fastai.conv_learner import *\n    from sklearn.model_selection import train_test_split \n    PATH = \"../input/dogs-vs-cats-redux-kernels-edition/\"\n    sz=224\n    root_prefix = PATH\n    train_filenames = os.listdir('%s/train/' % (root_prefix))\n    test_filenames  = os.listdir('%s/test/'  % (root_prefix))\n    my_train = train_filenames\n    my_train, my_cv = train_test_split(train_filenames, test_size=0.1, random_state=0)\n    \n    from pathlib import Path\n    !cp -as \"$(pwd)/../input/\" \"$(pwd)/COPY\"\n    root_prefix = 'COPY'\n    \n    def remove_and_create_class(dirname):\n        if os.path.exists(dirname):\n            shutil.rmtree(dirname)\n        os.mkdir(dirname)\n        os.mkdir(dirname+'/cat')\n        os.mkdir(dirname+'/dog')\n    \n    remove_and_create_class('%s/train' % (root_prefix))\n    remove_and_create_class('%s/valid' % (root_prefix))\n    \n    for filename in filter(lambda x: x.split(\".\")[0] == \"cat\", my_train):\n        os.symlink('%s/train/' % (root_prefix)+filename, '%s/train/cat/' % (root_prefix)+filename)\n    for filename in filter(lambda x: x.split(\".\")[0] == \"dog\", my_train):\n        os.symlink('%s/train/' % (root_prefix)+filename, '%s/train/dog/' % (root_prefix)+filename)\n    for filename in filter(lambda x: x.split(\".\")[0] == \"cat\", my_cv):\n        os.symlink('%s/train/' % (root_prefix)+filename, '%s/valid/cat/' % (root_prefix)+filename)\n    for filename in filter(lambda x: x.split(\".\")[0] == \"dog\", my_cv):\n        os.symlink('%s/train/' % (root_prefix)+filename, '%s/valid/dog/' % (root_prefix)+filename)",
      "votes": null
    },
    {
      "id": "315185",
      "postDate": "04/16/2018 20:47:46",
      "content": "<p>You can install custom packages in the \"Settings\" tab within the Kernel editor.</p>",
      "rawMarkdown": "You can install custom packages in the \"Settings\" tab within the Kernel editor.",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 315101,
      "author_name": "",
      "author_url": "",
      "post_date": "04/16/2018 18:39:13",
      "content": "<p>create your own dataset with the required structure and upload it on kaggle, but you still will not be able to use fastai in kernels, since it is not installed</p>",
      "votes": null,
      "replies": [
        {
          "id": 315185,
          "author_name": "paultimothymooney",
          "author_url": "",
          "post_date": "04/16/2018 20:47:46",
          "content": "<p>You can install custom packages in the \"Settings\" tab within the Kernel editor.</p>",
          "votes": null,
          "replies": []
        }
      ]
    },
    {
      "id": 315184,
      "author_name": "paultimothymooney",
      "author_url": "",
      "post_date": "04/16/2018 20:46:14",
      "content": "<p>You can reorganize your data by creating symbolic links.  </p>\n\n<pre><code>from fastai.conv_learner import *\nfrom sklearn.model_selection import train_test_split \nPATH = \"../input/dogs-vs-cats-redux-kernels-edition/\"\nsz=224\nroot_prefix = PATH\ntrain_filenames = os.listdir('%s/train/' % (root_prefix))\ntest_filenames  = os.listdir('%s/test/'  % (root_prefix))\nmy_train = train_filenames\nmy_train, my_cv = train_test_split(train_filenames, test_size=0.1, random_state=0)\n\nfrom pathlib import Path\n!cp -as \"$(pwd)/../input/\" \"$(pwd)/COPY\"\nroot_prefix = 'COPY'\n\ndef remove_and_create_class(dirname):\n    if os.path.exists(dirname):\n        shutil.rmtree(dirname)\n    os.mkdir(dirname)\n    os.mkdir(dirname+'/cat')\n    os.mkdir(dirname+'/dog')\n\nremove_and_create_class('%s/train' % (root_prefix))\nremove_and_create_class('%s/valid' % (root_prefix))\n\nfor filename in filter(lambda x: x.split(\".\")[0] == \"cat\", my_train):\n    os.symlink('%s/train/' % (root_prefix)+filename, '%s/train/cat/' % (root_prefix)+filename)\nfor filename in filter(lambda x: x.split(\".\")[0] == \"dog\", my_train):\n    os.symlink('%s/train/' % (root_prefix)+filename, '%s/train/dog/' % (root_prefix)+filename)\nfor filename in filter(lambda x: x.split(\".\")[0] == \"cat\", my_cv):\n    os.symlink('%s/train/' % (root_prefix)+filename, '%s/valid/cat/' % (root_prefix)+filename)\nfor filename in filter(lambda x: x.split(\".\")[0] == \"dog\", my_cv):\n    os.symlink('%s/train/' % (root_prefix)+filename, '%s/valid/dog/' % (root_prefix)+filename)\n</code></pre>",
      "votes": null,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "315051": "I wanted to try fast.ai with this dataset, but the implementation of fast.ai needs the data to be stored in two different folders, each one of them containing either the cats or the dogs. However, I cannot try to make this because I am not allowed to create directories or move files from one directory to another. Any thoughts on how can I do it without having to change the fast.ai code?",
    "315101": "create your own dataset with the required structure and upload it on kaggle, but you still will not be able to use fastai in kernels, since it is not installed",
    "315184": "You can reorganize your data by creating symbolic links.  \n\n    from fastai.conv_learner import *\n    from sklearn.model_selection import train_test_split \n    PATH = \"../input/dogs-vs-cats-redux-kernels-edition/\"\n    sz=224\n    root_prefix = PATH\n    train_filenames = os.listdir('%s/train/' % (root_prefix))\n    test_filenames  = os.listdir('%s/test/'  % (root_prefix))\n    my_train = train_filenames\n    my_train, my_cv = train_test_split(train_filenames, test_size=0.1, random_state=0)\n    \n    from pathlib import Path\n    !cp -as \"$(pwd)/../input/\" \"$(pwd)/COPY\"\n    root_prefix = 'COPY'\n    \n    def remove_and_create_class(dirname):\n        if os.path.exists(dirname):\n            shutil.rmtree(dirname)\n        os.mkdir(dirname)\n        os.mkdir(dirname+'/cat')\n        os.mkdir(dirname+'/dog')\n    \n    remove_and_create_class('%s/train' % (root_prefix))\n    remove_and_create_class('%s/valid' % (root_prefix))\n    \n    for filename in filter(lambda x: x.split(\".\")[0] == \"cat\", my_train):\n        os.symlink('%s/train/' % (root_prefix)+filename, '%s/train/cat/' % (root_prefix)+filename)\n    for filename in filter(lambda x: x.split(\".\")[0] == \"dog\", my_train):\n        os.symlink('%s/train/' % (root_prefix)+filename, '%s/train/dog/' % (root_prefix)+filename)\n    for filename in filter(lambda x: x.split(\".\")[0] == \"cat\", my_cv):\n        os.symlink('%s/train/' % (root_prefix)+filename, '%s/valid/cat/' % (root_prefix)+filename)\n    for filename in filter(lambda x: x.split(\".\")[0] == \"dog\", my_cv):\n        os.symlink('%s/train/' % (root_prefix)+filename, '%s/valid/dog/' % (root_prefix)+filename)",
    "315185": "You can install custom packages in the \"Settings\" tab within the Kernel editor."
  },
  "source": "meta"
}