{"cells":[{"metadata":{"_cell_guid":"6bc77023-f135-4588-9c01-d0f7164b5504","_uuid":"7810e72daa92a82f8423dda602b9c9bded989016"},"cell_type":"markdown","source":"**Reorganize Data with Symbolic Links**"},{"metadata":{"_cell_guid":"bb8b6f14-3121-4a46-85a5-0c8587f173f1","_uuid":"cc835a50435863bc6484f94ee2b752a6f99ad101"},"cell_type":"markdown","source":"In computing, a symbolic link (also symlink or soft link) is a term for any file that contains a reference to another file or directory in the form of an absolute or relative path and that affects pathname resolution (Source: https://en.wikipedia.org/wiki/Symbolic_link).  Here we will use symbolic links to reorganize our data.\n\nOriginally the \"Dogs vs Cats\" images were organized inside of two folders titled \"Train\" and \"Test\" and the images of dogs and cats were easily identifiable based off of the file name for each image.  We want to reorganize the images such that now they will be organized inside of two folders titled \"Train\" and \"Valid\" and such that the images of dogs and cats will be easily identifiable based off of whether they are in a subfolder that is titled \"Dog\" or a subfolder that is titled \"Cat\".  This is the [format](http://wiki.fast.ai/index.php/Lesson_1_Notes#Data_Structure) that is required for completing exercises in the 2018 [FAST.AI](http://www.fast.ai/) Deep Learning course.\n"},{"metadata":{"_cell_guid":"b7bb304f-82af-4d58-8b51-8ed0d5dfceba","_uuid":"3a19acdb0f0ef786e42eba297da1069a0b554dcd"},"cell_type":"markdown","source":"*Step 1: Describe Original Data Organization*"},{"metadata":{"_cell_guid":"4c25482f-8822-4a5c-940b-47bffa9e960e","_uuid":"b61c42ec0c28c7ddd4ecd3c7679797e981240860","trusted":true},"cell_type":"code","source":"import os\nprint(os.listdir(\"../input\"))","execution_count":17,"outputs":[]},{"metadata":{"_cell_guid":"3949a84e-13b9-4ff3-82e2-7b45502757f3","_uuid":"644b889cbea73bf9757b7ce980f4756184c7a5ad","trusted":true,"_kg_hide-output":true},"cell_type":"code","source":"print(os.listdir(\"../input/train\"))","execution_count":18,"outputs":[]},{"metadata":{"_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","trusted":true},"cell_type":"code","source":"from sklearn.model_selection import train_test_split\nPATH = \"../input/\"\nroot_prefix = PATH\ntrain_filenames = os.listdir('%s/train/' % (root_prefix))\nprint(\"Sample of Training Data:\", train_filenames[0:10])\ntest_filenames  = os.listdir('%s/test/'  % (root_prefix))\nprint(\"\\nSample of Testing Data:\", test_filenames[0:10])","execution_count":19,"outputs":[]},{"metadata":{"_cell_guid":"a476a6f2-fc05-4156-8d98-3ea04bb92a83","_uuid":"6c77c0ee6e170ce1ed6773d87e13f1408d083756","trusted":true},"cell_type":"code","source":"my_train = train_filenames\nmy_train, my_cv = train_test_split(train_filenames, test_size=0.1, random_state=0)\nprint(\"Number of Training Images:\",len(my_train))\nprint(\"Number of Testing Images:\", len(my_cv))","execution_count":20,"outputs":[]},{"metadata":{"_cell_guid":"a0eb90d2-44f1-48f6-a271-2fd016e761f0","_uuid":"8c8838d72d74cb9b5e4ca4becc37a6d17d9f960f"},"cell_type":"markdown","source":"*Step 2: Reorganize Data Using Symbolic Links*"},{"metadata":{"_cell_guid":"e4282d97-746c-41b1-895e-80fdad060e41","_uuid":"b289fc0f5be391292a888d7950f06aee7830a3b8","trusted":true,"collapsed":true},"cell_type":"code","source":"import shutil\nfrom pathlib import Path\n# Make symlinks\n!cp -as \"$(pwd)/../input/\" \"$(pwd)/COPY\"\nroot_prefix = 'COPY'\n\ndef remove_and_create_class(dirname):\n    if os.path.exists(dirname):\n        shutil.rmtree(dirname)\n    os.mkdir(dirname)\n    os.mkdir(dirname+'/cat')\n    os.mkdir(dirname+'/dog')\n\nremove_and_create_class('%s/train' % (root_prefix))\nremove_and_create_class('%s/valid' % (root_prefix))\n\nfor filename in filter(lambda x: x.split(\".\")[0] == \"cat\", my_train):\n    os.symlink('%s/train/' % (root_prefix)+filename, '%s/train/cat/' % (root_prefix)+filename)\nfor filename in filter(lambda x: x.split(\".\")[0] == \"dog\", my_train):\n    os.symlink('%s/train/' % (root_prefix)+filename, '%s/train/dog/' % (root_prefix)+filename)\nfor filename in filter(lambda x: x.split(\".\")[0] == \"cat\", my_cv):\n    os.symlink('%s/train/' % (root_prefix)+filename, '%s/valid/cat/' % (root_prefix)+filename)\nfor filename in filter(lambda x: x.split(\".\")[0] == \"dog\", my_cv):\n    os.symlink('%s/train/' % (root_prefix)+filename, '%s/valid/dog/' % (root_prefix)+filename)","execution_count":21,"outputs":[]},{"metadata":{"_cell_guid":"099c458f-7cd6-41a6-bedb-21cb3a48687b","_uuid":"24171c6bf6fb174fb26fceb5a2c11e77778d0a29"},"cell_type":"markdown","source":"*Step 3: Describe New Data Organization*"},{"metadata":{"_cell_guid":"e69c166c-46b5-4c99-ad79-b80cd7e877cf","_uuid":"f00d4049677b6f3724c57a188d438430814e056b","trusted":true},"cell_type":"code","source":"PATH = 'COPY'\nprint(os.listdir('COPY/train'))\nprint(os.listdir('COPY/valid'))","execution_count":22,"outputs":[]},{"metadata":{"_cell_guid":"6195f35c-15cb-438c-aa2d-b77a67100741","_uuid":"5c48b15005b6482ad0c9fc0f26a7cda4b60a7252","trusted":true,"_kg_hide-output":true},"cell_type":"code","source":"print(os.listdir('COPY/valid/cat'))","execution_count":23,"outputs":[]},{"metadata":{"_cell_guid":"6928cb1d-dd15-44a0-8614-0d308d61c233","_uuid":"ec55e0f83df81e3d0cb862c5ce3b14d0fdf1ac3d","trusted":true,"collapsed":true},"cell_type":"code","source":"# Remove symlinks before committing\n!rm -rf \"$(pwd)/COPY\"","execution_count":24,"outputs":[]},{"metadata":{"_cell_guid":"91ca2205-7e06-4047-8d86-b90c63d9af55","_uuid":"598361f38b53a219c27b9f7647832e018e139a00"},"cell_type":"markdown","source":"*Step 4: Proceed with analysis using your newly reorganized data*"}],"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"name":"python","version":"3.6.5","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"}},"nbformat":4,"nbformat_minor":1}