{
  "id": 332027,
  "title": "How to download this data and then divide into train and validation all on Google colab",
  "url": "/competitions/dogs-vs-cats-redux-kernels-edition/discussion/332027",
  "author_name": "",
  "post_date": "2022-06-20T03:21:32.817738500Z",
  "votes": 1,
  "comment_count": 1,
  "views": 0,
  "content": "<p>I am a beginner in Deep learning. I found a link called API for downloading this dataset <code>kaggle competitions download -c dogs-vs-cats-redux-kernels-edition</code> Now my question is, how to download this data onto the Google-drive, then how to divide this data into train and test? </p>",
  "messages": [
    {
      "id": "1826093",
      "postDate": "06/20/2022 03:21:32",
      "content": "<p>I am a beginner in Deep learning. I found a link called API for downloading this dataset <code>kaggle competitions download -c dogs-vs-cats-redux-kernels-edition</code> Now my question is, how to download this data onto the Google-drive, then how to divide this data into train and test? </p>",
      "rawMarkdown": "I am a beginner in Deep learning. I found a link called API for downloading this dataset `kaggle competitions download -c dogs-vs-cats-redux-kernels-edition` Now my question is, how to download this data onto the Google-drive, then how to divide this data into train and test?",
      "votes": null
    },
    {
      "id": "1942714",
      "postDate": "09/16/2022 22:40:40",
      "content": "<p>**To download the Data:<br>\nimport os<br>\nos.environ['KAGGLE_USERNAME'] = 'username'                                     # username from the json file<br>\nos.environ['KAGGLE_KEY'] = 'API  Key'                   # key from the json file<br>\n!kaggle competitions download -c dogs-vs-cats-redux-kernels-edition</p>\n<p>** To Devide the data: <br>\nYou can make individual directories for train test and validation and add those images in them on the basis of the categoriation and </p>\n<h1>Creating Train, Test and Validaiton directory</h1>\n<p>train_dir = os.path.join(base_dir, 'train')<br>\nos.mkdir(train_dir)<br>\nvalidation_dir = os.path.join(base_dir, 'validation')<br>\nos.mkdir(validation_dir)<br>\ntest_dir = os.path.join(base_dir, 'test')<br>\nos.mkdir(test_dir)</p>\n<h1>Training directory for Cats and Dogs</h1>\n<p>train_cats_dir = os.path.join(train_dir, 'cats')<br>\nos.mkdir(train_cats_dir)<br>\ntrain_dogs_dir = os.path.join(train_dir, 'dogs')<br>\nos.mkdir(train_dogs_dir)</p>\n<h1>Validation directory for Cats and Dogs</h1>\n<p>validation_cats_dir = os.path.join(validation_dir, 'cats')<br>\nos.mkdir(validation_cats_dir)<br>\nvalidation_dogs_dir = os.path.join(validation_dir, 'dogs')<br>\nos.mkdir(validation_dogs_dir)</p>\n<h1>Test directory for Cats and Dogs</h1>\n<p>test_cats_dir = os.path.join(test_dir, 'cats')<br>\nos.mkdir(test_cats_dir)<br>\ntest_dogs_dir = os.path.join(test_dir, 'dogs')<br>\nos.mkdir(test_dogs_dir)</p>\n<p>import shutil<br>\ndef copyfiles(a,b,c,d): <br>\n  fnames = [a.format(i) for i in range(b,c)]<br>\n  for fname in fnames:<br>\n    src = os.path.join(original_dataset_dir,fname)<br>\n    dst = os.path.join(d, fname)<br>\n    shutil.copyfile(src,dst)</p>\n<p>copyfiles('cat.{}.jpg',0,9000,train_cats_dir)<br>\ncopyfiles('cat.{}.jpg',9000,11500,validation_cats_dir)<br>\ncopyfiles('cat.{}.jpg',11500,12500,test_cats_dir)<br>\ncopyfiles('dog.{}.jpg',0,9000,train_dogs_dir)<br>\ncopyfiles('dog.{}.jpg',9000,11500,validation_dogs_dir)<br>\ncopyfiles('dog.{}.jpg',11500,12500,test_dogs_dir)<br>\n**<br>\nAfter this you can directly pass the data from the directories, you can use my code for reference**</p>",
      "rawMarkdown": "**To download the Data:\nimport os\nos.environ['KAGGLE_USERNAME'] = 'username'                                     # username from the json file\nos.environ['KAGGLE_KEY'] = 'API  Key'                   # key from the json file\n!kaggle competitions download -c dogs-vs-cats-redux-kernels-edition\n\n\n** To Devide the data: \nYou can make individual directories for train test and validation and add those images in them on the basis of the categoriation and \n\n# Creating Train, Test and Validaiton directory\ntrain_dir = os.path.join(base_dir, 'train')\nos.mkdir(train_dir)\nvalidation_dir = os.path.join(base_dir, 'validation')\nos.mkdir(validation_dir)\ntest_dir = os.path.join(base_dir, 'test')\nos.mkdir(test_dir)\n\n# Training directory for Cats and Dogs\ntrain_cats_dir = os.path.join(train_dir, 'cats')\nos.mkdir(train_cats_dir)\ntrain_dogs_dir = os.path.join(train_dir, 'dogs')\nos.mkdir(train_dogs_dir)\n\n# Validation directory for Cats and Dogs\nvalidation_cats_dir = os.path.join(validation_dir, 'cats')\nos.mkdir(validation_cats_dir)\nvalidation_dogs_dir = os.path.join(validation_dir, 'dogs')\nos.mkdir(validation_dogs_dir)\n\n# Test directory for Cats and Dogs\ntest_cats_dir = os.path.join(test_dir, 'cats')\nos.mkdir(test_cats_dir)\ntest_dogs_dir = os.path.join(test_dir, 'dogs')\nos.mkdir(test_dogs_dir)\n\nimport shutil\ndef copyfiles(a,b,c,d): \n  fnames = [a.format(i) for i in range(b,c)]\n  for fname in fnames:\n    src = os.path.join(original_dataset_dir,fname)\n    dst = os.path.join(d, fname)\n    shutil.copyfile(src,dst)\n\ncopyfiles('cat.{}.jpg',0,9000,train_cats_dir)\ncopyfiles('cat.{}.jpg',9000,11500,validation_cats_dir)\ncopyfiles('cat.{}.jpg',11500,12500,test_cats_dir)\ncopyfiles('dog.{}.jpg',0,9000,train_dogs_dir)\ncopyfiles('dog.{}.jpg',9000,11500,validation_dogs_dir)\ncopyfiles('dog.{}.jpg',11500,12500,test_dogs_dir)\n**\nAfter this you can directly pass the data from the directories, you can use my code for reference**",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 1942714,
      "author_name": "talibilat",
      "author_url": "",
      "post_date": "09/16/2022 22:40:40",
      "content": "<p>**To download the Data:<br>\nimport os<br>\nos.environ['KAGGLE_USERNAME'] = 'username'                                     # username from the json file<br>\nos.environ['KAGGLE_KEY'] = 'API  Key'                   # key from the json file<br>\n!kaggle competitions download -c dogs-vs-cats-redux-kernels-edition</p>\n<p>** To Devide the data: <br>\nYou can make individual directories for train test and validation and add those images in them on the basis of the categoriation and </p>\n<h1>Creating Train, Test and Validaiton directory</h1>\n<p>train_dir = os.path.join(base_dir, 'train')<br>\nos.mkdir(train_dir)<br>\nvalidation_dir = os.path.join(base_dir, 'validation')<br>\nos.mkdir(validation_dir)<br>\ntest_dir = os.path.join(base_dir, 'test')<br>\nos.mkdir(test_dir)</p>\n<h1>Training directory for Cats and Dogs</h1>\n<p>train_cats_dir = os.path.join(train_dir, 'cats')<br>\nos.mkdir(train_cats_dir)<br>\ntrain_dogs_dir = os.path.join(train_dir, 'dogs')<br>\nos.mkdir(train_dogs_dir)</p>\n<h1>Validation directory for Cats and Dogs</h1>\n<p>validation_cats_dir = os.path.join(validation_dir, 'cats')<br>\nos.mkdir(validation_cats_dir)<br>\nvalidation_dogs_dir = os.path.join(validation_dir, 'dogs')<br>\nos.mkdir(validation_dogs_dir)</p>\n<h1>Test directory for Cats and Dogs</h1>\n<p>test_cats_dir = os.path.join(test_dir, 'cats')<br>\nos.mkdir(test_cats_dir)<br>\ntest_dogs_dir = os.path.join(test_dir, 'dogs')<br>\nos.mkdir(test_dogs_dir)</p>\n<p>import shutil<br>\ndef copyfiles(a,b,c,d): <br>\n  fnames = [a.format(i) for i in range(b,c)]<br>\n  for fname in fnames:<br>\n    src = os.path.join(original_dataset_dir,fname)<br>\n    dst = os.path.join(d, fname)<br>\n    shutil.copyfile(src,dst)</p>\n<p>copyfiles('cat.{}.jpg',0,9000,train_cats_dir)<br>\ncopyfiles('cat.{}.jpg',9000,11500,validation_cats_dir)<br>\ncopyfiles('cat.{}.jpg',11500,12500,test_cats_dir)<br>\ncopyfiles('dog.{}.jpg',0,9000,train_dogs_dir)<br>\ncopyfiles('dog.{}.jpg',9000,11500,validation_dogs_dir)<br>\ncopyfiles('dog.{}.jpg',11500,12500,test_dogs_dir)<br>\n**<br>\nAfter this you can directly pass the data from the directories, you can use my code for reference**</p>",
      "votes": null,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "1826093": "I am a beginner in Deep learning. I found a link called API for downloading this dataset `kaggle competitions download -c dogs-vs-cats-redux-kernels-edition` Now my question is, how to download this data onto the Google-drive, then how to divide this data into train and test?",
    "1942714": "**To download the Data:\nimport os\nos.environ['KAGGLE_USERNAME'] = 'username'                                     # username from the json file\nos.environ['KAGGLE_KEY'] = 'API  Key'                   # key from the json file\n!kaggle competitions download -c dogs-vs-cats-redux-kernels-edition\n\n\n** To Devide the data: \nYou can make individual directories for train test and validation and add those images in them on the basis of the categoriation and \n\n# Creating Train, Test and Validaiton directory\ntrain_dir = os.path.join(base_dir, 'train')\nos.mkdir(train_dir)\nvalidation_dir = os.path.join(base_dir, 'validation')\nos.mkdir(validation_dir)\ntest_dir = os.path.join(base_dir, 'test')\nos.mkdir(test_dir)\n\n# Training directory for Cats and Dogs\ntrain_cats_dir = os.path.join(train_dir, 'cats')\nos.mkdir(train_cats_dir)\ntrain_dogs_dir = os.path.join(train_dir, 'dogs')\nos.mkdir(train_dogs_dir)\n\n# Validation directory for Cats and Dogs\nvalidation_cats_dir = os.path.join(validation_dir, 'cats')\nos.mkdir(validation_cats_dir)\nvalidation_dogs_dir = os.path.join(validation_dir, 'dogs')\nos.mkdir(validation_dogs_dir)\n\n# Test directory for Cats and Dogs\ntest_cats_dir = os.path.join(test_dir, 'cats')\nos.mkdir(test_cats_dir)\ntest_dogs_dir = os.path.join(test_dir, 'dogs')\nos.mkdir(test_dogs_dir)\n\nimport shutil\ndef copyfiles(a,b,c,d): \n  fnames = [a.format(i) for i in range(b,c)]\n  for fname in fnames:\n    src = os.path.join(original_dataset_dir,fname)\n    dst = os.path.join(d, fname)\n    shutil.copyfile(src,dst)\n\ncopyfiles('cat.{}.jpg',0,9000,train_cats_dir)\ncopyfiles('cat.{}.jpg',9000,11500,validation_cats_dir)\ncopyfiles('cat.{}.jpg',11500,12500,test_cats_dir)\ncopyfiles('dog.{}.jpg',0,9000,train_dogs_dir)\ncopyfiles('dog.{}.jpg',9000,11500,validation_dogs_dir)\ncopyfiles('dog.{}.jpg',11500,12500,test_dogs_dir)\n**\nAfter this you can directly pass the data from the directories, you can use my code for reference**"
  },
  "source": "meta"
}