{
  "id": 53620,
  "title": "Does multi-gpu mode help speed up training using matterport maskrcnn ?",
  "url": "/competitions/data-science-bowl-2018/discussion/53620",
  "author_name": "",
  "post_date": "2018-04-02T13:16:26.813082200Z",
  "votes": null,
  "comment_count": 3,
  "views": 0,
  "content": "<p>Hi, I found using 2 gtx1080ti gpu, one epoch need about 10 minutes, while using 1 gpu one epoch only need 8 mintues. Anyone also face this problem ?</p>",
  "messages": [
    {
      "id": "307789",
      "postDate": "04/02/2018 13:16:26",
      "content": "<p>Hi, I found using 2 gtx1080ti gpu, one epoch need about 10 minutes, while using 1 gpu one epoch only need 8 mintues. Anyone also face this problem ?</p>",
      "rawMarkdown": "Hi, I found using 2 gtx1080ti gpu, one epoch need about 10 minutes, while using 1 gpu one epoch only need 8 mintues. Anyone also face this problem ?",
      "votes": null
    },
    {
      "id": "308284",
      "postDate": "04/03/2018 09:15:04",
      "content": "<p>Make sure you have <code>multiprocessing</code> and number of workers set accordingly so your CPU is not the bottleneck. </p>\n\n<p>With 2 x 1080 Tis I am doing ~180 secs per epoch (one epoch = 600 samples).</p>",
      "rawMarkdown": "Make sure you have `multiprocessing` and number of workers set accordingly so your CPU is not the bottleneck. \n\nWith 2 x 1080 Tis I am doing ~180 secs per epoch (one epoch = 600 samples).",
      "votes": null
    },
    {
      "id": "309056",
      "postDate": "04/04/2018 15:11:10",
      "content": "<p>@Andres Torrubia, could you tell me how to set <strong>multiprocessing</strong> and number of workers. I use the matterport's mask_rcnn code.</p>\n\n<p>With 2 x 1080 tis I am doing about ~1000s per epoch(one epoch = 600 samples), it's quiet slow than yours. </p>\n\n<p>Epoch 51/60\n600/600 [==============================] - 1033s 2s/step - loss: 0.5418 - rpn_class_loss: 0.0102 - rpn_bbox_loss: 0.1260 - mrcnn_class_loss: 0.0670 - mrcnn_bbox_loss: 0.1166 - mrcnn_mask_loss: 0.2219 - val_loss: 1.0873 - val_rpn_class_loss: 0.0030 - val_rpn_bbox_loss: 0.3494 - val_mrcnn_class_loss: 0.0443 - val_mrcnn_bbox_loss: 0.3419 - val_mrcnn_mask_loss: 0.3486</p>\n\n<p>The envrionment are shown below:\nTensorflow-gpu : 1.7.0\nCUDA : 9.0\nCudnn : 7.0.5</p>\n\n<p>If you have any suggestion please kindly let me know. Thanks</p>",
      "rawMarkdown": "Andres Torrubia, could you tell me how to set **multiprocessing** and number of workers. I use the matterport's mask_rcnn code.\n\nWith 2 x 1080 tis I am doing about ~1000s per epoch(one epoch = 600 samples), it's quiet slow than yours. \n\nEpoch 51/60\n600/600 [==============================] - 1033s 2s/step - loss: 0.5418 - rpn_class_loss: 0.0102 - rpn_bbox_loss: 0.1260 - mrcnn_class_loss: 0.0670 - mrcnn_bbox_loss: 0.1166 - mrcnn_mask_loss: 0.2219 - val_loss: 1.0873 - val_rpn_class_loss: 0.0030 - val_rpn_bbox_loss: 0.3494 - val_mrcnn_class_loss: 0.0443 - val_mrcnn_bbox_loss: 0.3419 - val_mrcnn_mask_loss: 0.3486\n\n\nThe envrionment are shown below:\nTensorflow-gpu : 1.7.0\nCUDA : 9.0\nCudnn : 7.0.5\n\nIf you have any suggestion please kindly let me know. Thanks",
      "votes": null
    },
    {
      "id": "309149",
      "postDate": "04/04/2018 18:16:38",
      "content": "<p>I believe multiprocessing as-is does not work on Windows, we have:</p>\n\n<pre><code>         if os.name is 'nt':\n            workers = 1\n            use_multiprocessing = False\n        else:\n            workers = multiprocessing.cpu_count() - 2\n            use_multiprocessing = True\n\n        self.keras_model.fit_generator(\n            train_generator,\n            initial_epoch=self.epoch,\n            epochs=epochs,\n            steps_per_epoch=self.config.STEPS_PER_EPOCH,\n            callbacks=callbacks,\n            validation_data=val_generator if val_dataset is not None else None,\n            validation_steps=self.config.VALIDATION_STEPS if val_dataset is not None else 0,\n            max_queue_size=32,\n            workers=workers,\n            use_multiprocessing=use_multiprocessing,\n        )\n        self.epoch = max(self.epoch, epochs)\n</code></pre>\n\n<p>Hope it helps.</p>",
      "rawMarkdown": "I believe multiprocessing as-is does not work on Windows, we have:\n\n             if os.name is 'nt':\n                workers = 1\n                use_multiprocessing = False\n            else:\n                workers = multiprocessing.cpu_count() - 2\n                use_multiprocessing = True\n    \n            self.keras_model.fit_generator(\n                train_generator,\n                initial_epoch=self.epoch,\n                epochs=epochs,\n                steps_per_epoch=self.config.STEPS_PER_EPOCH,\n                callbacks=callbacks,\n                validation_data=val_generator if val_dataset is not None else None,\n                validation_steps=self.config.VALIDATION_STEPS if val_dataset is not None else 0,\n                max_queue_size=32,\n                workers=workers,\n                use_multiprocessing=use_multiprocessing,\n            )\n            self.epoch = max(self.epoch, epochs)\n\nHope it helps.",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 308284,
      "author_name": "antorsae",
      "author_url": "",
      "post_date": "04/03/2018 09:15:04",
      "content": "<p>Make sure you have <code>multiprocessing</code> and number of workers set accordingly so your CPU is not the bottleneck. </p>\n\n<p>With 2 x 1080 Tis I am doing ~180 secs per epoch (one epoch = 600 samples).</p>",
      "votes": null,
      "replies": [
        {
          "id": 309056,
          "author_name": "chiaosun",
          "author_url": "",
          "post_date": "04/04/2018 15:11:10",
          "content": "<p>@Andres Torrubia, could you tell me how to set <strong>multiprocessing</strong> and number of workers. I use the matterport's mask_rcnn code.</p>\n\n<p>With 2 x 1080 tis I am doing about ~1000s per epoch(one epoch = 600 samples), it's quiet slow than yours. </p>\n\n<p>Epoch 51/60\n600/600 [==============================] - 1033s 2s/step - loss: 0.5418 - rpn_class_loss: 0.0102 - rpn_bbox_loss: 0.1260 - mrcnn_class_loss: 0.0670 - mrcnn_bbox_loss: 0.1166 - mrcnn_mask_loss: 0.2219 - val_loss: 1.0873 - val_rpn_class_loss: 0.0030 - val_rpn_bbox_loss: 0.3494 - val_mrcnn_class_loss: 0.0443 - val_mrcnn_bbox_loss: 0.3419 - val_mrcnn_mask_loss: 0.3486</p>\n\n<p>The envrionment are shown below:\nTensorflow-gpu : 1.7.0\nCUDA : 9.0\nCudnn : 7.0.5</p>\n\n<p>If you have any suggestion please kindly let me know. Thanks</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 309149,
          "author_name": "antorsae",
          "author_url": "",
          "post_date": "04/04/2018 18:16:38",
          "content": "<p>I believe multiprocessing as-is does not work on Windows, we have:</p>\n\n<pre><code>         if os.name is 'nt':\n            workers = 1\n            use_multiprocessing = False\n        else:\n            workers = multiprocessing.cpu_count() - 2\n            use_multiprocessing = True\n\n        self.keras_model.fit_generator(\n            train_generator,\n            initial_epoch=self.epoch,\n            epochs=epochs,\n            steps_per_epoch=self.config.STEPS_PER_EPOCH,\n            callbacks=callbacks,\n            validation_data=val_generator if val_dataset is not None else None,\n            validation_steps=self.config.VALIDATION_STEPS if val_dataset is not None else 0,\n            max_queue_size=32,\n            workers=workers,\n            use_multiprocessing=use_multiprocessing,\n        )\n        self.epoch = max(self.epoch, epochs)\n</code></pre>\n\n<p>Hope it helps.</p>",
          "votes": null,
          "replies": []
        }
      ]
    }
  ],
  "raw_markdown_by_id": {
    "307789": "Hi, I found using 2 gtx1080ti gpu, one epoch need about 10 minutes, while using 1 gpu one epoch only need 8 mintues. Anyone also face this problem ?",
    "308284": "Make sure you have `multiprocessing` and number of workers set accordingly so your CPU is not the bottleneck. \n\nWith 2 x 1080 Tis I am doing ~180 secs per epoch (one epoch = 600 samples).",
    "309056": "Andres Torrubia, could you tell me how to set **multiprocessing** and number of workers. I use the matterport's mask_rcnn code.\n\nWith 2 x 1080 tis I am doing about ~1000s per epoch(one epoch = 600 samples), it's quiet slow than yours. \n\nEpoch 51/60\n600/600 [==============================] - 1033s 2s/step - loss: 0.5418 - rpn_class_loss: 0.0102 - rpn_bbox_loss: 0.1260 - mrcnn_class_loss: 0.0670 - mrcnn_bbox_loss: 0.1166 - mrcnn_mask_loss: 0.2219 - val_loss: 1.0873 - val_rpn_class_loss: 0.0030 - val_rpn_bbox_loss: 0.3494 - val_mrcnn_class_loss: 0.0443 - val_mrcnn_bbox_loss: 0.3419 - val_mrcnn_mask_loss: 0.3486\n\n\nThe envrionment are shown below:\nTensorflow-gpu : 1.7.0\nCUDA : 9.0\nCudnn : 7.0.5\n\nIf you have any suggestion please kindly let me know. Thanks",
    "309149": "I believe multiprocessing as-is does not work on Windows, we have:\n\n             if os.name is 'nt':\n                workers = 1\n                use_multiprocessing = False\n            else:\n                workers = multiprocessing.cpu_count() - 2\n                use_multiprocessing = True\n    \n            self.keras_model.fit_generator(\n                train_generator,\n                initial_epoch=self.epoch,\n                epochs=epochs,\n                steps_per_epoch=self.config.STEPS_PER_EPOCH,\n                callbacks=callbacks,\n                validation_data=val_generator if val_dataset is not None else None,\n                validation_steps=self.config.VALIDATION_STEPS if val_dataset is not None else 0,\n                max_queue_size=32,\n                workers=workers,\n                use_multiprocessing=use_multiprocessing,\n            )\n            self.epoch = max(self.epoch, epochs)\n\nHope it helps."
  },
  "source": "meta"
}