{
  "id": 150972,
  "title": "How to train faster-RCNN with different backbones?",
  "url": "/competitions/global-wheat-detection/discussion/150972",
  "author_name": "",
  "post_date": "2020-05-14T00:17:35.163115500Z",
  "votes": 6,
  "comment_count": 8,
  "views": 0,
  "content": "<p>I am new to machine learning, I am having difficulties in using different backbones(like ResNeXt, DenseNet) for Faster-RCNN in PyTorch. Can anyone help? </p>",
  "messages": [
    {
      "id": "846710",
      "postDate": "05/14/2020 00:17:35",
      "content": "<p>I am new to machine learning, I am having difficulties in using different backbones(like ResNeXt, DenseNet) for Faster-RCNN in PyTorch. Can anyone help? </p>",
      "rawMarkdown": "I am new to machine learning, I am having difficulties in using different backbones(like ResNeXt, DenseNet) for Faster-RCNN in PyTorch. Can anyone help?",
      "votes": null
    },
    {
      "id": "846934",
      "postDate": "05/14/2020 04:32:43",
      "content": "<p>The backbones usually have two parts: 1, feature extraction module; 2, classification head. If the backbone stores the feature extraction part as an internal variable (<code>self.features = nn.Sequential(..)</code> or something similar, then you can do this:</p>\n\n<p>```\nbackbone = torchvision.models.mobilenet_v2(pretrained=True).features</p>\n\n<h1>FasterRCNN needs to know the number of output channels in a backbone.</h1>\n\n<p>backbone.out_channels = 1280\n```</p>\n\n<p>If the backbone does not store the feature extraction part as a variable, than you could do something like this:\n```\nclass MyEfficientNet(EfficientNet):</p>\n\n<pre><code>def __init__(self, blocks_args=None, global_params=None):\n    super().__init__(blocks_args, global_params)\n\ndef forward(self, inputs):\n    # Modify the forward method, so that it returns only the features.\n    return super().extract_features(inputs)\n</code></pre>\n\n<p>backbone = MyEfficientNet.from_pretrained(model_name='efficientnet-b0', num_classes=2)\nbackbone.out_channels = 1280  # EfficientNet-B0\n```</p>\n\n<p>Once you have the backbone:\n<code>\nmodel = FasterRCNN(backbone, num_classes=2, ...extra params...)\n</code></p>\n\n<p>See <a href=\"https://pytorch.org/tutorials/intermediate/torchvision_tutorial.html\">this doc</a> for more information.</p>",
      "rawMarkdown": "The backbones usually have two parts: 1, feature extraction module; 2, classification head. If the backbone stores the feature extraction part as an internal variable (`self.features = nn.Sequential(..)` or something similar, then you can do this:\n\n```\nbackbone = torchvision.models.mobilenet_v2(pretrained=True).features\n# FasterRCNN needs to know the number of output channels in a backbone.\nbackbone.out_channels = 1280\n```\n\nIf the backbone does not store the feature extraction part as a variable, than you could do something like this:\n```\nclass MyEfficientNet(EfficientNet):\n\n    def __init__(self, blocks_args=None, global_params=None):\n        super().__init__(blocks_args, global_params)\n\n    def forward(self, inputs):\n        # Modify the forward method, so that it returns only the features.\n        return super().extract_features(inputs)\n\nbackbone = MyEfficientNet.from_pretrained(model_name='efficientnet-b0', num_classes=2)\nbackbone.out_channels = 1280  # EfficientNet-B0\n```\n\nOnce you have the backbone:\n```\nmodel = FasterRCNN(backbone, num_classes=2, ...extra params...)\n```\n\nSee [this doc](https://pytorch.org/tutorials/intermediate/torchvision_tutorial.html) for more information.",
      "votes": null
    },
    {
      "id": "847039",
      "postDate": "05/14/2020 06:21:13",
      "content": "<p>Thanks <a href=\"/pestipeti\">@pestipeti</a>. So for ResNext this should work \n```\nclass ResNextFeatures(nn.Module):\n    def <strong>init</strong>(self):\n        super(ResNextFeatures, self).<strong>init</strong>()\n        base_model = torchvision.models.resnext50_32x4d(pretrained=True)</p>\n\n<pre><code>    self.seq1 = nn.Sequential(base_model.conv1, \n                              base_model.bn1,\n                              base_model.relu,\n                              base_model.maxpool)\n    self.seq2 = nn.Sequential(base_model.layer1, \n                              base_model.layer2,\n                              base_model.layer3,\n                              base_model.layer4)\n\n    self.out_channels = 2048\n\ndef forward(self, x):\n    x = self.seq1(x)\n    x = self.seq2(x)\n\n    return x\n</code></pre>\n\n<p><code>``\n</code>backbone = ResNextFeatures()`</p>\n\n<p>&amp; after that \n```\nanchor_generator = AnchorGenerator(sizes=((8, 16 , 32, 64),),\n                                   aspect_ratios=((0.5, 1.0, 2.0)))</p>\n\n<p>roi_pooler = torchvision.ops.MultiScaleRoIAlign(featmap_names=[0],\n                                                output_size=32,\n                                                sampling_ratio=2)</p>\n\n<p>model = FasterRCNN(backbone,\n                   num_classes=2,\n                   rpn_anchor_generator=anchor_generator,\n                   box_roi_pool=roi_pooler)\n```</p>\n\n<p>but I getting an error with this, may be some of the parameters that I have used are wrong, please correct me! Also,  can you please explain a bit more about <code>torchvision.ops.MultiScaleRoIAlign</code> parameters. Thanks in advance</p>",
      "rawMarkdown": "Thanks @pestipeti. So for ResNext this should work \n```\nclass ResNextFeatures(nn.Module):\n    def __init__(self):\n        super(ResNextFeatures, self).__init__()\n        base_model = torchvision.models.resnext50_32x4d(pretrained=True)\n        \n        self.seq1 = nn.Sequential(base_model.conv1, \n                                  base_model.bn1,\n                                  base_model.relu,\n                                  base_model.maxpool)\n        self.seq2 = nn.Sequential(base_model.layer1, \n                                  base_model.layer2,\n                                  base_model.layer3,\n                                  base_model.layer4)\n        \n        self.out_channels = 2048\n    \n    def forward(self, x):\n        x = self.seq1(x)\n        x = self.seq2(x)\n\n        return x\n```\n`backbone = ResNextFeatures()`\n\n&amp; after that \n```\nanchor_generator = AnchorGenerator(sizes=((8, 16 , 32, 64),),\n                                   aspect_ratios=((0.5, 1.0, 2.0)))\n\nroi_pooler = torchvision.ops.MultiScaleRoIAlign(featmap_names=[0],\n                                                output_size=32,\n                                                sampling_ratio=2)\n\nmodel = FasterRCNN(backbone,\n                   num_classes=2,\n                   rpn_anchor_generator=anchor_generator,\n                   box_roi_pool=roi_pooler)\n```\n\nbut I getting an error with this, may be some of the parameters that I have used are wrong, please correct me! Also,  can you please explain a bit more about `torchvision.ops.MultiScaleRoIAlign` parameters. Thanks in advance",
      "votes": null
    },
    {
      "id": "847062",
      "postDate": "05/14/2020 06:34:19",
      "content": "<p>I am not sure. Your backbone class seems fine. Try to create FasterRCNN without custom anchor/pooler first. </p>",
      "rawMarkdown": "I am not sure. Your backbone class seems fine. Try to create FasterRCNN without custom anchor/pooler first.",
      "votes": null
    },
    {
      "id": "847124",
      "postDate": "05/14/2020 07:28:13",
      "content": "<p>Okay</p>",
      "rawMarkdown": "Okay",
      "votes": null
    },
    {
      "id": "848456",
      "postDate": "05/15/2020 01:43:43",
      "content": "<p>I found this code at <a href=\"https://stackoverflow.com/questions/58362892/resnet-18-as-backbone-in-faster-r-cnn\">https://stackoverflow.com/questions/58362892/resnet-18-as-backbone-in-faster-r-cnn</a></p>\n\n<p><code>\nresnext_net = torchvision.models.resnext50_32x4d(pretrained=True)\nmodules = list(resnext_net.children())[:-2]\nbackbone = nn.Sequential(*modules)\nbackbone.out_channels = 2048\nmodel = FasterRCNN(backbone, num_classes=2, **kwargs)\n</code></p>\n\n<p>This might help you. I have not tested the final trained model yet but the training seems to work fine. </p>\n\n<p>Although i must point out that i get a slightly lower accuracy for the dataset of this competition while training compared to fine tuning the available fasterrcnn_resnet50_fpn model.</p>\n\n<p>Maybe i am doing something wrong idk.</p>",
      "rawMarkdown": "I found this code at https://stackoverflow.com/questions/58362892/resnet-18-as-backbone-in-faster-r-cnn\n\n```\nresnext_net = torchvision.models.resnext50_32x4d(pretrained=True)\nmodules = list(resnext_net.children())[:-2]\nbackbone = nn.Sequential(*modules)\nbackbone.out_channels = 2048\nmodel = FasterRCNN(backbone, num_classes=2, **kwargs)\n```\n\nThis might help you. I have not tested the final trained model yet but the training seems to work fine. \n\nAlthough i must point out that i get a slightly lower accuracy for the dataset of this competition while training compared to fine tuning the available fasterrcnn_resnet50_fpn model.\n\nMaybe i am doing something wrong idk.",
      "votes": null
    },
    {
      "id": "848638",
      "postDate": "05/15/2020 05:57:47",
      "content": "<p>I also trained FasterCNN with ResNext50 as the backbone. I noticed 2 things. \n1. Accuracy is lower may be we need to train it more.\n2. It's taking too much time, for a single epoch while fine-tuning it was taking around 6 min. but for training using ResNext50 for a single epoch it is taking 45 min. I don't know why this is happening.</p>",
      "rawMarkdown": "I also trained FasterCNN with ResNext50 as the backbone. I noticed 2 things. \n1. Accuracy is lower may be we need to train it more.\n2. It's taking too much time, for a single epoch while fine-tuning it was taking around 6 min. but for training using ResNext50 for a single epoch it is taking 45 min. I don't know why this is happening.",
      "votes": null
    },
    {
      "id": "849541",
      "postDate": "05/15/2020 21:46:21",
      "content": "<ol>\n<li><p>Yes, but i trained it with image augmentations also and for a good 30 Epochs what i noticed was that accuracy stops increasing after 8-9 Epochs.</p></li>\n<li><p>I think its common for ResNext to take more time but 45 mins is too much if you're getting 6 on fine-tuning. I get around the same time +1~2 mins.</p></li>\n</ol>\n\n<p>Maybe if we train it on a different dataset first like the fasterrcnnresnet50fpn was trained on and then fine-tune it for this dataset it might work. Although thats a long process and requires a good gpu (i dont have a good gpu)</p>",
      "rawMarkdown": "1. Yes, but i trained it with image augmentations also and for a good 30 Epochs what i noticed was that accuracy stops increasing after 8-9 Epochs.\n\n2. I think its common for ResNext to take more time but 45 mins is too much if you're getting 6 on fine-tuning. I get around the same time +1~2 mins.\n\nMaybe if we train it on a different dataset first like the fasterrcnnresnet50fpn was trained on and then fine-tune it for this dataset it might work. Although thats a long process and requires a good gpu (i dont have a good gpu)",
      "votes": null
    },
    {
      "id": "849552",
      "postDate": "05/15/2020 22:14:15",
      "content": "<p>I forgot about the batch_size that I am using, while training ResNext my batch_size was 1 as google colab can't handle more than that that's why I think it took 45 min.</p>",
      "rawMarkdown": "I forgot about the batch_size that I am using, while training ResNext my batch_size was 1 as google colab can't handle more than that that's why I think it took 45 min.",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 846934,
      "author_name": "pestipeti",
      "author_url": "",
      "post_date": "05/14/2020 04:32:43",
      "content": "<p>The backbones usually have two parts: 1, feature extraction module; 2, classification head. If the backbone stores the feature extraction part as an internal variable (<code>self.features = nn.Sequential(..)</code> or something similar, then you can do this:</p>\n\n<p>```\nbackbone = torchvision.models.mobilenet_v2(pretrained=True).features</p>\n\n<h1>FasterRCNN needs to know the number of output channels in a backbone.</h1>\n\n<p>backbone.out_channels = 1280\n```</p>\n\n<p>If the backbone does not store the feature extraction part as a variable, than you could do something like this:\n```\nclass MyEfficientNet(EfficientNet):</p>\n\n<pre><code>def __init__(self, blocks_args=None, global_params=None):\n    super().__init__(blocks_args, global_params)\n\ndef forward(self, inputs):\n    # Modify the forward method, so that it returns only the features.\n    return super().extract_features(inputs)\n</code></pre>\n\n<p>backbone = MyEfficientNet.from_pretrained(model_name='efficientnet-b0', num_classes=2)\nbackbone.out_channels = 1280  # EfficientNet-B0\n```</p>\n\n<p>Once you have the backbone:\n<code>\nmodel = FasterRCNN(backbone, num_classes=2, ...extra params...)\n</code></p>\n\n<p>See <a href=\"https://pytorch.org/tutorials/intermediate/torchvision_tutorial.html\">this doc</a> for more information.</p>",
      "votes": null,
      "replies": [
        {
          "id": 847039,
          "author_name": "imvivek14",
          "author_url": "",
          "post_date": "05/14/2020 06:21:13",
          "content": "<p>Thanks <a href=\"/pestipeti\">@pestipeti</a>. So for ResNext this should work \n```\nclass ResNextFeatures(nn.Module):\n    def <strong>init</strong>(self):\n        super(ResNextFeatures, self).<strong>init</strong>()\n        base_model = torchvision.models.resnext50_32x4d(pretrained=True)</p>\n\n<pre><code>    self.seq1 = nn.Sequential(base_model.conv1, \n                              base_model.bn1,\n                              base_model.relu,\n                              base_model.maxpool)\n    self.seq2 = nn.Sequential(base_model.layer1, \n                              base_model.layer2,\n                              base_model.layer3,\n                              base_model.layer4)\n\n    self.out_channels = 2048\n\ndef forward(self, x):\n    x = self.seq1(x)\n    x = self.seq2(x)\n\n    return x\n</code></pre>\n\n<p><code>``\n</code>backbone = ResNextFeatures()`</p>\n\n<p>&amp; after that \n```\nanchor_generator = AnchorGenerator(sizes=((8, 16 , 32, 64),),\n                                   aspect_ratios=((0.5, 1.0, 2.0)))</p>\n\n<p>roi_pooler = torchvision.ops.MultiScaleRoIAlign(featmap_names=[0],\n                                                output_size=32,\n                                                sampling_ratio=2)</p>\n\n<p>model = FasterRCNN(backbone,\n                   num_classes=2,\n                   rpn_anchor_generator=anchor_generator,\n                   box_roi_pool=roi_pooler)\n```</p>\n\n<p>but I getting an error with this, may be some of the parameters that I have used are wrong, please correct me! Also,  can you please explain a bit more about <code>torchvision.ops.MultiScaleRoIAlign</code> parameters. Thanks in advance</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 847062,
          "author_name": "pestipeti",
          "author_url": "",
          "post_date": "05/14/2020 06:34:19",
          "content": "<p>I am not sure. Your backbone class seems fine. Try to create FasterRCNN without custom anchor/pooler first. </p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 847124,
          "author_name": "imvivek14",
          "author_url": "",
          "post_date": "05/14/2020 07:28:13",
          "content": "<p>Okay</p>",
          "votes": null,
          "replies": []
        }
      ]
    },
    {
      "id": 848456,
      "author_name": "johndoexxx",
      "author_url": "",
      "post_date": "05/15/2020 01:43:43",
      "content": "<p>I found this code at <a href=\"https://stackoverflow.com/questions/58362892/resnet-18-as-backbone-in-faster-r-cnn\">https://stackoverflow.com/questions/58362892/resnet-18-as-backbone-in-faster-r-cnn</a></p>\n\n<p><code>\nresnext_net = torchvision.models.resnext50_32x4d(pretrained=True)\nmodules = list(resnext_net.children())[:-2]\nbackbone = nn.Sequential(*modules)\nbackbone.out_channels = 2048\nmodel = FasterRCNN(backbone, num_classes=2, **kwargs)\n</code></p>\n\n<p>This might help you. I have not tested the final trained model yet but the training seems to work fine. </p>\n\n<p>Although i must point out that i get a slightly lower accuracy for the dataset of this competition while training compared to fine tuning the available fasterrcnn_resnet50_fpn model.</p>\n\n<p>Maybe i am doing something wrong idk.</p>",
      "votes": null,
      "replies": [
        {
          "id": 848638,
          "author_name": "imvivek14",
          "author_url": "",
          "post_date": "05/15/2020 05:57:47",
          "content": "<p>I also trained FasterCNN with ResNext50 as the backbone. I noticed 2 things. \n1. Accuracy is lower may be we need to train it more.\n2. It's taking too much time, for a single epoch while fine-tuning it was taking around 6 min. but for training using ResNext50 for a single epoch it is taking 45 min. I don't know why this is happening.</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 849541,
          "author_name": "johndoexxx",
          "author_url": "",
          "post_date": "05/15/2020 21:46:21",
          "content": "<ol>\n<li><p>Yes, but i trained it with image augmentations also and for a good 30 Epochs what i noticed was that accuracy stops increasing after 8-9 Epochs.</p></li>\n<li><p>I think its common for ResNext to take more time but 45 mins is too much if you're getting 6 on fine-tuning. I get around the same time +1~2 mins.</p></li>\n</ol>\n\n<p>Maybe if we train it on a different dataset first like the fasterrcnnresnet50fpn was trained on and then fine-tune it for this dataset it might work. Although thats a long process and requires a good gpu (i dont have a good gpu)</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 849552,
          "author_name": "imvivek14",
          "author_url": "",
          "post_date": "05/15/2020 22:14:15",
          "content": "<p>I forgot about the batch_size that I am using, while training ResNext my batch_size was 1 as google colab can't handle more than that that's why I think it took 45 min.</p>",
          "votes": null,
          "replies": []
        }
      ]
    }
  ],
  "raw_markdown_by_id": {
    "846710": "I am new to machine learning, I am having difficulties in using different backbones(like ResNeXt, DenseNet) for Faster-RCNN in PyTorch. Can anyone help?",
    "846934": "The backbones usually have two parts: 1, feature extraction module; 2, classification head. If the backbone stores the feature extraction part as an internal variable (`self.features = nn.Sequential(..)` or something similar, then you can do this:\n\n```\nbackbone = torchvision.models.mobilenet_v2(pretrained=True).features\n# FasterRCNN needs to know the number of output channels in a backbone.\nbackbone.out_channels = 1280\n```\n\nIf the backbone does not store the feature extraction part as a variable, than you could do something like this:\n```\nclass MyEfficientNet(EfficientNet):\n\n    def __init__(self, blocks_args=None, global_params=None):\n        super().__init__(blocks_args, global_params)\n\n    def forward(self, inputs):\n        # Modify the forward method, so that it returns only the features.\n        return super().extract_features(inputs)\n\nbackbone = MyEfficientNet.from_pretrained(model_name='efficientnet-b0', num_classes=2)\nbackbone.out_channels = 1280  # EfficientNet-B0\n```\n\nOnce you have the backbone:\n```\nmodel = FasterRCNN(backbone, num_classes=2, ...extra params...)\n```\n\nSee [this doc](https://pytorch.org/tutorials/intermediate/torchvision_tutorial.html) for more information.",
    "847039": "Thanks @pestipeti. So for ResNext this should work \n```\nclass ResNextFeatures(nn.Module):\n    def __init__(self):\n        super(ResNextFeatures, self).__init__()\n        base_model = torchvision.models.resnext50_32x4d(pretrained=True)\n        \n        self.seq1 = nn.Sequential(base_model.conv1, \n                                  base_model.bn1,\n                                  base_model.relu,\n                                  base_model.maxpool)\n        self.seq2 = nn.Sequential(base_model.layer1, \n                                  base_model.layer2,\n                                  base_model.layer3,\n                                  base_model.layer4)\n        \n        self.out_channels = 2048\n    \n    def forward(self, x):\n        x = self.seq1(x)\n        x = self.seq2(x)\n\n        return x\n```\n`backbone = ResNextFeatures()`\n\n&amp; after that \n```\nanchor_generator = AnchorGenerator(sizes=((8, 16 , 32, 64),),\n                                   aspect_ratios=((0.5, 1.0, 2.0)))\n\nroi_pooler = torchvision.ops.MultiScaleRoIAlign(featmap_names=[0],\n                                                output_size=32,\n                                                sampling_ratio=2)\n\nmodel = FasterRCNN(backbone,\n                   num_classes=2,\n                   rpn_anchor_generator=anchor_generator,\n                   box_roi_pool=roi_pooler)\n```\n\nbut I getting an error with this, may be some of the parameters that I have used are wrong, please correct me! Also,  can you please explain a bit more about `torchvision.ops.MultiScaleRoIAlign` parameters. Thanks in advance",
    "847062": "I am not sure. Your backbone class seems fine. Try to create FasterRCNN without custom anchor/pooler first.",
    "847124": "Okay",
    "848456": "I found this code at https://stackoverflow.com/questions/58362892/resnet-18-as-backbone-in-faster-r-cnn\n\n```\nresnext_net = torchvision.models.resnext50_32x4d(pretrained=True)\nmodules = list(resnext_net.children())[:-2]\nbackbone = nn.Sequential(*modules)\nbackbone.out_channels = 2048\nmodel = FasterRCNN(backbone, num_classes=2, **kwargs)\n```\n\nThis might help you. I have not tested the final trained model yet but the training seems to work fine. \n\nAlthough i must point out that i get a slightly lower accuracy for the dataset of this competition while training compared to fine tuning the available fasterrcnn_resnet50_fpn model.\n\nMaybe i am doing something wrong idk.",
    "848638": "I also trained FasterCNN with ResNext50 as the backbone. I noticed 2 things. \n1. Accuracy is lower may be we need to train it more.\n2. It's taking too much time, for a single epoch while fine-tuning it was taking around 6 min. but for training using ResNext50 for a single epoch it is taking 45 min. I don't know why this is happening.",
    "849541": "1. Yes, but i trained it with image augmentations also and for a good 30 Epochs what i noticed was that accuracy stops increasing after 8-9 Epochs.\n\n2. I think its common for ResNext to take more time but 45 mins is too much if you're getting 6 on fine-tuning. I get around the same time +1~2 mins.\n\nMaybe if we train it on a different dataset first like the fasterrcnnresnet50fpn was trained on and then fine-tune it for this dataset it might work. Although thats a long process and requires a good gpu (i dont have a good gpu)",
    "849552": "I forgot about the batch_size that I am using, while training ResNext my batch_size was 1 as google colab can't handle more than that that's why I think it took 45 min."
  },
  "source": "meta"
}