{
  "id": 44645,
  "title": "How to Improve Pretrained Models",
  "url": "/competitions/dog-breed-identification/discussion/44645",
  "author_name": "",
  "post_date": "2017-11-30T16:26:45.592435400Z",
  "votes": 3,
  "comment_count": 2,
  "views": 0,
  "content": "<p>Hi all.</p>\n\n<p>I am new to machine learning and am trying to figure out how to improve on pretrained models. My first attempt was to use a pretrained Resnet model through Pytorch with just one linear layer on the end, as per this awesome post:</p>\n\n<p><a href=\"https://www.kaggle.com/pvlima/use-pretrained-pytorch-models4\">https://www.kaggle.com/pvlima/use-pretrained-pytorch-models4</a></p>\n\n<p>I modified the code to predict for all 120 classes and got a little over 87% accuracy on my validation set. However, I can’t seem to make it any better! I have tried adding more layers to the end of Resnet, VGG, and Inception pretrained network, but can only manage to get about 60% accuracy this way. I have tried playing with the learning rate, dropout, and number of epochs, but it is just not fitting as well as the out of the box version. I have included my general approach to modifying the VGG and Resnet models provided by Pytorch below. </p>\n\n<p>Any ideas what to try next?</p>\n\n<h1>Resnet with one linear fully connected layer with no activation</h1>\n\n<pre><code>resnet = models.resnet152(pretrained=True)\nnum_features = resnet.fc.in_features\nresnet.fc = torch.nn.Linear(num_features, 120)\n</code></pre>\n\n<h1>Resnet with multiple trainable layers:</h1>\n\n<pre><code>resnet = models.resnet152(pretrained=True)\nfor param in resnet.parameters():\n    param.requires_grad = False\nnum_features = resnet.fc.in_features\nfc_layers = nn.Sequential(\n                nn.Linear(num_features, 4096),\n                nn.ReLU(inplace=True),\n                nn.Dropout(p=0.1),\n                nn.Linear(4096, 120),\n                nn.ReLU(inplace=True),\n                nn.Dropout(p=0.1),\n            )\nresnet.fc = fc_layers\n</code></pre>\n\n<h1>VGG with multiple trainable layers:</h1>\n\n<pre><code>class Net(nn.Module):\ndef __init__(self, original_model):\n    super(Net, self).__init__()\n    self.pretrained = nn.Sequential(*list(original_model.children())[:-1])\n    self.finetuned = nn.Sequential(\n        nn.Linear(512 * 7 * 7, 4096),\n        nn.ReLU(inplace=True),\n        nn.Dropout(p=0.5),\n        nn.Linear(4096, 4096),\n        nn.ReLU(inplace=True),\n        nn.Dropout(p=0.5),\n        nn.Linear(4096, 120),\n        nn.Softmax()\n    )\n\ndef forward(self, x):\n    x = self.pretrained(x)\n    x = x.view(x.size(0), -1)\n    x = self.finetuned(x)\n    return x\n\ndef num_flat_features(self, x):\n    size = x.size()[1:]  # all dimensions except the batch dimension\n    num_features = 1\n    for s in size:\n        num_features *= s\n    return num_features\n\npretrained = models.vgg16(pretrained=True)\nfor param in pretrained.parameters():\n    param.requires_grad = False\n\nnet = Net(pretrained)\n</code></pre>",
  "messages": [
    {
      "id": "251052",
      "postDate": "11/30/2017 16:26:45",
      "content": "<p>Hi all.</p>\n\n<p>I am new to machine learning and am trying to figure out how to improve on pretrained models. My first attempt was to use a pretrained Resnet model through Pytorch with just one linear layer on the end, as per this awesome post:</p>\n\n<p><a href=\"https://www.kaggle.com/pvlima/use-pretrained-pytorch-models4\">https://www.kaggle.com/pvlima/use-pretrained-pytorch-models4</a></p>\n\n<p>I modified the code to predict for all 120 classes and got a little over 87% accuracy on my validation set. However, I can’t seem to make it any better! I have tried adding more layers to the end of Resnet, VGG, and Inception pretrained network, but can only manage to get about 60% accuracy this way. I have tried playing with the learning rate, dropout, and number of epochs, but it is just not fitting as well as the out of the box version. I have included my general approach to modifying the VGG and Resnet models provided by Pytorch below. </p>\n\n<p>Any ideas what to try next?</p>\n\n<h1>Resnet with one linear fully connected layer with no activation</h1>\n\n<pre><code>resnet = models.resnet152(pretrained=True)\nnum_features = resnet.fc.in_features\nresnet.fc = torch.nn.Linear(num_features, 120)\n</code></pre>\n\n<h1>Resnet with multiple trainable layers:</h1>\n\n<pre><code>resnet = models.resnet152(pretrained=True)\nfor param in resnet.parameters():\n    param.requires_grad = False\nnum_features = resnet.fc.in_features\nfc_layers = nn.Sequential(\n                nn.Linear(num_features, 4096),\n                nn.ReLU(inplace=True),\n                nn.Dropout(p=0.1),\n                nn.Linear(4096, 120),\n                nn.ReLU(inplace=True),\n                nn.Dropout(p=0.1),\n            )\nresnet.fc = fc_layers\n</code></pre>\n\n<h1>VGG with multiple trainable layers:</h1>\n\n<pre><code>class Net(nn.Module):\ndef __init__(self, original_model):\n    super(Net, self).__init__()\n    self.pretrained = nn.Sequential(*list(original_model.children())[:-1])\n    self.finetuned = nn.Sequential(\n        nn.Linear(512 * 7 * 7, 4096),\n        nn.ReLU(inplace=True),\n        nn.Dropout(p=0.5),\n        nn.Linear(4096, 4096),\n        nn.ReLU(inplace=True),\n        nn.Dropout(p=0.5),\n        nn.Linear(4096, 120),\n        nn.Softmax()\n    )\n\ndef forward(self, x):\n    x = self.pretrained(x)\n    x = x.view(x.size(0), -1)\n    x = self.finetuned(x)\n    return x\n\ndef num_flat_features(self, x):\n    size = x.size()[1:]  # all dimensions except the batch dimension\n    num_features = 1\n    for s in size:\n        num_features *= s\n    return num_features\n\npretrained = models.vgg16(pretrained=True)\nfor param in pretrained.parameters():\n    param.requires_grad = False\n\nnet = Net(pretrained)\n</code></pre>",
      "rawMarkdown": "Hi all.\n\nI am new to machine learning and am trying to figure out how to improve on pretrained models. My first attempt was to use a pretrained Resnet model through Pytorch with just one linear layer on the end, as per this awesome post:\n\nhttps://www.kaggle.com/pvlima/use-pretrained-pytorch-models4\n\nI modified the code to predict for all 120 classes and got a little over 87% accuracy on my validation set. However, I can’t seem to make it any better! I have tried adding more layers to the end of Resnet, VGG, and Inception pretrained network, but can only manage to get about 60% accuracy this way. I have tried playing with the learning rate, dropout, and number of epochs, but it is just not fitting as well as the out of the box version. I have included my general approach to modifying the VGG and Resnet models provided by Pytorch below. \n\nAny ideas what to try next?\n\n# Resnet with one linear fully connected layer with no activation\n\n    resnet = models.resnet152(pretrained=True)\n    num_features = resnet.fc.in_features\n    resnet.fc = torch.nn.Linear(num_features, 120)\n\n# Resnet with multiple trainable layers:\n\n    resnet = models.resnet152(pretrained=True)\n    for param in resnet.parameters():\n        param.requires_grad = False\n    num_features = resnet.fc.in_features\n    fc_layers = nn.Sequential(\n                    nn.Linear(num_features, 4096),\n                    nn.ReLU(inplace=True),\n                    nn.Dropout(p=0.1),\n                    nn.Linear(4096, 120),\n                    nn.ReLU(inplace=True),\n                    nn.Dropout(p=0.1),\n                )\n    resnet.fc = fc_layers\n\n# VGG with multiple trainable layers:\n    class Net(nn.Module):\n    def __init__(self, original_model):\n        super(Net, self).__init__()\n        self.pretrained = nn.Sequential(*list(original_model.children())[:-1])\n        self.finetuned = nn.Sequential(\n            nn.Linear(512 * 7 * 7, 4096),\n            nn.ReLU(inplace=True),\n            nn.Dropout(p=0.5),\n            nn.Linear(4096, 4096),\n            nn.ReLU(inplace=True),\n            nn.Dropout(p=0.5),\n            nn.Linear(4096, 120),\n            nn.Softmax()\n        )\n    \n    def forward(self, x):\n        x = self.pretrained(x)\n        x = x.view(x.size(0), -1)\n        x = self.finetuned(x)\n        return x\n    \n    def num_flat_features(self, x):\n        size = x.size()[1:]  # all dimensions except the batch dimension\n        num_features = 1\n        for s in size:\n            num_features *= s\n        return num_features\n\n    pretrained = models.vgg16(pretrained=True)\n    for param in pretrained.parameters():\n        param.requires_grad = False\n    \n    net = Net(pretrained)",
      "votes": null
    },
    {
      "id": "254154",
      "postDate": "12/06/2017 10:20:30",
      "content": "<p>You can also try data augmentation doing different transformations on the dataset (flip, random crop, etc).</p>",
      "rawMarkdown": "You can also try data augmentation doing different transformations on the dataset (flip, random crop, etc).",
      "votes": null
    },
    {
      "id": "254436",
      "postDate": "12/07/2017 00:02:36",
      "content": "<p>Thank you for your suggestion. I have been running the following transforms and normalization on the models mentioned above:</p>\n\n<pre><code>normalize = transforms.Normalize(\n     mean=[0.485, 0.456, 0.406],\n     std=[0.229, 0.224, 0.225]\n)\ndata_transforms = transforms.Compose([\n      transforms.Scale(224),\n      transforms.CenterCrop(224),\n      transforms.RandomHorizontalFlip(),\n      transforms.ToTensor(),\n      normalize\n])\n</code></pre>",
      "rawMarkdown": "Thank you for your suggestion. I have been running the following transforms and normalization on the models mentioned above:\n\n    normalize = transforms.Normalize(\n         mean=[0.485, 0.456, 0.406],\n         std=[0.229, 0.224, 0.225]\n    )\n    data_transforms = transforms.Compose([\n          transforms.Scale(224),\n          transforms.CenterCrop(224),\n          transforms.RandomHorizontalFlip(),\n          transforms.ToTensor(),\n          normalize\n    ])",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 254154,
      "author_name": "gntoni",
      "author_url": "",
      "post_date": "12/06/2017 10:20:30",
      "content": "<p>You can also try data augmentation doing different transformations on the dataset (flip, random crop, etc).</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 254436,
      "author_name": "vedder",
      "author_url": "",
      "post_date": "12/07/2017 00:02:36",
      "content": "<p>Thank you for your suggestion. I have been running the following transforms and normalization on the models mentioned above:</p>\n\n<pre><code>normalize = transforms.Normalize(\n     mean=[0.485, 0.456, 0.406],\n     std=[0.229, 0.224, 0.225]\n)\ndata_transforms = transforms.Compose([\n      transforms.Scale(224),\n      transforms.CenterCrop(224),\n      transforms.RandomHorizontalFlip(),\n      transforms.ToTensor(),\n      normalize\n])\n</code></pre>",
      "votes": null,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "251052": "Hi all.\n\nI am new to machine learning and am trying to figure out how to improve on pretrained models. My first attempt was to use a pretrained Resnet model through Pytorch with just one linear layer on the end, as per this awesome post:\n\nhttps://www.kaggle.com/pvlima/use-pretrained-pytorch-models4\n\nI modified the code to predict for all 120 classes and got a little over 87% accuracy on my validation set. However, I can’t seem to make it any better! I have tried adding more layers to the end of Resnet, VGG, and Inception pretrained network, but can only manage to get about 60% accuracy this way. I have tried playing with the learning rate, dropout, and number of epochs, but it is just not fitting as well as the out of the box version. I have included my general approach to modifying the VGG and Resnet models provided by Pytorch below. \n\nAny ideas what to try next?\n\n# Resnet with one linear fully connected layer with no activation\n\n    resnet = models.resnet152(pretrained=True)\n    num_features = resnet.fc.in_features\n    resnet.fc = torch.nn.Linear(num_features, 120)\n\n# Resnet with multiple trainable layers:\n\n    resnet = models.resnet152(pretrained=True)\n    for param in resnet.parameters():\n        param.requires_grad = False\n    num_features = resnet.fc.in_features\n    fc_layers = nn.Sequential(\n                    nn.Linear(num_features, 4096),\n                    nn.ReLU(inplace=True),\n                    nn.Dropout(p=0.1),\n                    nn.Linear(4096, 120),\n                    nn.ReLU(inplace=True),\n                    nn.Dropout(p=0.1),\n                )\n    resnet.fc = fc_layers\n\n# VGG with multiple trainable layers:\n    class Net(nn.Module):\n    def __init__(self, original_model):\n        super(Net, self).__init__()\n        self.pretrained = nn.Sequential(*list(original_model.children())[:-1])\n        self.finetuned = nn.Sequential(\n            nn.Linear(512 * 7 * 7, 4096),\n            nn.ReLU(inplace=True),\n            nn.Dropout(p=0.5),\n            nn.Linear(4096, 4096),\n            nn.ReLU(inplace=True),\n            nn.Dropout(p=0.5),\n            nn.Linear(4096, 120),\n            nn.Softmax()\n        )\n    \n    def forward(self, x):\n        x = self.pretrained(x)\n        x = x.view(x.size(0), -1)\n        x = self.finetuned(x)\n        return x\n    \n    def num_flat_features(self, x):\n        size = x.size()[1:]  # all dimensions except the batch dimension\n        num_features = 1\n        for s in size:\n            num_features *= s\n        return num_features\n\n    pretrained = models.vgg16(pretrained=True)\n    for param in pretrained.parameters():\n        param.requires_grad = False\n    \n    net = Net(pretrained)",
    "254154": "You can also try data augmentation doing different transformations on the dataset (flip, random crop, etc).",
    "254436": "Thank you for your suggestion. I have been running the following transforms and normalization on the models mentioned above:\n\n    normalize = transforms.Normalize(\n         mean=[0.485, 0.456, 0.406],\n         std=[0.229, 0.224, 0.225]\n    )\n    data_transforms = transforms.Compose([\n          transforms.Scale(224),\n          transforms.CenterCrop(224),\n          transforms.RandomHorizontalFlip(),\n          transforms.ToTensor(),\n          normalize\n    ])"
  },
  "source": "meta"
}