{
  "id": 24638,
  "title": "Training and validation loss stagnates at 0.6931",
  "url": "/competitions/dogs-vs-cats-redux-kernels-edition/discussion/24638",
  "author_name": "prajwalkr",
  "post_date": "2016-10-22T07:44:47.530000",
  "votes": 2,
  "comment_count": 6,
  "views": 504,
  "content": "<p>I am trying to finetune a pretrained AlexNet in Keras, but despite trying out various learning rates, optimizers and regularization my loss always stagnates to 0.6931 after a couple of epochs. </p>\n\n<pre><code>Epoch 1/5000\n\n14992/14992 [==============================] - 362s - loss: 8.2028 - val_loss: 0.6932\n\nEpoch 2/5000\n\n14992/14992 [==============================] - 363s - loss: 0.6982 - val_loss: 0.6931\n\nEpoch 3/5000\n\n14976/14992 [============================&gt;.] - ETA: 0s - loss: 0.6937Epoch 00002: val_loss did not improve\n\nEpoch 4/5000\n\n14976/14992 [============================&gt;.] - ETA: 0s - loss: 0.6938Epoch 00003: val_loss did not improve\n\nEpoch 5/5000\n\n14976/14992 [============================&gt;.] - ETA: 0s - loss: 0.6934Epoch 00004: val_loss did not improve\n\nEpoch 6/5000\n\n14992/14992 [==============================] - 383s - loss: 0.6936 - val_loss: 0.6930\n\nEpoch 7/5000\n\n14976/14992 [============================&gt;.] - ETA: 0s - loss: 0.6936Epoch 00006: val_loss did not improve\n\nEpoch 8/5000\n\n14976/14992 [============================&gt;.] - ETA: 0s - loss: 0.6934Epoch 00007: val_loss did not improve\n</code></pre>\n\n<p>I've let it train till 80 epochs, but the val_loss is exactly the same, not even a fraction of a change, as Keras says. What could be the problem?</p>\n\n<p>Thank you.</p>",
  "messages": [
    {
      "id": 144898,
      "postDate": "2016-11-15T20:21:35.543Z",
      "content": "<p>Shuffle the data and labels (retaining correspondence). </p>",
      "rawMarkdown": "Shuffle the data and labels (retaining correspondence). ",
      "votes": 1
    },
    {
      "id": 140682,
      "postDate": "2016-10-22T07:44:47.530Z",
      "content": "<p>I am trying to finetune a pretrained AlexNet in Keras, but despite trying out various learning rates, optimizers and regularization my loss always stagnates to 0.6931 after a couple of epochs. </p>\n\n<pre><code>Epoch 1/5000\n\n14992/14992 [==============================] - 362s - loss: 8.2028 - val_loss: 0.6932\n\nEpoch 2/5000\n\n14992/14992 [==============================] - 363s - loss: 0.6982 - val_loss: 0.6931\n\nEpoch 3/5000\n\n14976/14992 [============================&gt;.] - ETA: 0s - loss: 0.6937Epoch 00002: val_loss did not improve\n\nEpoch 4/5000\n\n14976/14992 [============================&gt;.] - ETA: 0s - loss: 0.6938Epoch 00003: val_loss did not improve\n\nEpoch 5/5000\n\n14976/14992 [============================&gt;.] - ETA: 0s - loss: 0.6934Epoch 00004: val_loss did not improve\n\nEpoch 6/5000\n\n14992/14992 [==============================] - 383s - loss: 0.6936 - val_loss: 0.6930\n\nEpoch 7/5000\n\n14976/14992 [============================&gt;.] - ETA: 0s - loss: 0.6936Epoch 00006: val_loss did not improve\n\nEpoch 8/5000\n\n14976/14992 [============================&gt;.] - ETA: 0s - loss: 0.6934Epoch 00007: val_loss did not improve\n</code></pre>\n\n<p>I've let it train till 80 epochs, but the val_loss is exactly the same, not even a fraction of a change, as Keras says. What could be the problem?</p>\n\n<p>Thank you.</p>",
      "rawMarkdown": "I am trying to finetune a pretrained AlexNet in Keras, but despite trying out various learning rates, optimizers and regularization my loss always stagnates to 0.6931 after a couple of epochs. \r\n\r\n    Epoch 1/5000\r\n    \r\n    14992/14992 [==============================] - 362s - loss: 8.2028 - val_loss: 0.6932\r\n    \r\n    Epoch 2/5000\r\n    \r\n    14992/14992 [==============================] - 363s - loss: 0.6982 - val_loss: 0.6931\r\n    \r\n    Epoch 3/5000\r\n    \r\n    14976/14992 [============================>.] - ETA: 0s - loss: 0.6937Epoch 00002: val_loss did not improve\r\n    \r\n    Epoch 4/5000\r\n    \r\n    14976/14992 [============================>.] - ETA: 0s - loss: 0.6938Epoch 00003: val_loss did not improve\r\n    \r\n    Epoch 5/5000\r\n    \r\n    14976/14992 [============================>.] - ETA: 0s - loss: 0.6934Epoch 00004: val_loss did not improve\r\n    \r\n    Epoch 6/5000\r\n    \r\n    14992/14992 [==============================] - 383s - loss: 0.6936 - val_loss: 0.6930\r\n    \r\n    Epoch 7/5000\r\n    \r\n    14976/14992 [============================>.] - ETA: 0s - loss: 0.6936Epoch 00006: val_loss did not improve\r\n    \r\n    Epoch 8/5000\r\n    \r\n    14976/14992 [============================>.] - ETA: 0s - loss: 0.6934Epoch 00007: val_loss did not improve\r\n\r\n\r\nI've let it train till 80 epochs, but the val_loss is exactly the same, not even a fraction of a change, as Keras says. What could be the problem?\r\n\r\nThank you.",
      "votes": 2
    },
    {
      "id": 164042,
      "postDate": "2017-02-27T13:13:21.643Z",
      "content": "<p>I'm having the same problem using Mxnet. If it's not a problem of the dataset, it is possible that the model is not yet converging. Probably more computational power is needed. Are you implementing it on a CPU or GPU? </p>",
      "rawMarkdown": "I'm having the same problem using Mxnet. If it's not a problem of the dataset, it is possible that the model is not yet converging. Probably more computational power is needed. Are you implementing it on a CPU or GPU? "
    },
    {
      "id": 159662,
      "postDate": "2017-02-03T07:17:00.680Z",
      "content": "<p>loss around 0.69, means you got a random prediction. I think your model have not converge yet.  </p>",
      "rawMarkdown": "loss around 0.69, means you got a random prediction. I think your model have not converge yet.  "
    },
    {
      "id": 140743,
      "postDate": "2016-10-22T18:50:40.760Z",
      "content": "<p>I've figured out that this happens irrespective of the CNN I use. Further, when I tested on the Kaggle test set, I see that almost <strong>all my predictions are very close to 0.0</strong> for some models or <strong>all my predictions are very close to 1.0</strong> for others.</p>\n\n<p>That is my CNN models are always being biased to predicting just one class almost all the time. How do I prevent this? </p>",
      "rawMarkdown": "I've figured out that this happens irrespective of the CNN I use. Further, when I tested on the Kaggle test set, I see that almost **all my predictions are very close to 0.0** for some models or **all my predictions are very close to 1.0** for others.\r\n\r\nThat is my CNN models are always being biased to predicting just one class almost all the time. How do I prevent this? \r\n",
      "replies": [
        {
          "id": 163775,
          "postDate": "2017-02-25T21:32:06.753Z",
          "content": "<p>If it happens with all CNNs, it could be a problem with your training data - make sure that images are imported correctly and correspond to the right labels.</p>\n\n<p>If it only happens on fine-tuning, make sure that all but a few layers are fixed (layer.trainable=False) and choose a small learning rate. As you start with a pre-trained net, loss should be small even at the start. </p>\n\n<p>A good CNN will on this task often return values close to 0.0 or 1.0, this is probably not your problem. </p>",
          "rawMarkdown": "If it happens with all CNNs, it could be a problem with your training data - make sure that images are imported correctly and correspond to the right labels.\n\nIf it only happens on fine-tuning, make sure that all but a few layers are fixed (layer.trainable=False) and choose a small learning rate. As you start with a pre-trained net, loss should be small even at the start. \n\nA good CNN will on this task often return values close to 0.0 or 1.0, this is probably not your problem. "
        }
      ]
    },
    {
      "id": 150740,
      "postDate": "2016-12-16T14:43:02.017Z",
      "rawMarkdown": "",
      "isDeleted": true
    }
  ],
  "comments": [
    {
      "id": 144898,
      "author_name": "latent4g",
      "author_url": "",
      "post_date": "2016-11-15T20:21:35.543000",
      "content": "<p>Shuffle the data and labels (retaining correspondence). </p>",
      "votes": 1,
      "replies": []
    },
    {
      "id": 164042,
      "author_name": "Michele Vitali",
      "author_url": "",
      "post_date": "2017-02-27T13:13:21.643000",
      "content": "<p>I'm having the same problem using Mxnet. If it's not a problem of the dataset, it is possible that the model is not yet converging. Probably more computational power is needed. Are you implementing it on a CPU or GPU? </p>",
      "votes": 0,
      "replies": []
    },
    {
      "id": 159662,
      "author_name": "LifeMatrix",
      "author_url": "",
      "post_date": "2017-02-03T07:17:00.680000",
      "content": "<p>loss around 0.69, means you got a random prediction. I think your model have not converge yet.  </p>",
      "votes": 0,
      "replies": []
    },
    {
      "id": 140743,
      "author_name": "prajwalkr",
      "author_url": "",
      "post_date": "2016-10-22T18:50:40.760000",
      "content": "<p>I've figured out that this happens irrespective of the CNN I use. Further, when I tested on the Kaggle test set, I see that almost <strong>all my predictions are very close to 0.0</strong> for some models or <strong>all my predictions are very close to 1.0</strong> for others.</p>\n\n<p>That is my CNN models are always being biased to predicting just one class almost all the time. How do I prevent this? </p>",
      "votes": 0,
      "replies": [
        {
          "id": 163775,
          "author_name": "Peter Grenholm",
          "author_url": "",
          "post_date": "2017-02-25T21:32:06.753000",
          "content": "<p>If it happens with all CNNs, it could be a problem with your training data - make sure that images are imported correctly and correspond to the right labels.</p>\n\n<p>If it only happens on fine-tuning, make sure that all but a few layers are fixed (layer.trainable=False) and choose a small learning rate. As you start with a pre-trained net, loss should be small even at the start. </p>\n\n<p>A good CNN will on this task often return values close to 0.0 or 1.0, this is probably not your problem. </p>",
          "votes": 0,
          "replies": []
        }
      ]
    },
    {
      "id": 150740,
      "author_name": "",
      "author_url": "",
      "post_date": "2016-12-16T14:43:02.017000",
      "content": "",
      "votes": 0,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "144898": "Shuffle the data and labels (retaining correspondence). ",
    "140682": "I am trying to finetune a pretrained AlexNet in Keras, but despite trying out various learning rates, optimizers and regularization my loss always stagnates to 0.6931 after a couple of epochs. \r\n\r\n    Epoch 1/5000\r\n    \r\n    14992/14992 [==============================] - 362s - loss: 8.2028 - val_loss: 0.6932\r\n    \r\n    Epoch 2/5000\r\n    \r\n    14992/14992 [==============================] - 363s - loss: 0.6982 - val_loss: 0.6931\r\n    \r\n    Epoch 3/5000\r\n    \r\n    14976/14992 [============================>.] - ETA: 0s - loss: 0.6937Epoch 00002: val_loss did not improve\r\n    \r\n    Epoch 4/5000\r\n    \r\n    14976/14992 [============================>.] - ETA: 0s - loss: 0.6938Epoch 00003: val_loss did not improve\r\n    \r\n    Epoch 5/5000\r\n    \r\n    14976/14992 [============================>.] - ETA: 0s - loss: 0.6934Epoch 00004: val_loss did not improve\r\n    \r\n    Epoch 6/5000\r\n    \r\n    14992/14992 [==============================] - 383s - loss: 0.6936 - val_loss: 0.6930\r\n    \r\n    Epoch 7/5000\r\n    \r\n    14976/14992 [============================>.] - ETA: 0s - loss: 0.6936Epoch 00006: val_loss did not improve\r\n    \r\n    Epoch 8/5000\r\n    \r\n    14976/14992 [============================>.] - ETA: 0s - loss: 0.6934Epoch 00007: val_loss did not improve\r\n\r\n\r\nI've let it train till 80 epochs, but the val_loss is exactly the same, not even a fraction of a change, as Keras says. What could be the problem?\r\n\r\nThank you.",
    "164042": "I'm having the same problem using Mxnet. If it's not a problem of the dataset, it is possible that the model is not yet converging. Probably more computational power is needed. Are you implementing it on a CPU or GPU? ",
    "159662": "loss around 0.69, means you got a random prediction. I think your model have not converge yet.  ",
    "140743": "I've figured out that this happens irrespective of the CNN I use. Further, when I tested on the Kaggle test set, I see that almost **all my predictions are very close to 0.0** for some models or **all my predictions are very close to 1.0** for others.\r\n\r\nThat is my CNN models are always being biased to predicting just one class almost all the time. How do I prevent this? \r\n",
    "150740": ""
  }
}