{
  "id": 12658,
  "title": "Big difference between logLoss on validation set and that on test set",
  "url": "/competitions/datasciencebowl/discussion/12658",
  "author_name": "",
  "post_date": "2015-03-02T15:08:08.030Z",
  "votes": null,
  "comment_count": 3,
  "views": 1075,
  "content": "<p>Dear all,</p>\n<p>I'm wondering if anyone has met with similar problem when using Caffe to do the prediction.</p>\n<p>I random choose 10% from training set as validation set, which is only used to select the model. After local training, I got a model with logLoss 1.62. But when I made prediction and submitted, I got a logLoss 3.00. The way I make prediction using Caffe is from <a href=\"http://nbviewer.ipython.org/github/BVLC/caffe/blob/master/examples/classification.ipynb\">this link</a>.&nbsp;</p>\n<p>My code is like the following:</p>\n<p>net = caffe.Classifier(MODEL_FILE, PRETRAINTED, image_dims = (64,64))</p>\n<p>## read in test images into inputimage</p>\n<p>prediction[i*batchSize:(i+1)*batchSize,:] = net.predict(input_image[i*batchSize:(i+1)*batchSize])</p>\n<p>So anyone has any idea of why it is like this.</p>\n<p>Thanks.</p>",
  "messages": [
    {
      "id": "65242",
      "postDate": "03/02/2015 15:08:08",
      "content": "<p>Dear all,</p>\n<p>I'm wondering if anyone has met with similar problem when using Caffe to do the prediction.</p>\n<p>I random choose 10% from training set as validation set, which is only used to select the model. After local training, I got a model with logLoss 1.62. But when I made prediction and submitted, I got a logLoss 3.00. The way I make prediction using Caffe is from <a href=\"http://nbviewer.ipython.org/github/BVLC/caffe/blob/master/examples/classification.ipynb\">this link</a>.&nbsp;</p>\n<p>My code is like the following:</p>\n<p>net = caffe.Classifier(MODEL_FILE, PRETRAINTED, image_dims = (64,64))</p>\n<p>## read in test images into inputimage</p>\n<p>prediction[i*batchSize:(i+1)*batchSize,:] = net.predict(input_image[i*batchSize:(i+1)*batchSize])</p>\n<p>So anyone has any idea of why it is like this.</p>\n<p>Thanks.</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "65276",
      "postDate": "03/02/2015 18:31:45",
      "content": "<p>Does your validation set label distribution matches whole set label distribution? Have you took random 10% or 10% from every label? And are you not mixing labels in submission file? (For example in acantharia_protist column you write acantharia_protist_big_center predictions)</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "65289",
      "postDate": "03/02/2015 20:41:12",
      "content": "<p>[quote=Pikqu;65276]</p>\n<p>Does your validation set label distribution matches whole set label distribution? Have you took random 10% or 10% from every label? And are you not mixing labels in submission file? (For example in acantharia_protist column you write acantharia_protist_big_center predictions)</p>\n<p>[/quote]</p>\n<p>Thanks!</p>\n<p>I had the same problem...Now I check my code for submission and I may have swapped the class names!</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "65296",
      "postDate": "03/02/2015 21:40:49",
      "content": "<p>@Pikqu: Thanks. I choose 10% from each label. The reason is the order of columns. Solved now. Thanks again!</p>",
      "rawMarkdown": "",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 65276,
      "author_name": "pikquu",
      "author_url": "",
      "post_date": "03/02/2015 18:31:45",
      "content": "<p>Does your validation set label distribution matches whole set label distribution? Have you took random 10% or 10% from every label? And are you not mixing labels in submission file? (For example in acantharia_protist column you write acantharia_protist_big_center predictions)</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 65289,
      "author_name": "elenacuoco",
      "author_url": "",
      "post_date": "03/02/2015 20:41:12",
      "content": "<p>[quote=Pikqu;65276]</p>\n<p>Does your validation set label distribution matches whole set label distribution? Have you took random 10% or 10% from every label? And are you not mixing labels in submission file? (For example in acantharia_protist column you write acantharia_protist_big_center predictions)</p>\n<p>[/quote]</p>\n<p>Thanks!</p>\n<p>I had the same problem...Now I check my code for submission and I may have swapped the class names!</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 65296,
      "author_name": "negelis",
      "author_url": "",
      "post_date": "03/02/2015 21:40:49",
      "content": "<p>@Pikqu: Thanks. I choose 10% from each label. The reason is the order of columns. Solved now. Thanks again!</p>",
      "votes": null,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "65242": "",
    "65276": "",
    "65289": "",
    "65296": ""
  },
  "source": "meta"
}