{
  "id": 50005,
  "title": "Dataset distribution between test and train set",
  "url": "/competitions/acoustic-scene-2018/discussion/50005",
  "author_name": "",
  "post_date": "2018-02-18T01:21:00.387650Z",
  "votes": null,
  "comment_count": 5,
  "views": 0,
  "content": "<p>Hello, I appreciate if any idea shared.</p>\n\n<p>I've trained a network which finally showed validation accuracy 0.74 or so, but test result was found to be very bad 0.42. Simply wondering what's wrong.</p>\n\n<ul>\n<li>Used crossvalidation_train.csv for train/valid split.</li>\n<li>Training accuracy got almost close to 1.0, and validation accuracy is 0.74. Train/valid split seems to be OK.</li>\n</ul>\n\n<p>Then I made submission results as follows:</p>\n\n<pre><code>X_test = np.load('X_test.npy')\nmodel.load_weights(WEIGHT_FILE)\ny_preds = model.predict(X_test)\nwith open('submit.csv', 'w') as f:\n    f.writelines(['Id,Scene_label\\n'])\n    f.writelines(['%d,%s\\n' % (i, int2label[np.argmax(pred)]) for i, pred in enumerate(y_preds)])\n</code></pre>\n\n<p>If it is not problem with programming failure, maybe train/valid/test data distribution would be rather different than traditional MNIST/CIFAR-10 datasets,  or ...</p>\n\n<p>Thank you.</p>",
  "messages": [
    {
      "id": "284704",
      "postDate": "02/18/2018 01:21:00",
      "content": "<p>Hello, I appreciate if any idea shared.</p>\n\n<p>I've trained a network which finally showed validation accuracy 0.74 or so, but test result was found to be very bad 0.42. Simply wondering what's wrong.</p>\n\n<ul>\n<li>Used crossvalidation_train.csv for train/valid split.</li>\n<li>Training accuracy got almost close to 1.0, and validation accuracy is 0.74. Train/valid split seems to be OK.</li>\n</ul>\n\n<p>Then I made submission results as follows:</p>\n\n<pre><code>X_test = np.load('X_test.npy')\nmodel.load_weights(WEIGHT_FILE)\ny_preds = model.predict(X_test)\nwith open('submit.csv', 'w') as f:\n    f.writelines(['Id,Scene_label\\n'])\n    f.writelines(['%d,%s\\n' % (i, int2label[np.argmax(pred)]) for i, pred in enumerate(y_preds)])\n</code></pre>\n\n<p>If it is not problem with programming failure, maybe train/valid/test data distribution would be rather different than traditional MNIST/CIFAR-10 datasets,  or ...</p>\n\n<p>Thank you.</p>",
      "rawMarkdown": "Hello, I appreciate if any idea shared.\n\nI've trained a network which finally showed validation accuracy 0.74 or so, but test result was found to be very bad 0.42. Simply wondering what's wrong.\n\n- Used crossvalidation_train.csv for train/valid split.\n- Training accuracy got almost close to 1.0, and validation accuracy is 0.74. Train/valid split seems to be OK.\n\nThen I made submission results as follows:\n\n    X_test = np.load('X_test.npy')\n    model.load_weights(WEIGHT_FILE)\n    y_preds = model.predict(X_test)\n    with open('submit.csv', 'w') as f:\n        f.writelines(['Id,Scene_label\\n'])\n        f.writelines(['%d,%s\\n' % (i, int2label[np.argmax(pred)]) for i, pred in enumerate(y_preds)])\n\nIf it is not problem with programming failure, maybe train/valid/test data distribution would be rather different than traditional MNIST/CIFAR-10 datasets,  or ...\n\nThank you.",
      "votes": null
    },
    {
      "id": "284716",
      "postDate": "02/18/2018 03:56:53",
      "content": "<p>Double checked with validation set as follows, it seems to be fine...</p>\n\n<pre><code>y_valid_preds = model.predict(X_valid)\ny_valid_pred_cls = [np.argmax(pred) for pred in y_valid_preds]\ny_valid_refs = [np.argmax(y) for y in y_valid]\n\nvalid_results = [result == ref for result, ref in zip(y_valid_pred_cls, y_valid_refs)]\n\n# accuracy\nprint(np.sum(valid_results)/len(valid_results))\n# 0.7662337662337663\n\n# double check answers\n[print(result, '\\t', ref) for result, ref in zip(y_valid_pred_cls[:10], y_valid_refs[:10])]\n# 10     10\n# 5      11\n# 5      5\n# 14     13\n# 5      5\n# 11     11\n# 5      11\n# 12     12\n# 11     11\n# 1      1\n</code></pre>",
      "rawMarkdown": "Double checked with validation set as follows, it seems to be fine...\n\n    y_valid_preds = model.predict(X_valid)\n    y_valid_pred_cls = [np.argmax(pred) for pred in y_valid_preds]\n    y_valid_refs = [np.argmax(y) for y in y_valid]\n    \n    valid_results = [result == ref for result, ref in zip(y_valid_pred_cls, y_valid_refs)]\n    \n    # accuracy\n    print(np.sum(valid_results)/len(valid_results))\n    # 0.7662337662337663\n    \n    # double check answers\n    [print(result, '\\t', ref) for result, ref in zip(y_valid_pred_cls[:10], y_valid_refs[:10])]\n    # 10 \t 10\n    # 5 \t 11\n    # 5 \t 5\n    # 14 \t 13\n    # 5 \t 5\n    # 11 \t 11\n    # 5 \t 11\n    # 12 \t 12\n    # 11 \t 11\n    # 1 \t 1",
      "votes": null
    },
    {
      "id": "284736",
      "postDate": "02/18/2018 06:22:02",
      "content": "<p>When training accuracy is almost close to 1 and validation accuracy is lagging far behind (i.e. 0.74 in your case), in general, it suggests your model is overfitting to the training set.</p>",
      "rawMarkdown": "When training accuracy is almost close to 1 and validation accuracy is lagging far behind (i.e. 0.74 in your case), in general, it suggests your model is overfitting to the training set.",
      "votes": null
    },
    {
      "id": "284970",
      "postDate": "02/19/2018 00:43:23",
      "content": "<p>Thanks, yes basically it is overfitted, I need to improve that. I found my mistake on preprocessing test set...\nI forgot to normalize test set that have made test set differs from train/valid sets for sure...</p>\n\n<pre><code># Normalize dataset\nvalue_max = np.max(np.vstack([X_train, X_valid]))\nX_train = X_train / value_max\nX_valid = X_valid / value_max\nX_test = X_test / value_max  # &lt;=== THIS WAS MISSING\n</code></pre>\n\n<p>Now I got test accuracy over 0.71, looks reasonable comparing with train/valid accuracy (close to 1 and 0.74 respectively).</p>",
      "rawMarkdown": "Thanks, yes basically it is overfitted, I need to improve that. I found my mistake on preprocessing test set...\nI forgot to normalize test set that have made test set differs from train/valid sets for sure...\n\n    # Normalize dataset\n    value_max = np.max(np.vstack([X_train, X_valid]))\n    X_train = X_train / value_max\n    X_valid = X_valid / value_max\n    X_test = X_test / value_max  # &lt;=== THIS WAS MISSING\n\nNow I got test accuracy over 0.71, looks reasonable comparing with train/valid accuracy (close to 1 and 0.74 respectively).",
      "votes": null
    },
    {
      "id": "285873",
      "postDate": "02/21/2018 06:25:30",
      "content": "<p>Glad you solved it. The distribution of classes should indeed be very even between train/validation/testing.</p>\n\n<p>And I think most people are seeing very coherent accuracies between the proposed local cross-validation split and the public leaderboard.</p>\n\n<p>Good luck!</p>\n\n<p>Heikki</p>",
      "rawMarkdown": "Glad you solved it. The distribution of classes should indeed be very even between train/validation/testing.\n\nAnd I think most people are seeing very coherent accuracies between the proposed local cross-validation split and the public leaderboard.\n\nGood luck!\n\nHeikki",
      "votes": null
    },
    {
      "id": "286525",
      "postDate": "02/22/2018 10:14:11",
      "content": "<p>Thanks for your comment, now I see consistent results with more attempts.\nI'll try some more!</p>",
      "rawMarkdown": "Thanks for your comment, now I see consistent results with more attempts.\nI'll try some more!",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 284716,
      "author_name": "daisukelab",
      "author_url": "",
      "post_date": "02/18/2018 03:56:53",
      "content": "<p>Double checked with validation set as follows, it seems to be fine...</p>\n\n<pre><code>y_valid_preds = model.predict(X_valid)\ny_valid_pred_cls = [np.argmax(pred) for pred in y_valid_preds]\ny_valid_refs = [np.argmax(y) for y in y_valid]\n\nvalid_results = [result == ref for result, ref in zip(y_valid_pred_cls, y_valid_refs)]\n\n# accuracy\nprint(np.sum(valid_results)/len(valid_results))\n# 0.7662337662337663\n\n# double check answers\n[print(result, '\\t', ref) for result, ref in zip(y_valid_pred_cls[:10], y_valid_refs[:10])]\n# 10     10\n# 5      11\n# 5      5\n# 14     13\n# 5      5\n# 11     11\n# 5      11\n# 12     12\n# 11     11\n# 1      1\n</code></pre>",
      "votes": null,
      "replies": []
    },
    {
      "id": 284736,
      "author_name": "aashita",
      "author_url": "",
      "post_date": "02/18/2018 06:22:02",
      "content": "<p>When training accuracy is almost close to 1 and validation accuracy is lagging far behind (i.e. 0.74 in your case), in general, it suggests your model is overfitting to the training set.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 284970,
      "author_name": "daisukelab",
      "author_url": "",
      "post_date": "02/19/2018 00:43:23",
      "content": "<p>Thanks, yes basically it is overfitted, I need to improve that. I found my mistake on preprocessing test set...\nI forgot to normalize test set that have made test set differs from train/valid sets for sure...</p>\n\n<pre><code># Normalize dataset\nvalue_max = np.max(np.vstack([X_train, X_valid]))\nX_train = X_train / value_max\nX_valid = X_valid / value_max\nX_test = X_test / value_max  # &lt;=== THIS WAS MISSING\n</code></pre>\n\n<p>Now I got test accuracy over 0.71, looks reasonable comparing with train/valid accuracy (close to 1 and 0.74 respectively).</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 285873,
      "author_name": "mahehu",
      "author_url": "",
      "post_date": "02/21/2018 06:25:30",
      "content": "<p>Glad you solved it. The distribution of classes should indeed be very even between train/validation/testing.</p>\n\n<p>And I think most people are seeing very coherent accuracies between the proposed local cross-validation split and the public leaderboard.</p>\n\n<p>Good luck!</p>\n\n<p>Heikki</p>",
      "votes": null,
      "replies": [
        {
          "id": 286525,
          "author_name": "daisukelab",
          "author_url": "",
          "post_date": "02/22/2018 10:14:11",
          "content": "<p>Thanks for your comment, now I see consistent results with more attempts.\nI'll try some more!</p>",
          "votes": null,
          "replies": []
        }
      ]
    }
  ],
  "raw_markdown_by_id": {
    "284704": "Hello, I appreciate if any idea shared.\n\nI've trained a network which finally showed validation accuracy 0.74 or so, but test result was found to be very bad 0.42. Simply wondering what's wrong.\n\n- Used crossvalidation_train.csv for train/valid split.\n- Training accuracy got almost close to 1.0, and validation accuracy is 0.74. Train/valid split seems to be OK.\n\nThen I made submission results as follows:\n\n    X_test = np.load('X_test.npy')\n    model.load_weights(WEIGHT_FILE)\n    y_preds = model.predict(X_test)\n    with open('submit.csv', 'w') as f:\n        f.writelines(['Id,Scene_label\\n'])\n        f.writelines(['%d,%s\\n' % (i, int2label[np.argmax(pred)]) for i, pred in enumerate(y_preds)])\n\nIf it is not problem with programming failure, maybe train/valid/test data distribution would be rather different than traditional MNIST/CIFAR-10 datasets,  or ...\n\nThank you.",
    "284716": "Double checked with validation set as follows, it seems to be fine...\n\n    y_valid_preds = model.predict(X_valid)\n    y_valid_pred_cls = [np.argmax(pred) for pred in y_valid_preds]\n    y_valid_refs = [np.argmax(y) for y in y_valid]\n    \n    valid_results = [result == ref for result, ref in zip(y_valid_pred_cls, y_valid_refs)]\n    \n    # accuracy\n    print(np.sum(valid_results)/len(valid_results))\n    # 0.7662337662337663\n    \n    # double check answers\n    [print(result, '\\t', ref) for result, ref in zip(y_valid_pred_cls[:10], y_valid_refs[:10])]\n    # 10 \t 10\n    # 5 \t 11\n    # 5 \t 5\n    # 14 \t 13\n    # 5 \t 5\n    # 11 \t 11\n    # 5 \t 11\n    # 12 \t 12\n    # 11 \t 11\n    # 1 \t 1",
    "284736": "When training accuracy is almost close to 1 and validation accuracy is lagging far behind (i.e. 0.74 in your case), in general, it suggests your model is overfitting to the training set.",
    "284970": "Thanks, yes basically it is overfitted, I need to improve that. I found my mistake on preprocessing test set...\nI forgot to normalize test set that have made test set differs from train/valid sets for sure...\n\n    # Normalize dataset\n    value_max = np.max(np.vstack([X_train, X_valid]))\n    X_train = X_train / value_max\n    X_valid = X_valid / value_max\n    X_test = X_test / value_max  # &lt;=== THIS WAS MISSING\n\nNow I got test accuracy over 0.71, looks reasonable comparing with train/valid accuracy (close to 1 and 0.74 respectively).",
    "285873": "Glad you solved it. The distribution of classes should indeed be very even between train/validation/testing.\n\nAnd I think most people are seeing very coherent accuracies between the proposed local cross-validation split and the public leaderboard.\n\nGood luck!\n\nHeikki",
    "286525": "Thanks for your comment, now I see consistent results with more attempts.\nI'll try some more!"
  },
  "source": "meta"
}