{
  "id": 95098,
  "title": "why leader board score better than cv score?",
  "url": "/competitions/champs-scalar-coupling/discussion/95098",
  "author_name": "",
  "post_date": "2019-06-10T01:29:09.723483700Z",
  "votes": 5,
  "comment_count": 3,
  "views": 0,
  "content": "<p>So I just enter this competition a few days ago. I don't know if anyone else has the same issue. It seems that my leader board score is constantly better than my cv validation score. It's quite odd. Hope some one can shed some light.\nThis is how I calculate the validation score:\nfor i,t in enumerate(typelist): #type list is a list of unique molecule types\n    sum_of_score += np.log(mean_absolute_error(target_of_current_type , out_of_fold_predictions_of_current_type))</p>\n\n<p>score=sum_of_score /len(typelist)</p>\n\n<p>mean_absolute_error is from sklearn</p>",
  "messages": [
    {
      "id": "548827",
      "postDate": "06/10/2019 01:29:09",
      "content": "<p>So I just enter this competition a few days ago. I don't know if anyone else has the same issue. It seems that my leader board score is constantly better than my cv validation score. It's quite odd. Hope some one can shed some light.\nThis is how I calculate the validation score:\nfor i,t in enumerate(typelist): #type list is a list of unique molecule types\n    sum_of_score += np.log(mean_absolute_error(target_of_current_type , out_of_fold_predictions_of_current_type))</p>\n\n<p>score=sum_of_score /len(typelist)</p>\n\n<p>mean_absolute_error is from sklearn</p>",
      "rawMarkdown": "So I just enter this competition a few days ago. I don't know if anyone else has the same issue. It seems that my leader board score is constantly better than my cv validation score. It's quite odd. Hope some one can shed some light.\nThis is how I calculate the validation score:\nfor i,t in enumerate(typelist): #type list is a list of unique molecule types\n    sum\\_of\\_score += np.log(mean\\_absolute\\_error(target\\_of\\_current\\_type , out\\_of\\_fold\\_predictions\\_of\\_current_type))\n\nscore=sum\\_of\\_score /len(typelist)\n\nmean\\_absolute\\_error is from sklearn",
      "votes": null
    },
    {
      "id": "548893",
      "postDate": "06/10/2019 04:24:48",
      "content": "<p>It seems to me that your calculation for the validation score is not wrong. Same as you, my LB score is always higher than CV score. (ex. a subimission: CV: -0.237 -&gt; LB: -0.466)</p>\n\n<p>I also want to know why these things happen...</p>",
      "rawMarkdown": "It seems to me that your calculation for the validation score is not wrong. Same as you, my LB score is always higher than CV score. (ex. a subimission: CV: -0.237 -&gt; LB: -0.466)\n\nI also want to know why these things happen...",
      "votes": null
    },
    {
      "id": "551520",
      "postDate": "06/12/2019 20:03:13",
      "content": "<p>Maybe Kaggle implemented the evaluation metric wrong like log of individual absolute errors or MAE of logs etc? There was such an issue in this competition and was corrected later: <a href=\"https://www.kaggle.com/c/ga-customer-revenue-prediction\">https://www.kaggle.com/c/ga-customer-revenue-prediction</a></p>",
      "rawMarkdown": "Maybe Kaggle implemented the evaluation metric wrong like log of individual absolute errors or MAE of logs etc? There was such an issue in this competition and was corrected later: https://www.kaggle.com/c/ga-customer-revenue-prediction",
      "votes": null
    },
    {
      "id": "551620",
      "postDate": "06/12/2019 23:59:05",
      "content": "<p>Oh, the nightmare again? Maybe need to validate it, if so.</p>",
      "rawMarkdown": "Oh, the nightmare again? Maybe need to validate it, if so.",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 548893,
      "author_name": "takaishikawa",
      "author_url": "",
      "post_date": "06/10/2019 04:24:48",
      "content": "<p>It seems to me that your calculation for the validation score is not wrong. Same as you, my LB score is always higher than CV score. (ex. a subimission: CV: -0.237 -&gt; LB: -0.466)</p>\n\n<p>I also want to know why these things happen...</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 551520,
      "author_name": "aerdem4",
      "author_url": "",
      "post_date": "06/12/2019 20:03:13",
      "content": "<p>Maybe Kaggle implemented the evaluation metric wrong like log of individual absolute errors or MAE of logs etc? There was such an issue in this competition and was corrected later: <a href=\"https://www.kaggle.com/c/ga-customer-revenue-prediction\">https://www.kaggle.com/c/ga-customer-revenue-prediction</a></p>",
      "votes": null,
      "replies": [
        {
          "id": 551620,
          "author_name": "takaishikawa",
          "author_url": "",
          "post_date": "06/12/2019 23:59:05",
          "content": "<p>Oh, the nightmare again? Maybe need to validate it, if so.</p>",
          "votes": null,
          "replies": []
        }
      ]
    }
  ],
  "raw_markdown_by_id": {
    "548827": "So I just enter this competition a few days ago. I don't know if anyone else has the same issue. It seems that my leader board score is constantly better than my cv validation score. It's quite odd. Hope some one can shed some light.\nThis is how I calculate the validation score:\nfor i,t in enumerate(typelist): #type list is a list of unique molecule types\n    sum\\_of\\_score += np.log(mean\\_absolute\\_error(target\\_of\\_current\\_type , out\\_of\\_fold\\_predictions\\_of\\_current_type))\n\nscore=sum\\_of\\_score /len(typelist)\n\nmean\\_absolute\\_error is from sklearn",
    "548893": "It seems to me that your calculation for the validation score is not wrong. Same as you, my LB score is always higher than CV score. (ex. a subimission: CV: -0.237 -&gt; LB: -0.466)\n\nI also want to know why these things happen...",
    "551520": "Maybe Kaggle implemented the evaluation metric wrong like log of individual absolute errors or MAE of logs etc? There was such an issue in this competition and was corrected later: https://www.kaggle.com/c/ga-customer-revenue-prediction",
    "551620": "Oh, the nightmare again? Maybe need to validate it, if so."
  },
  "source": "meta"
}