{
  "id": 105304,
  "title": "KFold vs GroupKFold",
  "url": "/competitions/champs-scalar-coupling/discussion/105304",
  "author_name": "",
  "post_date": "2019-08-22T08:19:24.733848900Z",
  "votes": 1,
  "comment_count": 3,
  "views": 0,
  "content": "<p>I used KFold CV instead of GroupKFold by 'molecule_name' because of a mistake, then my LB is much worse than CV.\nLater, I found this problem and retrained the model, then LB seems consistent with CV.\nWhy KFold results have such a big gap with LB? Is this normal? </p>",
  "messages": [
    {
      "id": "605286",
      "postDate": "08/22/2019 08:19:24",
      "content": "<p>I used KFold CV instead of GroupKFold by 'molecule_name' because of a mistake, then my LB is much worse than CV.\nLater, I found this problem and retrained the model, then LB seems consistent with CV.\nWhy KFold results have such a big gap with LB? Is this normal? </p>",
      "rawMarkdown": "I used KFold CV instead of GroupKFold by 'molecule_name' because of a mistake, then my LB is much worse than CV.\nLater, I found this problem and retrained the model, then LB seems consistent with CV.\nWhy KFold results have such a big gap with LB? Is this normal?",
      "votes": null
    },
    {
      "id": "605460",
      "postDate": "08/22/2019 11:48:22",
      "content": "<p>Likely because some of the data points within the same molecule are highly correlated. The extreme example would be a symmetric molecule, where there's two coupling constants that are exactly the same, but one of them will be in your training set and the other in the validation set. This makes the score on the validation set lower than what the score on the separate test set would be.</p>",
      "rawMarkdown": "Likely because some of the data points within the same molecule are highly correlated. The extreme example would be a symmetric molecule, where there's two coupling constants that are exactly the same, but one of them will be in your training set and the other in the validation set. This makes the score on the validation set lower than what the score on the separate test set would be.",
      "votes": null
    },
    {
      "id": "605545",
      "postDate": "08/22/2019 13:27:27",
      "content": "<p>Thank you! </p>",
      "rawMarkdown": "Thank you!",
      "votes": null
    },
    {
      "id": "607023",
      "postDate": "08/24/2019 13:11:04",
      "content": "<p>Holy crap - I never thought that the same molecule would be in train AND test - now I know where I am going wrong thanks!</p>",
      "rawMarkdown": "Holy crap - I never thought that the same molecule would be in train AND test - now I know where I am going wrong thanks!",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 605460,
      "author_name": "larsbratholm",
      "author_url": "",
      "post_date": "08/22/2019 11:48:22",
      "content": "<p>Likely because some of the data points within the same molecule are highly correlated. The extreme example would be a symmetric molecule, where there's two coupling constants that are exactly the same, but one of them will be in your training set and the other in the validation set. This makes the score on the validation set lower than what the score on the separate test set would be.</p>",
      "votes": null,
      "replies": [
        {
          "id": 605545,
          "author_name": "pku1400010735",
          "author_url": "",
          "post_date": "08/22/2019 13:27:27",
          "content": "<p>Thank you! </p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 607023,
          "author_name": "scirpus",
          "author_url": "",
          "post_date": "08/24/2019 13:11:04",
          "content": "<p>Holy crap - I never thought that the same molecule would be in train AND test - now I know where I am going wrong thanks!</p>",
          "votes": null,
          "replies": []
        }
      ]
    }
  ],
  "raw_markdown_by_id": {
    "605286": "I used KFold CV instead of GroupKFold by 'molecule_name' because of a mistake, then my LB is much worse than CV.\nLater, I found this problem and retrained the model, then LB seems consistent with CV.\nWhy KFold results have such a big gap with LB? Is this normal?",
    "605460": "Likely because some of the data points within the same molecule are highly correlated. The extreme example would be a symmetric molecule, where there's two coupling constants that are exactly the same, but one of them will be in your training set and the other in the validation set. This makes the score on the validation set lower than what the score on the separate test set would be.",
    "605545": "Thank you!",
    "607023": "Holy crap - I never thought that the same molecule would be in train AND test - now I know where I am going wrong thanks!"
  },
  "source": "meta"
}