{
  "id": 96724,
  "title": "puzzling problem with CV/LB",
  "url": "/competitions/champs-scalar-coupling/discussion/96724",
  "author_name": "",
  "post_date": "2019-06-22T13:42:10.487054100Z",
  "votes": null,
  "comment_count": 5,
  "views": 0,
  "content": "<p>I have strange problem with CV/LB score, maybe you could help.\nI have kernel with features like that:</p>\n\n<p><code>\ntrain_g = train.groupby('molecule_name')\ntest_g = test.groupby('molecule_name')\ntrain[\"molecule_count\"] = train_g[\"id\"].transform(\"count\")\ntest[\"molecule_count\"] = test_g[\"id\"].transform(\"count\")\ntrain[\"molecule_distance_max\"] = train_g[\"distance\"].transform(\"max\")\ntest[\"molecule_distance_max\"] = test_g[\"distance\"].transform(\"max\")\ntrain[\"molecule_distance_min\"] = train_g[\"distance\"].transform(\"min\")\ntest[\"molecule_distance_min\"] = test_g[\"distance\"].transform(\"min\")\ntrain[\"molecule_distance_mean\"] = train_g[\"distance\"].transform(\"mean\")\ntest[\"molecule_distance_mean\"] = test_g[\"distance\"].transform(\"mean\")\ntrain[\"molecule_distance_div_max\"] = train['distance'] / train[\"molecule_distance_max\"]\ntest[\"molecule_distance_div_max\"] = test['distance'] / test[\"molecule_distance_max\"]\ntrain[\"molecule_distance_div_min\"] = train['distance'] / train[\"molecule_distance_min\"]\ntest[\"molecule_distance_div_min\"] = test['distance'] / test[\"molecule_distance_min\"]\ntrain[\"molecule_distance_div_mean\"] = train['distance'] / train[\"molecule_distance_mean\"]\ntest[\"molecule_distance_div_mean\"] = test['distance'] / test[\"molecule_distance_mean\"]\n</code></p>\n\n<p>and:\n```</p>\n\n<p>train_t_g = train.groupby('type')\ntest_t_g = test.groupby('type')\ntrain[\"type_distance_mean\"] = train_t_g[\"distance\"].transform(\"mean\")\ntest[\"type_distance_mean\"] = test_t_g[\"distance\"].transform(\"mean\")\ntrain[\"type_distance_div_mean\"] = train['distance'] / train[\"type_distance_mean\"]\ntest[\"type_distance_div_mean\"] = test['distance'] / test[\"type_distance_mean\"]\n```</p>\n\n<p>I use simple train/test split and my validation score is:</p>\n\n<p>[16569] training's l1: 1.35908  valid_0's l1: 1.74419</p>\n\n<p>on LB my score is 0.460</p>\n\n<p>now please notice what is different here:</p>\n\n<p>```\ntrain_g = train.groupby('molecule_name')\ntest_g = test.groupby('molecule_name')\ntrain[\"molecule_count\"] = train_g[\"id\"].transform(\"count\")\ntest[\"molecule_count\"] = test_g[\"id\"].transform(\"count\")\ntrain[\"molecule_distance_max\"] = train_g[\"distance\"].transform(\"max\")\ntest[\"molecule_distance_max\"] = test_g[\"distance\"].transform(\"max\")\ntrain[\"molecule_distance_min\"] = train_g[\"distance\"].transform(\"min\")\ntest[\"molecule_distance_min\"] = test_g[\"distance\"].transform(\"min\")\ntrain[\"molecule_distance_mean\"] = train_g[\"distance\"].transform(\"mean\")\ntest[\"molecule_distance_mean\"] = test_g[\"distance\"].transform(\"mean\")\ntrain[\"molecule_distance_div_max\"] = train['distance'] / train[\"molecule_distance_max\"]\ntest[\"molecule_distance_div_max\"] = test['distance'] / test[\"molecule_distance_max\"]\ntrain[\"molecule_distance_div_min\"] = train['distance'] / train[\"molecule_distance_min\"]\ntest[\"molecule_distance_div_min\"] = test['distance'] / test[\"molecule_distance_min\"]\ntrain[\"molecule_distance_div_mean\"] = train['distance'] / train[\"molecule_distance_mean\"]\ntest[\"molecule_distance_div_mean\"] = test['distance'] / test[\"molecule_distance_mean\"]</p>\n\n<p>train_t_g = train.groupby('type')\ntest_t_g = test.groupby('type')</p>\n\n<p>train[\"type_distance_mean\"] = train_t_g[\"distance\"].transform(\"mean\")\ntest[\"type_distance_mean\"] = test_t_g[\"distance\"].transform(\"mean\")\ntrain[\"type_distance_div_mean\"] = train['distance'] / train[\"type_distance_mean\"]\ntest[\"type_distance_div_mean\"] = test['distance'] / test[\"type_distance_mean\"]</p>\n\n<p>train[\"type_distance_min\"] = train_t_g[\"distance\"].transform(\"min\")\ntest[\"type_distance_min\"] = test_t_g[\"distance\"].transform(\"min\")\ntrain[\"type_distance_div_min\"] = train['distance'] / train[\"type_distance_min\"]\ntest[\"type_distance_div_min\"] = test['distance'] / test[\"type_distance_min\"]</p>\n\n<p>train[\"type_distance_max\"] = train_t_g[\"distance\"].transform(\"max\")\ntest[\"type_distance_max\"] = test_t_g[\"distance\"].transform(\"max\")\ntrain[\"type_distance_div_max\"] = train['distance'] / train[\"type_distance_max\"]\ntest[\"type_distance_div_max\"] = test['distance'] / test[\"type_distance_max\"]\n```</p>\n\n<p>I just added min and max distances per type.</p>\n\n<p>The local result is:</p>\n\n<p>[16051] training's l1: 1.3594   valid_1's l1: 1.73379</p>\n\n<p>but LB is 0.989</p>\n\n<p>so LB is much worse than previous one, looks like some heavy overfitting</p>\n\n<p>this is my <strong>second time with this case</strong> because I have seen similar result before in different kernel so I was searching for code or logic error but wasn't able to find one</p>\n\n<p>do you have any clues what may be wrong?</p>\n\n<p>in the beginning i was trying to use features only available for train set so this kind of result was expected, but this time I just calc distances, I am puzzled where can be the error</p>",
  "messages": [
    {
      "id": "558515",
      "postDate": "06/22/2019 13:42:10",
      "content": "<p>I have strange problem with CV/LB score, maybe you could help.\nI have kernel with features like that:</p>\n\n<p><code>\ntrain_g = train.groupby('molecule_name')\ntest_g = test.groupby('molecule_name')\ntrain[\"molecule_count\"] = train_g[\"id\"].transform(\"count\")\ntest[\"molecule_count\"] = test_g[\"id\"].transform(\"count\")\ntrain[\"molecule_distance_max\"] = train_g[\"distance\"].transform(\"max\")\ntest[\"molecule_distance_max\"] = test_g[\"distance\"].transform(\"max\")\ntrain[\"molecule_distance_min\"] = train_g[\"distance\"].transform(\"min\")\ntest[\"molecule_distance_min\"] = test_g[\"distance\"].transform(\"min\")\ntrain[\"molecule_distance_mean\"] = train_g[\"distance\"].transform(\"mean\")\ntest[\"molecule_distance_mean\"] = test_g[\"distance\"].transform(\"mean\")\ntrain[\"molecule_distance_div_max\"] = train['distance'] / train[\"molecule_distance_max\"]\ntest[\"molecule_distance_div_max\"] = test['distance'] / test[\"molecule_distance_max\"]\ntrain[\"molecule_distance_div_min\"] = train['distance'] / train[\"molecule_distance_min\"]\ntest[\"molecule_distance_div_min\"] = test['distance'] / test[\"molecule_distance_min\"]\ntrain[\"molecule_distance_div_mean\"] = train['distance'] / train[\"molecule_distance_mean\"]\ntest[\"molecule_distance_div_mean\"] = test['distance'] / test[\"molecule_distance_mean\"]\n</code></p>\n\n<p>and:\n```</p>\n\n<p>train_t_g = train.groupby('type')\ntest_t_g = test.groupby('type')\ntrain[\"type_distance_mean\"] = train_t_g[\"distance\"].transform(\"mean\")\ntest[\"type_distance_mean\"] = test_t_g[\"distance\"].transform(\"mean\")\ntrain[\"type_distance_div_mean\"] = train['distance'] / train[\"type_distance_mean\"]\ntest[\"type_distance_div_mean\"] = test['distance'] / test[\"type_distance_mean\"]\n```</p>\n\n<p>I use simple train/test split and my validation score is:</p>\n\n<p>[16569] training's l1: 1.35908  valid_0's l1: 1.74419</p>\n\n<p>on LB my score is 0.460</p>\n\n<p>now please notice what is different here:</p>\n\n<p>```\ntrain_g = train.groupby('molecule_name')\ntest_g = test.groupby('molecule_name')\ntrain[\"molecule_count\"] = train_g[\"id\"].transform(\"count\")\ntest[\"molecule_count\"] = test_g[\"id\"].transform(\"count\")\ntrain[\"molecule_distance_max\"] = train_g[\"distance\"].transform(\"max\")\ntest[\"molecule_distance_max\"] = test_g[\"distance\"].transform(\"max\")\ntrain[\"molecule_distance_min\"] = train_g[\"distance\"].transform(\"min\")\ntest[\"molecule_distance_min\"] = test_g[\"distance\"].transform(\"min\")\ntrain[\"molecule_distance_mean\"] = train_g[\"distance\"].transform(\"mean\")\ntest[\"molecule_distance_mean\"] = test_g[\"distance\"].transform(\"mean\")\ntrain[\"molecule_distance_div_max\"] = train['distance'] / train[\"molecule_distance_max\"]\ntest[\"molecule_distance_div_max\"] = test['distance'] / test[\"molecule_distance_max\"]\ntrain[\"molecule_distance_div_min\"] = train['distance'] / train[\"molecule_distance_min\"]\ntest[\"molecule_distance_div_min\"] = test['distance'] / test[\"molecule_distance_min\"]\ntrain[\"molecule_distance_div_mean\"] = train['distance'] / train[\"molecule_distance_mean\"]\ntest[\"molecule_distance_div_mean\"] = test['distance'] / test[\"molecule_distance_mean\"]</p>\n\n<p>train_t_g = train.groupby('type')\ntest_t_g = test.groupby('type')</p>\n\n<p>train[\"type_distance_mean\"] = train_t_g[\"distance\"].transform(\"mean\")\ntest[\"type_distance_mean\"] = test_t_g[\"distance\"].transform(\"mean\")\ntrain[\"type_distance_div_mean\"] = train['distance'] / train[\"type_distance_mean\"]\ntest[\"type_distance_div_mean\"] = test['distance'] / test[\"type_distance_mean\"]</p>\n\n<p>train[\"type_distance_min\"] = train_t_g[\"distance\"].transform(\"min\")\ntest[\"type_distance_min\"] = test_t_g[\"distance\"].transform(\"min\")\ntrain[\"type_distance_div_min\"] = train['distance'] / train[\"type_distance_min\"]\ntest[\"type_distance_div_min\"] = test['distance'] / test[\"type_distance_min\"]</p>\n\n<p>train[\"type_distance_max\"] = train_t_g[\"distance\"].transform(\"max\")\ntest[\"type_distance_max\"] = test_t_g[\"distance\"].transform(\"max\")\ntrain[\"type_distance_div_max\"] = train['distance'] / train[\"type_distance_max\"]\ntest[\"type_distance_div_max\"] = test['distance'] / test[\"type_distance_max\"]\n```</p>\n\n<p>I just added min and max distances per type.</p>\n\n<p>The local result is:</p>\n\n<p>[16051] training's l1: 1.3594   valid_1's l1: 1.73379</p>\n\n<p>but LB is 0.989</p>\n\n<p>so LB is much worse than previous one, looks like some heavy overfitting</p>\n\n<p>this is my <strong>second time with this case</strong> because I have seen similar result before in different kernel so I was searching for code or logic error but wasn't able to find one</p>\n\n<p>do you have any clues what may be wrong?</p>\n\n<p>in the beginning i was trying to use features only available for train set so this kind of result was expected, but this time I just calc distances, I am puzzled where can be the error</p>",
      "rawMarkdown": "I have strange problem with CV/LB score, maybe you could help.\nI have kernel with features like that:\n\n```\ntrain_g = train.groupby('molecule_name')\ntest_g = test.groupby('molecule_name')\ntrain[\"molecule_count\"] = train_g[\"id\"].transform(\"count\")\ntest[\"molecule_count\"] = test_g[\"id\"].transform(\"count\")\ntrain[\"molecule_distance_max\"] = train_g[\"distance\"].transform(\"max\")\ntest[\"molecule_distance_max\"] = test_g[\"distance\"].transform(\"max\")\ntrain[\"molecule_distance_min\"] = train_g[\"distance\"].transform(\"min\")\ntest[\"molecule_distance_min\"] = test_g[\"distance\"].transform(\"min\")\ntrain[\"molecule_distance_mean\"] = train_g[\"distance\"].transform(\"mean\")\ntest[\"molecule_distance_mean\"] = test_g[\"distance\"].transform(\"mean\")\ntrain[\"molecule_distance_div_max\"] = train['distance'] / train[\"molecule_distance_max\"]\ntest[\"molecule_distance_div_max\"] = test['distance'] / test[\"molecule_distance_max\"]\ntrain[\"molecule_distance_div_min\"] = train['distance'] / train[\"molecule_distance_min\"]\ntest[\"molecule_distance_div_min\"] = test['distance'] / test[\"molecule_distance_min\"]\ntrain[\"molecule_distance_div_mean\"] = train['distance'] / train[\"molecule_distance_mean\"]\ntest[\"molecule_distance_div_mean\"] = test['distance'] / test[\"molecule_distance_mean\"]\n```\n\nand:\n```\n\ntrain_t_g = train.groupby('type')\ntest_t_g = test.groupby('type')\ntrain[\"type_distance_mean\"] = train_t_g[\"distance\"].transform(\"mean\")\ntest[\"type_distance_mean\"] = test_t_g[\"distance\"].transform(\"mean\")\ntrain[\"type_distance_div_mean\"] = train['distance'] / train[\"type_distance_mean\"]\ntest[\"type_distance_div_mean\"] = test['distance'] / test[\"type_distance_mean\"]\n```\n\nI use simple train/test split and my validation score is:\n\n[16569]\ttraining's l1: 1.35908\tvalid_0's l1: 1.74419\n\non LB my score is 0.460\n\nnow please notice what is different here:\n\n```\ntrain_g = train.groupby('molecule_name')\ntest_g = test.groupby('molecule_name')\ntrain[\"molecule_count\"] = train_g[\"id\"].transform(\"count\")\ntest[\"molecule_count\"] = test_g[\"id\"].transform(\"count\")\ntrain[\"molecule_distance_max\"] = train_g[\"distance\"].transform(\"max\")\ntest[\"molecule_distance_max\"] = test_g[\"distance\"].transform(\"max\")\ntrain[\"molecule_distance_min\"] = train_g[\"distance\"].transform(\"min\")\ntest[\"molecule_distance_min\"] = test_g[\"distance\"].transform(\"min\")\ntrain[\"molecule_distance_mean\"] = train_g[\"distance\"].transform(\"mean\")\ntest[\"molecule_distance_mean\"] = test_g[\"distance\"].transform(\"mean\")\ntrain[\"molecule_distance_div_max\"] = train['distance'] / train[\"molecule_distance_max\"]\ntest[\"molecule_distance_div_max\"] = test['distance'] / test[\"molecule_distance_max\"]\ntrain[\"molecule_distance_div_min\"] = train['distance'] / train[\"molecule_distance_min\"]\ntest[\"molecule_distance_div_min\"] = test['distance'] / test[\"molecule_distance_min\"]\ntrain[\"molecule_distance_div_mean\"] = train['distance'] / train[\"molecule_distance_mean\"]\ntest[\"molecule_distance_div_mean\"] = test['distance'] / test[\"molecule_distance_mean\"]\n\ntrain_t_g = train.groupby('type')\ntest_t_g = test.groupby('type')\n\ntrain[\"type_distance_mean\"] = train_t_g[\"distance\"].transform(\"mean\")\ntest[\"type_distance_mean\"] = test_t_g[\"distance\"].transform(\"mean\")\ntrain[\"type_distance_div_mean\"] = train['distance'] / train[\"type_distance_mean\"]\ntest[\"type_distance_div_mean\"] = test['distance'] / test[\"type_distance_mean\"]\n\ntrain[\"type_distance_min\"] = train_t_g[\"distance\"].transform(\"min\")\ntest[\"type_distance_min\"] = test_t_g[\"distance\"].transform(\"min\")\ntrain[\"type_distance_div_min\"] = train['distance'] / train[\"type_distance_min\"]\ntest[\"type_distance_div_min\"] = test['distance'] / test[\"type_distance_min\"]\n\ntrain[\"type_distance_max\"] = train_t_g[\"distance\"].transform(\"max\")\ntest[\"type_distance_max\"] = test_t_g[\"distance\"].transform(\"max\")\ntrain[\"type_distance_div_max\"] = train['distance'] / train[\"type_distance_max\"]\ntest[\"type_distance_div_max\"] = test['distance'] / test[\"type_distance_max\"]\n```\n\nI just added min and max distances per type.\n\nThe local result is:\n\n[16051]\ttraining's l1: 1.3594\tvalid_1's l1: 1.73379\n\nbut LB is 0.989\n\nso LB is much worse than previous one, looks like some heavy overfitting\n\nthis is my **second time with this case** because I have seen similar result before in different kernel so I was searching for code or logic error but wasn't able to find one\n\ndo you have any clues what may be wrong?\n\nin the beginning i was trying to use features only available for train set so this kind of result was expected, but this time I just calc distances, I am puzzled where can be the error",
      "votes": null
    },
    {
      "id": "558573",
      "postDate": "06/22/2019 15:25:43",
      "content": "<p>Two things. First, the \"training's l1\" and \"valid's l1\" scores are unweighted mean absolute error, whereas the LB score is a weighted log of the MAE so that the fit has to be equally good on all types regardless of their proportions in the test set. See the competition description: <a href=\"https://www.kaggle.com/c/champs-scalar-coupling/overview/evaluation\">https://www.kaggle.com/c/champs-scalar-coupling/overview/evaluation</a>.</p>\n\n<p>Second, you should not be using a simple train/val split. The test set does not contain any molecules that are in the train set, so your train and val sets should also have disjoint molecules (or else your CV score will badly underestimate your LB score). See my kernel for an example: <a href=\"https://www.kaggle.com/robertburbidge/distance-features\">https://www.kaggle.com/robertburbidge/distance-features</a>.</p>",
      "rawMarkdown": "Two things. First, the \"training's l1\" and \"valid's l1\" scores are unweighted mean absolute error, whereas the LB score is a weighted log of the MAE so that the fit has to be equally good on all types regardless of their proportions in the test set. See the competition description: https://www.kaggle.com/c/champs-scalar-coupling/overview/evaluation.\n\nSecond, you should not be using a simple train/val split. The test set does not contain any molecules that are in the train set, so your train and val sets should also have disjoint molecules (or else your CV score will badly underestimate your LB score). See my kernel for an example: https://www.kaggle.com/robertburbidge/distance-features.",
      "votes": null
    },
    {
      "id": "558668",
      "postDate": "06/22/2019 18:09:02",
      "content": "<p>There are valid points, however, why I observe this problem only when I use min/max distance per type not per molecule?</p>",
      "rawMarkdown": "There are valid points, however, why I observe this problem only when I use min/max distance per type not per molecule?",
      "votes": null
    },
    {
      "id": "559764",
      "postDate": "06/24/2019 14:59:58",
      "content": "<p>I did additional test this time with kfold:</p>\n\n<p>```\nfrom sklearn.model_selection import GroupKFold\nfolds = 5</p>\n\n<p>kfold = GroupKFold(n_splits = folds)\nsplits = kfold.split(X, y, train['molecule_name'])\n```</p>\n\n<p>still exactly same observation, when I add type_distance_max features I see huge drop in LB score</p>",
      "rawMarkdown": "I did additional test this time with kfold:\n\n```\nfrom sklearn.model_selection import GroupKFold\nfolds = 5\n\nkfold = GroupKFold(n_splits = folds)\nsplits = kfold.split(X, y, train['molecule_name'])\n```\n\nstill exactly same observation, when I add type_distance_max features I see huge drop in LB score",
      "votes": null
    },
    {
      "id": "559850",
      "postDate": "06/24/2019 17:15:55",
      "content": "<blockquote>\n  <p>I just added min and max distances per type.</p>\n</blockquote>\n\n<p>IMHO, the issue is that you compute them separately for train and test.  </p>",
      "rawMarkdown": "&gt; I just added min and max distances per type.\n\nIMHO, the issue is that you compute them separately for train and test.",
      "votes": null
    },
    {
      "id": "560390",
      "postDate": "06/25/2019 11:25:53",
      "content": "<p>You are right! And for molecule it doesnt matter because they are separated, thanks!</p>",
      "rawMarkdown": "You are right! And for molecule it doesnt matter because they are separated, thanks!",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 558573,
      "author_name": "robertburbidge",
      "author_url": "",
      "post_date": "06/22/2019 15:25:43",
      "content": "<p>Two things. First, the \"training's l1\" and \"valid's l1\" scores are unweighted mean absolute error, whereas the LB score is a weighted log of the MAE so that the fit has to be equally good on all types regardless of their proportions in the test set. See the competition description: <a href=\"https://www.kaggle.com/c/champs-scalar-coupling/overview/evaluation\">https://www.kaggle.com/c/champs-scalar-coupling/overview/evaluation</a>.</p>\n\n<p>Second, you should not be using a simple train/val split. The test set does not contain any molecules that are in the train set, so your train and val sets should also have disjoint molecules (or else your CV score will badly underestimate your LB score). See my kernel for an example: <a href=\"https://www.kaggle.com/robertburbidge/distance-features\">https://www.kaggle.com/robertburbidge/distance-features</a>.</p>",
      "votes": null,
      "replies": [
        {
          "id": 558668,
          "author_name": "jacekpoplawski",
          "author_url": "",
          "post_date": "06/22/2019 18:09:02",
          "content": "<p>There are valid points, however, why I observe this problem only when I use min/max distance per type not per molecule?</p>",
          "votes": null,
          "replies": []
        }
      ]
    },
    {
      "id": 559764,
      "author_name": "jacekpoplawski",
      "author_url": "",
      "post_date": "06/24/2019 14:59:58",
      "content": "<p>I did additional test this time with kfold:</p>\n\n<p>```\nfrom sklearn.model_selection import GroupKFold\nfolds = 5</p>\n\n<p>kfold = GroupKFold(n_splits = folds)\nsplits = kfold.split(X, y, train['molecule_name'])\n```</p>\n\n<p>still exactly same observation, when I add type_distance_max features I see huge drop in LB score</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 559850,
      "author_name": "cpmpml",
      "author_url": "",
      "post_date": "06/24/2019 17:15:55",
      "content": "<blockquote>\n  <p>I just added min and max distances per type.</p>\n</blockquote>\n\n<p>IMHO, the issue is that you compute them separately for train and test.  </p>",
      "votes": null,
      "replies": [
        {
          "id": 560390,
          "author_name": "jacekpoplawski",
          "author_url": "",
          "post_date": "06/25/2019 11:25:53",
          "content": "<p>You are right! And for molecule it doesnt matter because they are separated, thanks!</p>",
          "votes": null,
          "replies": []
        }
      ]
    }
  ],
  "raw_markdown_by_id": {
    "558515": "I have strange problem with CV/LB score, maybe you could help.\nI have kernel with features like that:\n\n```\ntrain_g = train.groupby('molecule_name')\ntest_g = test.groupby('molecule_name')\ntrain[\"molecule_count\"] = train_g[\"id\"].transform(\"count\")\ntest[\"molecule_count\"] = test_g[\"id\"].transform(\"count\")\ntrain[\"molecule_distance_max\"] = train_g[\"distance\"].transform(\"max\")\ntest[\"molecule_distance_max\"] = test_g[\"distance\"].transform(\"max\")\ntrain[\"molecule_distance_min\"] = train_g[\"distance\"].transform(\"min\")\ntest[\"molecule_distance_min\"] = test_g[\"distance\"].transform(\"min\")\ntrain[\"molecule_distance_mean\"] = train_g[\"distance\"].transform(\"mean\")\ntest[\"molecule_distance_mean\"] = test_g[\"distance\"].transform(\"mean\")\ntrain[\"molecule_distance_div_max\"] = train['distance'] / train[\"molecule_distance_max\"]\ntest[\"molecule_distance_div_max\"] = test['distance'] / test[\"molecule_distance_max\"]\ntrain[\"molecule_distance_div_min\"] = train['distance'] / train[\"molecule_distance_min\"]\ntest[\"molecule_distance_div_min\"] = test['distance'] / test[\"molecule_distance_min\"]\ntrain[\"molecule_distance_div_mean\"] = train['distance'] / train[\"molecule_distance_mean\"]\ntest[\"molecule_distance_div_mean\"] = test['distance'] / test[\"molecule_distance_mean\"]\n```\n\nand:\n```\n\ntrain_t_g = train.groupby('type')\ntest_t_g = test.groupby('type')\ntrain[\"type_distance_mean\"] = train_t_g[\"distance\"].transform(\"mean\")\ntest[\"type_distance_mean\"] = test_t_g[\"distance\"].transform(\"mean\")\ntrain[\"type_distance_div_mean\"] = train['distance'] / train[\"type_distance_mean\"]\ntest[\"type_distance_div_mean\"] = test['distance'] / test[\"type_distance_mean\"]\n```\n\nI use simple train/test split and my validation score is:\n\n[16569]\ttraining's l1: 1.35908\tvalid_0's l1: 1.74419\n\non LB my score is 0.460\n\nnow please notice what is different here:\n\n```\ntrain_g = train.groupby('molecule_name')\ntest_g = test.groupby('molecule_name')\ntrain[\"molecule_count\"] = train_g[\"id\"].transform(\"count\")\ntest[\"molecule_count\"] = test_g[\"id\"].transform(\"count\")\ntrain[\"molecule_distance_max\"] = train_g[\"distance\"].transform(\"max\")\ntest[\"molecule_distance_max\"] = test_g[\"distance\"].transform(\"max\")\ntrain[\"molecule_distance_min\"] = train_g[\"distance\"].transform(\"min\")\ntest[\"molecule_distance_min\"] = test_g[\"distance\"].transform(\"min\")\ntrain[\"molecule_distance_mean\"] = train_g[\"distance\"].transform(\"mean\")\ntest[\"molecule_distance_mean\"] = test_g[\"distance\"].transform(\"mean\")\ntrain[\"molecule_distance_div_max\"] = train['distance'] / train[\"molecule_distance_max\"]\ntest[\"molecule_distance_div_max\"] = test['distance'] / test[\"molecule_distance_max\"]\ntrain[\"molecule_distance_div_min\"] = train['distance'] / train[\"molecule_distance_min\"]\ntest[\"molecule_distance_div_min\"] = test['distance'] / test[\"molecule_distance_min\"]\ntrain[\"molecule_distance_div_mean\"] = train['distance'] / train[\"molecule_distance_mean\"]\ntest[\"molecule_distance_div_mean\"] = test['distance'] / test[\"molecule_distance_mean\"]\n\ntrain_t_g = train.groupby('type')\ntest_t_g = test.groupby('type')\n\ntrain[\"type_distance_mean\"] = train_t_g[\"distance\"].transform(\"mean\")\ntest[\"type_distance_mean\"] = test_t_g[\"distance\"].transform(\"mean\")\ntrain[\"type_distance_div_mean\"] = train['distance'] / train[\"type_distance_mean\"]\ntest[\"type_distance_div_mean\"] = test['distance'] / test[\"type_distance_mean\"]\n\ntrain[\"type_distance_min\"] = train_t_g[\"distance\"].transform(\"min\")\ntest[\"type_distance_min\"] = test_t_g[\"distance\"].transform(\"min\")\ntrain[\"type_distance_div_min\"] = train['distance'] / train[\"type_distance_min\"]\ntest[\"type_distance_div_min\"] = test['distance'] / test[\"type_distance_min\"]\n\ntrain[\"type_distance_max\"] = train_t_g[\"distance\"].transform(\"max\")\ntest[\"type_distance_max\"] = test_t_g[\"distance\"].transform(\"max\")\ntrain[\"type_distance_div_max\"] = train['distance'] / train[\"type_distance_max\"]\ntest[\"type_distance_div_max\"] = test['distance'] / test[\"type_distance_max\"]\n```\n\nI just added min and max distances per type.\n\nThe local result is:\n\n[16051]\ttraining's l1: 1.3594\tvalid_1's l1: 1.73379\n\nbut LB is 0.989\n\nso LB is much worse than previous one, looks like some heavy overfitting\n\nthis is my **second time with this case** because I have seen similar result before in different kernel so I was searching for code or logic error but wasn't able to find one\n\ndo you have any clues what may be wrong?\n\nin the beginning i was trying to use features only available for train set so this kind of result was expected, but this time I just calc distances, I am puzzled where can be the error",
    "558573": "Two things. First, the \"training's l1\" and \"valid's l1\" scores are unweighted mean absolute error, whereas the LB score is a weighted log of the MAE so that the fit has to be equally good on all types regardless of their proportions in the test set. See the competition description: https://www.kaggle.com/c/champs-scalar-coupling/overview/evaluation.\n\nSecond, you should not be using a simple train/val split. The test set does not contain any molecules that are in the train set, so your train and val sets should also have disjoint molecules (or else your CV score will badly underestimate your LB score). See my kernel for an example: https://www.kaggle.com/robertburbidge/distance-features.",
    "558668": "There are valid points, however, why I observe this problem only when I use min/max distance per type not per molecule?",
    "559764": "I did additional test this time with kfold:\n\n```\nfrom sklearn.model_selection import GroupKFold\nfolds = 5\n\nkfold = GroupKFold(n_splits = folds)\nsplits = kfold.split(X, y, train['molecule_name'])\n```\n\nstill exactly same observation, when I add type_distance_max features I see huge drop in LB score",
    "559850": "&gt; I just added min and max distances per type.\n\nIMHO, the issue is that you compute them separately for train and test.",
    "560390": "You are right! And for molecule it doesnt matter because they are separated, thanks!"
  },
  "source": "meta"
}