{
  "id": 66959,
  "title": "CV/LB after update",
  "url": "/competitions/ga-customer-revenue-prediction/discussion/66959",
  "author_name": "",
  "post_date": "2018-09-27T05:03:17.880478700Z",
  "votes": 12,
  "comment_count": 8,
  "views": 0,
  "content": "<p>What is your CV/LB scores after <a href=\"https://www.kaggle.com/c/ga-customer-revenue-prediction/discussion/66737#latest-394205\">Leaderboard Update</a>?</p>\n\n<p>Using simple 5-fold, my cv started at .6154 at session-level and .5930 at user-level with LB .4434. After some new features it is around .5643 and .5573 with .4388 LB. However, sometimes good improvements on CV doesnt translate to any LB improvement, which is a bit worrisome.</p>",
  "messages": [
    {
      "id": "394570",
      "postDate": "09/27/2018 05:03:17",
      "content": "<p>What is your CV/LB scores after <a href=\"https://www.kaggle.com/c/ga-customer-revenue-prediction/discussion/66737#latest-394205\">Leaderboard Update</a>?</p>\n\n<p>Using simple 5-fold, my cv started at .6154 at session-level and .5930 at user-level with LB .4434. After some new features it is around .5643 and .5573 with .4388 LB. However, sometimes good improvements on CV doesnt translate to any LB improvement, which is a bit worrisome.</p>",
      "rawMarkdown": "What is your CV/LB scores after [Leaderboard Update][1]?\n\nUsing simple 5-fold, my cv started at .6154 at session-level and .5930 at user-level with LB .4434. After some new features it is around .5643 and .5573 with .4388 LB. However, sometimes good improvements on CV doesnt translate to any LB improvement, which is a bit worrisome.\n\n\n  [1]: https://www.kaggle.com/c/ga-customer-revenue-prediction/discussion/66737#latest-394205",
      "votes": null
    },
    {
      "id": "394598",
      "postDate": "09/27/2018 05:52:45",
      "content": "<p>So do I, need some ways to get hook with LB, \nCV: 1.55543   LB:1.4446;\nCV: 1.590926  LB:1.4314</p>",
      "rawMarkdown": "So do I, need some ways to get hook with LB, \nCV: 1.55543   LB:1.4446;\nCV: 1.590926  LB:1.4314",
      "votes": null
    },
    {
      "id": "394812",
      "postDate": "09/27/2018 13:52:21",
      "content": "<p>By simple 5-fold, do you use shuffle and fix the random state? When I used 10 fold, I get better CV but also does not translate to LB improvement.</p>",
      "rawMarkdown": "By simple 5-fold, do you use shuffle and fix the random state? When I used 10 fold, I get better CV but also does not translate to LB improvement.",
      "votes": null
    },
    {
      "id": "395121",
      "postDate": "09/28/2018 02:02:37",
      "content": "<p>Yes, just normal KFold with suffle and a fixed random seed</p>",
      "rawMarkdown": "Yes, just normal KFold with suffle and a fixed random seed",
      "votes": null
    },
    {
      "id": "395122",
      "postDate": "09/28/2018 02:07:18",
      "content": "<p>Same here. I think my CV and LB are more in accordance with each other after LB update but also sometimes my improvement in my CV leads to a worse LB. Maybe a better validation strategy or something else should be clarified.</p>",
      "rawMarkdown": "Same here. I think my CV and LB are more in accordance with each other after LB update but also sometimes my improvement in my CV leads to a worse LB. Maybe a better validation strategy or something else should be clarified.",
      "votes": null
    },
    {
      "id": "395417",
      "postDate": "09/28/2018 14:45:25",
      "content": "<p>No idea why my lb is really bad... lol\n5Fold CV 1.5487 LB 1.4532\n5Fold CV 1.5516 LB 1.4512</p>",
      "rawMarkdown": "No idea why my lb is really bad... lol\n5Fold CV 1.5487 LB 1.4532\n5Fold CV 1.5516 LB 1.4512",
      "votes": null
    },
    {
      "id": "395428",
      "postDate": "09/28/2018 15:03:25",
      "content": "<p>In fact, my CV keeps going down with new features added, but LB keeps rising. The lowest LB score was with no feature engineering at all</p>",
      "rawMarkdown": "In fact, my CV keeps going down with new features added, but LB keeps rising. The lowest LB score was with no feature engineering at all",
      "votes": null
    },
    {
      "id": "395539",
      "postDate": "09/28/2018 19:19:32",
      "content": "<p>Model is overfitting?</p>",
      "rawMarkdown": "Model is overfitting?",
      "votes": null
    },
    {
      "id": "395561",
      "postDate": "09/28/2018 20:30:26",
      "content": "<p>That would be the most obvious conclusion. But at the moment I do not understand where does it happen.</p>",
      "rawMarkdown": "That would be the most obvious conclusion. But at the moment I do not understand where does it happen.",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 394598,
      "author_name": "marcuslin",
      "author_url": "",
      "post_date": "09/27/2018 05:52:45",
      "content": "<p>So do I, need some ways to get hook with LB, \nCV: 1.55543   LB:1.4446;\nCV: 1.590926  LB:1.4314</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 394812,
      "author_name": "everswind",
      "author_url": "",
      "post_date": "09/27/2018 13:52:21",
      "content": "<p>By simple 5-fold, do you use shuffle and fix the random state? When I used 10 fold, I get better CV but also does not translate to LB improvement.</p>",
      "votes": null,
      "replies": [
        {
          "id": 395121,
          "author_name": "jsaguiar",
          "author_url": "",
          "post_date": "09/28/2018 02:02:37",
          "content": "<p>Yes, just normal KFold with suffle and a fixed random seed</p>",
          "votes": null,
          "replies": []
        }
      ]
    },
    {
      "id": 395122,
      "author_name": "syoya1997",
      "author_url": "",
      "post_date": "09/28/2018 02:07:18",
      "content": "<p>Same here. I think my CV and LB are more in accordance with each other after LB update but also sometimes my improvement in my CV leads to a worse LB. Maybe a better validation strategy or something else should be clarified.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 395417,
      "author_name": "kingychiu",
      "author_url": "",
      "post_date": "09/28/2018 14:45:25",
      "content": "<p>No idea why my lb is really bad... lol\n5Fold CV 1.5487 LB 1.4532\n5Fold CV 1.5516 LB 1.4512</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 395428,
      "author_name": "mlisovyi",
      "author_url": "",
      "post_date": "09/28/2018 15:03:25",
      "content": "<p>In fact, my CV keeps going down with new features added, but LB keeps rising. The lowest LB score was with no feature engineering at all</p>",
      "votes": null,
      "replies": [
        {
          "id": 395539,
          "author_name": "everswind",
          "author_url": "",
          "post_date": "09/28/2018 19:19:32",
          "content": "<p>Model is overfitting?</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 395561,
          "author_name": "mlisovyi",
          "author_url": "",
          "post_date": "09/28/2018 20:30:26",
          "content": "<p>That would be the most obvious conclusion. But at the moment I do not understand where does it happen.</p>",
          "votes": null,
          "replies": []
        }
      ]
    }
  ],
  "raw_markdown_by_id": {
    "394570": "What is your CV/LB scores after [Leaderboard Update][1]?\n\nUsing simple 5-fold, my cv started at .6154 at session-level and .5930 at user-level with LB .4434. After some new features it is around .5643 and .5573 with .4388 LB. However, sometimes good improvements on CV doesnt translate to any LB improvement, which is a bit worrisome.\n\n\n  [1]: https://www.kaggle.com/c/ga-customer-revenue-prediction/discussion/66737#latest-394205",
    "394598": "So do I, need some ways to get hook with LB, \nCV: 1.55543   LB:1.4446;\nCV: 1.590926  LB:1.4314",
    "394812": "By simple 5-fold, do you use shuffle and fix the random state? When I used 10 fold, I get better CV but also does not translate to LB improvement.",
    "395121": "Yes, just normal KFold with suffle and a fixed random seed",
    "395122": "Same here. I think my CV and LB are more in accordance with each other after LB update but also sometimes my improvement in my CV leads to a worse LB. Maybe a better validation strategy or something else should be clarified.",
    "395417": "No idea why my lb is really bad... lol\n5Fold CV 1.5487 LB 1.4532\n5Fold CV 1.5516 LB 1.4512",
    "395428": "In fact, my CV keeps going down with new features added, but LB keeps rising. The lowest LB score was with no feature engineering at all",
    "395539": "Model is overfitting?",
    "395561": "That would be the most obvious conclusion. But at the moment I do not understand where does it happen."
  },
  "source": "meta"
}