{
  "id": 131838,
  "title": "QWK Score Vs OOF Scores",
  "url": "/competitions/google-cloud-ncaa-march-madness-2020-division-1-mens-tournament/discussion/131838",
  "author_name": "",
  "post_date": "2020-02-22T04:06:18.700537200Z",
  "votes": 1,
  "comment_count": 1,
  "views": 0,
  "content": "<p>Hi ,</p>\n\n<p>My question is not related to this specific competition  , but a general question about stability of scores between each OOF (Out of fold) and QWK (Quadratic wieghted Kappa) score</p>\n\n<p>Currently i have only 493 records and  24 columns tabular data , target value is classification binary 1 or 0 and an imbalanced set so i balance it using class_weights parameter in catboost</p>\n\n<p>I fit the model based using Catboost algorithm  , question here based on above number of records when i fit model using OOF , each fold give me different score and each fold is stopping by over-fitting as i am using parameter od_type=\"Iter\" and early_stopping_rounds=500. I am getting QWK 0.56 which is acceptable for me  but not sure when i see OOF score . Should i consider only QWK or OOF or both?</p>\n\n<p>Any your valuable thoughts would give me a direction please feel free to share any</p>\n\n<p><img src=\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F213385%2Fba94cda6195c6bce2986464389aa4766%2FMetrics.PNG?generation=1582344289708424&amp;alt=media\" alt=\"\"></p>",
  "messages": [
    {
      "id": "753345",
      "postDate": "02/22/2020 04:06:18",
      "content": "<p>Hi ,</p>\n\n<p>My question is not related to this specific competition  , but a general question about stability of scores between each OOF (Out of fold) and QWK (Quadratic wieghted Kappa) score</p>\n\n<p>Currently i have only 493 records and  24 columns tabular data , target value is classification binary 1 or 0 and an imbalanced set so i balance it using class_weights parameter in catboost</p>\n\n<p>I fit the model based using Catboost algorithm  , question here based on above number of records when i fit model using OOF , each fold give me different score and each fold is stopping by over-fitting as i am using parameter od_type=\"Iter\" and early_stopping_rounds=500. I am getting QWK 0.56 which is acceptable for me  but not sure when i see OOF score . Should i consider only QWK or OOF or both?</p>\n\n<p>Any your valuable thoughts would give me a direction please feel free to share any</p>\n\n<p><img src=\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F213385%2Fba94cda6195c6bce2986464389aa4766%2FMetrics.PNG?generation=1582344289708424&amp;alt=media\" alt=\"\"></p>",
      "rawMarkdown": "Hi ,\n\nMy question is not related to this specific competition  , but a general question about stability of scores between each OOF (Out of fold) and QWK (Quadratic wieghted Kappa) score\n\nCurrently i have only 493 records and  24 columns tabular data , target value is classification binary 1 or 0 and an imbalanced set so i balance it using class_weights parameter in catboost\n\nI fit the model based using Catboost algorithm  , question here based on above number of records when i fit model using OOF , each fold give me different score and each fold is stopping by over-fitting as i am using parameter od_type=\"Iter\" and early_stopping_rounds=500. I am getting QWK 0.56 which is acceptable for me  but not sure when i see OOF score . Should i consider only QWK or OOF or both?\n\nAny your valuable thoughts would give me a direction please feel free to share any\n\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F213385%2Fba94cda6195c6bce2986464389aa4766%2FMetrics.PNG?generation=1582344289708424&amp;alt=media)",
      "votes": null
    },
    {
      "id": "753731",
      "postDate": "02/22/2020 15:51:42",
      "content": "<p>We have data from the seasons starting from 1985-2019 right now. Which is quite alot , and most of the old data is pretty irrelevant. That could be what's causing overfit? Also make sure  you're not using any leaked data since we already have the results of the LB.</p>",
      "rawMarkdown": "We have data from the seasons starting from 1985-2019 right now. Which is quite alot , and most of the old data is pretty irrelevant. That could be what's causing overfit? Also make sure  you're not using any leaked data since we already have the results of the LB.",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 753731,
      "author_name": "p4rallax",
      "author_url": "",
      "post_date": "02/22/2020 15:51:42",
      "content": "<p>We have data from the seasons starting from 1985-2019 right now. Which is quite alot , and most of the old data is pretty irrelevant. That could be what's causing overfit? Also make sure  you're not using any leaked data since we already have the results of the LB.</p>",
      "votes": null,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "753345": "Hi ,\n\nMy question is not related to this specific competition  , but a general question about stability of scores between each OOF (Out of fold) and QWK (Quadratic wieghted Kappa) score\n\nCurrently i have only 493 records and  24 columns tabular data , target value is classification binary 1 or 0 and an imbalanced set so i balance it using class_weights parameter in catboost\n\nI fit the model based using Catboost algorithm  , question here based on above number of records when i fit model using OOF , each fold give me different score and each fold is stopping by over-fitting as i am using parameter od_type=\"Iter\" and early_stopping_rounds=500. I am getting QWK 0.56 which is acceptable for me  but not sure when i see OOF score . Should i consider only QWK or OOF or both?\n\nAny your valuable thoughts would give me a direction please feel free to share any\n\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F213385%2Fba94cda6195c6bce2986464389aa4766%2FMetrics.PNG?generation=1582344289708424&amp;alt=media)",
    "753731": "We have data from the seasons starting from 1985-2019 right now. Which is quite alot , and most of the old data is pretty irrelevant. That could be what's causing overfit? Also make sure  you're not using any leaked data since we already have the results of the LB."
  },
  "source": "meta"
}