{
  "id": 235026,
  "title": "Local CV is probably meaningless?",
  "url": "/competitions/coleridgeinitiative-show-us-the-data/discussion/235026",
  "author_name": "",
  "post_date": "2021-04-27T11:19:58.756231600Z",
  "votes": 5,
  "comment_count": 1,
  "views": 0,
  "content": "<p>Hi there! Not sure if this has been discussed, if yes then please pardon the duplication.</p>\n<p>As stated in the competition description as well as in several discussion threads, we are aware that even the training set is NOT fully labelled, i.e., there are true labels not given.</p>\n<p>In that case, even if we have an excellent model it is not possible to accurately validate its predictions, because if the model predicts extra \"true labels\" that are not given, they will be treated as false-positives in your own CV.</p>\n<p>Any thought about this? Thanks in advance. Cheers!</p>",
  "messages": [
    {
      "id": "1285909",
      "postDate": "04/27/2021 11:19:58",
      "content": "<p>Hi there! Not sure if this has been discussed, if yes then please pardon the duplication.</p>\n<p>As stated in the competition description as well as in several discussion threads, we are aware that even the training set is NOT fully labelled, i.e., there are true labels not given.</p>\n<p>In that case, even if we have an excellent model it is not possible to accurately validate its predictions, because if the model predicts extra \"true labels\" that are not given, they will be treated as false-positives in your own CV.</p>\n<p>Any thought about this? Thanks in advance. Cheers!</p>",
      "rawMarkdown": "Hi there! Not sure if this has been discussed, if yes then please pardon the duplication.\n\nAs stated in the competition description as well as in several discussion threads, we are aware that even the training set is NOT fully labelled, i.e., there are true labels not given.\n\nIn that case, even if we have an excellent model it is not possible to accurately validate its predictions, because if the model predicts extra \"true labels\" that are not given, they will be treated as false-positives in your own CV.\n\nAny thought about this? Thanks in advance. Cheers!",
      "votes": null
    },
    {
      "id": "1286378",
      "postDate": "04/27/2021 22:38:49",
      "content": "<p>It is certainly something to consider. I wouldn't say meaningless but rather an imperfect indicator of our models performance, like the public LB. If the local CV correlated closely with the Public LB, we can be more confident about our models performances. For instance my last submissions got: </p>\n<table>\n<thead>\n<tr>\n<th>CV</th>\n<th>Public LB</th>\n</tr>\n</thead>\n<tbody>\n<tr>\n<td>0.126</td>\n<td>0.151</td>\n</tr>\n<tr>\n<td>0.153</td>\n<td>0.170</td>\n</tr>\n<tr>\n<td>0.162</td>\n<td>0.179</td>\n</tr>\n</tbody>\n</table>",
      "rawMarkdown": "It is certainly something to consider. I wouldn't say meaningless but rather an imperfect indicator of our models performance, like the public LB. If the local CV correlated closely with the Public LB, we can be more confident about our models performances. For instance my last submissions got: \n\n| CV | Public LB |\n| --- | --- \n| 0.126 | 0.151 |\n| 0.153 | 0.170 |\n| 0.162 | 0.179 |",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 1286378,
      "author_name": "hannes82",
      "author_url": "",
      "post_date": "04/27/2021 22:38:49",
      "content": "<p>It is certainly something to consider. I wouldn't say meaningless but rather an imperfect indicator of our models performance, like the public LB. If the local CV correlated closely with the Public LB, we can be more confident about our models performances. For instance my last submissions got: </p>\n<table>\n<thead>\n<tr>\n<th>CV</th>\n<th>Public LB</th>\n</tr>\n</thead>\n<tbody>\n<tr>\n<td>0.126</td>\n<td>0.151</td>\n</tr>\n<tr>\n<td>0.153</td>\n<td>0.170</td>\n</tr>\n<tr>\n<td>0.162</td>\n<td>0.179</td>\n</tr>\n</tbody>\n</table>",
      "votes": null,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "1285909": "Hi there! Not sure if this has been discussed, if yes then please pardon the duplication.\n\nAs stated in the competition description as well as in several discussion threads, we are aware that even the training set is NOT fully labelled, i.e., there are true labels not given.\n\nIn that case, even if we have an excellent model it is not possible to accurately validate its predictions, because if the model predicts extra \"true labels\" that are not given, they will be treated as false-positives in your own CV.\n\nAny thought about this? Thanks in advance. Cheers!",
    "1286378": "It is certainly something to consider. I wouldn't say meaningless but rather an imperfect indicator of our models performance, like the public LB. If the local CV correlated closely with the Public LB, we can be more confident about our models performances. For instance my last submissions got: \n\n| CV | Public LB |\n| --- | --- \n| 0.126 | 0.151 |\n| 0.153 | 0.170 |\n| 0.162 | 0.179 |"
  },
  "source": "meta"
}