{ "id": 234042, "title": "Is manual labeling allowed in this competition?", "url": "/competitions/coleridgeinitiative-show-us-the-data/discussion/234042", "author_name": "", "post_date": "2021-04-22T11:18:23.545792900Z", "votes": 25, "comment_count": 8, "views": 0, "content": "

My understandings with manual labeling in a competition is that we can, in general, do whatever we want with the training data as long as we leave the test data alone, so I'm sorry if I'm beating a straw man here.
\nThe reason I'm asking this question is that a portion of the training data here was intentionally left unlabeled, it's a challenge we have to overcome, and simply re-label the data seems to be a cheap way to achieve that.
\nI think confirmation from the hosts will be great for us before we commit further. Is manual labeling allowed and do we need to declare it in the external data thread?

", "messages": [ { "id": "1280801", "postDate": "04/22/2021 11:18:23", "content": "

My understandings with manual labeling in a competition is that we can, in general, do whatever we want with the training data as long as we leave the test data alone, so I'm sorry if I'm beating a straw man here.
\nThe reason I'm asking this question is that a portion of the training data here was intentionally left unlabeled, it's a challenge we have to overcome, and simply re-label the data seems to be a cheap way to achieve that.
\nI think confirmation from the hosts will be great for us before we commit further. Is manual labeling allowed and do we need to declare it in the external data thread?

", "rawMarkdown": "My understandings with manual labeling in a competition is that we can, in general, do whatever we want with the *training* data as long as we leave the test data alone, so I'm sorry if I'm beating a straw man here.\nThe reason I'm asking this question is that a portion of the training data here was **intentionally** left unlabeled, it's a challenge we have to overcome, and simply re-label the data seems to be a cheap way to achieve that.\nI think confirmation from the hosts will be great for us before we commit further. Is manual labeling allowed and do we need to declare it in the external data thread?", "votes": null }, { "id": "1280900", "postDate": "04/22/2021 13:12:57", "content": "", "rawMarkdown": "You can **modify training data** i.e. manually labeling, changing labels, etc. You can also delete/ignore training data as per your modeling strategy. You don't need to declare any of these changes.\n- If you actually use any external data, then you need to post links in the [official external data thread](https://www.kaggle.com/c/coleridgeinitiative-show-us-the-data/discussion/232414).", "votes": null }, { "id": "1280904", "postDate": "04/22/2021 13:23:14", "content": "

Yes, I figure this will be canon if the hosts won't reply to this topic.

", "rawMarkdown": "Yes, I figure this will be canon if the hosts won't reply to this topic.", "votes": null }, { "id": "1281479", "postDate": "04/23/2021 02:07:07", "content": "

In Rules-5, hand labeling is described as follows.

\n
Submissions may not use or incorporate information from hand labeling or human prediction of the validation dataset or test data records.\n
\n

Does what you ask in this thread means whether hand labeling can be used in training or text matching if we don't change a submission directly?

", "rawMarkdown": "In [Rules-5](https://www.kaggle.com/c/coleridgeinitiative-show-us-the-data/rules), hand labeling is described as follows.\n\n```\nSubmissions may not use or incorporate information from hand labeling or human prediction of the validation dataset or test data records.\n```\n\nDoes what you ask in this thread means whether hand labeling can be used in training or text matching if we don't change a submission directly?", "votes": null }, { "id": "1281487", "postDate": "04/23/2021 02:25:58", "content": "

What that rule stated is that you cannot use manually extracted information from the test set for whatever purpose, directly or indirectly modifying the submission. It is redundant in a kernel competition since the test is hidden anyway.
\nThat left us with the training, which I understand can be messed with. However, my concern is that in this competition the host gave us incomplete training data and wanted us to build a solution that generalize well to exhaustedly labeled test data, manually labeling the train data seems to defeat that purpose.
\nAs you can see, this competition already went through two changes, first the metrics and then the requirement to declare external data, so I'm asking this to avoid another awkward situation @philculliton .

", "rawMarkdown": "What that rule stated is that you cannot use manually extracted information from the test set for whatever purpose, directly or indirectly modifying the submission. It is redundant in a kernel competition since the test is hidden anyway. \nThat left us with the training, which I understand can be messed with. However, my concern is that in this competition the host gave us incomplete training data and wanted us to build a solution that generalize well to exhaustedly labeled test data, manually labeling the train data seems to defeat that purpose. \nAs you can see, this competition already went through two changes, first the metrics and then the requirement to declare external data, so I'm asking this to avoid another awkward situation @philculliton .", "votes": null }, { "id": "1281849", "postDate": "04/23/2021 11:12:44", "content": "

@philculliton can you clarify?

", "rawMarkdown": "philculliton can you clarify?", "votes": null }, { "id": "1286353", "postDate": "04/27/2021 21:17:56", "content": "

I would love a clarification of this @philculliton

", "rawMarkdown": "I would love a clarification of this @philculliton", "votes": null }, { "id": "1286890", "postDate": "04/28/2021 14:00:58", "content": "

Great, another hand-labelling competition.

\n

Good thing is that the test data is not really available, otherwise Kaggle could've allowed people to annotate it such as in Hubmap …

", "rawMarkdown": "Great, another hand-labelling competition.\n\nGood thing is that the test data is not really available, otherwise Kaggle could've allowed people to annotate it such as in [Hubmap](https://www.kaggle.com/c/hubmap-kidney-segmentation/discussion/227616) ...", "votes": null }, { "id": "1336521", "postDate": "06/05/2021 02:30:45", "content": "

update: I got 0.505 by QA model。
\n（At the same time, I look forward to finding other teammates to improve the ranking.）

\n

QA method of recording process：
\nhttps://www.kaggle.com/c/coleridgeinitiative-show-us-the-data/discussion/238226

", "rawMarkdown": "update: I got 0.505 by QA model。\n（At the same time, I look forward to finding other teammates to improve the ranking.）\n\nQA method of recording process：\n[https://www.kaggle.com/c/coleridgeinitiative-show-us-the-data/discussion/238226](url)", "votes": null } ], "comments": [ { "id": 1280900, "author_name": "sirishks", "author_url": "", "post_date": "04/22/2021 13:12:57", "content": "", "votes": null, "replies": [ { "id": 1280904, "author_name": "suicaokhoailang", "author_url": "", "post_date": "04/22/2021 13:23:14", "content": "

Yes, I figure this will be canon if the hosts won't reply to this topic.

", "votes": null, "replies": [] } ] }, { "id": 1281479, "author_name": "spshota", "author_url": "", "post_date": "04/23/2021 02:07:07", "content": "

In Rules-5, hand labeling is described as follows.

\n
Submissions may not use or incorporate information from hand labeling or human prediction of the validation dataset or test data records.\n
\n

Does what you ask in this thread means whether hand labeling can be used in training or text matching if we don't change a submission directly?

", "votes": null, "replies": [ { "id": 1281487, "author_name": "suicaokhoailang", "author_url": "", "post_date": "04/23/2021 02:25:58", "content": "

What that rule stated is that you cannot use manually extracted information from the test set for whatever purpose, directly or indirectly modifying the submission. It is redundant in a kernel competition since the test is hidden anyway.
\nThat left us with the training, which I understand can be messed with. However, my concern is that in this competition the host gave us incomplete training data and wanted us to build a solution that generalize well to exhaustedly labeled test data, manually labeling the train data seems to defeat that purpose.
\nAs you can see, this competition already went through two changes, first the metrics and then the requirement to declare external data, so I'm asking this to avoid another awkward situation @philculliton .

", "votes": null, "replies": [] } ] }, { "id": 1281849, "author_name": "dathudeptrai", "author_url": "", "post_date": "04/23/2021 11:12:44", "content": "

@philculliton can you clarify?

", "votes": null, "replies": [] }, { "id": 1286353, "author_name": "manuelsh", "author_url": "", "post_date": "04/27/2021 21:17:56", "content": "

I would love a clarification of this @philculliton

", "votes": null, "replies": [] }, { "id": 1286890, "author_name": "theoviel", "author_url": "", "post_date": "04/28/2021 14:00:58", "content": "

Great, another hand-labelling competition.

\n

Good thing is that the test data is not really available, otherwise Kaggle could've allowed people to annotate it such as in Hubmap …

", "votes": null, "replies": [] }, { "id": 1336521, "author_name": "vanle73", "author_url": "", "post_date": "06/05/2021 02:30:45", "content": "

update: I got 0.505 by QA model。
\n（At the same time, I look forward to finding other teammates to improve the ranking.）

\n

QA method of recording process：
\nhttps://www.kaggle.com/c/coleridgeinitiative-show-us-the-data/discussion/238226

", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "1280801": "My understandings with manual labeling in a competition is that we can, in general, do whatever we want with the *training* data as long as we leave the test data alone, so I'm sorry if I'm beating a straw man here.\nThe reason I'm asking this question is that a portion of the training data here was **intentionally** left unlabeled, it's a challenge we have to overcome, and simply re-label the data seems to be a cheap way to achieve that.\nI think confirmation from the hosts will be great for us before we commit further. Is manual labeling allowed and do we need to declare it in the external data thread?", "1280900": "You can **modify training data** i.e. manually labeling, changing labels, etc. You can also delete/ignore training data as per your modeling strategy. You don't need to declare any of these changes.\n- If you actually use any external data, then you need to post links in the [official external data thread](https://www.kaggle.com/c/coleridgeinitiative-show-us-the-data/discussion/232414).", "1280904": "Yes, I figure this will be canon if the hosts won't reply to this topic.", "1281479": "In [Rules-5](https://www.kaggle.com/c/coleridgeinitiative-show-us-the-data/rules), hand labeling is described as follows.\n\n```\nSubmissions may not use or incorporate information from hand labeling or human prediction of the validation dataset or test data records.\n```\n\nDoes what you ask in this thread means whether hand labeling can be used in training or text matching if we don't change a submission directly?", "1281487": "What that rule stated is that you cannot use manually extracted information from the test set for whatever purpose, directly or indirectly modifying the submission. It is redundant in a kernel competition since the test is hidden anyway. \nThat left us with the training, which I understand can be messed with. However, my concern is that in this competition the host gave us incomplete training data and wanted us to build a solution that generalize well to exhaustedly labeled test data, manually labeling the train data seems to defeat that purpose. \nAs you can see, this competition already went through two changes, first the metrics and then the requirement to declare external data, so I'm asking this to avoid another awkward situation @philculliton .", "1281849": "philculliton can you clarify?", "1286353": "I would love a clarification of this @philculliton", "1286890": "Great, another hand-labelling competition.\n\nGood thing is that the test data is not really available, otherwise Kaggle could've allowed people to annotate it such as in [Hubmap](https://www.kaggle.com/c/hubmap-kidney-segmentation/discussion/227616) ...", "1336521": "update: I got 0.505 by QA model。\n（At the same time, I look forward to finding other teammates to improve the ranking.）\n\nQA method of recording process：\n[https://www.kaggle.com/c/coleridgeinitiative-show-us-the-data/discussion/238226](url)" }, "source": "meta" }