Due to the existence of unlabeled labels in the data set, the current method I use is not to use span training without labels, and to divide the training set and the verification set so that they have no duplicate labels, but the effect of the verification set and the effect of the test set are not Correlation. Therefore, how to avoid the unlabeled span 's impact is also one of the problems that need to be solved urgently.
", "messages": [ { "id": "1341370", "postDate": "06/08/2021 15:57:47", "content": "Due to the existence of unlabeled labels in the data set, the current method I use is not to use span training without labels, and to divide the training set and the verification set so that they have no duplicate labels, but the effect of the verification set and the effect of the test set are not Correlation. Therefore, how to avoid the unlabeled span 's impact is also one of the problems that need to be solved urgently.
", "rawMarkdown": "Due to the existence of unlabeled labels in the data set, the current method I use is not to use span training without labels, and to divide the training set and the verification set so that they have no duplicate labels, but the effect of the verification set and the effect of the test set are not Correlation. Therefore, how to avoid the unlabeled span 's impact is also one of the problems that need to be solved urgently.", "votes": null }, { "id": "1342254", "postDate": "06/09/2021 09:51:42", "content": "I do a KFold split on unique train IDS. Thats why I can think of. But the problem is dataset is highly imbalanced from all sense.
", "rawMarkdown": "I do a KFold split on unique train IDS. Thats why I can think of. But the problem is dataset is highly imbalanced from all sense.", "votes": null }, { "id": "1348804", "postDate": "06/14/2021 09:25:46", "content": "Well, you should also be aware of very similar labels/labels with identical parts inside. Imho, cross-validation should be about K Fold on top of clusters, bot only unique labels.
\nIdk how to cope with unlabeled data, Imho this can't be solved w/o hand labelling. Currently I try to stick to recall as a main metric to learn (unique labels from test set, no simiilar data from train is involved), not precision. I do understand that this is the opposite of the f0.5 metric, however this might be the way to include unlabeled data's impact into my CV.
\nAlso LB shows that NER trained this way is performing better than precision maxed one.
I agree. Cross validation on unique labels works well than what I proposed earlier. Due to time constraints I am not looking into this anymore.
\nTo deal with highly imbalance , one thing is . We have only 118 unique train labels. Of that many are repeating.
\nDo N folds, but make sure repeated labels should be sampled only to a limit. Let's say each in fold sample max 10 duplicate labels. And next fold, sample max 10 again, but exclude previously sampled.
\nIt helps.
", "rawMarkdown": "I agree. Cross validation on unique labels works well than what I proposed earlier. Due to time constraints I am not looking into this anymore.\n\nTo deal with highly imbalance , one thing is . We have only 118 unique train labels. Of that many are repeating. \n\nDo N folds, but make sure repeated labels should be sampled only to a limit. Let's say each in fold sample max 10 duplicate labels. And next fold, sample max 10 again, but exclude previously sampled. \n\nIt helps.", "votes": null } ], "comments": [ { "id": 1342254, "author_name": "s4sarath", "author_url": "", "post_date": "06/09/2021 09:51:42", "content": "I do a KFold split on unique train IDS. Thats why I can think of. But the problem is dataset is highly imbalanced from all sense.
", "votes": null, "replies": [] }, { "id": 1348804, "author_name": "lightnessofbein", "author_url": "", "post_date": "06/14/2021 09:25:46", "content": "Well, you should also be aware of very similar labels/labels with identical parts inside. Imho, cross-validation should be about K Fold on top of clusters, bot only unique labels.
\nIdk how to cope with unlabeled data, Imho this can't be solved w/o hand labelling. Currently I try to stick to recall as a main metric to learn (unique labels from test set, no simiilar data from train is involved), not precision. I do understand that this is the opposite of the f0.5 metric, however this might be the way to include unlabeled data's impact into my CV.
\nAlso LB shows that NER trained this way is performing better than precision maxed one.
I agree. Cross validation on unique labels works well than what I proposed earlier. Due to time constraints I am not looking into this anymore.
\nTo deal with highly imbalance , one thing is . We have only 118 unique train labels. Of that many are repeating.
\nDo N folds, but make sure repeated labels should be sampled only to a limit. Let's say each in fold sample max 10 duplicate labels. And next fold, sample max 10 again, but exclude previously sampled.
\nIt helps.
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "1341370": "Due to the existence of unlabeled labels in the data set, the current method I use is not to use span training without labels, and to divide the training set and the verification set so that they have no duplicate labels, but the effect of the verification set and the effect of the test set are not Correlation. Therefore, how to avoid the unlabeled span 's impact is also one of the problems that need to be solved urgently.", "1342254": "I do a KFold split on unique train IDS. Thats why I can think of. But the problem is dataset is highly imbalanced from all sense.", "1348804": "Well, you should also be aware of very similar labels/labels with identical parts inside. Imho, cross-validation should be about K Fold on top of clusters, bot only unique labels.\nIdk how to cope with unlabeled data, Imho this can't be solved w/o hand labelling. Currently I try to stick to recall as a main metric to learn (unique labels from test set, no simiilar data from train is involved), not precision. I do understand that this is the opposite of the f0.5 metric, however this might be the way to include unlabeled data's impact into my CV.\nAlso LB shows that NER trained this way is performing better than precision maxed one.", "1348892": "I agree. Cross validation on unique labels works well than what I proposed earlier. Due to time constraints I am not looking into this anymore.\n\nTo deal with highly imbalance , one thing is . We have only 118 unique train labels. Of that many are repeating. \n\nDo N folds, but make sure repeated labels should be sampled only to a limit. Let's say each in fold sample max 10 duplicate labels. And next fold, sample max 10 again, but exclude previously sampled. \n\nIt helps." }, "source": "meta" }