Thank you for hosting this awesome competition and congrats to the winners.
\nThis problem is interesting and challenging. The data and annotations are high quality. It's a great learning and practicing experience working on this competition.
\nI started this competition compeletely ignoring those shared NER baselines. I thought in an \"object detection\" task, segmentation + posprocess approaches can not do better than object detection approaches. I'm surprised by the postprocess ideas in top solutions.
\nOverall, what I made is a YOLO-like text span detector. I share my code in this notebook.
\nCheckpoints are shared here.
\nTo reproduce the checkpoints, check the code and configs here.
\nAutoModelForTokenClassification.from_pretrained(num_labels=1 + 2 + num_classes, ...)\n1 for objectness (or span-ness?. fg/bg classification)
\n2 for regression (distance from fg location to first and last index of corresponding span)
\nnum_classes for discourse type classification
\nNetwork logits is in shape (num_tokens, 10). This is inconvenient (decoding output, ensembling models with different tokenizers). So I aggregate the logits to (num_words, 10) using RoIAlign.
\nThe problem of this formulation is how to define positive for objectness training. In object detection, center of object is a natural choice of positve. In text span detection, I found the first word of span a good choice of positive.
\nIn addition, I assign lowest cost word in each span as positive during training. This is inspired by YOLOX.
\nI randomly replace tokens with mask token during training.
\nFor other details, please check the code.
\nThe only post process is nms.
\nI used one cycle policy in training and averaged weights of the last few epochs.
\nTo ensemble different models/folds, I simply averaged outputs of models.
\nWBF ensemble does not work in local validation, and I didn't figure out why. I think I did something wrong here.
| backbone | \nValidation | \nPublic LB | \nPrivate LB | \n
|---|---|---|---|
| google/bigbird-roberta-base | \n0.685~0.69 | \n\n | \n |
| allenai/longformer-base-4096 | \n0.685~0.69 | \n\n | \n |
| allenai/longformer-large-4096 | \n0.70~0.71 | \n\n | \n |
| microsoft/deberta-base | \n0.70~0.705 | \n\n | \n |
| microsoft/deberta-large | \n0.715~0.72 | \n\n | \n |
| microsoft/deberta-xlarge | \n0.715~0.72 | \n\n | \n |
| microsoft/deberta-large + microsoft/deberta-xlarge | \n0.723 | \n0.714 | \n0.732 | \n
The best combination is deberta-large + deberta-xlarge. Ensembling more does not help.
\nIn the final submission, I used 2 weights (2/5 folds) each model. Submission time is around 2 hours.
This solution should be #1
\nMore seriously, it is very elegant and I'm pretty sure that there is room for improvement - perhaps it would outperform other approaches if combined with a features+boosting model :)
", "votes": 15, "replies": [ { "id": 1726052, "author_name": "CroDoc", "author_url": "", "post_date": "2022-03-17T15:59:03.777000", "content": "Submission time 2h ha ha … crazy…
", "votes": 1, "replies": [] }, { "id": 1726247, "author_name": "Ken Krige", "author_url": "", "post_date": "2022-03-17T19:41:31.397000", "content": "\n\nThis solution should be #1
\n
Agree
", "votes": 0, "replies": [] }, { "id": 1727404, "author_name": "tascj", "author_url": "", "post_date": "2022-03-18T00:34:41.197000", "content": "Thank you.
\nYes there is room for improvement. I didn't try ensemble until the last day because I thought WBF would just work, however it did not work as expected. There are TPs in bigbird model output that deberta models does not catch, but I haven't found a proper merging method without adding too many FPs.
\nThe post process ideas in top solutions are something I would never came up with. My model using low threshold can't achieve the recall shared in the 1st solution. There's a lot to learn and try.
", "votes": 2, "replies": [] } ] }, { "id": 1725409, "author_name": "hxcc", "author_url": "", "post_date": "2022-03-17T04:33:18.033000", "content": "love how you tackle an NLP problem with CV methods and made it work. Simple, elegant, very little ensembling, fast inference time. My favourite solution so far! Thanks for sharing!
", "votes": 9, "replies": [] }, { "id": 1725867, "author_name": "CroDoc", "author_url": "", "post_date": "2022-03-17T13:41:43.177000", "content": "Your work on this competition is just WOW! Well done!
", "votes": 1, "replies": [] }, { "id": 1725383, "author_name": "gezi", "author_url": "", "post_date": "2022-03-17T03:44:16.553000", "content": "Really elegant and creative work! Want to have a try😃
", "votes": 1, "replies": [] }, { "id": 1725362, "author_name": "KhanhVD", "author_url": "", "post_date": "2022-03-17T03:15:23.437000", "content": "Interesting solution! Congrats on 6th place and thanks for sharing your approach.
", "votes": 1, "replies": [] }, { "id": 1848146, "author_name": "Mr Nagrom", "author_url": "", "post_date": "2022-07-08T11:55:03.963000", "content": "Great model !!
\nI had thought of the same segmentation approach with YOLO but I failed to make my model as efficient as yours and I finally find myself at the 1937 position of the competition. But this is my first real competition on Kaggle.
Your code is a real inspiration to me. Thanks for sharing !!!
", "votes": 0, "replies": [] }, { "id": 1746097, "author_name": "gagaa", "author_url": "", "post_date": "2022-04-05T13:55:11.767000", "content": "thanks for sharing! you mention \"In text span detection, I found the first word of span a good choice of positive.\",in this situation is there no possible to predict firest location of span?so num_labels=1 + 1 + num_classes
", "votes": 0, "replies": [ { "id": 1748013, "author_name": "tascj", "author_url": "", "post_date": "2022-04-07T08:49:57.353000", "content": "Yes. 1 is enough for single positive assignment, 2 is used for the extra dynamically assigned positive.
", "votes": 0, "replies": [] } ] }, { "id": 1731304, "author_name": "GUNER", "author_url": "", "post_date": "2022-03-22T08:15:12.473000", "content": "One (more) tiny :-) question on 'dynamic_pozitive' targets :
\nAs the first word is always the target and the 'dynamic_pozitive' is an add on without a fixed grid-location (it is calculated based on minimum loss-rule) ,, wouldn't the contribution of 'dynamic_pozitive' targets diminish as you train? Since, the first word is always getting trained on and it dominates the network in time (,so: soon, 'first word' also dominates the 'min-loss' calculations (as the word with minimum loss)?
I had the same concern. The cost is not equal to loss, and there are random masking augmentation so in practice it does not happen and dynamic positive improves score. Actually in practice there could be multiple words with lower cost compared to the first word of span.
\nAnyway, in the worst case it just falls back to single positive. I think it's fine.
", "votes": 1, "replies": [] }, { "id": 1732120, "author_name": "GUNER", "author_url": "", "post_date": "2022-03-23T03:18:48.940000", "content": "Thanks again.
", "votes": 0, "replies": [] } ] }, { "id": 1731257, "author_name": "tyaba", "author_url": "", "post_date": "2022-03-22T06:48:21.077000", "content": "Thank you for sharing your wonderful solution!
\nI would like to know what fg and bg refer to.
foreground and background
", "votes": 1, "replies": [] }, { "id": 1732100, "author_name": "tyaba", "author_url": "", "post_date": "2022-03-23T02:09:34.597000", "content": "Thank you for your replying!
", "votes": 0, "replies": [] } ] }, { "id": 1730428, "author_name": "Mohammad Al-Azawi", "author_url": "", "post_date": "2022-03-21T08:36:51.313000", "content": "interesting
", "votes": 0, "replies": [] }, { "id": 1729931, "author_name": "Aruna S", "author_url": "", "post_date": "2022-03-20T17:36:26.533000", "content": "Well done !! Congratulations @tascj0 !!🤩🤩
", "votes": 0, "replies": [] }, { "id": 1729167, "author_name": "akshay kumar sahu", "author_url": "", "post_date": "2022-03-19T17:30:48.297000", "content": "well done, get this to top
", "votes": 0, "replies": [] }, { "id": 1728624, "author_name": "Niek van der Zwaag", "author_url": "", "post_date": "2022-03-19T05:06:44.323000", "content": "Creative solution, thanks for sharing your thoughts and process 🙌
", "votes": 0, "replies": [] }, { "id": 1727980, "author_name": "GUNER", "author_url": "", "post_date": "2022-03-18T13:53:44.310000", "content": "'''Network logits is in shape (num_tokens, 10)'''
\nBecause the model creates output for each token (CV-pixel equivalent), isn't it more like a segmentation rathen than a detection job, thus falling not very far from NER approaches?
\n\nisn't it more like a segmentation rathen than a detection job
\n
No. obj_loss is computed using all words output, reg_loss and cls_loss are computed using output at positive locations, others ignored. I suggest reading YOLO papers and source code to understand the training target design.
", "votes": 1, "replies": [] }, { "id": 1728571, "author_name": "GUNER", "author_url": "", "post_date": "2022-03-19T03:35:48.377000", "content": "Got it, thanks.
", "votes": 0, "replies": [] } ] }, { "id": 1727963, "author_name": "GUNER", "author_url": "", "post_date": "2022-03-18T13:32:11.773000", "content": "'''In addition, I assign lowest cost word in each span as positive during training.'''
\nWe are talking about the ground truth span and creating 'an additional positive' (in addition to the first word of the span) within the ground truth span, so that an additional loss will be calculated, right?
\nWhat is the reference, when you calculate 'lowest cost word'? in other words what is this other word 'lowest cost' of/to?
\nthank you.
\n\nadditional loss will be calculated, right?
\n
The more positive you have, the more \"samples\" you have to compute cls_loss and reg_loss. So yes.
\n\n\nwhat is this other word 'lowest cost' of/to?
\n
The lowest cost word of the span. It might be the first or not. The cost is computed using network output of current sample&iteration. Please check the code for details.
", "votes": 1, "replies": [] }, { "id": 1728568, "author_name": "GUNER", "author_url": "", "post_date": "2022-03-19T03:34:44.717000", "content": "Thank you.
", "votes": 0, "replies": [] } ] }, { "id": 1727874, "author_name": "patriot", "author_url": "", "post_date": "2022-03-18T12:05:21.700000", "content": "Thank you for the original solution! There is much to learn from you.
\n\n\nI used one cycle policy in training and averaged weights of the last few epochs
\n
Is this something like SWA / EMA that averages weights in the middle of learning? Or did you average after learning like model soups?
", "votes": 0, "replies": [ { "id": 1728481, "author_name": "tascj", "author_url": "", "post_date": "2022-03-19T00:27:44.167000", "content": "It's SWA, average of weights of different epochs in single hyperparameters setting. Model soups seems to be average of different weights from same pretrain using different hyperparameters.
", "votes": 1, "replies": [] } ] }, { "id": 1727724, "author_name": "Patrick Chan", "author_url": "", "post_date": "2022-03-18T07:55:39.700000", "content": "MasterStroke… or should I say Grandmaster strokes! Well done and very innovative indeed crafting CV techniques into the NLP world! Thks very much for the share. Much room for improvement with ensembling and post processing… Should get much higher ranking… Any idea why the WBF did not work in your case?
", "votes": 0, "replies": [ { "id": 1728495, "author_name": "tascj", "author_url": "", "post_date": "2022-03-19T00:38:34.090000", "content": "\n\nAny idea why the WBF did not work in your case?
\n
Unfortunately no.
", "votes": 0, "replies": [] }, { "id": 1728895, "author_name": "tascj", "author_url": "", "post_date": "2022-03-19T11:50:41.033000", "content": "@drpatrickchan It seems to be just wrong hyperparameters. A low iou_thr (0.2) is necessary to ensemble my models. Didn't try iou_thr lower than 0.5.
\nThanks to the post processing code shared by @cdeotte, I checked the difference and soon found the problem. Different score thresholds for different classes also helps a bit after WBF, while it does not work for my models before WBF.
\nThe ensemble of 4models (longformer large + 3 deberta) x 2folds could get 0.720 public and 0.737 private (iou_thr=0.2, score_thr=0.3 for all classes).
", "votes": 5, "replies": [] } ] }, { "id": 1727582, "author_name": "kwchen", "author_url": "", "post_date": "2022-03-18T04:59:49.853000", "content": "Congratulations and wow this solution is amazing!
\nFrom the YOLO paper:
\n\nIt weights localization er\u0002ror equally with classification error which may not be ideal.
\n
Did you try tuning the weights between classification loss and regression loss? I feel like you could boost performance a lot with a bit of tuning.
", "votes": 0, "replies": [ { "id": 1727681, "author_name": "tascj", "author_url": "", "post_date": "2022-03-18T07:14:59.430000", "content": "Thank you.
\nI tried a few settings, the loss weights between classification and regression does not matter much.
\nThe loss weight of objectness is important.
\nFollowing YOLOX, I use
num_total_samples = obj_target.sum()\n obj_loss = F.binary_cross_entropy_with_logits(\n obj_pred, obj_target, reduction='sum') / num_total_samples\nIf I haven't read the source code of YOLOX before, I may use
\n obj_loss = F.binary_cross_entropy_with_logits(\n obj_pred, obj_target, reduction='mean')\nSo interesting - well done. I wish I could have spent more time on this competition, but definitely going to experiment with your code.
", "votes": 0, "replies": [] }, { "id": 1725662, "author_name": "SchenbergZ", "author_url": "", "post_date": "2022-03-17T09:49:02.183000", "content": "Congrats! 2 questions: what is your first output head objectness meaning? Is your inference result only based on the 2nd output-head calculation and the discource-type based on 3rd output head?
", "votes": 0, "replies": [ { "id": 1725700, "author_name": "tascj", "author_url": "", "post_date": "2022-03-17T10:50:09.603000", "content": "Thank you.
\n\n\nwhat is your first output head objectness meaning?
\n
Objectness target is a vector of shape (num_words, ). For each ground truth span, you have at least one positive word (1 in the objectness) as representative of the span. Others words in the span are negative (0 in the objectness vector).
\n\n\nIs your inference result only based on the 2nd output-head calculation and the discource-type based on 3rd output head?
\n
All three heads' output are used in inference.
\nI suggest reading YOLO papers and source code to understand the training target design and how to decode raw prediction to target in inference.
", "votes": 2, "replies": [] } ] }, { "id": 1725611, "author_name": "GUNER", "author_url": "", "post_date": "2022-03-17T08:55:40.917000", "content": "You nailed it without hitting the nail a thousand times :-)
\nYou should be the one!
this tip help me a lot,thank you
", "votes": 1, "replies": [] } ], "comment_count": 40, "messages": [ { "id": 1725353, "postDate": "2022-03-17T03:02:29.593Z", "content": "Thank you for hosting this awesome competition and congrats to the winners.
\nThis problem is interesting and challenging. The data and annotations are high quality. It's a great learning and practicing experience working on this competition.
\nI started this competition compeletely ignoring those shared NER baselines. I thought in an \"object detection\" task, segmentation + posprocess approaches can not do better than object detection approaches. I'm surprised by the postprocess ideas in top solutions.
\nOverall, what I made is a YOLO-like text span detector. I share my code in this notebook.
\nCheckpoints are shared here.
\nTo reproduce the checkpoints, check the code and configs here.
\nAutoModelForTokenClassification.from_pretrained(num_labels=1 + 2 + num_classes, ...)\n1 for objectness (or span-ness?. fg/bg classification)
\n2 for regression (distance from fg location to first and last index of corresponding span)
\nnum_classes for discourse type classification
\nNetwork logits is in shape (num_tokens, 10). This is inconvenient (decoding output, ensembling models with different tokenizers). So I aggregate the logits to (num_words, 10) using RoIAlign.
\nThe problem of this formulation is how to define positive for objectness training. In object detection, center of object is a natural choice of positve. In text span detection, I found the first word of span a good choice of positive.
\nIn addition, I assign lowest cost word in each span as positive during training. This is inspired by YOLOX.
\nI randomly replace tokens with mask token during training.
\nFor other details, please check the code.
\nThe only post process is nms.
\nI used one cycle policy in training and averaged weights of the last few epochs.
\nTo ensemble different models/folds, I simply averaged outputs of models.
\nWBF ensemble does not work in local validation, and I didn't figure out why. I think I did something wrong here.
| backbone | \nValidation | \nPublic LB | \nPrivate LB | \n
|---|---|---|---|
| google/bigbird-roberta-base | \n0.685~0.69 | \n\n | \n |
| allenai/longformer-base-4096 | \n0.685~0.69 | \n\n | \n |
| allenai/longformer-large-4096 | \n0.70~0.71 | \n\n | \n |
| microsoft/deberta-base | \n0.70~0.705 | \n\n | \n |
| microsoft/deberta-large | \n0.715~0.72 | \n\n | \n |
| microsoft/deberta-xlarge | \n0.715~0.72 | \n\n | \n |
| microsoft/deberta-large + microsoft/deberta-xlarge | \n0.723 | \n0.714 | \n0.732 | \n
The best combination is deberta-large + deberta-xlarge. Ensembling more does not help.
\nIn the final submission, I used 2 weights (2/5 folds) each model. Submission time is around 2 hours.
This solution should be #1
\nMore seriously, it is very elegant and I'm pretty sure that there is room for improvement - perhaps it would outperform other approaches if combined with a features+boosting model :)
", "rawMarkdown": "This solution should be #1\n\nMore seriously, it is very elegant and I'm pretty sure that there is room for improvement - perhaps it would outperform other approaches if combined with a features+boosting model :)", "votes": 15, "replies": [ { "id": 1726052, "postDate": "2022-03-17T15:59:03.777Z", "content": "Submission time 2h ha ha … crazy…
", "rawMarkdown": "Submission time 2h ha ha ... crazy...", "votes": 1 }, { "id": 1726247, "postDate": "2022-03-17T19:41:31.397Z", "content": "\n\nThis solution should be #1
\n
Agree
", "rawMarkdown": "> This solution should be #1\n\nAgree" }, { "id": 1727404, "postDate": "2022-03-18T00:34:41.197Z", "content": "Thank you.
\nYes there is room for improvement. I didn't try ensemble until the last day because I thought WBF would just work, however it did not work as expected. There are TPs in bigbird model output that deberta models does not catch, but I haven't found a proper merging method without adding too many FPs.
\nThe post process ideas in top solutions are something I would never came up with. My model using low threshold can't achieve the recall shared in the 1st solution. There's a lot to learn and try.
", "rawMarkdown": "Thank you.\n\nYes there is room for improvement. I didn't try ensemble until the last day because I thought WBF would just work, however it did not work as expected. There are TPs in bigbird model output that deberta models does not catch, but I haven't found a proper merging method without adding too many FPs.\n\nThe post process ideas in top solutions are something I would never came up with. My model using low threshold can't achieve the recall shared in the 1st solution. There's a lot to learn and try.", "votes": 2 } ] }, { "id": 1725409, "postDate": "2022-03-17T04:33:18.033Z", "content": "love how you tackle an NLP problem with CV methods and made it work. Simple, elegant, very little ensembling, fast inference time. My favourite solution so far! Thanks for sharing!
", "rawMarkdown": "love how you tackle an NLP problem with CV methods and made it work. Simple, elegant, very little ensembling, fast inference time. My favourite solution so far! Thanks for sharing!", "votes": 9 }, { "id": 1725867, "postDate": "2022-03-17T13:41:43.177Z", "content": "Your work on this competition is just WOW! Well done!
", "rawMarkdown": "Your work on this competition is just WOW! Well done!", "votes": 1 }, { "id": 1725383, "postDate": "2022-03-17T03:44:16.553Z", "content": "Really elegant and creative work! Want to have a try😃
", "rawMarkdown": "Really elegant and creative work! Want to have a try😃", "votes": 1 }, { "id": 1725362, "postDate": "2022-03-17T03:15:23.437Z", "content": "Interesting solution! Congrats on 6th place and thanks for sharing your approach.
", "rawMarkdown": "Interesting solution! Congrats on 6th place and thanks for sharing your approach.", "votes": 1 }, { "id": 1848146, "postDate": "2022-07-08T11:55:03.963Z", "content": "Great model !!
\nI had thought of the same segmentation approach with YOLO but I failed to make my model as efficient as yours and I finally find myself at the 1937 position of the competition. But this is my first real competition on Kaggle.
Your code is a real inspiration to me. Thanks for sharing !!!
", "rawMarkdown": "Great model !!\nI had thought of the same segmentation approach with YOLO but I failed to make my model as efficient as yours and I finally find myself at the 1937 position of the competition. But this is my first real competition on Kaggle.\n\nYour code is a real inspiration to me. Thanks for sharing !!!" }, { "id": 1746097, "postDate": "2022-04-05T13:55:11.767Z", "content": "thanks for sharing! you mention \"In text span detection, I found the first word of span a good choice of positive.\",in this situation is there no possible to predict firest location of span?so num_labels=1 + 1 + num_classes
", "rawMarkdown": "thanks for sharing! you mention \"In text span detection, I found the first word of span a good choice of positive.\",in this situation is there no possible to predict firest location of span?so num_labels=1 + 1 + num_classes", "replies": [ { "id": 1748013, "postDate": "2022-04-07T08:49:57.353Z", "content": "Yes. 1 is enough for single positive assignment, 2 is used for the extra dynamically assigned positive.
", "rawMarkdown": "Yes. 1 is enough for single positive assignment, 2 is used for the extra dynamically assigned positive." } ] }, { "id": 1731304, "postDate": "2022-03-22T08:15:12.473Z", "content": "One (more) tiny :-) question on 'dynamic_pozitive' targets :
\nAs the first word is always the target and the 'dynamic_pozitive' is an add on without a fixed grid-location (it is calculated based on minimum loss-rule) ,, wouldn't the contribution of 'dynamic_pozitive' targets diminish as you train? Since, the first word is always getting trained on and it dominates the network in time (,so: soon, 'first word' also dominates the 'min-loss' calculations (as the word with minimum loss)?
I had the same concern. The cost is not equal to loss, and there are random masking augmentation so in practice it does not happen and dynamic positive improves score. Actually in practice there could be multiple words with lower cost compared to the first word of span.
\nAnyway, in the worst case it just falls back to single positive. I think it's fine.
", "rawMarkdown": "I had the same concern. The cost is not equal to loss, and there are random masking augmentation so in practice it does not happen and dynamic positive improves score. Actually in practice there could be multiple words with lower cost compared to the first word of span.\n\nAnyway, in the worst case it just falls back to single positive. I think it's fine.", "votes": 1 }, { "id": 1732120, "postDate": "2022-03-23T03:18:48.940Z", "content": "Thanks again.
", "rawMarkdown": "Thanks again." } ] }, { "id": 1731257, "postDate": "2022-03-22T06:48:21.077Z", "content": "Thank you for sharing your wonderful solution!
\nI would like to know what fg and bg refer to.
foreground and background
", "rawMarkdown": "foreground and background", "votes": 1 }, { "id": 1732100, "postDate": "2022-03-23T02:09:34.597Z", "content": "Thank you for your replying!
", "rawMarkdown": "Thank you for your replying!" } ] }, { "id": 1730428, "postDate": "2022-03-21T08:36:51.313Z", "content": "interesting
", "rawMarkdown": "interesting " }, { "id": 1729931, "postDate": "2022-03-20T17:36:26.533Z", "content": "Well done !! Congratulations @tascj0 !!🤩🤩
", "rawMarkdown": "Well done !! Congratulations @tascj0 !!🤩🤩" }, { "id": 1729167, "postDate": "2022-03-19T17:30:48.297Z", "content": "well done, get this to top
", "rawMarkdown": "well done, get this to top" }, { "id": 1728624, "postDate": "2022-03-19T05:06:44.323Z", "content": "Creative solution, thanks for sharing your thoughts and process 🙌
", "rawMarkdown": "Creative solution, thanks for sharing your thoughts and process 🙌" }, { "id": 1727980, "postDate": "2022-03-18T13:53:44.310Z", "content": "'''Network logits is in shape (num_tokens, 10)'''
\nBecause the model creates output for each token (CV-pixel equivalent), isn't it more like a segmentation rathen than a detection job, thus falling not very far from NER approaches?
\n\nisn't it more like a segmentation rathen than a detection job
\n
No. obj_loss is computed using all words output, reg_loss and cls_loss are computed using output at positive locations, others ignored. I suggest reading YOLO papers and source code to understand the training target design.
", "rawMarkdown": "> isn't it more like a segmentation rathen than a detection job\n\nNo. obj_loss is computed using all words output, reg_loss and cls_loss are computed using output at positive locations, others ignored. I suggest reading YOLO papers and source code to understand the training target design.", "votes": 1 }, { "id": 1728571, "postDate": "2022-03-19T03:35:48.377Z", "content": "Got it, thanks.
", "rawMarkdown": "Got it, thanks. " } ] }, { "id": 1727963, "postDate": "2022-03-18T13:32:11.773Z", "content": "'''In addition, I assign lowest cost word in each span as positive during training.'''
\nWe are talking about the ground truth span and creating 'an additional positive' (in addition to the first word of the span) within the ground truth span, so that an additional loss will be calculated, right?
\nWhat is the reference, when you calculate 'lowest cost word'? in other words what is this other word 'lowest cost' of/to?
\nthank you.
\n\nadditional loss will be calculated, right?
\n
The more positive you have, the more \"samples\" you have to compute cls_loss and reg_loss. So yes.
\n\n\nwhat is this other word 'lowest cost' of/to?
\n
The lowest cost word of the span. It might be the first or not. The cost is computed using network output of current sample&iteration. Please check the code for details.
", "rawMarkdown": "> additional loss will be calculated, right?\n\nThe more positive you have, the more \"samples\" you have to compute cls_loss and reg_loss. So yes.\n\n> what is this other word 'lowest cost' of/to?\n\nThe lowest cost word of the span. It might be the first or not. The cost is computed using network output of current sample&iteration. Please check the code for details.", "votes": 1 }, { "id": 1728568, "postDate": "2022-03-19T03:34:44.717Z", "content": "Thank you.
", "rawMarkdown": "Thank you." } ] }, { "id": 1727874, "postDate": "2022-03-18T12:05:21.700Z", "content": "Thank you for the original solution! There is much to learn from you.
\n\n\nI used one cycle policy in training and averaged weights of the last few epochs
\n
Is this something like SWA / EMA that averages weights in the middle of learning? Or did you average after learning like model soups?
", "rawMarkdown": "Thank you for the original solution! There is much to learn from you.\n> I used one cycle policy in training and averaged weights of the last few epochs\n\nIs this something like SWA / EMA that averages weights in the middle of learning? Or did you average after learning like [model soups](https://arxiv.org/abs/2203.05482)?", "replies": [ { "id": 1728481, "postDate": "2022-03-19T00:27:44.167Z", "content": "It's SWA, average of weights of different epochs in single hyperparameters setting. Model soups seems to be average of different weights from same pretrain using different hyperparameters.
", "rawMarkdown": "It's SWA, average of weights of different epochs in single hyperparameters setting. Model soups seems to be average of different weights from same pretrain using different hyperparameters.", "votes": 1 } ] }, { "id": 1727724, "postDate": "2022-03-18T07:55:39.700Z", "content": "MasterStroke… or should I say Grandmaster strokes! Well done and very innovative indeed crafting CV techniques into the NLP world! Thks very much for the share. Much room for improvement with ensembling and post processing… Should get much higher ranking… Any idea why the WBF did not work in your case?
", "rawMarkdown": "MasterStroke... or should I say Grandmaster strokes! Well done and very innovative indeed crafting CV techniques into the NLP world! Thks very much for the share. Much room for improvement with ensembling and post processing... Should get much higher ranking... Any idea why the WBF did not work in your case?", "replies": [ { "id": 1728495, "postDate": "2022-03-19T00:38:34.090Z", "content": "\n\nAny idea why the WBF did not work in your case?
\n
Unfortunately no.
", "rawMarkdown": "> Any idea why the WBF did not work in your case?\n\nUnfortunately no." }, { "id": 1728895, "postDate": "2022-03-19T11:50:41.033Z", "content": "@drpatrickchan It seems to be just wrong hyperparameters. A low iou_thr (0.2) is necessary to ensemble my models. Didn't try iou_thr lower than 0.5.
\nThanks to the post processing code shared by @cdeotte, I checked the difference and soon found the problem. Different score thresholds for different classes also helps a bit after WBF, while it does not work for my models before WBF.
\nThe ensemble of 4models (longformer large + 3 deberta) x 2folds could get 0.720 public and 0.737 private (iou_thr=0.2, score_thr=0.3 for all classes).
", "rawMarkdown": "@drpatrickchan It seems to be just wrong hyperparameters. A low iou_thr (0.2) is necessary to ensemble my models. Didn't try iou_thr lower than 0.5.\n\nThanks to the post processing code shared by @cdeotte, I checked the difference and soon found the problem. Different score thresholds for different classes also helps a bit after WBF, while it does not work for my models before WBF.\n\nThe ensemble of 4models (longformer large + 3 deberta) x 2folds could get 0.720 public and 0.737 private (iou_thr=0.2, score_thr=0.3 for all classes).", "votes": 5 } ] }, { "id": 1727582, "postDate": "2022-03-18T04:59:49.853Z", "content": "Congratulations and wow this solution is amazing!
\nFrom the YOLO paper:
\n\nIt weights localization er\u0002ror equally with classification error which may not be ideal.
\n
Did you try tuning the weights between classification loss and regression loss? I feel like you could boost performance a lot with a bit of tuning.
", "rawMarkdown": "Congratulations and wow this solution is amazing!\nFrom the YOLO paper:\n> It weights localization er\u0002ror equally with classification error which may not be ideal.\n\nDid you try tuning the weights between classification loss and regression loss? I feel like you could boost performance a lot with a bit of tuning.", "replies": [ { "id": 1727681, "postDate": "2022-03-18T07:14:59.430Z", "content": "Thank you.
\nI tried a few settings, the loss weights between classification and regression does not matter much.
\nThe loss weight of objectness is important.
\nFollowing YOLOX, I use
num_total_samples = obj_target.sum()\n obj_loss = F.binary_cross_entropy_with_logits(\n obj_pred, obj_target, reduction='sum') / num_total_samples\nIf I haven't read the source code of YOLOX before, I may use
\n obj_loss = F.binary_cross_entropy_with_logits(\n obj_pred, obj_target, reduction='mean')\nSo interesting - well done. I wish I could have spent more time on this competition, but definitely going to experiment with your code.
", "rawMarkdown": "So interesting - well done. I wish I could have spent more time on this competition, but definitely going to experiment with your code." }, { "id": 1725662, "postDate": "2022-03-17T09:49:02.183Z", "content": "Congrats! 2 questions: what is your first output head objectness meaning? Is your inference result only based on the 2nd output-head calculation and the discource-type based on 3rd output head?
", "rawMarkdown": "Congrats! 2 questions: what is your first output head *objectness* meaning? Is your inference result only based on the 2nd output-head calculation and the discource-type based on 3rd output head?", "replies": [ { "id": 1725700, "postDate": "2022-03-17T10:50:09.603Z", "content": "Thank you.
\n\n\nwhat is your first output head objectness meaning?
\n
Objectness target is a vector of shape (num_words, ). For each ground truth span, you have at least one positive word (1 in the objectness) as representative of the span. Others words in the span are negative (0 in the objectness vector).
\n\n\nIs your inference result only based on the 2nd output-head calculation and the discource-type based on 3rd output head?
\n
All three heads' output are used in inference.
\nI suggest reading YOLO papers and source code to understand the training target design and how to decode raw prediction to target in inference.
", "rawMarkdown": "Thank you.\n\n> what is your first output head objectness meaning? \n\nObjectness target is a vector of shape (num_words, ). For each ground truth span, you have at least one positive word (1 in the objectness) as representative of the span. Others words in the span are negative (0 in the objectness vector).\n\n> Is your inference result only based on the 2nd output-head calculation and the discource-type based on 3rd output head?\n\nAll three heads' output are used in inference.\n\nI suggest reading YOLO papers and source code to understand the training target design and how to decode raw prediction to target in inference.", "votes": 2 } ] }, { "id": 1725611, "postDate": "2022-03-17T08:55:40.917Z", "content": "You nailed it without hitting the nail a thousand times :-)
\nYou should be the one!
this tip help me a lot,thank you
", "rawMarkdown": "this tip help me a lot,thank you", "votes": 1 } ], "complete": true, "source": "live", "fetched_at": "2026-10-19T12:35:58Z", "competition_id": 31779, "competition_slug": "feedback-prize-2021", "team_id": 8169219, "solution_url": "/competitions/feedback-prize-2021/writeups/rist-tascj-6th-place-solution-a-yolo-like-text-spa" }