\n
![\"1647309894393-e37bd0b0-7aac-45e6-a7d0-d749e3a10fb2.png\"](\"https://i.postimg.cc/wT8rPY0R/1647309894393-e37bd0b0-7aac-45e6-a7d0-d749e3a10fb2.png\")
\nWe put the bert training code here, Because the kaggle online resources are insufficient, you need to copy it to your own machine for training
First of all, I would like to thank the competition organizers for this great competition. At the same time, I am very grateful to many brothers for providing very good notebooks and discussions. I learned a lot from them and applied them to our final solution.
\nIn this competition, because each token needs to be combined into a sentence as the final result, in order to reduce post-processing, we divide our solution into two stages
\nFirst thanks to the excellent notebooks and discussions @cdeotte @abhishek @hengck23 and others
\nhttps://www.kaggle.com/cdeotte/tensorflow-longformer-ner-cv-0-633
\nhttps://www.kaggle.com/abhishek/two-longformers-are-better-than-1
\nhttps://www.kaggle.com/hengck23/1-birdformer-1-longformer-one-fold
\nhttps://www.kaggle.com/c/feedback-prize-2021/discussion/297591
\nhttps://www.kaggle.com/c/feedback-prize-2021/discussion/308992
\netc…
we tried various pretrain models, since the max length of some models is 512, for these models, we choose the method of segmented prediction and splicing, Finally we choice longformer-large, roberta-large, deberta-xxlarge, distilbart_mnli_12_9, bart_large_finetuned_squadv1 for ensemble. This stage takes 7 hours online, and cv score 0.712, lb score 0.706 with post-processing.
\nhere is each pretrain model cv score:
\n
\nWe put the bert training code here, Because the kaggle online resources are insufficient, you need to copy it to your own machine for training
Thanks @chasembowers for the excellent notebook https://www.kaggle.com/chasembowers/sequence-postprocessing-v2-67-lb
\nwe first recall as many candidate samples as possible by lowering the threshold. On the training set, we recall three million samples to achieve a mean of 95% of recalls, the recalls for each class are
\n| class | \nrecall | \n
|---|---|
| Claim | \n0.938 | \n
| Concluding Statement | \n0.972 | \n
| Counterclaim | \n0.906 | \n
| Evidence | \n0.974 | \n
| Lead | \n0.970 | \n
| Position | \n0.928 | \n
| Rebuttal | \n0.895 | \n
in addition, after getting the recall samples, we select sample with high boundary threshold and choice 65% length with the highest probability of the current class as a new sample, this method can help improve score about 0.008. Finally, We made about 170 features for lgb training, and select some samples as the final submission. This stage takes 1.5 hours online, and cv score 0.748, lb score 0.742.
\nWe tested our lgb on the 5 fold longformer model, and the score increased from 0.697 to 0.727. Because lgb is not trained on this prediction, the improvement will be lower than the actual. At the same time, we uploaded our model ensemble results here. If you are interested, you can replace the prediction results with your own to see how much the cv score can improve.
\nuseful attempt:
\n1、adversarial learning (awp/fgm): cv increase 0.01，lb 5fold ensemble increase 0.003.
\n2、model ensemble: single model lb 0.691, model ensemble 0.706, longformer and deberta ensemble increase most.
\n3、lgb sentence prediction: cv increase 0.036, lb increase 0.036, among this, select sample with high boundary threshold and choice 65% length with the highest probability of the current class as a new sample can increase 0.008
useless attempt:
\n1、add paragraph information to input
\n2、back translation
\n3、adjust the weight according to the position of the sentence in which the word is located
\n4、lgb model use overlap percentage as label to train
\n5、stage 2 use bert to predict and ensemble with lgb
code:
\nlongformer train: https://www.kaggle.com/wht1996/feedback-nn-train
\nlgb train: https://www.kaggle.com/wht1996/feedback-lgb-train
\n5fold longfomer with post-processing(lb 0.697): https://www.kaggle.com/wht1996/feedback-longformer-5fold-0-697
\n5fold longfomer with lgb(lb 0.727): https://www.kaggle.com/wht1996/feedback-two-stage-lb0-727
\ncv ensemble with lgb(cv 0.747): https://www.kaggle.com/wht1996/feedback-two-stage-cv0-747
Our code and data is published on GitHub here
", "comments": [ { "id": 1724125, "author_name": "hengck23", "author_url": "", "post_date": "2022-03-16T03:15:16.237000", "content": "[posiition/] Congratulations on you and yuor taem! [/posiition]
\n[claim] It is a well-deserved suesccs.[/claim]
\n[evidence]There have been much epinxteerms dnoe and thnak you for srihnag yuor rsteuls. [/evidence]
\nIt is ncie to see new mdeohts lkie araisdeavrl lenirang.
\n[conclusion]I wlil try it out![/conclusion]
![\"thumbs-up-icon-126802\"](\"https://i.ibb.co/TPsXyQH/thumbs-up-icon-126802.png\"){kind=icon}
Congrats @wht1996 and team. Great work!
", "votes": 3, "replies": [] }, { "id": 1724029, "author_name": "老肥", "author_url": "", "post_date": "2022-03-16T01:23:05.237000", "content": "lgb great again.
\nAmazing!
", "votes": 4, "replies": [] }, { "id": 1725089, "author_name": "@kaggleqrdl", "author_url": "", "post_date": "2022-03-16T19:46:53.890000", "content": "Very very nice architecture. Intuitions were excellent. I am looking forward to further and deeper analysis of this code as it is 'productized' for the OS community. I strongly encourage you to put this all on Github and contribute a paper to Arvix. You've done great work here.
", "votes": 1, "replies": [ { "id": 1725099, "author_name": "@kaggleqrdl", "author_url": "", "post_date": "2022-03-16T19:55:29.567000", "content": "One question, which pipeline would you say best describes your code:
\nBert (low threshold, high recall) -> span selection -> span manipulation.
\nor
\nBert (low threshold, high recall) -> span manipulation -> span selection
\nBack propegating through the pipeline is an interesting problem space.
", "votes": 0, "replies": [] }, { "id": 1725333, "author_name": "(⊙﹏⊙)", "author_url": "", "post_date": "2022-03-17T02:24:58.760000", "content": "I think it is Bert (low threshold, high recall) -> span manipulation -> span selection
", "votes": 0, "replies": [] } ] }, { "id": 1724102, "author_name": "jenny", "author_url": "", "post_date": "2022-03-16T02:50:36.430000", "content": "Congrats for your big win with such a huge score-lead on LB!
\nCan't believe yyds-LGB in the 2nd stage can bring such considerable improvement.
Because the kaggle online resources are insufficient, you need to copy it to your own machine for training
BTW: can you briefly explain your local resource and running time for training? Thanks!
", "votes": 1, "replies": [ { "id": 1724136, "author_name": "(⊙﹏⊙)", "author_url": "", "post_date": "2022-03-16T03:30:33.463000", "content": "A longformer-large model requires 4 A100s to train for about 5 hours, and a deberta-xxlarge model requires 4 A100s to train for about 24 hours
", "votes": 2, "replies": [] }, { "id": 1724186, "author_name": "Nicholas Broad", "author_url": "", "post_date": "2022-03-16T04:17:35.160000", "content": "4x A100 for 24 hours on jarvislabs would be 4*$2.39/hr*24 hours = $230
\nI guess your prize money will cover this 😁
", "votes": 1, "replies": [] } ] }, { "id": 1724057, "author_name": "小白Lan", "author_url": "", "post_date": "2022-03-16T02:10:10.177000", "content": "my dear dalao, plz accept my knees….LGB YYDS！
", "votes": 1, "replies": [] }, { "id": 1724047, "author_name": "Ethan", "author_url": "", "post_date": "2022-03-16T01:45:58.507000", "content": "Congrats! I learn a lot from your solution!
", "votes": 1, "replies": [] }, { "id": 1724041, "author_name": "chizhu", "author_url": "", "post_date": "2022-03-16T01:39:52.670000", "content": "LGB yyds!!!
", "votes": 1, "replies": [] }, { "id": 1724032, "author_name": "RB", "author_url": "", "post_date": "2022-03-16T01:24:50.797000", "content": "Congratulations and thank you for the writeup . Could you please elaborate on
\n\n\nsince the max length of some models is 512, for these models, we choose the method of segmented prediction and splicing
\n
what does segmented prediction and splicing mean ? Is this same as window striding ?
", "votes": 1, "replies": [ { "id": 1724040, "author_name": "(⊙﹏⊙)", "author_url": "", "post_date": "2022-03-16T01:39:16.427000", "content": "like this![\"image.png\"](\"https://i.postimg.cc/y8kP32W8/image.png\")
How do you combine the two sequences? Averaging the overlap?
", "votes": 1, "replies": [] }, { "id": 1724062, "author_name": "(⊙﹏⊙)", "author_url": "", "post_date": "2022-03-16T02:14:43.553000", "content": "It's similar to the simple average. A little optimization is added. See the code for details.
\nhttps://www.kaggle.com/wht1996/feedback-nn-train
if i == 0: \n preds = logits\nelse:\n if i == (loops-1):\n if i==1: \n inter_length = loops_end[0]-loops_start[-1]\n weight = torch.floor_divide(torch.arange(inter_length), inter_length-1).reshape(1,-1,1).to(device)\n intersection = preds[:,start:]*(1-weight) + logits[:,:inter_length]*(weight)\n preds = torch.cat([preds[:,:start],intersection,logits[:,inter_length:]],dim=1)\n else: \n preds = torch.cat([preds[:,:start],logits],dim=1)\n else:\n preds = torch.cat([preds,logits[:,duplicate_cnt:]],dim=1)\nWould you be able to explain the AWP model or point to a resource explaining it?
\nedit: never mind I found this: https://arxiv.org/abs/2004.05884
congrats!
", "votes": 0, "replies": [] }, { "id": 1783633, "author_name": "Zheng Zhao", "author_url": "", "post_date": "2022-05-10T14:39:14.167000", "content": "Congrats! I learn a lot from your solution!
", "votes": 0, "replies": [] }, { "id": 1737281, "author_name": "Shilida", "author_url": "", "post_date": "2022-03-28T09:50:47.867000", "content": "LGB yyds Amazing!!
", "votes": 0, "replies": [] }, { "id": 1737218, "author_name": "Jyotirmoy Bandyopadhayaya", "author_url": "", "post_date": "2022-03-28T08:35:41.017000", "content": "Great job. Well Done on your submission for the same
", "votes": 0, "replies": [] }, { "id": 1736775, "author_name": "Parv Yadav", "author_url": "", "post_date": "2022-03-27T18:06:20.300000", "content": "Congrats @wht1996 and thanks for sharing!
", "votes": 0, "replies": [] }, { "id": 1736639, "author_name": "egoriya", "author_url": "", "post_date": "2022-03-27T14:56:29.980000", "content": "Nice job man!
", "votes": 0, "replies": [] }, { "id": 1735169, "author_name": "stallone", "author_url": "", "post_date": "2022-03-25T23:49:20.417000", "content": "nice solution
", "votes": 0, "replies": [] }, { "id": 1734126, "author_name": "HieuGN", "author_url": "", "post_date": "2022-03-25T02:40:43.347000", "content": "perfect !!!
", "votes": 0, "replies": [] }, { "id": 1734032, "author_name": "Swarup Sahoo", "author_url": "", "post_date": "2022-03-24T22:30:30.790000", "content": "Thank you for the details! Great analysis.
", "votes": 0, "replies": [] }, { "id": 1733763, "author_name": "m_ouballa", "author_url": "", "post_date": "2022-03-24T15:37:29.827000", "content": "Congrats @wht1996 and team. Great work!
", "votes": 0, "replies": [] }, { "id": 1733638, "author_name": "Veronika Chernova", "author_url": "", "post_date": "2022-03-24T13:22:43.100000", "content": "Thank you for sharing!
\nI try to use some code from your notebook.
\nWhat does kfold_ids.pkl mean? I understand that there are 5 folds, but I can't understand why each fold in this file have two arrays with ids.
One is the id of the training set and the other is the id of the validation set
", "votes": 0, "replies": [] } ] }, { "id": 1733555, "author_name": "yin huaihong", "author_url": "", "post_date": "2022-03-24T11:58:46.643000", "content": "Congratulations! many thanks for your sharing
", "votes": 0, "replies": [] }, { "id": 1733071, "author_name": " KyleChen陳柏元", "author_url": "", "post_date": "2022-03-24T01:46:41.463000", "content": "Looks great!
", "votes": 0, "replies": [] }, { "id": 1733031, "author_name": "JotaCe7", "author_url": "", "post_date": "2022-03-24T00:37:43.023000", "content": "Hello dear Kaggle community! My name is Jose!
", "votes": 0, "replies": [] }, { "id": 1732497, "author_name": "Ashish Kapoor", "author_url": "", "post_date": "2022-03-23T12:57:43.840000", "content": "congrats!
", "votes": 0, "replies": [] }, { "id": 1732431, "author_name": "Tarek Ghanoum", "author_url": "", "post_date": "2022-03-23T11:31:10.483000", "content": "Great work!
", "votes": 0, "replies": [] }, { "id": 1732258, "author_name": "Hans Ma", "author_url": "", "post_date": "2022-03-23T07:08:31.807000", "content": "Amazing!!!!!
", "votes": 0, "replies": [] }, { "id": 1731975, "author_name": "Beatriz Velasco", "author_url": "", "post_date": "2022-03-22T22:51:14.747000", "content": "congratsss
", "votes": 0, "replies": [] }, { "id": 1731731, "author_name": "Scott Obsniuk", "author_url": "", "post_date": "2022-03-22T16:31:25.983000", "content": "👍👍👍👍👍👍👍👍👍👍👍👍👍👍
", "votes": 0, "replies": [] }, { "id": 1731235, "author_name": "", "author_url": "", "post_date": "2022-03-22T06:14:20.220000", "content": "", "votes": 0, "replies": [] }, { "id": 1731179, "author_name": "", "author_url": "", "post_date": "2022-03-22T04:23:55.260000", "content": "", "votes": 0, "replies": [] }, { "id": 1730935, "author_name": "", "author_url": "", "post_date": "2022-03-21T19:39:32.420000", "content": "", "votes": 0, "replies": [] }, { "id": 1730915, "author_name": "", "author_url": "", "post_date": "2022-03-21T18:47:22.387000", "content": "", "votes": 0, "replies": [] }, { "id": 1730464, "author_name": "", "author_url": "", "post_date": "2022-03-21T09:43:48.383000", "content": "", "votes": 0, "replies": [] }, { "id": 1729621, "author_name": "", "author_url": "", "post_date": "2022-03-20T09:46:37.513000", "content": "", "votes": 0, "replies": [] }, { "id": 1728094, "author_name": "", "author_url": "", "post_date": "2022-03-18T15:22:53.930000", "content": "", "votes": 0, "replies": [] }, { "id": 1727754, "author_name": "", "author_url": "", "post_date": "2022-03-18T08:41:44.650000", "content": "", "votes": 0, "replies": [] }, { "id": 1727725, "author_name": "", "author_url": "", "post_date": "2022-03-18T07:56:29.773000", "content": "", "votes": 0, "replies": [] }, { "id": 1727555, "author_name": "", "author_url": "", "post_date": "2022-03-18T03:53:02.887000", "content": "", "votes": 0, "replies": [] }, { "id": 1727538, "author_name": "", "author_url": "", "post_date": "2022-03-18T03:30:31.477000", "content": "", "votes": 0, "replies": [] }, { "id": 1727495, "author_name": "", "author_url": "", "post_date": "2022-03-18T02:41:14.513000", "content": "", "votes": 0, "replies": [] }, { "id": 1726023, "author_name": "", "author_url": "", "post_date": "2022-03-17T15:31:08.877000", "content": "", "votes": 0, "replies": [] }, { "id": 1725597, "author_name": "", "author_url": "", "post_date": "2022-03-17T08:45:08.913000", "content": "", "votes": 0, "replies": [] }, { "id": 1725295, "author_name": "", "author_url": "", "post_date": "2022-03-17T01:37:37.910000", "content": "", "votes": 0, "replies": [ { "id": 1725711, "author_name": "", "author_url": "", "post_date": "2022-03-17T11:02:02.467000", "content": "", "votes": 1, "replies": [] }, { "id": 1725752, "author_name": "", "author_url": "", "post_date": "2022-03-17T12:06:45.177000", "content": "", "votes": 0, "replies": [] } ] }, { "id": 1725291, "author_name": "", "author_url": "", "post_date": "2022-03-17T01:36:13.180000", "content": "", "votes": 0, "replies": [ { "id": 1725714, "author_name": "", "author_url": "", "post_date": "2022-03-17T11:02:59.143000", "content": "", "votes": 0, "replies": [] } ] }, { "id": 1725218, "author_name": "", "author_url": "", "post_date": "2022-03-16T23:24:17.200000", "content": "", "votes": 0, "replies": [] }, { "id": 1724914, "author_name": "", "author_url": "", "post_date": "2022-03-16T16:17:54.900000", "content": "", "votes": 0, "replies": [] }, { "id": 1724904, "author_name": "", "author_url": "", "post_date": "2022-03-16T16:03:22.357000", "content": "", "votes": 0, "replies": [] }, { "id": 1724883, "author_name": "", "author_url": "", "post_date": "2022-03-16T15:45:20.797000", "content": "", "votes": 0, "replies": [] }, { "id": 1724881, "author_name": "", "author_url": "", "post_date": "2022-03-16T15:45:10.580000", "content": "", "votes": 0, "replies": [] }, { "id": 1724804, "author_name": "", "author_url": "", "post_date": "2022-03-16T14:24:00.730000", "content": "", "votes": 0, "replies": [] }, { "id": 1724675, "author_name": "", "author_url": "", "post_date": "2022-03-16T12:45:25.840000", "content": "", "votes": 0, "replies": [] }, { "id": 1724650, "author_name": "", "author_url": "", "post_date": "2022-03-16T12:19:00.823000", "content": "", "votes": 0, "replies": [] }, { "id": 1724649, "author_name": "", "author_url": "", "post_date": "2022-03-16T12:18:21.917000", "content": "", "votes": 0, "replies": [] }, { "id": 1724640, "author_name": "", "author_url": "", "post_date": "2022-03-16T12:12:34.590000", "content": "", "votes": 0, "replies": [ { "id": 1725341, "author_name": "", "author_url": "", "post_date": "2022-03-17T02:44:09.807000", "content": "", "votes": 0, "replies": [] } ] }, { "id": 1724638, "author_name": "", "author_url": "", "post_date": "2022-03-16T12:07:15.940000", "content": "", "votes": 0, "replies": [] }, { "id": 1724492, "author_name": "", "author_url": "", "post_date": "2022-03-16T09:26:40.760000", "content": "", "votes": 0, "replies": [ { "id": 1724508, "author_name": "", "author_url": "", "post_date": "2022-03-16T09:33:33.827000", "content": "", "votes": 2, "replies": [] } ] }, { "id": 1724423, "author_name": "", "author_url": "", "post_date": "2022-03-16T08:37:02.843000", "content": "", "votes": 0, "replies": [] }, { "id": 1724328, "author_name": "", "author_url": "", "post_date": "2022-03-16T07:30:59.993000", "content": "", "votes": 0, "replies": [ { "id": 1724669, "author_name": "", "author_url": "", "post_date": "2022-03-16T12:36:48.267000", "content": "", "votes": 3, "replies": [] }, { "id": 1724786, "author_name": "", "author_url": "", "post_date": "2022-03-16T14:07:04.743000", "content": "", "votes": 0, "replies": [] }, { "id": 1724821, "author_name": "", "author_url": "", "post_date": "2022-03-16T14:37:39.893000", "content": "", "votes": 2, "replies": [] } ] }, { "id": 1724303, "author_name": "", "author_url": "", "post_date": "2022-03-16T06:56:30.830000", "content": "", "votes": 0, "replies": [] }, { "id": 1724293, "author_name": "", "author_url": "", "post_date": "2022-03-16T06:45:06.577000", "content": "", "votes": 0, "replies": [] }, { "id": 1724196, "author_name": "", "author_url": "", "post_date": "2022-03-16T04:30:50.277000", "content": "", "votes": 0, "replies": [] }, { "id": 1724149, "author_name": "", "author_url": "", "post_date": "2022-03-16T03:42:44.340000", "content": "", "votes": 0, "replies": [] }, { "id": 1724094, "author_name": "", "author_url": "", "post_date": "2022-03-16T02:44:58.140000", "content": "", "votes": 0, "replies": [] }, { "id": 1724079, "author_name": "", "author_url": "", "post_date": "2022-03-16T02:27:43.923000", "content": "", "votes": 0, "replies": [ { "id": 1724087, "author_name": "", "author_url": "", "post_date": "2022-03-16T02:38:29.100000", "content": "", "votes": 2, "replies": [] }, { "id": 1724092, "author_name": "", "author_url": "", "post_date": "2022-03-16T02:43:36.947000", "content": "", "votes": 0, "replies": [] }, { "id": 1724098, "author_name": "", "author_url": "", "post_date": "2022-03-16T02:49:20.113000", "content": "", "votes": 7, "replies": [] } ] }, { "id": 1724078, "author_name": "", "author_url": "", "post_date": "2022-03-16T02:27:03.823000", "content": "", "votes": 0, "replies": [] }, { "id": 1724076, "author_name": "", "author_url": "", "post_date": "2022-03-16T02:26:41.213000", "content": "", "votes": 0, "replies": [] }, { "id": 1724070, "author_name": "", "author_url": "", "post_date": "2022-03-16T02:20:18.457000", "content": "", "votes": 0, "replies": [ { "id": 1724075, "author_name": "", "author_url": "", "post_date": "2022-03-16T02:26:23.767000", "content": "", "votes": 0, "replies": [] } ] }, { "id": 1724068, "author_name": "", "author_url": "", "post_date": "2022-03-16T02:18:55.607000", "content": "", "votes": 0, "replies": [ { "id": 1724074, "author_name": "", "author_url": "", "post_date": "2022-03-16T02:25:32.627000", "content": "", "votes": 1, "replies": [] }, { "id": 1724108, "author_name": "", "author_url": "", "post_date": "2022-03-16T02:58:18.173000", "content": "", "votes": 0, "replies": [] }, { "id": 1724116, "author_name": "", "author_url": "", "post_date": "2022-03-16T03:06:23.817000", "content": "", "votes": 0, "replies": [] }, { "id": 1729217, "author_name": "", "author_url": "", "post_date": "2022-03-19T19:17:41.713000", "content": "", "votes": 0, "replies": [] } ] }, { "id": 1724054, "author_name": "", "author_url": "", "post_date": "2022-03-16T02:08:59.837000", "content": "", "votes": 0, "replies": [] }, { "id": 1724034, "author_name": "", "author_url": "", "post_date": "2022-03-16T01:31:23.517000", "content": "", "votes": 0, "replies": [ { "id": 1724044, "author_name": "", "author_url": "", "post_date": "2022-03-16T01:42:16.263000", "content": "", "votes": 2, "replies": [] }, { "id": 1724069, "author_name": "", "author_url": "", "post_date": "2022-03-16T02:19:02.250000", "content": "", "votes": 0, "replies": [] } ] }, { "id": 1724033, "author_name": "", "author_url": "", "post_date": "2022-03-16T01:28:52.567000", "content": "", "votes": 0, "replies": [] }, { "id": 1724027, "author_name": "", "author_url": "", "post_date": "2022-03-16T01:19:45.477000", "content": "", "votes": 0, "replies": [] }, { "id": 1730523, "author_name": "", "author_url": "", "post_date": "2022-03-21T11:40:08.870000", "content": "", "votes": 0, "replies": [] }, { "id": 1727798, "author_name": "", "author_url": "", "post_date": "2022-03-18T10:11:49.657000", "content": "", "votes": 0, "replies": [] }, { "id": 1724287, "author_name": "", "author_url": "", "post_date": "2022-03-16T06:37:11.883000", "content": "", "votes": 1, "replies": [] }, { "id": 1775333, "author_name": "", "author_url": "", "post_date": "2022-05-02T22:42:52.560000", "content": "", "votes": 0, "replies": [] }, { "id": 1734066, "author_name": "", "author_url": "", "post_date": "2022-03-25T00:02:09.497000", "content": "", "votes": 0, "replies": [] }, { "id": 1733659, "author_name": "", "author_url": "", "post_date": "2022-03-24T13:47:04.867000", "content": "", "votes": 0, "replies": [] }, { "id": 1733022, "author_name": "", "author_url": "", "post_date": "2022-03-24T00:17:04.427000", "content": "", "votes": 0, "replies": [] }, { "id": 1731300, "author_name": "", "author_url": "", "post_date": "2022-03-22T08:07:06.050000", "content": "", "votes": 0, "replies": [] }, { "id": 1729809, "author_name": "", "author_url": "", "post_date": "2022-03-20T14:46:32.483000", "content": "", "votes": 0, "replies": [] }, { "id": 1728032, "author_name": "", "author_url": "", "post_date": "2022-03-18T14:36:17.583000", "content": "", "votes": 0, "replies": [] }, { "id": 1727578, "author_name": "", "author_url": "", "post_date": "2022-03-18T04:44:22.513000", "content": "", "votes": 0, "replies": [] }, { "id": 1726232, "author_name": "", "author_url": "", "post_date": "2022-03-17T19:30:10.317000", "content": "", "votes": 0, "replies": [] }, { "id": 1725332, "author_name": "", "author_url": "", "post_date": "2022-03-17T02:24:05.563000", "content": "", "votes": 0, "replies": [] }, { "id": 1724977, "author_name": "", "author_url": "", "post_date": "2022-03-16T17:27:40.073000", "content": "", "votes": 0, "replies": [] }, { "id": 1724111, "author_name": "", "author_url": "", "post_date": "2022-03-16T03:01:21.963000", "content": "", "votes": 0, "replies": [] } ], "comment_count": 114, "messages": [ { "id": 1724024, "postDate": "2022-03-16T01:14:05.163Z", "content": "First of all, I would like to thank the competition organizers for this great competition. At the same time, I am very grateful to many brothers for providing very good notebooks and discussions. I learned a lot from them and applied them to our final solution.
\nIn this competition, because each token needs to be combined into a sentence as the final result, in order to reduce post-processing, we divide our solution into two stages
\nFirst thanks to the excellent notebooks and discussions @cdeotte @abhishek @hengck23 and others
\nhttps://www.kaggle.com/cdeotte/tensorflow-longformer-ner-cv-0-633
\nhttps://www.kaggle.com/abhishek/two-longformers-are-better-than-1
\nhttps://www.kaggle.com/hengck23/1-birdformer-1-longformer-one-fold
\nhttps://www.kaggle.com/c/feedback-prize-2021/discussion/297591
\nhttps://www.kaggle.com/c/feedback-prize-2021/discussion/308992
\netc…
we tried various pretrain models, since the max length of some models is 512, for these models, we choose the method of segmented prediction and splicing, Finally we choice longformer-large, roberta-large, deberta-xxlarge, distilbart_mnli_12_9, bart_large_finetuned_squadv1 for ensemble. This stage takes 7 hours online, and cv score 0.712, lb score 0.706 with post-processing.
\nhere is each pretrain model cv score:
\n
\nWe put the bert training code here, Because the kaggle online resources are insufficient, you need to copy it to your own machine for training
Thanks @chasembowers for the excellent notebook https://www.kaggle.com/chasembowers/sequence-postprocessing-v2-67-lb
\nwe first recall as many candidate samples as possible by lowering the threshold. On the training set, we recall three million samples to achieve a mean of 95% of recalls, the recalls for each class are
\n| class | \nrecall | \n
|---|---|
| Claim | \n0.938 | \n
| Concluding Statement | \n0.972 | \n
| Counterclaim | \n0.906 | \n
| Evidence | \n0.974 | \n
| Lead | \n0.970 | \n
| Position | \n0.928 | \n
| Rebuttal | \n0.895 | \n
in addition, after getting the recall samples, we select sample with high boundary threshold and choice 65% length with the highest probability of the current class as a new sample, this method can help improve score about 0.008. Finally, We made about 170 features for lgb training, and select some samples as the final submission. This stage takes 1.5 hours online, and cv score 0.748, lb score 0.742.
\nWe tested our lgb on the 5 fold longformer model, and the score increased from 0.697 to 0.727. Because lgb is not trained on this prediction, the improvement will be lower than the actual. At the same time, we uploaded our model ensemble results here. If you are interested, you can replace the prediction results with your own to see how much the cv score can improve.
\nuseful attempt:
\n1、adversarial learning (awp/fgm): cv increase 0.01，lb 5fold ensemble increase 0.003.
\n2、model ensemble: single model lb 0.691, model ensemble 0.706, longformer and deberta ensemble increase most.
\n3、lgb sentence prediction: cv increase 0.036, lb increase 0.036, among this, select sample with high boundary threshold and choice 65% length with the highest probability of the current class as a new sample can increase 0.008
useless attempt:
\n1、add paragraph information to input
\n2、back translation
\n3、adjust the weight according to the position of the sentence in which the word is located
\n4、lgb model use overlap percentage as label to train
\n5、stage 2 use bert to predict and ensemble with lgb
code:
\nlongformer train: https://www.kaggle.com/wht1996/feedback-nn-train
\nlgb train: https://www.kaggle.com/wht1996/feedback-lgb-train
\n5fold longfomer with post-processing(lb 0.697): https://www.kaggle.com/wht1996/feedback-longformer-5fold-0-697
\n5fold longfomer with lgb(lb 0.727): https://www.kaggle.com/wht1996/feedback-two-stage-lb0-727
\ncv ensemble with lgb(cv 0.747): https://www.kaggle.com/wht1996/feedback-two-stage-cv0-747
Our code and data is published on GitHub here
", "rawMarkdown": "First of all, I would like to thank the competition organizers for this great competition. At the same time, I am very grateful to many brothers for providing very good notebooks and discussions. I learned a lot from them and applied them to our final solution. \n\nIn this competition, because each token needs to be combined into a sentence as the final result, in order to reduce post-processing, we divide our solution into two stages\n\n# stage 1: bert token prediction\nFirst thanks to the excellent notebooks and discussions @cdeotte @abhishek @hengck23 and others\nhttps://www.kaggle.com/cdeotte/tensorflow-longformer-ner-cv-0-633 \nhttps://www.kaggle.com/abhishek/two-longformers-are-better-than-1\nhttps://www.kaggle.com/hengck23/1-birdformer-1-longformer-one-fold\nhttps://www.kaggle.com/c/feedback-prize-2021/discussion/297591\nhttps://www.kaggle.com/c/feedback-prize-2021/discussion/308992\netc...\n\nwe tried various pretrain models, since the max length of some models is 512, for these models, we choose the method of segmented prediction and splicing, Finally we choice longformer-large, roberta-large, deberta-xxlarge, distilbart_mnli_12_9, bart_large_finetuned_squadv1 for ensemble. This stage takes 7 hours online, and **cv score 0.712**, **lb score 0.706** with post-processing.\n\nhere is each pretrain model cv score:\n[](https://postimg.cc/c6hTg2qd)\nWe put the bert training code [here](https://www.kaggle.com/wht1996/feedback-nn-train), Because the kaggle online resources are insufficient, you need to copy it to your own machine for training\n\n# stage 2: lgb sentence prediction\nThanks @chasembowers for the excellent notebook https://www.kaggle.com/chasembowers/sequence-postprocessing-v2-67-lb\n\nwe first recall as many candidate samples as possible by lowering the threshold. On the training set, we recall three million samples to achieve a mean of 95% of recalls, the recalls for each class are \n| class | recall |\n| --- | --- |\n| Claim | 0.938 |\n| Concluding Statement | 0.972 |\n| Counterclaim | 0.906 |\n| Evidence | 0.974 |\n| Lead | 0.970 |\n| Position | 0.928 |\n| Rebuttal | 0.895 |\n\nin addition, after getting the recall samples, we select sample with high boundary threshold and choice 65% length with the highest probability of the current class as a new sample, this method can help improve score about 0.008. Finally, We made about 170 features for lgb training, and select some samples as the final submission. This stage takes 1.5 hours online, and **cv score 0.748**, **lb score 0.742**.\n\nWe tested our lgb on the 5 fold longformer model, and the score increased from [0.697](https://www.kaggle.com/wht1996/feedback-longformer-5fold-0-697) to [0.727](https://www.kaggle.com/wht1996/feedback-two-stage-lb0-727). Because lgb is not trained on this prediction, the improvement will be lower than the actual. At the same time, we uploaded our model ensemble results [here](https://www.kaggle.com/wht1996/feedback-two-stage-cv748). If you are interested, you can replace the prediction results with your own to see how much the cv score can improve.\n\n# Summarize\n**useful attempt:**\n1、adversarial learning (awp/fgm): cv increase 0.01，lb 5fold ensemble increase 0.003.\n2、model ensemble: single model lb 0.691, model ensemble 0.706, longformer and deberta ensemble increase most.\n3、lgb sentence prediction: cv increase 0.036, lb increase 0.036, among this, select sample with high boundary threshold and choice 65% length with the highest probability of the current class as a new sample can increase 0.008\n\n**useless attempt:**\n1、add paragraph information to input\n2、back translation\n3、adjust the weight according to the position of the sentence in which the word is located\n4、lgb model use overlap percentage as label to train\n5、stage 2 use bert to predict and ensemble with lgb\n\n**code:**\nlongformer train: https://www.kaggle.com/wht1996/feedback-nn-train\nlgb train: https://www.kaggle.com/wht1996/feedback-lgb-train\n5fold longfomer with post-processing(lb 0.697): https://www.kaggle.com/wht1996/feedback-longformer-5fold-0-697\n5fold longfomer with lgb(lb 0.727): https://www.kaggle.com/wht1996/feedback-two-stage-lb0-727\ncv ensemble with lgb(cv 0.747): https://www.kaggle.com/wht1996/feedback-two-stage-cv0-747\n\nOur code and data is published on GitHub [here](https://github.com/antmachineintelligence/Feedback_1st)", "votes": 282 }, { "id": 1724125, "postDate": "2022-03-16T03:15:16.237Z", "content": "[posiition/] Congratulations on you and yuor taem! [/posiition]
\n[claim] It is a well-deserved suesccs.[/claim]
\n[evidence]There have been much epinxteerms dnoe and thnak you for srihnag yuor rsteuls. [/evidence]
\nIt is ncie to see new mdeohts lkie araisdeavrl lenirang.
\n[conclusion]I wlil try it out![/conclusion]
\n",
"votes": 42
},
{
"id": 1724206,
"postDate": "2022-03-16T04:43:08.710Z",
"content": "Congrats @wht1996 and team. Great work!
", "rawMarkdown": "Congrats @wht1996 and team. Great work!", "votes": 3 }, { "id": 1724029, "postDate": "2022-03-16T01:23:05.237Z", "content": "lgb great again.
\nAmazing!
", "rawMarkdown": "lgb great again.\n\nAmazing!", "votes": 4 }, { "id": 1725089, "postDate": "2022-03-16T19:46:53.890Z", "content": "Very very nice architecture. Intuitions were excellent. I am looking forward to further and deeper analysis of this code as it is 'productized' for the OS community. I strongly encourage you to put this all on Github and contribute a paper to Arvix. You've done great work here.
", "rawMarkdown": "Very very nice architecture. Intuitions were excellent. I am looking forward to further and deeper analysis of this code as it is 'productized' for the OS community. I strongly encourage you to put this all on Github and contribute a paper to Arvix. You've done great work here.", "votes": 1, "replies": [ { "id": 1725099, "postDate": "2022-03-16T19:55:29.567Z", "content": "One question, which pipeline would you say best describes your code:
\nBert (low threshold, high recall) -> span selection -> span manipulation.
\nor
\nBert (low threshold, high recall) -> span manipulation -> span selection
\nBack propegating through the pipeline is an interesting problem space.
", "rawMarkdown": "One question, which pipeline would you say best describes your code:\n\nBert (low threshold, high recall) -> span selection -> span manipulation. \n\nor\n\nBert (low threshold, high recall) -> span manipulation -> span selection\n\nBack propegating through the pipeline is an interesting problem space." }, { "id": 1725333, "postDate": "2022-03-17T02:24:58.760Z", "content": "I think it is Bert (low threshold, high recall) -> span manipulation -> span selection
", "rawMarkdown": "I think it is Bert (low threshold, high recall) -> span manipulation -> span selection" } ] }, { "id": 1724102, "postDate": "2022-03-16T02:50:36.430Z", "content": "Congrats for your big win with such a huge score-lead on LB!
\nCan't believe yyds-LGB in the 2nd stage can bring such considerable improvement.
Because the kaggle online resources are insufficient, you need to copy it to your own machine for training
BTW: can you briefly explain your local resource and running time for training? Thanks!
", "rawMarkdown": "Congrats for your big win with such a huge score-lead on LB! \nCan't believe yyds-LGB in the 2nd stage can bring such considerable improvement. \n\n```Because the kaggle online resources are insufficient, you need to copy it to your own machine for training```\n\nBTW: can you briefly explain your local resource and running time for training? Thanks!", "votes": 1, "replies": [ { "id": 1724136, "postDate": "2022-03-16T03:30:33.463Z", "content": "A longformer-large model requires 4 A100s to train for about 5 hours, and a deberta-xxlarge model requires 4 A100s to train for about 24 hours
", "rawMarkdown": "A longformer-large model requires 4 A100s to train for about 5 hours, and a deberta-xxlarge model requires 4 A100s to train for about 24 hours", "votes": 2 }, { "id": 1724186, "postDate": "2022-03-16T04:17:35.160Z", "content": "4x A100 for 24 hours on jarvislabs would be 4*$2.39/hr*24 hours = $230
\nI guess your prize money will cover this 😁
", "rawMarkdown": "4x A100 for 24 hours on jarvislabs would be 4\\*$2.39/hr\\*24 hours = $230\n\n# 😲\n\nI guess your prize money will cover this 😁", "votes": 1 } ] }, { "id": 1724057, "postDate": "2022-03-16T02:10:10.177Z", "content": "my dear dalao, plz accept my knees….LGB YYDS！
", "rawMarkdown": "my dear dalao, plz accept my knees....LGB YYDS！", "votes": 1 }, { "id": 1724047, "postDate": "2022-03-16T01:45:58.507Z", "content": "Congrats! I learn a lot from your solution!
", "rawMarkdown": "Congrats! I learn a lot from your solution!", "votes": 1 }, { "id": 1724041, "postDate": "2022-03-16T01:39:52.670Z", "content": "LGB yyds!!!
", "rawMarkdown": "LGB yyds!!!", "votes": 1 }, { "id": 1724032, "postDate": "2022-03-16T01:24:50.797Z", "content": "Congratulations and thank you for the writeup . Could you please elaborate on
\n\n\nsince the max length of some models is 512, for these models, we choose the method of segmented prediction and splicing
\n
what does segmented prediction and splicing mean ? Is this same as window striding ?
", "rawMarkdown": "Congratulations and thank you for the writeup . Could you please elaborate on \n> since the max length of some models is 512, for these models, we choose the method of segmented prediction and splicing\n\nwhat does segmented prediction and splicing mean ? Is this same as window striding ? ", "votes": 1, "replies": [ { "id": 1724040, "postDate": "2022-03-16T01:39:16.427Z", "content": "like this
How do you combine the two sequences? Averaging the overlap?
", "rawMarkdown": "How do you combine the two sequences? Averaging the overlap?", "votes": 1 }, { "id": 1724062, "postDate": "2022-03-16T02:14:43.553Z", "content": "It's similar to the simple average. A little optimization is added. See the code for details.
\nhttps://www.kaggle.com/wht1996/feedback-nn-train
if i == 0: \n preds = logits\nelse:\n if i == (loops-1):\n if i==1: \n inter_length = loops_end[0]-loops_start[-1]\n weight = torch.floor_divide(torch.arange(inter_length), inter_length-1).reshape(1,-1,1).to(device)\n intersection = preds[:,start:]*(1-weight) + logits[:,:inter_length]*(weight)\n preds = torch.cat([preds[:,:start],intersection,logits[:,inter_length:]],dim=1)\n else: \n preds = torch.cat([preds[:,:start],logits],dim=1)\n else:\n preds = torch.cat([preds,logits[:,duplicate_cnt:]],dim=1)\nWould you be able to explain the AWP model or point to a resource explaining it?
\nedit: never mind I found this: https://arxiv.org/abs/2004.05884
congrats!
", "rawMarkdown": "congrats!\n\n\n" }, { "id": 1783633, "postDate": "2022-05-10T14:39:14.167Z", "content": "Congrats! I learn a lot from your solution!
", "rawMarkdown": "Congrats! I learn a lot from your solution!" }, { "id": 1737281, "postDate": "2022-03-28T09:50:47.867Z", "content": "LGB yyds Amazing!!
", "rawMarkdown": "LGB yyds Amazing!!" }, { "id": 1737218, "postDate": "2022-03-28T08:35:41.017Z", "content": "Great job. Well Done on your submission for the same
", "rawMarkdown": "Great job. Well Done on your submission for the same " }, { "id": 1736775, "postDate": "2022-03-27T18:06:20.300Z", "content": "Congrats @wht1996 and thanks for sharing!
", "rawMarkdown": "Congrats @wht1996 and thanks for sharing!" }, { "id": 1736639, "postDate": "2022-03-27T14:56:29.980Z", "content": "Nice job man!
", "rawMarkdown": "Nice job man!" }, { "id": 1735169, "postDate": "2022-03-25T23:49:20.417Z", "content": "nice solution
", "rawMarkdown": "nice solution" }, { "id": 1734126, "postDate": "2022-03-25T02:40:43.347Z", "content": "perfect !!!
", "rawMarkdown": "perfect !!!" }, { "id": 1734032, "postDate": "2022-03-24T22:30:30.790Z", "content": "Thank you for the details! Great analysis.
", "rawMarkdown": "Thank you for the details! Great analysis." }, { "id": 1733763, "postDate": "2022-03-24T15:37:29.827Z", "content": "Congrats @wht1996 and team. Great work!
", "rawMarkdown": "Congrats @wht1996 and team. Great work!\n\n" }, { "id": 1733638, "postDate": "2022-03-24T13:22:43.100Z", "content": "Thank you for sharing!
\nI try to use some code from your notebook.
\nWhat does kfold_ids.pkl mean? I understand that there are 5 folds, but I can't understand why each fold in this file have two arrays with ids.
One is the id of the training set and the other is the id of the validation set
", "rawMarkdown": "One is the id of the training set and the other is the id of the validation set" } ] }, { "id": 1733555, "postDate": "2022-03-24T11:58:46.643Z", "content": "Congratulations! many thanks for your sharing
", "rawMarkdown": "Congratulations! many thanks for your sharing" }, { "id": 1733071, "postDate": "2022-03-24T01:46:41.463Z", "content": "Looks great!
", "rawMarkdown": "Looks great!" }, { "id": 1733031, "postDate": "2022-03-24T00:37:43.023Z", "content": "Hello dear Kaggle community! My name is Jose!
", "rawMarkdown": "Hello dear Kaggle community! My name is Jose!" }, { "id": 1732497, "postDate": "2022-03-23T12:57:43.840Z", "content": "congrats!
", "rawMarkdown": "congrats! " }, { "id": 1732431, "postDate": "2022-03-23T11:31:10.483Z", "content": "Great work!
", "rawMarkdown": "Great work!" }, { "id": 1732258, "postDate": "2022-03-23T07:08:31.807Z", "content": "Amazing!!!!!
", "rawMarkdown": "Amazing!!!!!" }, { "id": 1731975, "postDate": "2022-03-22T22:51:14.747Z", "content": "congratsss
", "rawMarkdown": "congratsss" }, { "id": 1731731, "postDate": "2022-03-22T16:31:25.983Z", "content": "👍👍👍👍👍👍👍👍👍👍👍👍👍👍
", "rawMarkdown": "👍👍👍👍👍👍👍👍👍👍👍👍👍👍" }, { "id": 1731235, "postDate": "2022-03-22T06:14:20.220Z", "content": "Great work 👍
", "rawMarkdown": "Great work 👍" }, { "id": 1731179, "postDate": "2022-03-22T04:23:55.260Z", "content": "Very very nice architecture. Intuitions were excellent.👍
", "rawMarkdown": "Very very nice architecture. Intuitions were excellent.👍" }, { "id": 1730935, "postDate": "2022-03-21T19:39:32.420Z", "content": "Great work!
", "rawMarkdown": "Great work!" }, { "id": 1730915, "postDate": "2022-03-21T18:47:22.387Z", "content": "Congratulations!
", "rawMarkdown": "Congratulations!" }, { "id": 1730464, "postDate": "2022-03-21T09:43:48.383Z", "content": "Very good architecture.
", "rawMarkdown": "Very good architecture." }, { "id": 1729621, "postDate": "2022-03-20T09:46:37.513Z", "content": "Very nice, good job!
", "rawMarkdown": "Very nice, good job!" }, { "id": 1728094, "postDate": "2022-03-18T15:22:53.930Z", "content": "interesting work!
", "rawMarkdown": "interesting work!" }, { "id": 1727754, "postDate": "2022-03-18T08:41:44.650Z", "content": "Looks good!
", "rawMarkdown": "Looks good!" }, { "id": 1727725, "postDate": "2022-03-18T07:56:29.773Z", "content": "nice study
", "rawMarkdown": "nice study" }, { "id": 1727555, "postDate": "2022-03-18T03:53:02.887Z", "content": "Interesting!
", "rawMarkdown": "Interesting!" }, { "id": 1727538, "postDate": "2022-03-18T03:30:31.477Z", "content": "Great work, many of us have thought to use LGB for rerank, but only your team figure out the key is recall rate. Congratulations!
", "rawMarkdown": "Great work, many of us have thought to use LGB for rerank, but only your team figure out the key is recall rate. Congratulations!" }, { "id": 1727495, "postDate": "2022-03-18T02:41:14.513Z", "content": "congrats! by the way, the resolution of the pic of cv score is very low even for the original pic, could you please fix it?
", "rawMarkdown": "congrats! by the way, the resolution of the pic of cv score is very low even for the original pic, could you please fix it?" }, { "id": 1726023, "postDate": "2022-03-17T15:31:08.877Z", "content": "Excellent!
", "rawMarkdown": "Excellent!" }, { "id": 1725597, "postDate": "2022-03-17T08:45:08.913Z", "content": "Big Congrats!!,
\nStaying calm and focused, you have given the best response to the 'speculators',
and a big thank you,
\nYou have shared 'both' your knowledge and code, which is not very common of those speculators :-)
Hi, @wht1996 . Could you please tell how do you ensemble different models?
\nThanks in advance!
", "rawMarkdown": "Hi, @wht1996 . Could you please tell how do you ensemble different models?\n\nThanks in advance!", "replies": [ { "id": 1725711, "postDate": "2022-03-17T11:02:02.467Z", "content": "you can refer to https://www.kaggle.com/hengck23/1-birdformer-1-longformer-one-fold
", "rawMarkdown": "you can refer to https://www.kaggle.com/hengck23/1-birdformer-1-longformer-one-fold", "votes": 1 }, { "id": 1725752, "postDate": "2022-03-17T12:06:45.177Z", "content": "Thanks a lot!
", "rawMarkdown": "Thanks a lot!" } ] }, { "id": 1725291, "postDate": "2022-03-17T01:36:13.180Z", "content": "Congrats and thanks for sharing !!!
\nDid you use any Data Augmentation methods to improve model performance?
we tried back translation, but it didn't work
", "rawMarkdown": "we tried back translation, but it didn't work" } ] }, { "id": 1725218, "postDate": "2022-03-16T23:24:17.200Z", "content": "Congratulations! Great job
", "rawMarkdown": "Congratulations! Great job" }, { "id": 1724914, "postDate": "2022-03-16T16:17:54.900Z", "content": "Thanks for sharing and congratulations👏 ..will wait to see your code unfortunately we could not get that much juice it of v3 .. But Learnt good techniques will use in the future from this
", "rawMarkdown": "Thanks for sharing and congratulations👏 ..will wait to see your code unfortunately we could not get that much juice it of v3 .. But Learnt good techniques will use in the future from this" }, { "id": 1724904, "postDate": "2022-03-16T16:03:22.357Z", "content": "Congratulations! Great work. I tried post-processing but it didn't succeed. I will study from your notebook. Thank you for sharing.
", "rawMarkdown": "Congratulations! Great work. I tried post-processing but it didn't succeed. I will study from your notebook. Thank you for sharing." }, { "id": 1724883, "postDate": "2022-03-16T15:45:20.797Z", "content": "Nice solution
", "rawMarkdown": "Nice solution" }, { "id": 1724881, "postDate": "2022-03-16T15:45:10.580Z", "content": "Very impressive, good job !
", "rawMarkdown": "Very impressive, good job !" }, { "id": 1724804, "postDate": "2022-03-16T14:24:00.730Z", "content": "Thanks for sharing! Great works! Can't wait to learn from yours achievements!👍
", "rawMarkdown": "Thanks for sharing! Great works! Can't wait to learn from yours achievements!👍" }, { "id": 1724675, "postDate": "2022-03-16T12:45:25.840Z", "content": "Congratulations and thanks for sharing your success with us.
\nEd
", "rawMarkdown": "Congratulations and thanks for sharing your success with us.\n\nEd" }, { "id": 1724650, "postDate": "2022-03-16T12:19:00.823Z", "content": "Impressive 👍
", "rawMarkdown": "Impressive 👍" }, { "id": 1724649, "postDate": "2022-03-16T12:18:21.917Z", "content": "Congratulation and thanks for the solution
", "rawMarkdown": "Congratulation and thanks for the solution" }, { "id": 1724640, "postDate": "2022-03-16T12:12:34.590Z", "content": "lgb stands for the light gradient boosting, right?
", "rawMarkdown": "lgb stands for the light gradient boosting, right?", "replies": [ { "id": 1725341, "postDate": "2022-03-17T02:44:09.807Z", "content": "yes, LightGBM
", "rawMarkdown": "yes, LightGBM" } ] }, { "id": 1724638, "postDate": "2022-03-16T12:07:15.940Z", "content": "I'm learning a lot! Thanks for the great explanation and code<(_ _)>
", "rawMarkdown": "I'm learning a lot! Thanks for the great explanation and code<(_ _)>" }, { "id": 1724492, "postDate": "2022-03-16T09:26:40.760Z", "content": "Congrats on the result!
\n\n\ncv score 0.712, lb score 0.706 with post-processing.
\n
Can you say a bit more on the postprocessing you use?
", "rawMarkdown": "Congrats on the result! \n\n> cv score 0.712, lb score 0.706 with post-processing.\n\n Can you say a bit more on the postprocessing you use?", "replies": [ { "id": 1724508, "postDate": "2022-03-16T09:33:33.827Z", "content": "You can refer to the after_deal function in this code https://www.kaggle.com/wht1996/feedback-longformer-5fold-0-697
", "rawMarkdown": "You can refer to the after_deal function in this code https://www.kaggle.com/wht1996/feedback-longformer-5fold-0-697", "votes": 2 } ] }, { "id": 1724423, "postDate": "2022-03-16T08:37:02.843Z", "content": "very impressive solution. Thanks for sharing
", "rawMarkdown": "very impressive solution. Thanks for sharing" }, { "id": 1724328, "postDate": "2022-03-16T07:30:59.993Z", "content": "Great job! Thanks for sharing and big congrats!
\nWhat kind of features did you create for the candidates?
", "rawMarkdown": "Great job! Thanks for sharing and big congrats! \n\nWhat kind of features did you create for the candidates?", "replies": [ { "id": 1724669, "postDate": "2022-03-16T12:36:48.267Z", "content": "Features we have made some conventional statistical features for the current sentence and before and after the sentence. There is nothing special. I think our core strength is not features, but how to get and process candidate samples.
", "rawMarkdown": "Features we have made some conventional statistical features for the current sentence and before and after the sentence. There is nothing special. I think our core strength is not features, but how to get and process candidate samples.", "votes": 3 }, { "id": 1724786, "postDate": "2022-03-16T14:07:04.743Z", "content": "Did you try training a separate bert-like model instead of using statistical features and LGB?
", "rawMarkdown": "Did you try training a separate bert-like model instead of using statistical features and LGB?" }, { "id": 1724821, "postDate": "2022-03-16T14:37:39.893Z", "content": "In stage 1, we used bert for token prediction. and In stage 2, we also tried to use bert for sentence prediction, but the result was not good, so we did not use it in the end
", "rawMarkdown": "In stage 1, we used bert for token prediction. and In stage 2, we also tried to use bert for sentence prediction, but the result was not good, so we did not use it in the end", "votes": 2 } ] }, { "id": 1724303, "postDate": "2022-03-16T06:56:30.830Z", "content": "Great job guys! So it was lgb training which gave an edge at the end to win🙂
", "rawMarkdown": "Great job guys! So it was lgb training which gave an edge at the end to win🙂" }, { "id": 1724293, "postDate": "2022-03-16T06:45:06.577Z", "content": "Congrats @wht1996. And thanks for sharing your ideas. Great work👍.
", "rawMarkdown": "Congrats @wht1996. And thanks for sharing your ideas. Great work👍." }, { "id": 1724196, "postDate": "2022-03-16T04:30:50.277Z", "content": "Congratulations on your win and thanks a lot for sharing your kernels 🙌
", "rawMarkdown": "Congratulations on your win and thanks a lot for sharing your kernels 🙌" }, { "id": 1724149, "postDate": "2022-03-16T03:42:44.340Z", "content": "Congratulations @wht1996 on the win , simple and awesome solution . We were also thinking on the same lines , we used a two layered NN to combine model predictions instead of LGB which was working better , but we didn't have enough powerful Longformer models to give us the boost 🐱
", "rawMarkdown": "Congratulations @wht1996 on the win , simple and awesome solution . We were also thinking on the same lines , we used a two layered NN to combine model predictions instead of LGB which was working better , but we didn't have enough powerful Longformer models to give us the boost 🐱" }, { "id": 1724094, "postDate": "2022-03-16T02:44:58.140Z", "content": "Congratulations! And thanks for sharing!
", "rawMarkdown": "Congratulations! And thanks for sharing!" }, { "id": 1724079, "postDate": "2022-03-16T02:27:43.923Z", "content": "Congrats and thanks for ur solution. I have a concern. I don’t see how you train the lgb models. You only provided lgb model files, which can be trained by anything. I would like to train using my data (end-to-end) to check if lgb can be that good. Could you show us for more transparency and confirm your impressive win?
", "rawMarkdown": "Congrats and thanks for ur solution. I have a concern. I don’t see how you train the lgb models. You only provided lgb model files, which can be trained by anything. I would like to train using my data (end-to-end) to check if lgb can be that good. Could you show us for more transparency and confirm your impressive win?", "replies": [ { "id": 1724087, "postDate": "2022-03-16T02:38:29.100Z", "content": "If you recall samples and do features according to https://www.kaggle.com/wht1996/feedback-two-stage-cv0-747, you can directly train a tree model of your own, so we don't provide additional training code for the tree model. Of course, I can also tidy up our lgb model training code if needed
\nbtw, our lgb model params are
params = {\n 'boosting': 'gbdt',\n 'objective': 'binary',\n 'metric': {'auc'},\n 'num_leaves': 15,\n 'min_data_in_leaf': 30,\n 'max_depth': 5,\n 'learning_rate': 0.03,\n \"feature_fraction\": 0.7,\n \"bagging_fraction\": 0.7,\n 'min_data_in_bin':15,\n \"lambda_l1\": 5,\n 'lambda_l2': 5,\n \"random_state\": 1996,\n \"num_threads\": num_jobs,\n }\nThanks a lot. I’ll try it out.
", "rawMarkdown": "Thanks a lot. I’ll try it out. " }, { "id": 1724098, "postDate": "2022-03-16T02:49:20.113Z", "content": "here is the lgb train code https://www.kaggle.com/wht1996/feedback-lgb-train
", "rawMarkdown": "here is the lgb train code https://www.kaggle.com/wht1996/feedback-lgb-train", "votes": 7 } ] }, { "id": 1724078, "postDate": "2022-03-16T02:27:03.823Z", "content": "Congrats :o
", "rawMarkdown": "Congrats :o" }, { "id": 1724076, "postDate": "2022-03-16T02:26:41.213Z", "content": "very impress solution, great job
", "rawMarkdown": "very impress solution, great job" }, { "id": 1724070, "postDate": "2022-03-16T02:20:18.457Z", "content": "Also, how do you train deberta xlarge? With local gpu?
\nBtw, congrats!
Yes, we used A100 for training
", "rawMarkdown": "Yes, we used A100 for training" } ] }, { "id": 1724068, "postDate": "2022-03-16T02:18:55.607Z", "content": "Hi I just can't understand your lgb methods. can you elaborate select sample with high boundary threshold and choice 65% length with the highest probability of the current class
", "rawMarkdown": "Hi I just can't understand your lgb methods. can you elaborate *select sample with high boundary threshold and choice 65% length with the highest probability of the current class*", "replies": [ { "id": 1724074, "postDate": "2022-03-16T02:25:32.627Z", "content": "Because the intersection of prediction and label only needs to be greater than 50%, if the probability of two boundaries of a sample is very large, I can be sure that the sample will not be longer, then I can make it shorter to improve robustness
", "rawMarkdown": "Because the intersection of prediction and label only needs to be greater than 50%, if the probability of two boundaries of a sample is very large, I can be sure that the sample will not be longer, then I can make it shorter to improve robustness", "votes": 1 }, { "id": 1724108, "postDate": "2022-03-16T02:58:18.173Z", "content": "Does probability of two boundaries of a sample mean that the prediction of a segment in an essay is somehow a specific class with high probabilities in the start token and the end token of this segment?
How do you get 3 million recall samples? Is that means you make prediction with the whole training data?
How to recall samples and how to get new samples according to the boundary can refer to the code https://www.kaggle.com/wht1996/feedback-two-stage-cv0-747
", "rawMarkdown": "How to recall samples and how to get new samples according to the boundary can refer to the code https://www.kaggle.com/wht1996/feedback-two-stage-cv0-747" }, { "id": 1729217, "postDate": "2022-03-19T19:17:41.713Z", "content": "Cool! One more question for running your training script of awp: for the adversial lr have you tuned it or just leave it as default?
", "rawMarkdown": "Cool! One more question for running your training script of awp: for the adversial lr have you tuned it or just leave it as default?" } ] }, { "id": 1724054, "postDate": "2022-03-16T02:08:59.837Z", "content": "Congrats! very clear way！
", "rawMarkdown": "Congrats! very clear way！" }, { "id": 1724034, "postDate": "2022-03-16T01:31:23.517Z", "content": "Thanks for sharing :) Im happy to watch ur write-up.
\nIs the cv score u mentioned in google sheets calculated w/ length and score filtering or w/o filtering?
I don't quite understand what is w/ length and score filtering, you can refer to https://www.kaggle.com/wht1996/feedback-two-stage-cv0-747
", "rawMarkdown": "I don't quite understand what is w/ length and score filtering, you can refer to https://www.kaggle.com/wht1996/feedback-two-stage-cv0-747", "votes": 2 }, { "id": 1724069, "postDate": "2022-03-16T02:19:02.250Z", "content": "I mean post-processing but I noticed this from ur notebook after leaving comments. thanks for sharing again and big congrats to 1st.
", "rawMarkdown": "I mean post-processing but I noticed this from ur notebook after leaving comments. thanks for sharing again and big congrats to 1st." } ] }, { "id": 1724033, "postDate": "2022-03-16T01:28:52.567Z", "content": "Amazing!!!
", "rawMarkdown": "Amazing!!!" }, { "id": 1724027, "postDate": "2022-03-16T01:19:45.477Z", "content": "Been waiting for a while for your team's solution write-up. Thanks for posting! As for the winner's code license, on the rules page it says \"WINNER LICENSE TYPE: OPEN SOURCE - MIT\" so I don't think posting it here would be a problem. Congrats for getting 1st place with such a big margin.
", "rawMarkdown": "Been waiting for a while for your team's solution write-up. Thanks for posting! As for the winner's code license, on the rules page it says \"WINNER LICENSE TYPE: OPEN SOURCE - MIT\" so I don't think posting it here would be a problem. Congrats for getting 1st place with such a big margin." }, { "id": 1730523, "postDate": "2022-03-21T11:40:08.870Z", "rawMarkdown": "", "isDeleted": true }, { "id": 1727798, "postDate": "2022-03-18T10:11:49.657Z", "rawMarkdown": "", "isDeleted": true }, { "id": 1724287, "postDate": "2022-03-16T06:37:11.883Z", "content": "Congratulations thank you for sharing🙌
", "rawMarkdown": "Congratulations thank you for sharing🙌", "votes": 1 }, { "id": 1775333, "postDate": "2022-05-02T22:42:52.560Z", "content": "Thanks you!! amazing!
", "rawMarkdown": "Thanks you!! amazing!" }, { "id": 1734066, "postDate": "2022-03-25T00:02:09.497Z", "content": "Thank you for sharing!
", "rawMarkdown": "Thank you for sharing!" }, { "id": 1733659, "postDate": "2022-03-24T13:47:04.867Z", "content": "Thanks for the information
", "rawMarkdown": "Thanks for the information" }, { "id": 1733022, "postDate": "2022-03-24T00:17:04.427Z", "content": "Congratulations and thanks for sharing.
", "rawMarkdown": "Congratulations and thanks for sharing." }, { "id": 1731300, "postDate": "2022-03-22T08:07:06.050Z", "content": "Thanks for your sharing
", "rawMarkdown": "Thanks for your sharing" }, { "id": 1729809, "postDate": "2022-03-20T14:46:32.483Z", "content": "Thank You for sharing 👍🏻
", "rawMarkdown": "Thank You for sharing 👍🏻" }, { "id": 1728032, "postDate": "2022-03-18T14:36:17.583Z", "content": "Congratulations thank you for sharing🙌
", "rawMarkdown": "Congratulations thank you for sharing🙌" }, { "id": 1727578, "postDate": "2022-03-18T04:44:22.513Z", "content": "Thank you for sharing!
", "rawMarkdown": "Thank you for sharing!" }, { "id": 1726232, "postDate": "2022-03-17T19:30:10.317Z", "content": "Thanks for sharing!
", "rawMarkdown": "Thanks for sharing!" }, { "id": 1725332, "postDate": "2022-03-17T02:24:05.563Z", "content": "Congrats! Thanks a lot for sharing.
", "rawMarkdown": "Congrats! Thanks a lot for sharing." }, { "id": 1724977, "postDate": "2022-03-16T17:27:40.073Z", "content": "congrats and thanks for your work
", "rawMarkdown": "congrats and thanks for your work" }, { "id": 1724111, "postDate": "2022-03-16T03:01:21.963Z", "content": "Thank you for sharing~
", "rawMarkdown": "Thank you for sharing~" } ], "complete": true, "source": "live", "fetched_at": "2026-10-19T12:38:22Z", "competition_id": 31779, "competition_slug": "feedback-prize-2021", "team_id": 7929735, "solution_url": "/competitions/feedback-prize-2021/writeups/world-peace-1st-solution-with-code-cv-0-748-lb-0-7" }