Thank you Georgia State University, The Learning Agency Lab, and Kaggle for a very well run competition. The data was high quality and interesting. It was great learning.
\nThanks also to Huggingface 🤗 . Where would NLP be without you.
\nUnfortunately, we missed out on the best of the postprocessing shown in some of the top approaches - WBF, GBM stackers, and Yolo-style span detector… really great work. Next time!
\nOur solution was blending a number of models (Big bird, Longformer, Deberta, Deberta-v2, Deberta-v3, Bart) all large models. We used a weighted average which was tuned with Optuna. Thresholds were also tuned with Optuna.
\nAs opposed to BIO tagging we used 9 model outputs - 7 classes & No span, as well one output to predict the initial token of each span. We found token dropout worked well to reduce overfitting.
\nTraining on very long sequences did not help too much. Using max 1250 tokens seemed to do well. For the shorter model it worked well to adjust the model to extend position token lengths, and just repeat the position embeddings 2X or 3X times.
", "comments": [ { "id": 1742461, "author_name": "Chris Deotte", "author_url": "", "post_date": "2022-04-01T22:39:06.800000", "content": "Congratulations Darragh and team!
\n\n\nAs opposed to BIO tagging we used 9 model outputs - 7 classes & No span, as well one output to predict the initial token of each span. We found token dropout worked well to reduce overfitting.
\n
This is a nice trick. I will try this for future NER tasks. Basically all the \"B\" labels from BIO are grouped into a single \"B\" label. I think the model would like that and learn better because now it gets more \"B\" label data. Is this your team's idea, or is this a common idea in NER NLP?
", "votes": 2, "replies": [ { "id": 1742828, "author_name": "Darragh", "author_url": "", "post_date": "2022-04-02T10:17:39.137000", "content": "When we initially merged both teams had been doing it this way, which made thing easier. The clovaai team, with BROS did a similar thing in their model, and that is where I had picked it up.
", "votes": 2, "replies": [] } ] } ], "comment_count": 2, "messages": [ { "id": 1739601, "postDate": "2022-03-30T07:21:05.383Z", "content": "Thank you Georgia State University, The Learning Agency Lab, and Kaggle for a very well run competition. The data was high quality and interesting. It was great learning.
\nThanks also to Huggingface 🤗 . Where would NLP be without you.
\nUnfortunately, we missed out on the best of the postprocessing shown in some of the top approaches - WBF, GBM stackers, and Yolo-style span detector… really great work. Next time!
\nOur solution was blending a number of models (Big bird, Longformer, Deberta, Deberta-v2, Deberta-v3, Bart) all large models. We used a weighted average which was tuned with Optuna. Thresholds were also tuned with Optuna.
\nAs opposed to BIO tagging we used 9 model outputs - 7 classes & No span, as well one output to predict the initial token of each span. We found token dropout worked well to reduce overfitting.
\nTraining on very long sequences did not help too much. Using max 1250 tokens seemed to do well. For the shorter model it worked well to adjust the model to extend position token lengths, and just repeat the position embeddings 2X or 3X times.
", "rawMarkdown": "Thank you Georgia State University, The Learning Agency Lab, and Kaggle for a very well run competition. The data was high quality and interesting. It was great learning. \n\nThanks also to Huggingface 🤗 . Where would NLP be without you.\n\nUnfortunately, we missed out on the best of the postprocessing shown in some of the top approaches - WBF, GBM stackers, and Yolo-style span detector… really great work. Next time! \n\nOur [solution](https://www.kaggle.com/code/darraghdog/new-baseline/notebook?scriptVersionId=91589401) was blending a number of models (Big bird, Longformer, Deberta, Deberta-v2, Deberta-v3, Bart) all large models. We used a weighted average which was tuned with Optuna. Thresholds were also tuned with Optuna. \n\nAs opposed to BIO tagging we used 9 model outputs - 7 classes & No span, as well one output to predict the initial token of each span. We found token dropout worked well to reduce overfitting. \n\nTraining on very long sequences did not help too much. Using max 1250 tokens seemed to do well. For the shorter model it worked well to adjust the model to extend position token lengths, and just repeat the position embeddings 2X or 3X times. ", "votes": 24 }, { "id": 1742461, "postDate": "2022-04-01T22:39:06.800Z", "content": "Congratulations Darragh and team!
\n\n\nAs opposed to BIO tagging we used 9 model outputs - 7 classes & No span, as well one output to predict the initial token of each span. We found token dropout worked well to reduce overfitting.
\n
This is a nice trick. I will try this for future NER tasks. Basically all the \"B\" labels from BIO are grouped into a single \"B\" label. I think the model would like that and learn better because now it gets more \"B\" label data. Is this your team's idea, or is this a common idea in NER NLP?
", "rawMarkdown": "Congratulations Darragh and team!\n\n>As opposed to BIO tagging we used 9 model outputs - 7 classes & No span, as well one output to predict the initial token of each span. We found token dropout worked well to reduce overfitting.\n\nThis is a nice trick. I will try this for future NER tasks. Basically all the \"B\" labels from BIO are grouped into a single \"B\" label. I think the model would like that and learn better because now it gets more \"B\" label data. Is this your team's idea, or is this a common idea in NER NLP?", "votes": 2, "replies": [ { "id": 1742828, "postDate": "2022-04-02T10:17:39.137Z", "content": "When we initially merged both teams had been doing it this way, which made thing easier. The clovaai team, with BROS did a similar thing in their model, and that is where I had picked it up.
", "rawMarkdown": "When we initially merged both teams had been doing it this way, which made thing easier. The clovaai team, with [BROS](https://github.com/clovaai/bros) did a similar thing in their model, and that is where I had picked it up. ", "votes": 2 } ] } ], "complete": true, "source": "live", "fetched_at": "2026-10-19T12:37:26Z", "competition_id": 31779, "competition_slug": "feedback-prize-2021", "team_id": 7940979, "solution_url": "/competitions/feedback-prize-2021/writeups/new-baseline-7th-place-solution" }