{
  "id": 313184,
  "title": "11th place solution",
  "content": "<p>First of all, thanks to competition organizers for hosting this competition.<br>\nThanks also to the community for sharing many ideas in Notebook and Discussion.</p>\n<h2>Summary</h2>\n<p>We ensembled 6 models and did post process. Next, we extracted candidate predictionstrings and predicted whether each predictionstring was TP or not using XGBoost, etc. Finally, we set a threshold for each class and cut off the right edge of the predictionstrings.</p>\n<h2>Models</h2>\n<p>We referred to the following notebook.<br><br>\n<a href=\"https://www.kaggle.com/cdeotte/tensorflow-longformer-ner-cv-0-633\" target=\"_blank\">https://www.kaggle.com/cdeotte/tensorflow-longformer-ner-cv-0-633</a><br>\n<br><br>\nThe following 6models were trained with different max_length depending on the model.</p>\n<ul>\n<li>longformer-large-4096 (max_length:2048)</li>\n<li>roberta-large (max_length:512)</li>\n<li>bart-large (max_length:512)</li>\n<li>funnel-transformer-large (max_length:512)</li>\n<li>distilbart-cnn-12-6 (max_length:512)</li>\n<li>deberta-large (max_length:1024)</li>\n</ul>\n<h2>Ensemble</h2>\n<p>Since different models use different tokenizers, once the prediction results were converted to the character level, they were reaggregated into longformer token units.<br>\nEnsemble with different weights for each of the following positions.</p>\n<ol>\n<li>0 ~ 512</li>\n<li>0 ~ Funnel-transformer-large-token-len</li>\n<li>Funnel-transformer-large-token-len ~ 1023</li>\n<li>1023 ~ <br><br>\nFunnel-transformer-large-token-len is the length of the 512 token of funnel-large when it is converted back to longformer tokenizer units. It was about 720.</li>\n</ol>\n<h2>Post Process</h2>\n<ul>\n<li>Multiply each prediction by the coefficient</li>\n<li>Ensemble with predictions of neighboring token</li>\n<li>Extracted candidates for predictionstrings with reference to the following notebook. These candidates were not only for the class with the highest prediction, but also for the second highest class. The some predictionstrings were extracted where the start was not B-token.<br><br>\n<a href=\"https://www.kaggle.com/cdeotte/tensorflow-longformer-ner-cv-0-633\" target=\"_blank\">https://www.kaggle.com/cdeotte/tensorflow-longformer-ner-cv-0-633</a></li>\n<li>The following 3 models predicted whether predictionstrings were TP or not. The features were created by aggregating the ensemble results for each predictstrings. A threshold was set for each class, and only predictstrings with predictions above the threshold were left.<ol>\n<li>XGBoost</li>\n<li>MLP</li>\n<li>LSTM</li></ol></li>\n<li>Finally, predictstrings length threshold and percentage of clipping were set for each class, and The right end of predictstrings that exceeded the threshold was deleted by the set percentage.</li>\n</ul>\n<h2>Edit</h2>\n<p>Inference notebook<br>\n<a href=\"https://www.kaggle.com/code/columbia2131/exp-038-ensemble-xgb-mlp-lstm-fe-fix-tail\" target=\"_blank\">https://www.kaggle.com/code/columbia2131/exp-038-ensemble-xgb-mlp-lstm-fe-fix-tail</a><br>\ntraining code<br>\n<a href=\"https://github.com/TakoiHirokazu/Feedback-Prize-Evaluating-Student-Writing\" target=\"_blank\">https://github.com/TakoiHirokazu/Feedback-Prize-Evaluating-Student-Writing</a><br>\npost process code<br>\n<a href=\"https://colab.research.google.com/drive/1J2NTaRSKi-X5SBXxvQtVJxokZIW0oq4L?usp=sharing\" target=\"_blank\">https://colab.research.google.com/drive/1J2NTaRSKi-X5SBXxvQtVJxokZIW0oq4L?usp=sharing</a><br>\n<a href=\"https://www.kaggle.com/code/columbia2131/tuning-exp038-right-edge-removing/notebook\" target=\"_blank\">https://www.kaggle.com/code/columbia2131/tuning-exp038-right-edge-removing/notebook</a></p>",
  "comments": [
    {
      "id": 1724680,
      "author_name": "Ed Welch",
      "author_url": "",
      "post_date": "2022-03-16T12:49:30.173000",
      "content": "<p>Congratulations and thanks for sharing your insights.   I had considered using XGBoost in a similar way.  </p>",
      "votes": 1,
      "replies": []
    },
    {
      "id": 1724533,
      "author_name": "SchenbergZ",
      "author_url": "",
      "post_date": "2022-03-16T09:57:10.900000",
      "content": "<p>Congrats. 1 question: as you have a xgb model for classification whether predictionstrings are TP or not, why is there still a threshold? </p>",
      "votes": 1,
      "replies": [
        {
          "id": 1724660,
          "author_name": "Takoi",
          "author_url": "",
          "post_date": "2022-03-16T12:27:47.577000",
          "content": "<p>Thanks, I can't filter predictionstrings without a threshold because the output of Xgboost is not 0 or 1, but numeric in the range 0 ~ 1.</p>",
          "votes": 3,
          "replies": []
        }
      ]
    },
    {
      "id": 1724121,
      "author_name": "Nicholas Broad",
      "author_url": "",
      "post_date": "2022-03-16T03:11:09.643000",
      "content": "<p>What were your CV for each model?</p>",
      "votes": 1,
      "replies": [
        {
          "id": 1724652,
          "author_name": "Takoi",
          "author_url": "",
          "post_date": "2022-03-16T12:20:48.093000",
          "content": "<p>Thank you for your comment. CV was measured before PP using the same method as below<br>\n<a href=\"https://www.kaggle.com/cdeotte/tensorflow-longformer-ner-cv-0-633\" target=\"_blank\">https://www.kaggle.com/cdeotte/tensorflow-longformer-ner-cv-0-633</a></p>\n<ul>\n<li>longformer-large-4096 (max_length:2048) : 0.6696</li>\n<li>roberta-large (max_length:512) : 0.6098</li>\n<li>bart-large (max_length:512) : 0.6073</li>\n<li>funnel-transformer-large (max_length:512) : 0.5984</li>\n<li>distilbart-cnn-12-6 (max_length:512) : 0.5957</li>\n<li>deberta-large (max_length:1024) : 0.6672</li>\n</ul>",
          "votes": 0,
          "replies": []
        },
        {
          "id": 1724672,
          "author_name": "Nicholas Broad",
          "author_url": "",
          "post_date": "2022-03-16T12:40:10.260000",
          "content": "<p>Thanks for this!</p>\n<p>Did you try 1024 max length for bart?</p>",
          "votes": 1,
          "replies": []
        },
        {
          "id": 1725407,
          "author_name": "Takoi",
          "author_url": "",
          "post_date": "2022-03-17T04:29:56.360000",
          "content": "<p>Yes, we tried, but the Public Score did not increase, so we did not adopt it in the end.</p>",
          "votes": 0,
          "replies": []
        }
      ]
    },
    {
      "id": 1724888,
      "author_name": "Ishan Mehta115",
      "author_url": "",
      "post_date": "2022-03-16T15:47:46.340000",
      "content": "<p>Thanks for sharing this</p>",
      "votes": 1,
      "replies": []
    },
    {
      "id": 1724080,
      "author_name": "van73",
      "author_url": "",
      "post_date": "2022-03-16T02:27:46.807000",
      "content": "<p>Thanks for sharing!</p>",
      "votes": 1,
      "replies": []
    }
  ],
  "comment_count": 9,
  "messages": [
    {
      "id": 1724072,
      "postDate": "2022-03-16T02:23:06.003Z",
      "content": "<p>First of all, thanks to competition organizers for hosting this competition.<br>\nThanks also to the community for sharing many ideas in Notebook and Discussion.</p>\n<h2>Summary</h2>\n<p>We ensembled 6 models and did post process. Next, we extracted candidate predictionstrings and predicted whether each predictionstring was TP or not using XGBoost, etc. Finally, we set a threshold for each class and cut off the right edge of the predictionstrings.</p>\n<h2>Models</h2>\n<p>We referred to the following notebook.<br><br>\n<a href=\"https://www.kaggle.com/cdeotte/tensorflow-longformer-ner-cv-0-633\" target=\"_blank\">https://www.kaggle.com/cdeotte/tensorflow-longformer-ner-cv-0-633</a><br>\n<br><br>\nThe following 6models were trained with different max_length depending on the model.</p>\n<ul>\n<li>longformer-large-4096 (max_length:2048)</li>\n<li>roberta-large (max_length:512)</li>\n<li>bart-large (max_length:512)</li>\n<li>funnel-transformer-large (max_length:512)</li>\n<li>distilbart-cnn-12-6 (max_length:512)</li>\n<li>deberta-large (max_length:1024)</li>\n</ul>\n<h2>Ensemble</h2>\n<p>Since different models use different tokenizers, once the prediction results were converted to the character level, they were reaggregated into longformer token units.<br>\nEnsemble with different weights for each of the following positions.</p>\n<ol>\n<li>0 ~ 512</li>\n<li>0 ~ Funnel-transformer-large-token-len</li>\n<li>Funnel-transformer-large-token-len ~ 1023</li>\n<li>1023 ~ <br><br>\nFunnel-transformer-large-token-len is the length of the 512 token of funnel-large when it is converted back to longformer tokenizer units. It was about 720.</li>\n</ol>\n<h2>Post Process</h2>\n<ul>\n<li>Multiply each prediction by the coefficient</li>\n<li>Ensemble with predictions of neighboring token</li>\n<li>Extracted candidates for predictionstrings with reference to the following notebook. These candidates were not only for the class with the highest prediction, but also for the second highest class. The some predictionstrings were extracted where the start was not B-token.<br><br>\n<a href=\"https://www.kaggle.com/cdeotte/tensorflow-longformer-ner-cv-0-633\" target=\"_blank\">https://www.kaggle.com/cdeotte/tensorflow-longformer-ner-cv-0-633</a></li>\n<li>The following 3 models predicted whether predictionstrings were TP or not. The features were created by aggregating the ensemble results for each predictstrings. A threshold was set for each class, and only predictstrings with predictions above the threshold were left.<ol>\n<li>XGBoost</li>\n<li>MLP</li>\n<li>LSTM</li></ol></li>\n<li>Finally, predictstrings length threshold and percentage of clipping were set for each class, and The right end of predictstrings that exceeded the threshold was deleted by the set percentage.</li>\n</ul>\n<h2>Edit</h2>\n<p>Inference notebook<br>\n<a href=\"https://www.kaggle.com/code/columbia2131/exp-038-ensemble-xgb-mlp-lstm-fe-fix-tail\" target=\"_blank\">https://www.kaggle.com/code/columbia2131/exp-038-ensemble-xgb-mlp-lstm-fe-fix-tail</a><br>\ntraining code<br>\n<a href=\"https://github.com/TakoiHirokazu/Feedback-Prize-Evaluating-Student-Writing\" target=\"_blank\">https://github.com/TakoiHirokazu/Feedback-Prize-Evaluating-Student-Writing</a><br>\npost process code<br>\n<a href=\"https://colab.research.google.com/drive/1J2NTaRSKi-X5SBXxvQtVJxokZIW0oq4L?usp=sharing\" target=\"_blank\">https://colab.research.google.com/drive/1J2NTaRSKi-X5SBXxvQtVJxokZIW0oq4L?usp=sharing</a><br>\n<a href=\"https://www.kaggle.com/code/columbia2131/tuning-exp038-right-edge-removing/notebook\" target=\"_blank\">https://www.kaggle.com/code/columbia2131/tuning-exp038-right-edge-removing/notebook</a></p>",
      "rawMarkdown": "First of all, thanks to competition organizers for hosting this competition.\nThanks also to the community for sharing many ideas in Notebook and Discussion.\n\n## Summary\nWe ensembled 6 models and did post process. Next, we extracted candidate predictionstrings and predicted whether each predictionstring was TP or not using XGBoost, etc. Finally, we set a threshold for each class and cut off the right edge of the predictionstrings.\n\n## Models\nWe referred to the following notebook.</br>\nhttps://www.kaggle.com/cdeotte/tensorflow-longformer-ner-cv-0-633\n</br>\nThe following 6models were trained with different max_length depending on the model.\n- longformer-large-4096 (max_length:2048)\n- roberta-large (max_length:512)\n- bart-large (max_length:512)\n- funnel-transformer-large (max_length:512)\n- distilbart-cnn-12-6 (max_length:512)\n- deberta-large (max_length:1024)\n\n## Ensemble\nSince different models use different tokenizers, once the prediction results were converted to the character level, they were reaggregated into longformer token units.\nEnsemble with different weights for each of the following positions.\n1. 0 ~ 512\n2. 0 ~ Funnel-transformer-large-token-len\n3. Funnel-transformer-large-token-len ~ 1023\n4. 1023 ~ </br>\nFunnel-transformer-large-token-len is the length of the 512 token of funnel-large when it is converted back to longformer tokenizer units. It was about 720.\n\n## Post Process\n- Multiply each prediction by the coefficient\n- Ensemble with predictions of neighboring token\n- Extracted candidates for predictionstrings with reference to the following notebook. These candidates were not only for the class with the highest prediction, but also for the second highest class. The some predictionstrings were extracted where the start was not B-token.</br>\nhttps://www.kaggle.com/cdeotte/tensorflow-longformer-ner-cv-0-633\n- The following 3 models predicted whether predictionstrings were TP or not. The features were created by aggregating the ensemble results for each predictstrings. A threshold was set for each class, and only predictstrings with predictions above the threshold were left.\n    1. XGBoost\n    2. MLP\n    3. LSTM\n- Finally, predictstrings length threshold and percentage of clipping were set for each class, and The right end of predictstrings that exceeded the threshold was deleted by the set percentage.\n\n## Edit\nInference notebook\nhttps://www.kaggle.com/code/columbia2131/exp-038-ensemble-xgb-mlp-lstm-fe-fix-tail\ntraining code\nhttps://github.com/TakoiHirokazu/Feedback-Prize-Evaluating-Student-Writing\npost process code\nhttps://colab.research.google.com/drive/1J2NTaRSKi-X5SBXxvQtVJxokZIW0oq4L?usp=sharing\nhttps://www.kaggle.com/code/columbia2131/tuning-exp038-right-edge-removing/notebook\n\n",
      "votes": 44
    },
    {
      "id": 1724680,
      "postDate": "2022-03-16T12:49:30.173Z",
      "content": "<p>Congratulations and thanks for sharing your insights.   I had considered using XGBoost in a similar way.  </p>",
      "rawMarkdown": "Congratulations and thanks for sharing your insights.   I had considered using XGBoost in a similar way.  ",
      "votes": 1
    },
    {
      "id": 1724533,
      "postDate": "2022-03-16T09:57:10.900Z",
      "content": "<p>Congrats. 1 question: as you have a xgb model for classification whether predictionstrings are TP or not, why is there still a threshold? </p>",
      "rawMarkdown": "Congrats. 1 question: as you have a xgb model for classification whether predictionstrings are TP or not, why is there still a threshold? ",
      "votes": 1,
      "replies": [
        {
          "id": 1724660,
          "postDate": "2022-03-16T12:27:47.577Z",
          "content": "<p>Thanks, I can't filter predictionstrings without a threshold because the output of Xgboost is not 0 or 1, but numeric in the range 0 ~ 1.</p>",
          "rawMarkdown": "Thanks, I can't filter predictionstrings without a threshold because the output of Xgboost is not 0 or 1, but numeric in the range 0 ~ 1.",
          "votes": 3
        }
      ]
    },
    {
      "id": 1724121,
      "postDate": "2022-03-16T03:11:09.643Z",
      "content": "<p>What were your CV for each model?</p>",
      "rawMarkdown": "What were your CV for each model?",
      "votes": 1,
      "replies": [
        {
          "id": 1724652,
          "postDate": "2022-03-16T12:20:48.093Z",
          "content": "<p>Thank you for your comment. CV was measured before PP using the same method as below<br>\n<a href=\"https://www.kaggle.com/cdeotte/tensorflow-longformer-ner-cv-0-633\" target=\"_blank\">https://www.kaggle.com/cdeotte/tensorflow-longformer-ner-cv-0-633</a></p>\n<ul>\n<li>longformer-large-4096 (max_length:2048) : 0.6696</li>\n<li>roberta-large (max_length:512) : 0.6098</li>\n<li>bart-large (max_length:512) : 0.6073</li>\n<li>funnel-transformer-large (max_length:512) : 0.5984</li>\n<li>distilbart-cnn-12-6 (max_length:512) : 0.5957</li>\n<li>deberta-large (max_length:1024) : 0.6672</li>\n</ul>",
          "rawMarkdown": "Thank you for your comment. CV was measured before PP using the same method as below\nhttps://www.kaggle.com/cdeotte/tensorflow-longformer-ner-cv-0-633\n- longformer-large-4096 (max_length:2048) : 0.6696\n- roberta-large (max_length:512) : 0.6098\n- bart-large (max_length:512) : 0.6073\n- funnel-transformer-large (max_length:512) : 0.5984\n- distilbart-cnn-12-6 (max_length:512) : 0.5957\n- deberta-large (max_length:1024) : 0.6672\n"
        },
        {
          "id": 1724672,
          "postDate": "2022-03-16T12:40:10.260Z",
          "content": "<p>Thanks for this!</p>\n<p>Did you try 1024 max length for bart?</p>",
          "rawMarkdown": "Thanks for this!\n\nDid you try 1024 max length for bart?",
          "votes": 1
        },
        {
          "id": 1725407,
          "postDate": "2022-03-17T04:29:56.360Z",
          "content": "<p>Yes, we tried, but the Public Score did not increase, so we did not adopt it in the end.</p>",
          "rawMarkdown": "Yes, we tried, but the Public Score did not increase, so we did not adopt it in the end."
        }
      ]
    },
    {
      "id": 1724888,
      "postDate": "2022-03-16T15:47:46.340Z",
      "content": "<p>Thanks for sharing this</p>",
      "rawMarkdown": "Thanks for sharing this",
      "votes": 1
    },
    {
      "id": 1724080,
      "postDate": "2022-03-16T02:27:46.807Z",
      "content": "<p>Thanks for sharing!</p>",
      "rawMarkdown": "Thanks for sharing!",
      "votes": 1
    }
  ],
  "complete": true,
  "source": "live",
  "fetched_at": "2026-10-19T12:36:03Z",
  "competition_id": 31779,
  "competition_slug": "feedback-prize-2021",
  "team_id": 7888852,
  "solution_url": "/competitions/feedback-prize-2021/writeups/11167-11th-place-solution"
}