{
  "id": 348160,
  "title": "Knowledge Injection Method Matters, 29th Writeup",
  "content": "<p>Like many top solutions, we tried the second stage GBDT model idea very early and it gave us a huge boost. However, the performance of our first-stage transformer model is much worse than the top teams.</p>\n<p>I used different token_type_id to inject discourse type and start/end info to the model, and only added special tokens to present the start/end of the whole essay. My input looks like the below:</p>\n<pre><code>token_ids:     [EBOS] other1 other2 lead1 lead2 other4 position1 position2 ... [EEOS]\ntokey_type_ids:  0    0      0       1       1       0       2       2         ...  0\n</code></pre>\n<p>Top teams mostly inject that info into the token_ids sequence, for example, wrapping the discourse text with type_start and type_end special tokens. After the competition, I tried this preprocessing method and it indeed bring significant performance gain.</p>\n<h2>What worked for us</h2>\n<ul>\n<li>MLM pretraining</li>\n<li>add topic info, then group by topic_id to aggregate scores as second-stage features</li>\n<li>mean+max pooling gave me some boost</li>\n<li>mean+max pooling of discourse element as element embedding, mean pooling of all the element embeddings of the same essay as essay embedding, concat [element embedding, essay embedding] before prediction head. This was inspired by the second stage aggregation.</li>\n</ul>\n<p>Special Thanks to Z by HP for sponsoring me a Z4 Workstation with dual A6000 GPUs.</p>",
  "comments": [
    {
      "id": 1919279,
      "author_name": "Apa",
      "author_url": "",
      "post_date": "2022-08-30T09:35:07.213000",
      "content": "<p>What does topic info refer to here?</p>",
      "votes": 0,
      "replies": [
        {
          "id": 1922429,
          "author_name": "Yuanhao",
          "author_url": "",
          "post_date": "2022-09-01T13:33:06.877000",
          "content": "<p>topic id.<br>\nWe trained a classification model to predict the 15 topics.</p>",
          "votes": 0,
          "replies": []
        }
      ]
    }
  ],
  "comment_count": 2,
  "messages": [
    {
      "id": 1915709,
      "postDate": "2022-08-27T08:27:13.503Z",
      "content": "<p>Like many top solutions, we tried the second stage GBDT model idea very early and it gave us a huge boost. However, the performance of our first-stage transformer model is much worse than the top teams.</p>\n<p>I used different token_type_id to inject discourse type and start/end info to the model, and only added special tokens to present the start/end of the whole essay. My input looks like the below:</p>\n<pre><code>token_ids:     [EBOS] other1 other2 lead1 lead2 other4 position1 position2 ... [EEOS]\ntokey_type_ids:  0    0      0       1       1       0       2       2         ...  0\n</code></pre>\n<p>Top teams mostly inject that info into the token_ids sequence, for example, wrapping the discourse text with type_start and type_end special tokens. After the competition, I tried this preprocessing method and it indeed bring significant performance gain.</p>\n<h2>What worked for us</h2>\n<ul>\n<li>MLM pretraining</li>\n<li>add topic info, then group by topic_id to aggregate scores as second-stage features</li>\n<li>mean+max pooling gave me some boost</li>\n<li>mean+max pooling of discourse element as element embedding, mean pooling of all the element embeddings of the same essay as essay embedding, concat [element embedding, essay embedding] before prediction head. This was inspired by the second stage aggregation.</li>\n</ul>\n<p>Special Thanks to Z by HP for sponsoring me a Z4 Workstation with dual A6000 GPUs.</p>",
      "rawMarkdown": "Like many top solutions, we tried the second stage GBDT model idea very early and it gave us a huge boost. However, the performance of our first-stage transformer model is much worse than the top teams.\n\nI used different token_type_id to inject discourse type and start/end info to the model, and only added special tokens to present the start/end of the whole essay. My input looks like the below:\n\n```\ntoken_ids:     [EBOS] other1 other2 lead1 lead2 other4 position1 position2 ... [EEOS]\ntokey_type_ids:  0    0      0       1       1       0       2       2         ...  0\n```\n\nTop teams mostly inject that info into the token_ids sequence, for example, wrapping the discourse text with type_start and type_end special tokens. After the competition, I tried this preprocessing method and it indeed bring significant performance gain.\n\n## What worked for us\n- MLM pretraining\n- add topic info, then group by topic_id to aggregate scores as second-stage features\n- mean+max pooling gave me some boost\n- mean+max pooling of discourse element as element embedding, mean pooling of all the element embeddings of the same essay as essay embedding, concat [element embedding, essay embedding] before prediction head. This was inspired by the second stage aggregation.\n\nSpecial Thanks to Z by HP for sponsoring me a Z4 Workstation with dual A6000 GPUs.",
      "votes": 8
    },
    {
      "id": 1919279,
      "postDate": "2022-08-30T09:35:07.213Z",
      "content": "<p>What does topic info refer to here?</p>",
      "rawMarkdown": "What does topic info refer to here?",
      "replies": [
        {
          "id": 1922429,
          "postDate": "2022-09-01T13:33:06.877Z",
          "content": "<p>topic id.<br>\nWe trained a classification model to predict the 15 topics.</p>",
          "rawMarkdown": "topic id.\nWe trained a classification model to predict the 15 topics."
        }
      ]
    }
  ],
  "complete": true,
  "source": "live",
  "fetched_at": "2026-10-19T17:54:09Z",
  "competition_id": 35308,
  "competition_slug": "feedback-prize-effectiveness",
  "team_id": 8819229,
  "solution_url": "/competitions/feedback-prize-effectiveness/writeups/lucky-stars-knowledge-injection-method-matters-29t"
}