{
  "id": 578431,
  "title": "1st Place Solution ",
  "content": "<p>Hi everyone,</p>\n<p>First, I’d like to thank the host for organizing such a great LLM competition.</p>\n<h1>Summary</h1>\n<p>For this competition, I used the <code>Qwen2.5-14B</code> and <code>Qwen3-14B</code> models as my base models and fine-tuned them specifically for text classification. To make the solution more robust and stable, for each model, I ensembled models that were trained across K-fold splits. On top of that, I did a weighted ensemble for Qwen2.5 and Qwen3.</p>\n<h1>Inputs</h1>\n<p>I used only the Math Question field and formatted it into the following prompt template:</p>\n<blockquote>\n  <p>prompt = f\"\"\"&lt;|im_start|&gt;user<br>\n  Your task is to classify each Math problem into one of these eight topics using a machine learning or NLP-based approach.<br>\n  0: Algebra<br>\n  1: Geometry and Trigonometry<br>\n  2: Calculus and Analysis<br>\n  3: Probability and Statistics<br>\n  4: Number Theory<br>\n  5: Combinatorics and Discrete Math<br>\n  6: Linear Algebra<br>\n  7: Abstract Algebra and Topology<br>\n  Your answer should be an integer that assigns the most appropriate topic category to the given Math problem based on its content and required reasoning.<br>\n  Math Problem: {MATH_QUESTION_HERE}<br>\n  Answer:\"\"\"</p>\n</blockquote>\n<h1>Model</h1>\n<p>Based on my recent experience in Kaggle competitions, 14B models strike a good balance for achieving competitive results, validating ideas, and aiming for a win. That’s why, when I joined this competition six days ago, I decided to stick with <code>Qwen2.5-14B</code> and the recently released <code>Qwen3-14B</code> as the base model and fine-tune them for text classification.</p>\n<p>I replaced the base Qwen3-14B model's <code>lm_head</code> with a new classification head that has the same size with embedding dim.</p>\n<p>Given the causal nature of decoder models, I used the last hidden state of the final token, <code>Answer:</code>, and fed it into a linear head for classification.</p>\n<h1>Cross-validation</h1>\n<p>Due to the class imbalance in the training set, I used StratifiedKFold (3 folds) to ensure that each fold preserved the overall class distribution. This helped maintain representative splits and allowed the model to be evaluated more reliably across all classes.</p>\n<h1>Training</h1>\n<ul>\n<li><p>To utlize the 4×L4 GPUs on Kaggle, I used Distributed Data Parallel (DDP) for faster training and rapid experimentation.</p></li>\n<li><p>For parameter-efficient fine-tuning, I applied vanilla LoRA. </p></li>\n<li><p>A key factor in this competition was tuning the learning rate. I used a Qwen2.5-0.5B model to efficiently search for optimal rates, with each fold taking only 6 minutes for training and validation on 4 GPUs. Interestingly, the settings from the smaller model transferred well to the larger 14B model.</p></li>\n<li><p>Differential learning rates didn’t work well in this competition, so I ended up using a quite large learning rate of 2e-4 for both the backbone and classification head, with a cosine decay schedule starting from 2e-4 and decaying to zero, without warm-up.</p></li>\n<li><p>To help the model focus on rare classes, I computed class weights and incorporated them into the cross-entropy loss. This, along with a good learning rate, was crucial for model performance.</p></li>\n</ul>\n<p>The training was performed using 3-fold cross-validation, with each fold trained for 3 epochs. Each epoch took 30 minutes for both training and validation.</p>\n<h1>Model Performance and Ensemble Results</h1>\n<table>\n<thead>\n<tr>\n<th>Model</th>\n<th>CV Score</th>\n<th>Public LB</th>\n<th>Private LB</th>\n</tr>\n</thead>\n<tbody>\n<tr>\n<td><strong>Qwen3-14B</strong></td>\n<td>0.909</td>\n<td>-</td>\n<td>-</td>\n</tr>\n<tr>\n<td><strong>Qwen2.5-14B</strong></td>\n<td>0.911</td>\n<td>-</td>\n<td>-</td>\n</tr>\n<tr>\n<td><strong>Ensemble</strong></td>\n<td>-</td>\n<td>0.9168</td>\n<td>0.9253</td>\n</tr>\n</tbody>\n</table>\n<h1>Inference</h1>\n<p>Multi-GPU inference was used to reduce inference time.</p>\n<p>During training, I noticed that 99% of the math questions were under 300 tokens, so I dynamically truncated inputs longer than 300 tokens. However, at inference time, I used the full input without truncation to fully leverage the math question content and maximize test-time compute.</p>\n<h1>Ensemble</h1>\n<p>For each model, I first averaged the K-fold models. Then, I performed a weighted average, giving <strong>0.6</strong> weight to <code>Qwen3-14B</code> and <strong>0.4</strong> weight to <code>Qwen2.5-14B</code> for the final predictions.</p>\n<h1>Code</h1>\n<p>Training Code for:</p>\n<ul>\n<li><strong>Qwen3-14B</strong>: <a href=\"https://www.kaggle.com/code/ducnh279/kacs1-fine-tuning-qwen3-14b\" target=\"_blank\">Qwen3-14B Fine-tuning</a></li>\n<li><strong>Qwen2.5-14B</strong>: <a href=\"https://www.kaggle.com/code/ducnh279/kacs1-fine-tuning-qwen2-5-14b\" target=\"_blank\">Qwen2.5-14B Fine-tuning</a></li>\n</ul>\n<p>Offline Inference: <a href=\"https://www.kaggle.com/code/ducnh279/kacs1-offline-inference?scriptVersionId=238990333\" target=\"_blank\">Offline Inference Code</a></p>",
  "comments": [
    {
      "id": 3199617,
      "author_name": "Taewoong Kim",
      "author_url": "",
      "post_date": "2025-05-11T08:33:15.327000",
      "content": "<p>Oh good ensemble! I used only qwen 2.5 32b. Cong your 1st!</p>",
      "votes": 4,
      "replies": [
        {
          "id": 3199624,
          "author_name": "ducnh279",
          "author_url": "",
          "post_date": "2025-05-11T08:45:57.163000",
          "content": "<p>I also trained Qwen 2.5 32B, but it performed similarly to the 14B model, so I assumed the 14B variant is enough! </p>",
          "votes": 1,
          "replies": []
        }
      ]
    },
    {
      "id": 3201375,
      "author_name": "Ali Habibullah",
      "author_url": "",
      "post_date": "2025-05-13T18:49:42.087000",
      "content": "<p>Nice solution! and congrats on getting first place.</p>",
      "votes": 1,
      "replies": [
        {
          "id": 3201648,
          "author_name": "ducnh279",
          "author_url": "",
          "post_date": "2025-05-14T07:44:49.457000",
          "content": "<p>Thank you, AIi!!!</p>",
          "votes": 0,
          "replies": []
        }
      ]
    },
    {
      "id": 3199989,
      "author_name": "Shapu",
      "author_url": "",
      "post_date": "2025-05-11T19:42:41.220000",
      "content": "<p>Great solution, thank you for sharing! I just have one question: x = self.backbone(**x).last_hidden_state[:, -1, :] seems to be taking the embedding of the final token, which in my case is 'Ġ', not 'Answer:' as I expected. I tried your code , am I missing something, or is there a mistake here?</p>",
      "votes": 1,
      "replies": [
        {
          "id": 3200008,
          "author_name": "ducnh279",
          "author_url": "",
          "post_date": "2025-05-11T20:31:15.957000",
          "content": "<p>I am not 100% sure I understand your question! I will answer in the way I think you are asking. Feel free to raise more questions or make clarifications if needed.</p>\n<blockquote>\n  <p>x = self.backbone(**x).last_hidden_state[:, -1, :] seems to be taking the embedding of the final token</p>\n</blockquote>\n<p>Yes, it's taking the embedding of the final token!</p>\n<p>'Answer:' is just a <strong>simplified</strong> example of the last token to help people get the idea. It really depends on your tokenizer and how it splits your input.</p>\n<p>Example:</p>\n<blockquote>\n  <p>text = \"What is 2 + 2? Answer:\"</p>\n</blockquote>\n<p>Tokenizer example (this depends on the tokenizer you're using):</p>\n<blockquote>\n  <p>input_ids = [1, 2, 3, 3, 4, 5, 6]<br>\n  tokens = ['What', 'is', '2', '+', '2', 'Answer', ':']</p>\n</blockquote>\n<p>Now, assuming 'last_hidden_state' is the output from the model: 'last_hidden_state[:, -1, :]' will give you the embedding for the last token, which in this case will be <code>':'</code>, <strong>not</strong> <code>'Answer:'</code>.</p>\n<p>The embedding for <code>':'</code> will then be fed into the linear head to produce the logits of the next token, and softmax will be applied to get predicted probabilities of next tokens, such as class labels: [0, 1, 2, 3, 4, 5, 6, 7].</p>\n<p>Be careful about assuming the last token is meaningful, it could just be punctuation or a subword, not the expected word (like 'Answer:').</p>\n<p>Note on padding &amp; truncation:<br>\nI set <code>tokenizer.padding_side='left'</code> to ensure that the input sequence always ends with 'Answer:'. However, if the input sequence exceeds the maximum token limit (e.g., 300 tokens), truncation may occur, which could cause 'Answer:' to be cut off or replaced by a random token at the end of the sequence, depending on the tokenizer’s behavior. </p>",
          "votes": 2,
          "replies": [
            {
              "id": 3200015,
              "author_name": "Shapu",
              "author_url": "",
              "post_date": "2025-05-11T21:08:51.557000",
              "content": "<p>Thank you for your clear explantation !  </p>",
              "votes": 1,
              "replies": []
            }
          ]
        }
      ]
    },
    {
      "id": 3199877,
      "author_name": "JF.XIE",
      "author_url": "",
      "post_date": "2025-05-11T15:51:30.363000",
      "content": "<p>I used the BERT model in this competition, but its performance is generally not as good as LLM. I'm studying much knowledges of LLM currently and your work is very useful me. Thanks for sharing!</p>",
      "votes": 1,
      "replies": [
        {
          "id": 3199889,
          "author_name": "ducnh279",
          "author_url": "",
          "post_date": "2025-05-11T16:21:46.503000",
          "content": "<p>Great to hear that! </p>",
          "votes": 0,
          "replies": []
        }
      ]
    },
    {
      "id": 3199708,
      "author_name": "Mohamed Eltayeb",
      "author_url": "",
      "post_date": "2025-05-11T11:18:51.673000",
      "content": "<p>Thanks for sharing your ideas! I honestly thought bert-like models might get to the top, but looks like i was mistaken haha</p>",
      "votes": 1,
      "replies": [
        {
          "id": 3199717,
          "author_name": "ducnh279",
          "author_url": "",
          "post_date": "2025-05-11T11:35:12.620000",
          "content": "<p>For math-related data, using decoder-based LLMs is currently the most effective approach. These models are typically trained on large-scale mathematical corpora during both pre-training and post-training stage, making them highly capable for downstream tasks.</p>\n<p>However, I believe there are still specific tasks where encoder-based models like BERT remain both effective and efficient, and can even outperform LLMs.</p>",
          "votes": 2,
          "replies": []
        }
      ]
    },
    {
      "id": 3199703,
      "author_name": "Faisal Alherran",
      "author_url": "",
      "post_date": "2025-05-11T11:06:02.063000",
      "content": "<p>Hi, I have a quick concern. Is it allowed to use an LLM as a classifier in this competition? We actually thought of this idea earlier but didn’t go ahead with it since the task was defined as a classification problem, not an LLM task. Just wanted to clarify if that approach is considered valid. Thanks!</p>",
      "votes": 1,
      "replies": [
        {
          "id": 3232785,
          "author_name": "Soory- KAUST",
          "author_url": "",
          "post_date": "2025-06-26T07:48:51.017000",
          "content": "<p>Nah, I think yes</p>",
          "votes": 0,
          "replies": []
        }
      ]
    },
    {
      "id": 3199700,
      "author_name": "joseph",
      "author_url": "",
      "post_date": "2025-05-11T10:56:40.563000",
      "content": "<p>Congrats, really appreciate the solution you shared!</p>",
      "votes": 1,
      "replies": []
    },
    {
      "id": 3199607,
      "author_name": "Mfahimhossen",
      "author_url": "",
      "post_date": "2025-05-11T08:22:48.353000",
      "content": "<p>Congratulations.  Thank you for sharing  idea. </p>",
      "votes": 1,
      "replies": []
    },
    {
      "id": 3199569,
      "author_name": "A.Abbas",
      "author_url": "",
      "post_date": "2025-05-11T06:39:11.080000",
      "content": "<p>Congrats on 1st place, thank you for sharing your solution</p>",
      "votes": 1,
      "replies": []
    },
    {
      "id": 3199478,
      "author_name": "Mansour",
      "author_url": "",
      "post_date": "2025-05-11T03:46:26.993000",
      "content": "<p>Congrats on your 1st place! Looking forward to learning from your solution.</p>",
      "votes": 1,
      "replies": []
    },
    {
      "id": 3199438,
      "author_name": "robin baldeo",
      "author_url": "",
      "post_date": "2025-05-11T02:07:15.307000",
      "content": "<p>Thanks for sharing, it was fun looking at your code. </p>",
      "votes": 1,
      "replies": []
    },
    {
      "id": 3199411,
      "author_name": "Denver Magtibay",
      "author_url": "",
      "post_date": "2025-05-10T23:28:05.257000",
      "content": "<p>Congrats! Thank you for sharing your idea! Will follow you!</p>",
      "votes": 1,
      "replies": []
    },
    {
      "id": 3284065,
      "author_name": "Diy Quang ",
      "author_url": "",
      "post_date": "2025-09-08T05:22:37.463000",
      "content": "<p>It seems the notebook is no longer available. Hope you can provide it again.</p>",
      "votes": 0,
      "replies": [
        {
          "id": 3286445,
          "author_name": "ducnh279",
          "author_url": "",
          "post_date": "2025-09-09T20:23:18.257000",
          "content": "<p>I opened again! Check it out!</p>",
          "votes": 0,
          "replies": []
        }
      ]
    },
    {
      "id": 3201832,
      "author_name": "Mehdi Abda",
      "author_url": "",
      "post_date": "2025-05-14T13:04:57.790000",
      "content": "<p>Congrats, and thanks for the solution!</p>",
      "votes": 1,
      "replies": []
    }
  ],
  "comment_count": 22,
  "messages": [
    {
      "id": 3199382,
      "postDate": "2025-05-10T21:39:13.343Z",
      "content": "<p>Hi everyone,</p>\n<p>First, I’d like to thank the host for organizing such a great LLM competition.</p>\n<h1>Summary</h1>\n<p>For this competition, I used the <code>Qwen2.5-14B</code> and <code>Qwen3-14B</code> models as my base models and fine-tuned them specifically for text classification. To make the solution more robust and stable, for each model, I ensembled models that were trained across K-fold splits. On top of that, I did a weighted ensemble for Qwen2.5 and Qwen3.</p>\n<h1>Inputs</h1>\n<p>I used only the Math Question field and formatted it into the following prompt template:</p>\n<blockquote>\n  <p>prompt = f\"\"\"&lt;|im_start|&gt;user<br>\n  Your task is to classify each Math problem into one of these eight topics using a machine learning or NLP-based approach.<br>\n  0: Algebra<br>\n  1: Geometry and Trigonometry<br>\n  2: Calculus and Analysis<br>\n  3: Probability and Statistics<br>\n  4: Number Theory<br>\n  5: Combinatorics and Discrete Math<br>\n  6: Linear Algebra<br>\n  7: Abstract Algebra and Topology<br>\n  Your answer should be an integer that assigns the most appropriate topic category to the given Math problem based on its content and required reasoning.<br>\n  Math Problem: {MATH_QUESTION_HERE}<br>\n  Answer:\"\"\"</p>\n</blockquote>\n<h1>Model</h1>\n<p>Based on my recent experience in Kaggle competitions, 14B models strike a good balance for achieving competitive results, validating ideas, and aiming for a win. That’s why, when I joined this competition six days ago, I decided to stick with <code>Qwen2.5-14B</code> and the recently released <code>Qwen3-14B</code> as the base model and fine-tune them for text classification.</p>\n<p>I replaced the base Qwen3-14B model's <code>lm_head</code> with a new classification head that has the same size with embedding dim.</p>\n<p>Given the causal nature of decoder models, I used the last hidden state of the final token, <code>Answer:</code>, and fed it into a linear head for classification.</p>\n<h1>Cross-validation</h1>\n<p>Due to the class imbalance in the training set, I used StratifiedKFold (3 folds) to ensure that each fold preserved the overall class distribution. This helped maintain representative splits and allowed the model to be evaluated more reliably across all classes.</p>\n<h1>Training</h1>\n<ul>\n<li><p>To utlize the 4×L4 GPUs on Kaggle, I used Distributed Data Parallel (DDP) for faster training and rapid experimentation.</p></li>\n<li><p>For parameter-efficient fine-tuning, I applied vanilla LoRA. </p></li>\n<li><p>A key factor in this competition was tuning the learning rate. I used a Qwen2.5-0.5B model to efficiently search for optimal rates, with each fold taking only 6 minutes for training and validation on 4 GPUs. Interestingly, the settings from the smaller model transferred well to the larger 14B model.</p></li>\n<li><p>Differential learning rates didn’t work well in this competition, so I ended up using a quite large learning rate of 2e-4 for both the backbone and classification head, with a cosine decay schedule starting from 2e-4 and decaying to zero, without warm-up.</p></li>\n<li><p>To help the model focus on rare classes, I computed class weights and incorporated them into the cross-entropy loss. This, along with a good learning rate, was crucial for model performance.</p></li>\n</ul>\n<p>The training was performed using 3-fold cross-validation, with each fold trained for 3 epochs. Each epoch took 30 minutes for both training and validation.</p>\n<h1>Model Performance and Ensemble Results</h1>\n<table>\n<thead>\n<tr>\n<th>Model</th>\n<th>CV Score</th>\n<th>Public LB</th>\n<th>Private LB</th>\n</tr>\n</thead>\n<tbody>\n<tr>\n<td><strong>Qwen3-14B</strong></td>\n<td>0.909</td>\n<td>-</td>\n<td>-</td>\n</tr>\n<tr>\n<td><strong>Qwen2.5-14B</strong></td>\n<td>0.911</td>\n<td>-</td>\n<td>-</td>\n</tr>\n<tr>\n<td><strong>Ensemble</strong></td>\n<td>-</td>\n<td>0.9168</td>\n<td>0.9253</td>\n</tr>\n</tbody>\n</table>\n<h1>Inference</h1>\n<p>Multi-GPU inference was used to reduce inference time.</p>\n<p>During training, I noticed that 99% of the math questions were under 300 tokens, so I dynamically truncated inputs longer than 300 tokens. However, at inference time, I used the full input without truncation to fully leverage the math question content and maximize test-time compute.</p>\n<h1>Ensemble</h1>\n<p>For each model, I first averaged the K-fold models. Then, I performed a weighted average, giving <strong>0.6</strong> weight to <code>Qwen3-14B</code> and <strong>0.4</strong> weight to <code>Qwen2.5-14B</code> for the final predictions.</p>\n<h1>Code</h1>\n<p>Training Code for:</p>\n<ul>\n<li><strong>Qwen3-14B</strong>: <a href=\"https://www.kaggle.com/code/ducnh279/kacs1-fine-tuning-qwen3-14b\" target=\"_blank\">Qwen3-14B Fine-tuning</a></li>\n<li><strong>Qwen2.5-14B</strong>: <a href=\"https://www.kaggle.com/code/ducnh279/kacs1-fine-tuning-qwen2-5-14b\" target=\"_blank\">Qwen2.5-14B Fine-tuning</a></li>\n</ul>\n<p>Offline Inference: <a href=\"https://www.kaggle.com/code/ducnh279/kacs1-offline-inference?scriptVersionId=238990333\" target=\"_blank\">Offline Inference Code</a></p>",
      "rawMarkdown": "Hi everyone,\n\nFirst, I’d like to thank the host for organizing such a great LLM competition.\n\n# Summary\nFor this competition, I used the ```Qwen2.5-14B``` and ```Qwen3-14B``` models as my base models and fine-tuned them specifically for text classification. To make the solution more robust and stable, for each model, I ensembled models that were trained across K-fold splits. On top of that, I did a weighted ensemble for Qwen2.5 and Qwen3.\n\n# Inputs\nI used only the Math Question field and formatted it into the following prompt template:\n\n>prompt = f\"\"\"<|im_start|>user\nYour task is to classify each Math problem into one of these eight topics using a machine learning or NLP-based approach.\n0: Algebra\n1: Geometry and Trigonometry\n2: Calculus and Analysis\n3: Probability and Statistics\n4: Number Theory\n5: Combinatorics and Discrete Math\n6: Linear Algebra\n7: Abstract Algebra and Topology\nYour answer should be an integer that assigns the most appropriate topic category to the given Math problem based on its content and required reasoning.\nMath Problem: {MATH_QUESTION_HERE}\nAnswer:\"\"\"\n\n# Model\nBased on my recent experience in Kaggle competitions, 14B models strike a good balance for achieving competitive results, validating ideas, and aiming for a win. That’s why, when I joined this competition six days ago, I decided to stick with ```Qwen2.5-14B``` and the recently released ```Qwen3-14B``` as the base model and fine-tune them for text classification.\n\nI replaced the base Qwen3-14B model's `lm_head` with a new classification head that has the same size with embedding dim.\n\nGiven the causal nature of decoder models, I used the last hidden state of the final token, ```Answer:```, and fed it into a linear head for classification.\n\n# Cross-validation\nDue to the class imbalance in the training set, I used StratifiedKFold (3 folds) to ensure that each fold preserved the overall class distribution. This helped maintain representative splits and allowed the model to be evaluated more reliably across all classes.\n\n# Training\n- To utlize the 4×L4 GPUs on Kaggle, I used Distributed Data Parallel (DDP) for faster training and rapid experimentation.\n\n- For parameter-efficient fine-tuning, I applied vanilla LoRA. \n\n- A key factor in this competition was tuning the learning rate. I used a Qwen2.5-0.5B model to efficiently search for optimal rates, with each fold taking only 6 minutes for training and validation on 4 GPUs. Interestingly, the settings from the smaller model transferred well to the larger 14B model.\n\n- Differential learning rates didn’t work well in this competition, so I ended up using a quite large learning rate of 2e-4 for both the backbone and classification head, with a cosine decay schedule starting from 2e-4 and decaying to zero, without warm-up.\n\n- To help the model focus on rare classes, I computed class weights and incorporated them into the cross-entropy loss. This, along with a good learning rate, was crucial for model performance.\n\nThe training was performed using 3-fold cross-validation, with each fold trained for 3 epochs. Each epoch took 30 minutes for both training and validation.\n\n# Model Performance and Ensemble Results\n\n| Model               | CV Score | Public LB | Private LB |\n|---------------------|----------|-----------|------------|\n| **Qwen3-14B**       | 0.909    | -         | -          |\n| **Qwen2.5-14B**     | 0.911    | -         | -          |\n| **Ensemble**        | -        | 0.9168    | 0.9253     |\n\n# Inference\nMulti-GPU inference was used to reduce inference time.\n\nDuring training, I noticed that 99% of the math questions were under 300 tokens, so I dynamically truncated inputs longer than 300 tokens. However, at inference time, I used the full input without truncation to fully leverage the math question content and maximize test-time compute.\n\n# Ensemble\nFor each model, I first averaged the K-fold models. Then, I performed a weighted average, giving **0.6** weight to `Qwen3-14B` and **0.4** weight to `Qwen2.5-14B` for the final predictions.\n\n# Code\nTraining Code for:\n- **Qwen3-14B**: [Qwen3-14B Fine-tuning](https://www.kaggle.com/code/ducnh279/kacs1-fine-tuning-qwen3-14b)\n- **Qwen2.5-14B**: [Qwen2.5-14B Fine-tuning](https://www.kaggle.com/code/ducnh279/kacs1-fine-tuning-qwen2-5-14b)\n\nOffline Inference: [Offline Inference Code](https://www.kaggle.com/code/ducnh279/kacs1-offline-inference?scriptVersionId=238990333)",
      "votes": 24
    },
    {
      "id": 3199617,
      "postDate": "2025-05-11T08:33:15.327Z",
      "content": "<p>Oh good ensemble! I used only qwen 2.5 32b. Cong your 1st!</p>",
      "rawMarkdown": "Oh good ensemble! I used only qwen 2.5 32b. Cong your 1st!",
      "votes": 4,
      "replies": [
        {
          "id": 3199624,
          "postDate": "2025-05-11T08:45:57.163Z",
          "content": "<p>I also trained Qwen 2.5 32B, but it performed similarly to the 14B model, so I assumed the 14B variant is enough! </p>",
          "rawMarkdown": "I also trained Qwen 2.5 32B, but it performed similarly to the 14B model, so I assumed the 14B variant is enough! ",
          "votes": 1
        }
      ]
    },
    {
      "id": 3201375,
      "postDate": "2025-05-13T18:49:42.087Z",
      "content": "<p>Nice solution! and congrats on getting first place.</p>",
      "rawMarkdown": "Nice solution! and congrats on getting first place.",
      "votes": 1,
      "replies": [
        {
          "id": 3201648,
          "postDate": "2025-05-14T07:44:49.457Z",
          "content": "<p>Thank you, AIi!!!</p>",
          "rawMarkdown": "Thank you, AIi!!!"
        }
      ]
    },
    {
      "id": 3199989,
      "postDate": "2025-05-11T19:42:41.220Z",
      "content": "<p>Great solution, thank you for sharing! I just have one question: x = self.backbone(**x).last_hidden_state[:, -1, :] seems to be taking the embedding of the final token, which in my case is 'Ġ', not 'Answer:' as I expected. I tried your code , am I missing something, or is there a mistake here?</p>",
      "rawMarkdown": "Great solution, thank you for sharing! I just have one question: x = self.backbone(**x).last_hidden_state[:, -1, :] seems to be taking the embedding of the final token, which in my case is 'Ġ', not 'Answer:' as I expected. I tried your code , am I missing something, or is there a mistake here?",
      "votes": 1,
      "replies": [
        {
          "id": 3200008,
          "postDate": "2025-05-11T20:31:15.957Z",
          "content": "<p>I am not 100% sure I understand your question! I will answer in the way I think you are asking. Feel free to raise more questions or make clarifications if needed.</p>\n<blockquote>\n  <p>x = self.backbone(**x).last_hidden_state[:, -1, :] seems to be taking the embedding of the final token</p>\n</blockquote>\n<p>Yes, it's taking the embedding of the final token!</p>\n<p>'Answer:' is just a <strong>simplified</strong> example of the last token to help people get the idea. It really depends on your tokenizer and how it splits your input.</p>\n<p>Example:</p>\n<blockquote>\n  <p>text = \"What is 2 + 2? Answer:\"</p>\n</blockquote>\n<p>Tokenizer example (this depends on the tokenizer you're using):</p>\n<blockquote>\n  <p>input_ids = [1, 2, 3, 3, 4, 5, 6]<br>\n  tokens = ['What', 'is', '2', '+', '2', 'Answer', ':']</p>\n</blockquote>\n<p>Now, assuming 'last_hidden_state' is the output from the model: 'last_hidden_state[:, -1, :]' will give you the embedding for the last token, which in this case will be <code>':'</code>, <strong>not</strong> <code>'Answer:'</code>.</p>\n<p>The embedding for <code>':'</code> will then be fed into the linear head to produce the logits of the next token, and softmax will be applied to get predicted probabilities of next tokens, such as class labels: [0, 1, 2, 3, 4, 5, 6, 7].</p>\n<p>Be careful about assuming the last token is meaningful, it could just be punctuation or a subword, not the expected word (like 'Answer:').</p>\n<p>Note on padding &amp; truncation:<br>\nI set <code>tokenizer.padding_side='left'</code> to ensure that the input sequence always ends with 'Answer:'. However, if the input sequence exceeds the maximum token limit (e.g., 300 tokens), truncation may occur, which could cause 'Answer:' to be cut off or replaced by a random token at the end of the sequence, depending on the tokenizer’s behavior. </p>",
          "rawMarkdown": "I am not 100% sure I understand your question! I will answer in the way I think you are asking. Feel free to raise more questions or make clarifications if needed.\n\n>x = self.backbone(**x).last_hidden_state[:, -1, :] seems to be taking the embedding of the final token\n\nYes, it's taking the embedding of the final token!\n\n\n'Answer:' is just a **simplified** example of the last token to help people get the idea. It really depends on your tokenizer and how it splits your input.\n\nExample:\n>text = \"What is 2 + 2? Answer:\"\n\nTokenizer example (this depends on the tokenizer you're using):\n>input_ids = [1, 2, 3, 3, 4, 5, 6]\n>tokens = ['What', 'is', '2', '+', '2', 'Answer', ':']\n\nNow, assuming 'last_hidden_state' is the output from the model: 'last_hidden_state[:, -1, :]' will give you the embedding for the last token, which in this case will be `':'`, **not** `'Answer:'`.\n\nThe embedding for `':'` will then be fed into the linear head to produce the logits of the next token, and softmax will be applied to get predicted probabilities of next tokens, such as class labels: [0, 1, 2, 3, 4, 5, 6, 7].\n\nBe careful about assuming the last token is meaningful, it could just be punctuation or a subword, not the expected word (like 'Answer:').\n\nNote on padding & truncation:\nI set `tokenizer.padding_side='left'` to ensure that the input sequence always ends with 'Answer:'. However, if the input sequence exceeds the maximum token limit (e.g., 300 tokens), truncation may occur, which could cause 'Answer:' to be cut off or replaced by a random token at the end of the sequence, depending on the tokenizer’s behavior. ",
          "votes": 2,
          "replies": [
            {
              "id": 3200015,
              "postDate": "2025-05-11T21:08:51.557Z",
              "content": "<p>Thank you for your clear explantation !  </p>",
              "rawMarkdown": "Thank you for your clear explantation !  ",
              "votes": 1
            }
          ]
        }
      ]
    },
    {
      "id": 3199877,
      "postDate": "2025-05-11T15:51:30.363Z",
      "content": "<p>I used the BERT model in this competition, but its performance is generally not as good as LLM. I'm studying much knowledges of LLM currently and your work is very useful me. Thanks for sharing!</p>",
      "rawMarkdown": "I used the BERT model in this competition, but its performance is generally not as good as LLM. I'm studying much knowledges of LLM currently and your work is very useful me. Thanks for sharing!",
      "votes": 1,
      "replies": [
        {
          "id": 3199889,
          "postDate": "2025-05-11T16:21:46.503Z",
          "content": "<p>Great to hear that! </p>",
          "rawMarkdown": "Great to hear that! "
        }
      ]
    },
    {
      "id": 3199708,
      "postDate": "2025-05-11T11:18:51.673Z",
      "content": "<p>Thanks for sharing your ideas! I honestly thought bert-like models might get to the top, but looks like i was mistaken haha</p>",
      "rawMarkdown": "Thanks for sharing your ideas! I honestly thought bert-like models might get to the top, but looks like i was mistaken haha",
      "votes": 1,
      "replies": [
        {
          "id": 3199717,
          "postDate": "2025-05-11T11:35:12.620Z",
          "content": "<p>For math-related data, using decoder-based LLMs is currently the most effective approach. These models are typically trained on large-scale mathematical corpora during both pre-training and post-training stage, making them highly capable for downstream tasks.</p>\n<p>However, I believe there are still specific tasks where encoder-based models like BERT remain both effective and efficient, and can even outperform LLMs.</p>",
          "rawMarkdown": "For math-related data, using decoder-based LLMs is currently the most effective approach. These models are typically trained on large-scale mathematical corpora during both pre-training and post-training stage, making them highly capable for downstream tasks.\n\nHowever, I believe there are still specific tasks where encoder-based models like BERT remain both effective and efficient, and can even outperform LLMs.",
          "votes": 2
        }
      ]
    },
    {
      "id": 3199703,
      "postDate": "2025-05-11T11:06:02.063Z",
      "content": "<p>Hi, I have a quick concern. Is it allowed to use an LLM as a classifier in this competition? We actually thought of this idea earlier but didn’t go ahead with it since the task was defined as a classification problem, not an LLM task. Just wanted to clarify if that approach is considered valid. Thanks!</p>",
      "rawMarkdown": "Hi, I have a quick concern. Is it allowed to use an LLM as a classifier in this competition? We actually thought of this idea earlier but didn’t go ahead with it since the task was defined as a classification problem, not an LLM task. Just wanted to clarify if that approach is considered valid. Thanks!",
      "votes": 1,
      "replies": [
        {
          "id": 3232785,
          "postDate": "2025-06-26T07:48:51.017Z",
          "content": "<p>Nah, I think yes</p>",
          "rawMarkdown": "Nah, I think yes"
        }
      ]
    },
    {
      "id": 3199700,
      "postDate": "2025-05-11T10:56:40.563Z",
      "content": "<p>Congrats, really appreciate the solution you shared!</p>",
      "rawMarkdown": "Congrats, really appreciate the solution you shared!",
      "votes": 1
    },
    {
      "id": 3199607,
      "postDate": "2025-05-11T08:22:48.353Z",
      "content": "<p>Congratulations.  Thank you for sharing  idea. </p>",
      "rawMarkdown": "Congratulations.  Thank you for sharing  idea. ",
      "votes": 1
    },
    {
      "id": 3199569,
      "postDate": "2025-05-11T06:39:11.080Z",
      "content": "<p>Congrats on 1st place, thank you for sharing your solution</p>",
      "rawMarkdown": "Congrats on 1st place, thank you for sharing your solution\n",
      "votes": 1
    },
    {
      "id": 3199478,
      "postDate": "2025-05-11T03:46:26.993Z",
      "content": "<p>Congrats on your 1st place! Looking forward to learning from your solution.</p>",
      "rawMarkdown": "Congrats on your 1st place! Looking forward to learning from your solution.",
      "votes": 1
    },
    {
      "id": 3199438,
      "postDate": "2025-05-11T02:07:15.307Z",
      "content": "<p>Thanks for sharing, it was fun looking at your code. </p>",
      "rawMarkdown": "Thanks for sharing, it was fun looking at your code. ",
      "votes": 1
    },
    {
      "id": 3199411,
      "postDate": "2025-05-10T23:28:05.257Z",
      "content": "<p>Congrats! Thank you for sharing your idea! Will follow you!</p>",
      "rawMarkdown": "Congrats! Thank you for sharing your idea! Will follow you!",
      "votes": 1
    },
    {
      "id": 3284065,
      "postDate": "2025-09-08T05:22:37.463Z",
      "content": "<p>It seems the notebook is no longer available. Hope you can provide it again.</p>",
      "rawMarkdown": "It seems the notebook is no longer available. Hope you can provide it again.",
      "replies": [
        {
          "id": 3286445,
          "postDate": "2025-09-09T20:23:18.257Z",
          "content": "<p>I opened again! Check it out!</p>",
          "rawMarkdown": "I opened again! Check it out!"
        }
      ]
    },
    {
      "id": 3201832,
      "postDate": "2025-05-14T13:04:57.790Z",
      "content": "<p>Congrats, and thanks for the solution!</p>",
      "rawMarkdown": "Congrats, and thanks for the solution!",
      "votes": 1
    }
  ],
  "complete": true,
  "source": "live",
  "fetched_at": "2026-10-19T22:54:05Z",
  "competition_id": 97669,
  "competition_slug": "classification-of-math-problems-by-kasut-academy",
  "team_id": 13769819,
  "solution_url": "/competitions/classification-of-math-problems-by-kasut-academy/writeups/ducnh279-1st-place-solution"
}