{
  "id": 578474,
  "title": "5th place solution",
  "content": "<p>Hi everyone,<br>\nFirst, I’d like to thank the host for organizing such a great LLM competition.</p>\n<h3>Problem Overview</h3>\n<p>The task is to classify mathematical questions into one of eight predefined categories .</p>\n<h2>This is a challenging NLP classification task that requires a deep understanding of both mathematical language and structure.</h2>\n<h3>Methodology</h3>\n<h4>1. <strong>Reasoning-Augmented Input Construction</strong></h4>\n<p>To enrich the input data, each question was paired with a brief, structured reasoning generated by the <code>deepseek-r1-distill-qwen-7b-awq-casperhansen/1</code> model via the <code>vllm</code> framework. This reasoning describes the problem’s mathematical nature without solving or labeling it.<br>\nPrompt used:</p>\n<pre><code> () -&gt; :\n    \n</code></pre>\n<p>The output of this prompt was stored in a new column, <code>Reasoning</code>, in both the train and test sets.</p>\n<h4>2. <strong>Input Formatting</strong></h4>\n<p>Each training example was transformed into a single input string by concatenating the original question with its reasoning:</p>\n<pre><code>train[] =  + train[] +  + train[]\ntest[] =  + test[] +  + test[]\n</code></pre>\n<p>This composite input helps the model leverage both the surface structure and the underlying semantics of the question.</p>\n<h4>3. <strong>Model Training</strong></h4>\n<p>I fine-tuned <code>deepseek-ai/deepseek-math-7b-base</code> on this reasoning-augmented dataset.</p>\n<ul>\n<li><strong>Tokenizer</strong>: Two special tokens were added:</li>\n</ul>\n<pre><code>   \n</code></pre>\n<ul>\n<li><strong>Training</strong>:<ul>\n<li>5-fold stratified cross-validation</li>\n<li>2 epochs per fold</li>\n<li>Fine-tuning via PERT (Parameter-Efficient Fine-Tuning)</li>\n<li>Trained on a single <strong>NVIDIA P100 GPU</strong></li></ul></li>\n</ul>\n<h4>4. <strong>Ensembling</strong></h4>\n<p>Final predictions were made by <strong>averaging the logits</strong> from all 5 folds (simple ensemble).</p>",
  "comments": [
    {
      "id": 3204176,
      "author_name": "Denver Magtibay",
      "author_url": "",
      "post_date": "2025-05-17T20:49:26.973000",
      "content": "<p>Congrats! Thanks for sharing ideas. More power!</p>",
      "votes": 1,
      "replies": []
    },
    {
      "id": 3199712,
      "author_name": "Mohamed Eltayeb",
      "author_url": "",
      "post_date": "2025-05-11T11:22:23.177000",
      "content": "<p>Thanks for sharing! I liked your solution so much. The best part is how efficient it is. Most of the tops are doing huge ensembles. Getting to this rank with a single model (sort of :) ) is amazing!</p>",
      "votes": 1,
      "replies": [
        {
          "id": 3199791,
          "author_name": "Shapu",
          "author_url": "",
          "post_date": "2025-05-11T13:21:58.673000",
          "content": "<p>Thank you, I just tried to minimize the compute time as much as possible.</p>",
          "votes": 1,
          "replies": []
        }
      ]
    },
    {
      "id": 3199793,
      "author_name": "Mfahimhossen",
      "author_url": "",
      "post_date": "2025-05-11T13:23:09.170000",
      "content": "<p>thank you for sharing</p>",
      "votes": 1,
      "replies": []
    }
  ],
  "comment_count": 4,
  "messages": [
    {
      "id": 3199596,
      "postDate": "2025-05-11T08:02:31.123Z",
      "content": "<p>Hi everyone,<br>\nFirst, I’d like to thank the host for organizing such a great LLM competition.</p>\n<h3>Problem Overview</h3>\n<p>The task is to classify mathematical questions into one of eight predefined categories .</p>\n<h2>This is a challenging NLP classification task that requires a deep understanding of both mathematical language and structure.</h2>\n<h3>Methodology</h3>\n<h4>1. <strong>Reasoning-Augmented Input Construction</strong></h4>\n<p>To enrich the input data, each question was paired with a brief, structured reasoning generated by the <code>deepseek-r1-distill-qwen-7b-awq-casperhansen/1</code> model via the <code>vllm</code> framework. This reasoning describes the problem’s mathematical nature without solving or labeling it.<br>\nPrompt used:</p>\n<pre><code> () -&gt; :\n    \n</code></pre>\n<p>The output of this prompt was stored in a new column, <code>Reasoning</code>, in both the train and test sets.</p>\n<h4>2. <strong>Input Formatting</strong></h4>\n<p>Each training example was transformed into a single input string by concatenating the original question with its reasoning:</p>\n<pre><code>train[] =  + train[] +  + train[]\ntest[] =  + test[] +  + test[]\n</code></pre>\n<p>This composite input helps the model leverage both the surface structure and the underlying semantics of the question.</p>\n<h4>3. <strong>Model Training</strong></h4>\n<p>I fine-tuned <code>deepseek-ai/deepseek-math-7b-base</code> on this reasoning-augmented dataset.</p>\n<ul>\n<li><strong>Tokenizer</strong>: Two special tokens were added:</li>\n</ul>\n<pre><code>   \n</code></pre>\n<ul>\n<li><strong>Training</strong>:<ul>\n<li>5-fold stratified cross-validation</li>\n<li>2 epochs per fold</li>\n<li>Fine-tuning via PERT (Parameter-Efficient Fine-Tuning)</li>\n<li>Trained on a single <strong>NVIDIA P100 GPU</strong></li></ul></li>\n</ul>\n<h4>4. <strong>Ensembling</strong></h4>\n<p>Final predictions were made by <strong>averaging the logits</strong> from all 5 folds (simple ensemble).</p>",
      "rawMarkdown": "\nHi everyone,\n\nFirst, I’d like to thank the host for organizing such a great LLM competition.\n\n### Problem Overview\n\nThe task is to classify mathematical questions into one of eight predefined categories .\n\nThis is a challenging NLP classification task that requires a deep understanding of both mathematical language and structure.\n\n---\n\n### Methodology\n\n#### 1. **Reasoning-Augmented Input Construction**\n\nTo enrich the input data, each question was paired with a brief, structured reasoning generated by the `deepseek-r1-distill-qwen-7b-awq-casperhansen/1` model via the `vllm` framework. This reasoning describes the problem’s mathematical nature without solving or labeling it.\n\nPrompt used:\n\n```python\ndef create_reasoning_prompt(question: str) -> str:\n    return f\"\"\"You are a mathematical assistant helping to analyze the structure of math problems. Given a question, briefly describe what it is asking and mention the main mathematical concepts or theorems likely involved.\n\nKeep your explanation short and focused less than 100 tokens. Do not solve or classify the problem.\n\n---\nProblem:\n{question}\n\nBrief analysis:\"\"\"\n```\n\nThe output of this prompt was stored in a new column, `Reasoning`, in both the train and test sets.\n\n#### 2. **Input Formatting**\n\nEach training example was transformed into a single input string by concatenating the original question with its reasoning:\n\n```python\ntrain['temp'] = \"[QUESTION] : \" + train['Question'] + \". [REASONING] : \" + train['Reasoning']\ntest['temp'] = \"[QUESTION] : \" + test['Question'] + \". [REASONING] : \" + test['Reasoning']\n```\n\nThis composite input helps the model leverage both the surface structure and the underlying semantics of the question.\n\n#### 3. **Model Training**\n\nI fine-tuned `deepseek-ai/deepseek-math-7b-base` on this reasoning-augmented dataset.\n\n* **Tokenizer**: Two special tokens were added:\n\n  ```json\n  \"additional_special_tokens\": [\"[QUESTION] :\", \"[REASONING] :\"]\n  ```\n* **Training**:\n\n  * 5-fold stratified cross-validation\n  * 2 epochs per fold\n  * Fine-tuning via PERT (Parameter-Efficient Fine-Tuning)\n  * Trained on a single **NVIDIA P100 GPU**\n\n#### 4. **Ensembling**\n\nFinal predictions were made by **averaging the logits** from all 5 folds (simple ensemble).\n\n\n",
      "votes": 5
    },
    {
      "id": 3204176,
      "postDate": "2025-05-17T20:49:26.973Z",
      "content": "<p>Congrats! Thanks for sharing ideas. More power!</p>",
      "rawMarkdown": "Congrats! Thanks for sharing ideas. More power!",
      "votes": 1
    },
    {
      "id": 3199712,
      "postDate": "2025-05-11T11:22:23.177Z",
      "content": "<p>Thanks for sharing! I liked your solution so much. The best part is how efficient it is. Most of the tops are doing huge ensembles. Getting to this rank with a single model (sort of :) ) is amazing!</p>",
      "rawMarkdown": "Thanks for sharing! I liked your solution so much. The best part is how efficient it is. Most of the tops are doing huge ensembles. Getting to this rank with a single model (sort of :) ) is amazing!",
      "votes": 1,
      "replies": [
        {
          "id": 3199791,
          "postDate": "2025-05-11T13:21:58.673Z",
          "content": "<p>Thank you, I just tried to minimize the compute time as much as possible.</p>",
          "rawMarkdown": "Thank you, I just tried to minimize the compute time as much as possible.",
          "votes": 1
        }
      ]
    },
    {
      "id": 3199793,
      "postDate": "2025-05-11T13:23:09.170Z",
      "content": "<p>thank you for sharing</p>",
      "rawMarkdown": "thank you for sharing",
      "votes": 1
    }
  ],
  "complete": true,
  "source": "live",
  "fetched_at": "2026-10-19T22:53:13Z",
  "competition_id": 97669,
  "competition_slug": "classification-of-math-problems-by-kasut-academy",
  "team_id": 13644358,
  "solution_url": "/competitions/classification-of-math-problems-by-kasut-academy/writeups/nfofuw-5th-place-solution"
}