{
  "id": 588009,
  "title": "What strategies boost you most in LB",
  "url": "/competitions/cmi-detect-behavior-with-sensor-data/discussion/588009",
  "author_name": "menglei",
  "post_date": "2025-07-03T20:35:22.722000",
  "votes": 12,
  "comment_count": 6,
  "views": 0,
  "content": "<p>I can think of the following general strategies that people use in Kaggle competitions: </p>\n<ol>\n<li>Initial model construction</li>\n<li>Feature engineering</li>\n<li>Data augmentation</li>\n<li>Loss-function design</li>\n<li>Fine-tuning (including architecture search)</li>\n<li>Model ensembling.</li>\n</ol>\n<p>Would be curious to see which of these areas you choose to focus on and how much payoff you ultimately get from that effort.</p>\n<p>For me, my time investment has looked like this (from most to least): <br>\nFine-tuning &gt; feature engineering &gt; loss-function design &gt; data augmentation &gt; initial model construction &gt; model ensembling.</p>\n<p>As for leaderboard gains:<br>\nFeature engineering (+0.08)&gt;data augmentation  (+0.04)&gt; Initial model selection (good model can give me a solid starting point)&gt;model ensembling(+0.02), fine-tuning (different architecture/hyperparams give me±0.02)&gt;loss-function design (hasn't give me any benefit yet, though I believe there is potential)</p>",
  "messages": [
    {
      "id": 3240467,
      "postDate": "2025-07-03T20:35:22.723Z",
      "content": "<p>I can think of the following general strategies that people use in Kaggle competitions: </p>\n<ol>\n<li>Initial model construction</li>\n<li>Feature engineering</li>\n<li>Data augmentation</li>\n<li>Loss-function design</li>\n<li>Fine-tuning (including architecture search)</li>\n<li>Model ensembling.</li>\n</ol>\n<p>Would be curious to see which of these areas you choose to focus on and how much payoff you ultimately get from that effort.</p>\n<p>For me, my time investment has looked like this (from most to least): <br>\nFine-tuning &gt; feature engineering &gt; loss-function design &gt; data augmentation &gt; initial model construction &gt; model ensembling.</p>\n<p>As for leaderboard gains:<br>\nFeature engineering (+0.08)&gt;data augmentation  (+0.04)&gt; Initial model selection (good model can give me a solid starting point)&gt;model ensembling(+0.02), fine-tuning (different architecture/hyperparams give me±0.02)&gt;loss-function design (hasn't give me any benefit yet, though I believe there is potential)</p>",
      "rawMarkdown": "I can think of the following general strategies that people use in Kaggle competitions: \n\n1. Initial model construction\n2. Feature engineering\n3. Data augmentation\n4. Loss-function design\n5. Fine-tuning (including architecture search)\n6. Model ensembling.\n\nWould be curious to see which of these areas you choose to focus on and how much payoff you ultimately get from that effort.\n\nFor me, my time investment has looked like this (from most to least): \nFine-tuning > feature engineering > loss-function design > data augmentation > initial model construction > model ensembling.\n\nAs for leaderboard gains:\nFeature engineering (+0.08)>data augmentation  (+0.04)> Initial model selection (good model can give me a solid starting point)>model ensembling(+0.02), fine-tuning (different architecture/hyperparams give me±0.02)>loss-function design (hasn't give me any benefit yet, though I believe there is potential)\n",
      "votes": 11
    },
    {
      "id": 3240478,
      "postDate": "2025-07-03T20:51:34.743Z",
      "content": "<p>Really cool breakdown! For me, feature engineering has consistently given the biggest boost, especially on tabular comps. Fine-tuning and good initial model choice help a lot too, but ensembling usually just gives that final push. Still experimenting with loss functions, haven’t cracked the magic there yet either!</p>",
      "rawMarkdown": "Really cool breakdown! For me, feature engineering has consistently given the biggest boost, especially on tabular comps. Fine-tuning and good initial model choice help a lot too, but ensembling usually just gives that final push. Still experimenting with loss functions, haven’t cracked the magic there yet either!",
      "votes": 1
    },
    {
      "id": 3241975,
      "postDate": "2025-07-05T12:09:34.413Z",
      "content": "<p>I agree about there being potential for improving performance with loss function design, but it's a new area for me. For now I have just been creating additional metrics for Local CV and I've been finding that helpful. </p>",
      "rawMarkdown": "I agree about there being potential for improving performance with loss function design, but it's a new area for me. For now I have just been creating additional metrics for Local CV and I've been finding that helpful. "
    },
    {
      "id": 3241369,
      "postDate": "2025-07-04T18:35:03.320Z",
      "content": "<p>What do you mean by data augmentation?</p>",
      "rawMarkdown": "What do you mean by data augmentation?",
      "replies": [
        {
          "id": 3241704,
          "postDate": "2025-07-05T04:53:30.187Z",
          "content": "<p>I used mixup (interpolation from two sequences) and it boosted my LB score.</p>",
          "rawMarkdown": "I used mixup (interpolation from two sequences) and it boosted my LB score.",
          "votes": 1,
          "replies": [
            {
              "id": 3242401,
              "postDate": "2025-07-05T20:40:20.067Z",
              "content": "<p>Oh thanks for the clarification!</p>",
              "rawMarkdown": "Oh thanks for the clarification!"
            }
          ]
        }
      ]
    },
    {
      "id": 3242280,
      "postDate": "2025-07-05T18:27:11.463Z",
      "rawMarkdown": "",
      "votes": 1,
      "isDeleted": true
    }
  ],
  "comments": [
    {
      "id": 3240478,
      "author_name": "Muhammad Faizan Khan",
      "author_url": "",
      "post_date": "2025-07-03T20:51:34.743000",
      "content": "<p>Really cool breakdown! For me, feature engineering has consistently given the biggest boost, especially on tabular comps. Fine-tuning and good initial model choice help a lot too, but ensembling usually just gives that final push. Still experimenting with loss functions, haven’t cracked the magic there yet either!</p>",
      "votes": 1,
      "replies": []
    },
    {
      "id": 3241975,
      "author_name": "Peter Moorhouse",
      "author_url": "",
      "post_date": "2025-07-05T12:09:34.413000",
      "content": "<p>I agree about there being potential for improving performance with loss function design, but it's a new area for me. For now I have just been creating additional metrics for Local CV and I've been finding that helpful. </p>",
      "votes": 0,
      "replies": []
    },
    {
      "id": 3241369,
      "author_name": "paperxd",
      "author_url": "",
      "post_date": "2025-07-04T18:35:03.320000",
      "content": "<p>What do you mean by data augmentation?</p>",
      "votes": 0,
      "replies": [
        {
          "id": 3241704,
          "author_name": "menglei",
          "author_url": "",
          "post_date": "2025-07-05T04:53:30.187000",
          "content": "<p>I used mixup (interpolation from two sequences) and it boosted my LB score.</p>",
          "votes": 1,
          "replies": [
            {
              "id": 3242401,
              "author_name": "paperxd",
              "author_url": "",
              "post_date": "2025-07-05T20:40:20.067000",
              "content": "<p>Oh thanks for the clarification!</p>",
              "votes": 0,
              "replies": []
            }
          ]
        }
      ]
    },
    {
      "id": 3242280,
      "author_name": "",
      "author_url": "",
      "post_date": "2025-07-05T18:27:11.463000",
      "content": "",
      "votes": 1,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "3240467": "I can think of the following general strategies that people use in Kaggle competitions: \n\n1. Initial model construction\n2. Feature engineering\n3. Data augmentation\n4. Loss-function design\n5. Fine-tuning (including architecture search)\n6. Model ensembling.\n\nWould be curious to see which of these areas you choose to focus on and how much payoff you ultimately get from that effort.\n\nFor me, my time investment has looked like this (from most to least): \nFine-tuning > feature engineering > loss-function design > data augmentation > initial model construction > model ensembling.\n\nAs for leaderboard gains:\nFeature engineering (+0.08)>data augmentation  (+0.04)> Initial model selection (good model can give me a solid starting point)>model ensembling(+0.02), fine-tuning (different architecture/hyperparams give me±0.02)>loss-function design (hasn't give me any benefit yet, though I believe there is potential)\n",
    "3240478": "Really cool breakdown! For me, feature engineering has consistently given the biggest boost, especially on tabular comps. Fine-tuning and good initial model choice help a lot too, but ensembling usually just gives that final push. Still experimenting with loss functions, haven’t cracked the magic there yet either!",
    "3241975": "I agree about there being potential for improving performance with loss function design, but it's a new area for me. For now I have just been creating additional metrics for Local CV and I've been finding that helpful. ",
    "3241369": "What do you mean by data augmentation?",
    "3242280": ""
  }
}