{
  "id": 583899,
  "title": "Sharing Results from Zero-Level Kaggle Agent (Claude Code) Experimentation",
  "url": "/competitions/cmi-detect-behavior-with-sensor-data/discussion/583899",
  "author_name": "",
  "post_date": "2025-06-10T09:33:24.948454200Z",
  "votes": 31,
  "comment_count": 6,
  "views": 0,
  "content": "<p>Inspired by <a href=\"https://www.kaggle.com/phalanx\" target=\"_blank\">@phalanx</a> 's post, I decided to try how high a score I could achieve using \"only\" a Kaggle agent! Instead of using Mastra as the agent application framework, I adopted Claude Code.</p>\n<p>At the start of this attempt, I was at zero level with almost no understanding of the competition.</p>\n<p><strong>Improvement Steps with Kaggle Agent</strong></p>\n<ol>\n<li>Use baseline with CV=0.772, LB=0.65 as exp_0</li>\n<li>Measure CV and LB for exp_i</li>\n<li>Copy exp_i directory as-is to exp_i+1</li>\n<li>Have Claude Code improve exp_i+1 based on exp_i results (Prompt: \"I've created exp_i+1. Please rewrite the contents. By the way, exp_i had CV=X, LB=Y.\")</li>\n<li>Return to step 2</li>\n</ol>\n<p><strong>Experimental Results Summary</strong></p>\n<table>\n<thead>\n<tr>\n<th>Experiment</th>\n<th>CV Score</th>\n<th>LB Score</th>\n<th>CV-LB Gap</th>\n<th>Main Approach</th>\n<th>Notes</th>\n</tr>\n</thead>\n<tbody>\n<tr>\n<td>exp_0</td>\n<td>0.772</td>\n<td>0.65</td>\n<td>+0.122</td>\n<td>Initial baseline</td>\n<td>First benchmark</td>\n</tr>\n<tr>\n<td>exp_1</td>\n<td>0.7662</td>\n<td>0.66</td>\n<td>+0.106</td>\n<td>Basic Transformer</td>\n<td>LB improved, CV slightly decreased</td>\n</tr>\n<tr>\n<td>exp_2</td>\n<td>0.7716</td>\n<td>0.63</td>\n<td>+0.142</td>\n<td>Enhanced Transformer</td>\n<td>CV improved but LB worsened</td>\n</tr>\n<tr>\n<td>exp_3</td>\n<td>0.7679</td>\n<td>0.63</td>\n<td>+0.138</td>\n<td>Generalization focus</td>\n<td>Enhanced regularization</td>\n</tr>\n<tr>\n<td>exp_4</td>\n<td>0.7629</td>\n<td>0.62</td>\n<td>+0.143</td>\n<td>Data augmentation</td>\n<td>Worst performance</td>\n</tr>\n<tr>\n<td>exp_5</td>\n<td>0.7746</td>\n<td>0.64</td>\n<td>+0.135</td>\n<td>CNN adoption</td>\n<td>Highest CV achieved</td>\n</tr>\n</tbody>\n</table>\n<p><strong>Summary</strong></p>\n<ul>\n<li>The Kaggle Agent (Claude Code) achieved LB improvement from 0.65 → 0.66, but struggled to improve further</li>\n<li>API usage cost was a few dollars</li>\n<li>Zero errors occurred</li>\n<li>When CV improved but LB decreased, the Kaggle Agent intelligently introduced overfitting countermeasures and other smart improvements</li>\n<li>Kaggle Agent is fun, so I recommend trying it! There are various approaches you can take, so there's room for creativity and innovation.</li>\n</ul>",
  "messages": [
    {
      "id": "3221020",
      "postDate": "06/10/2025 09:33:24",
      "content": "<p>Inspired by <a href=\"https://www.kaggle.com/phalanx\" target=\"_blank\">@phalanx</a> 's post, I decided to try how high a score I could achieve using \"only\" a Kaggle agent! Instead of using Mastra as the agent application framework, I adopted Claude Code.</p>\n<p>At the start of this attempt, I was at zero level with almost no understanding of the competition.</p>\n<p><strong>Improvement Steps with Kaggle Agent</strong></p>\n<ol>\n<li>Use baseline with CV=0.772, LB=0.65 as exp_0</li>\n<li>Measure CV and LB for exp_i</li>\n<li>Copy exp_i directory as-is to exp_i+1</li>\n<li>Have Claude Code improve exp_i+1 based on exp_i results (Prompt: \"I've created exp_i+1. Please rewrite the contents. By the way, exp_i had CV=X, LB=Y.\")</li>\n<li>Return to step 2</li>\n</ol>\n<p><strong>Experimental Results Summary</strong></p>\n<table>\n<thead>\n<tr>\n<th>Experiment</th>\n<th>CV Score</th>\n<th>LB Score</th>\n<th>CV-LB Gap</th>\n<th>Main Approach</th>\n<th>Notes</th>\n</tr>\n</thead>\n<tbody>\n<tr>\n<td>exp_0</td>\n<td>0.772</td>\n<td>0.65</td>\n<td>+0.122</td>\n<td>Initial baseline</td>\n<td>First benchmark</td>\n</tr>\n<tr>\n<td>exp_1</td>\n<td>0.7662</td>\n<td>0.66</td>\n<td>+0.106</td>\n<td>Basic Transformer</td>\n<td>LB improved, CV slightly decreased</td>\n</tr>\n<tr>\n<td>exp_2</td>\n<td>0.7716</td>\n<td>0.63</td>\n<td>+0.142</td>\n<td>Enhanced Transformer</td>\n<td>CV improved but LB worsened</td>\n</tr>\n<tr>\n<td>exp_3</td>\n<td>0.7679</td>\n<td>0.63</td>\n<td>+0.138</td>\n<td>Generalization focus</td>\n<td>Enhanced regularization</td>\n</tr>\n<tr>\n<td>exp_4</td>\n<td>0.7629</td>\n<td>0.62</td>\n<td>+0.143</td>\n<td>Data augmentation</td>\n<td>Worst performance</td>\n</tr>\n<tr>\n<td>exp_5</td>\n<td>0.7746</td>\n<td>0.64</td>\n<td>+0.135</td>\n<td>CNN adoption</td>\n<td>Highest CV achieved</td>\n</tr>\n</tbody>\n</table>\n<p><strong>Summary</strong></p>\n<ul>\n<li>The Kaggle Agent (Claude Code) achieved LB improvement from 0.65 → 0.66, but struggled to improve further</li>\n<li>API usage cost was a few dollars</li>\n<li>Zero errors occurred</li>\n<li>When CV improved but LB decreased, the Kaggle Agent intelligently introduced overfitting countermeasures and other smart improvements</li>\n<li>Kaggle Agent is fun, so I recommend trying it! There are various approaches you can take, so there's room for creativity and innovation.</li>\n</ul>",
      "rawMarkdown": "Inspired by @phalanx 's post, I decided to try how high a score I could achieve using \"only\" a Kaggle agent! Instead of using Mastra as the agent application framework, I adopted Claude Code.\n\nAt the start of this attempt, I was at zero level with almost no understanding of the competition.\n\n**Improvement Steps with Kaggle Agent**\n1. Use baseline with CV=0.772, LB=0.65 as exp_0\n2. Measure CV and LB for exp_i\n3. Copy exp_i directory as-is to exp_i+1\n4. Have Claude Code improve exp_i+1 based on exp_i results (Prompt: \"I've created exp_i+1. Please rewrite the contents. By the way, exp_i had CV=X, LB=Y.\")\n5. Return to step 2\n\n**Experimental Results Summary**\n\n| Experiment | CV Score | LB Score | CV-LB Gap | Main Approach | Notes |\n|------------|----------|----------|-----------|---------------|-------|\n| exp_0 | 0.772 | 0.65 | +0.122 | Initial baseline | First benchmark |\n| exp_1 | 0.7662 | 0.66 | +0.106 | Basic Transformer | LB improved, CV slightly decreased |\n| exp_2 | 0.7716 | 0.63 | +0.142 | Enhanced Transformer | CV improved but LB worsened |\n| exp_3 | 0.7679 | 0.63 | +0.138 | Generalization focus | Enhanced regularization |\n| exp_4 | 0.7629 | 0.62 | +0.143 | Data augmentation | Worst performance |\n| exp_5 | 0.7746 | 0.64 | +0.135 | CNN adoption | Highest CV achieved |\n\n**Summary**\n- The Kaggle Agent (Claude Code) achieved LB improvement from 0.65 → 0.66, but struggled to improve further\n- API usage cost was a few dollars\n- Zero errors occurred\n- When CV improved but LB decreased, the Kaggle Agent intelligently introduced overfitting countermeasures and other smart improvements\n- Kaggle Agent is fun, so I recommend trying it! There are various approaches you can take, so there's room for creativity and innovation.",
      "votes": null
    },
    {
      "id": "3221061",
      "postDate": "06/10/2025 10:56:29",
      "content": "<p><code>CV-LB Gap</code> seems to be quite large, relative to what other competitors are getting. </p>\n<p>Does the agent highlight this observation, and / or able to reduce the gap with a better \"CV\" scheme?</p>",
      "rawMarkdown": "`CV-LB Gap` seems to be quite large, relative to what other competitors are getting. \n\n\nDoes the agent highlight this observation, and / or able to reduce the gap with a better \"CV\" scheme?",
      "votes": null
    },
    {
      "id": "3221072",
      "postDate": "06/10/2025 11:12:33",
      "content": "<p>Thank you for your comment!<br>\nFrom observing the agent’s logs, there are signs that it attempted to reduce the gap through various strategies.<br>\nHowever, in practice, the gap hasn’t been significantly reduced.<br>\nIt seems that building a robust CV scheme will be key to making effective use of the Kaggle agent.</p>",
      "rawMarkdown": "Thank you for your comment!\nFrom observing the agent’s logs, there are signs that it attempted to reduce the gap through various strategies.\nHowever, in practice, the gap hasn’t been significantly reduced.\nIt seems that building a robust CV scheme will be key to making effective use of the Kaggle agent.",
      "votes": null
    },
    {
      "id": "3221188",
      "postDate": "06/10/2025 14:30:23",
      "content": "<p>Did you clarify the nature of this competition, particularly regarding the presence of null values in the TOF column for 50% of the sequences in your prompt? It seems the significant gap between the CV and LB scores might be attributed to training on the complete dataset without knowledge of the missing TOF data in the hidden test set.</p>",
      "rawMarkdown": "Did you clarify the nature of this competition, particularly regarding the presence of null values in the TOF column for 50% of the sequences in your prompt? It seems the significant gap between the CV and LB scores might be attributed to training on the complete dataset without knowledge of the missing TOF data in the hidden test set.",
      "votes": null
    },
    {
      "id": "3221192",
      "postDate": "06/10/2025 14:36:40",
      "content": "<p>Thank you for letting me know! I believe the agent probably isn’t handling it properly.<br>\nWhat I realized after actually trying the Kaggle agent is that it’s important, at the very least, for the agent to know the background information of the competition.</p>",
      "rawMarkdown": "Thank you for letting me know! I believe the agent probably isn’t handling it properly.\nWhat I realized after actually trying the Kaggle agent is that it’s important, at the very least, for the agent to know the background information of the competition.",
      "votes": null
    },
    {
      "id": "3223989",
      "postDate": "06/14/2025 07:34:33",
      "content": "<p>I wonder is the gap caused by that the actual test code have some features set to NULL. I have a similar gap</p>",
      "rawMarkdown": "I wonder is the gap caused by that the actual test code have some features set to NULL. I have a similar gap",
      "votes": null
    },
    {
      "id": "3226154",
      "postDate": "06/17/2025 10:12:48",
      "content": "<p>I guess the agent implementation for solving problems can get quite expensive</p>",
      "rawMarkdown": "I guess the agent implementation for solving problems can get quite expensive",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 3221061,
      "author_name": "gerwynng",
      "author_url": "",
      "post_date": "06/10/2025 10:56:29",
      "content": "<p><code>CV-LB Gap</code> seems to be quite large, relative to what other competitors are getting. </p>\n<p>Does the agent highlight this observation, and / or able to reduce the gap with a better \"CV\" scheme?</p>",
      "votes": null,
      "replies": [
        {
          "id": 3221072,
          "author_name": "isakatsuyoshi",
          "author_url": "",
          "post_date": "06/10/2025 11:12:33",
          "content": "<p>Thank you for your comment!<br>\nFrom observing the agent’s logs, there are signs that it attempted to reduce the gap through various strategies.<br>\nHowever, in practice, the gap hasn’t been significantly reduced.<br>\nIt seems that building a robust CV scheme will be key to making effective use of the Kaggle agent.</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 3223989,
          "author_name": "zhangyue199",
          "author_url": "",
          "post_date": "06/14/2025 07:34:33",
          "content": "<p>I wonder is the gap caused by that the actual test code have some features set to NULL. I have a similar gap</p>",
          "votes": null,
          "replies": []
        }
      ]
    },
    {
      "id": 3221188,
      "author_name": "hearttt",
      "author_url": "",
      "post_date": "06/10/2025 14:30:23",
      "content": "<p>Did you clarify the nature of this competition, particularly regarding the presence of null values in the TOF column for 50% of the sequences in your prompt? It seems the significant gap between the CV and LB scores might be attributed to training on the complete dataset without knowledge of the missing TOF data in the hidden test set.</p>",
      "votes": null,
      "replies": [
        {
          "id": 3221192,
          "author_name": "isakatsuyoshi",
          "author_url": "",
          "post_date": "06/10/2025 14:36:40",
          "content": "<p>Thank you for letting me know! I believe the agent probably isn’t handling it properly.<br>\nWhat I realized after actually trying the Kaggle agent is that it’s important, at the very least, for the agent to know the background information of the competition.</p>",
          "votes": null,
          "replies": []
        }
      ]
    },
    {
      "id": 3226154,
      "author_name": "sayedathar11",
      "author_url": "",
      "post_date": "06/17/2025 10:12:48",
      "content": "<p>I guess the agent implementation for solving problems can get quite expensive</p>",
      "votes": null,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "3221020": "Inspired by @phalanx 's post, I decided to try how high a score I could achieve using \"only\" a Kaggle agent! Instead of using Mastra as the agent application framework, I adopted Claude Code.\n\nAt the start of this attempt, I was at zero level with almost no understanding of the competition.\n\n**Improvement Steps with Kaggle Agent**\n1. Use baseline with CV=0.772, LB=0.65 as exp_0\n2. Measure CV and LB for exp_i\n3. Copy exp_i directory as-is to exp_i+1\n4. Have Claude Code improve exp_i+1 based on exp_i results (Prompt: \"I've created exp_i+1. Please rewrite the contents. By the way, exp_i had CV=X, LB=Y.\")\n5. Return to step 2\n\n**Experimental Results Summary**\n\n| Experiment | CV Score | LB Score | CV-LB Gap | Main Approach | Notes |\n|------------|----------|----------|-----------|---------------|-------|\n| exp_0 | 0.772 | 0.65 | +0.122 | Initial baseline | First benchmark |\n| exp_1 | 0.7662 | 0.66 | +0.106 | Basic Transformer | LB improved, CV slightly decreased |\n| exp_2 | 0.7716 | 0.63 | +0.142 | Enhanced Transformer | CV improved but LB worsened |\n| exp_3 | 0.7679 | 0.63 | +0.138 | Generalization focus | Enhanced regularization |\n| exp_4 | 0.7629 | 0.62 | +0.143 | Data augmentation | Worst performance |\n| exp_5 | 0.7746 | 0.64 | +0.135 | CNN adoption | Highest CV achieved |\n\n**Summary**\n- The Kaggle Agent (Claude Code) achieved LB improvement from 0.65 → 0.66, but struggled to improve further\n- API usage cost was a few dollars\n- Zero errors occurred\n- When CV improved but LB decreased, the Kaggle Agent intelligently introduced overfitting countermeasures and other smart improvements\n- Kaggle Agent is fun, so I recommend trying it! There are various approaches you can take, so there's room for creativity and innovation.",
    "3221061": "`CV-LB Gap` seems to be quite large, relative to what other competitors are getting. \n\n\nDoes the agent highlight this observation, and / or able to reduce the gap with a better \"CV\" scheme?",
    "3221072": "Thank you for your comment!\nFrom observing the agent’s logs, there are signs that it attempted to reduce the gap through various strategies.\nHowever, in practice, the gap hasn’t been significantly reduced.\nIt seems that building a robust CV scheme will be key to making effective use of the Kaggle agent.",
    "3221188": "Did you clarify the nature of this competition, particularly regarding the presence of null values in the TOF column for 50% of the sequences in your prompt? It seems the significant gap between the CV and LB scores might be attributed to training on the complete dataset without knowledge of the missing TOF data in the hidden test set.",
    "3221192": "Thank you for letting me know! I believe the agent probably isn’t handling it properly.\nWhat I realized after actually trying the Kaggle agent is that it’s important, at the very least, for the agent to know the background information of the competition.",
    "3223989": "I wonder is the gap caused by that the actual test code have some features set to NULL. I have a similar gap",
    "3226154": "I guess the agent implementation for solving problems can get quite expensive"
  },
  "source": "meta"
}