{
  "id": 639952,
  "title": "R2 scores vary significantly across different folds and seeds",
  "url": "/competitions/csiro-biomass/discussion/639952",
  "author_name": "Bull",
  "post_date": "2025-11-25T10:39:26.646000",
  "votes": 14,
  "comment_count": 9,
  "views": 0,
  "content": "<p>I'm using the fold splitting method referenced from <a href=\"https://www.kaggle.com/competitions/csiro-biomass/discussion/615401\" target=\"_blank\">https://www.kaggle.com/competitions/csiro-biomass/discussion/615401</a>.</p>\n<p>In my experiments, I tried changing the seed and fold and got the results shown below.  </p>\n<p>R2 scores vary significantly just by changing the fold and seed.  </p>\n<p>Has anyone else gotten similar results?  </p>\n<table>\n<thead>\n<tr>\n<th>Fold</th>\n<th>Seed 40</th>\n<th>Seed 41</th>\n<th>Seed 42</th>\n<th>Seed 43</th>\n</tr>\n</thead>\n<tbody>\n<tr>\n<td>0</td>\n<td>0.5244</td>\n<td>0.5276</td>\n<td>0.5222</td>\n<td>0.5545</td>\n</tr>\n<tr>\n<td>1</td>\n<td>0.4984</td>\n<td>0.5084</td>\n<td>0.4612</td>\n<td>0.5064</td>\n</tr>\n<tr>\n<td>2</td>\n<td>0.5992</td>\n<td>0.6229</td>\n<td>0.6122</td>\n<td>0.6206</td>\n</tr>\n<tr>\n<td>3</td>\n<td>0.6338</td>\n<td>0.6105</td>\n<td>0.6695</td>\n<td>0.6345</td>\n</tr>\n<tr>\n<td>4</td>\n<td>0.3770</td>\n<td>0.4268</td>\n<td>0.3674</td>\n<td>0.4636</td>\n</tr>\n</tbody>\n</table>",
  "messages": [
    {
      "id": 3347680,
      "postDate": "2025-11-25T10:39:26.647Z",
      "content": "<p>I'm using the fold splitting method referenced from <a href=\"https://www.kaggle.com/competitions/csiro-biomass/discussion/615401\" target=\"_blank\">https://www.kaggle.com/competitions/csiro-biomass/discussion/615401</a>.</p>\n<p>In my experiments, I tried changing the seed and fold and got the results shown below.  </p>\n<p>R2 scores vary significantly just by changing the fold and seed.  </p>\n<p>Has anyone else gotten similar results?  </p>\n<table>\n<thead>\n<tr>\n<th>Fold</th>\n<th>Seed 40</th>\n<th>Seed 41</th>\n<th>Seed 42</th>\n<th>Seed 43</th>\n</tr>\n</thead>\n<tbody>\n<tr>\n<td>0</td>\n<td>0.5244</td>\n<td>0.5276</td>\n<td>0.5222</td>\n<td>0.5545</td>\n</tr>\n<tr>\n<td>1</td>\n<td>0.4984</td>\n<td>0.5084</td>\n<td>0.4612</td>\n<td>0.5064</td>\n</tr>\n<tr>\n<td>2</td>\n<td>0.5992</td>\n<td>0.6229</td>\n<td>0.6122</td>\n<td>0.6206</td>\n</tr>\n<tr>\n<td>3</td>\n<td>0.6338</td>\n<td>0.6105</td>\n<td>0.6695</td>\n<td>0.6345</td>\n</tr>\n<tr>\n<td>4</td>\n<td>0.3770</td>\n<td>0.4268</td>\n<td>0.3674</td>\n<td>0.4636</td>\n</tr>\n</tbody>\n</table>",
      "rawMarkdown": "I'm using the fold splitting method referenced from https://www.kaggle.com/competitions/csiro-biomass/discussion/615401.\n\nIn my experiments, I tried changing the seed and fold and got the results shown below.  \n\nR2 scores vary significantly just by changing the fold and seed.  \n\nHas anyone else gotten similar results?  \n\n  | Fold | Seed 40 | Seed 41 | Seed 42 | Seed 43 |\n  |------|---------|---------|---------|---------|\n  | 0 | 0.5244 | 0.5276 | 0.5222 | 0.5545 |\n  | 1 | 0.4984 | 0.5084 | 0.4612 | 0.5064 |\n  | 2 | 0.5992 | 0.6229 | 0.6122 | 0.6206 |\n  | 3 | 0.6338 | 0.6105 | 0.6695 | 0.6345 |\n  | 4 | 0.3770 | 0.4268 | 0.3674 | 0.4636 |\n\n",
      "votes": 14
    },
    {
      "id": 3348132,
      "postDate": "2025-11-25T16:15:15.960Z",
      "content": "<p>This is exactly the case. The training dataset is simply too small: in one fold I get CV 0.58 with LB 0.60, and in another fold I get CV 0.75 but LB 0.53.</p>",
      "rawMarkdown": "This is exactly the case. The training dataset is simply too small: in one fold I get CV 0.58 with LB 0.60, and in another fold I get CV 0.75 but LB 0.53.",
      "votes": 4,
      "replies": [
        {
          "id": 3348535,
          "postDate": "2025-11-26T01:48:04.817Z",
          "content": "<p>for sure folds matter a lot .. conservative folds give less LB .. dunno if we all are overfitting or really generalizing right now :)</p>",
          "rawMarkdown": "for sure folds matter a lot .. conservative folds give less LB .. dunno if we all are overfitting or really generalizing right now :)",
          "votes": 1,
          "replies": [
            {
              "id": 3349156,
              "postDate": "2025-11-26T13:42:25.623Z",
              "rawMarkdown": "",
              "votes": -3,
              "isDeleted": true
            },
            {
              "id": 3349191,
              "postDate": "2025-11-26T14:12:34.953Z",
              "content": "<p>ya just the variability across folding strategies is bit confusing </p>",
              "rawMarkdown": "ya just the variability across folding strategies is bit confusing "
            },
            {
              "id": 3349225,
              "postDate": "2025-11-26T14:41:11.417Z",
              "content": "<p>well</p>\n<p>the private LB will be based on the <em>other</em> 43% soooo</p>\n<p>we should be worried 😵‍💫</p>",
              "rawMarkdown": "well\n\nthe private LB will be based on the *other* 43% soooo\n\nwe should be worried 😵‍💫"
            },
            {
              "id": 3349488,
              "postDate": "2025-11-26T16:45:41.447Z",
              "content": "<p>1100% chance we are over fitting when the train data contains 357 images.   </p>",
              "rawMarkdown": "1100% chance we are over fitting when the train data contains 357 images.   ",
              "votes": 3
            },
            {
              "id": 3352324,
              "postDate": "2025-11-29T06:12:06.090Z",
              "content": "<p>my loss curves are showing moderate overfitting, not just overfitting 😅</p>",
              "rawMarkdown": "my loss curves are showing moderate overfitting, not just overfitting 😅",
              "votes": 1
            }
          ]
        }
      ]
    },
    {
      "id": 3350916,
      "postDate": "2025-11-28T01:02:31.933Z",
      "content": "<p>I submitted 4fold*5seed=20models ensemble.</p>\n<p>CV:0.6627 LB:0.61</p>\n<p>CV:0.6001 LB:0.62</p>\n<p>Even with an ensemble of 20 models, the gap between LB and CV remains large.</p>",
      "rawMarkdown": "I submitted 4fold*5seed=20models ensemble.\n\nCV:0.6627 LB:0.61\n\nCV:0.6001 LB:0.62\n\nEven with an ensemble of 20 models, the gap between LB and CV remains large.",
      "votes": 1
    },
    {
      "id": 3349700,
      "postDate": "2025-11-26T22:45:10.390Z",
      "content": "<p>I think it’s a combination of the dataset size, it’s very small. Also, if you use the splitting methodology you shared, in some folds not all states are present.</p>",
      "rawMarkdown": "I think it’s a combination of the dataset size, it’s very small. Also, if you use the splitting methodology you shared, in some folds not all states are present.",
      "votes": 1
    }
  ],
  "comments": [
    {
      "id": 3348132,
      "author_name": "Zhecheng Li",
      "author_url": "",
      "post_date": "2025-11-25T16:15:15.960000",
      "content": "<p>This is exactly the case. The training dataset is simply too small: in one fold I get CV 0.58 with LB 0.60, and in another fold I get CV 0.75 but LB 0.53.</p>",
      "votes": 4,
      "replies": [
        {
          "id": 3348535,
          "author_name": "Gaurav Rawat",
          "author_url": "",
          "post_date": "2025-11-26T01:48:04.817000",
          "content": "<p>for sure folds matter a lot .. conservative folds give less LB .. dunno if we all are overfitting or really generalizing right now :)</p>",
          "votes": 1,
          "replies": [
            {
              "id": 3349156,
              "author_name": "",
              "author_url": "",
              "post_date": "2025-11-26T13:42:25.623000",
              "content": "",
              "votes": -3,
              "replies": []
            },
            {
              "id": 3349191,
              "author_name": "Gaurav Rawat",
              "author_url": "",
              "post_date": "2025-11-26T14:12:34.953000",
              "content": "<p>ya just the variability across folding strategies is bit confusing </p>",
              "votes": 0,
              "replies": []
            },
            {
              "id": 3349225,
              "author_name": "MD Mushfirat Mohaimin",
              "author_url": "",
              "post_date": "2025-11-26T14:41:11.417000",
              "content": "<p>well</p>\n<p>the private LB will be based on the <em>other</em> 43% soooo</p>\n<p>we should be worried 😵‍💫</p>",
              "votes": 0,
              "replies": []
            },
            {
              "id": 3349488,
              "author_name": "PC Jimmmy",
              "author_url": "",
              "post_date": "2025-11-26T16:45:41.447000",
              "content": "<p>1100% chance we are over fitting when the train data contains 357 images.   </p>",
              "votes": 3,
              "replies": []
            },
            {
              "id": 3352324,
              "author_name": "Geremie Yeo",
              "author_url": "",
              "post_date": "2025-11-29T06:12:06.090000",
              "content": "<p>my loss curves are showing moderate overfitting, not just overfitting 😅</p>",
              "votes": 1,
              "replies": []
            }
          ]
        }
      ]
    },
    {
      "id": 3350916,
      "author_name": "Bull",
      "author_url": "",
      "post_date": "2025-11-28T01:02:31.933000",
      "content": "<p>I submitted 4fold*5seed=20models ensemble.</p>\n<p>CV:0.6627 LB:0.61</p>\n<p>CV:0.6001 LB:0.62</p>\n<p>Even with an ensemble of 20 models, the gap between LB and CV remains large.</p>",
      "votes": 1,
      "replies": []
    },
    {
      "id": 3349700,
      "author_name": "Maximiliano Diaz Battan",
      "author_url": "",
      "post_date": "2025-11-26T22:45:10.390000",
      "content": "<p>I think it’s a combination of the dataset size, it’s very small. Also, if you use the splitting methodology you shared, in some folds not all states are present.</p>",
      "votes": 1,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "3347680": "I'm using the fold splitting method referenced from https://www.kaggle.com/competitions/csiro-biomass/discussion/615401.\n\nIn my experiments, I tried changing the seed and fold and got the results shown below.  \n\nR2 scores vary significantly just by changing the fold and seed.  \n\nHas anyone else gotten similar results?  \n\n  | Fold | Seed 40 | Seed 41 | Seed 42 | Seed 43 |\n  |------|---------|---------|---------|---------|\n  | 0 | 0.5244 | 0.5276 | 0.5222 | 0.5545 |\n  | 1 | 0.4984 | 0.5084 | 0.4612 | 0.5064 |\n  | 2 | 0.5992 | 0.6229 | 0.6122 | 0.6206 |\n  | 3 | 0.6338 | 0.6105 | 0.6695 | 0.6345 |\n  | 4 | 0.3770 | 0.4268 | 0.3674 | 0.4636 |\n\n",
    "3348132": "This is exactly the case. The training dataset is simply too small: in one fold I get CV 0.58 with LB 0.60, and in another fold I get CV 0.75 but LB 0.53.",
    "3350916": "I submitted 4fold*5seed=20models ensemble.\n\nCV:0.6627 LB:0.61\n\nCV:0.6001 LB:0.62\n\nEven with an ensemble of 20 models, the gap between LB and CV remains large.",
    "3349700": "I think it’s a combination of the dataset size, it’s very small. Also, if you use the splitting methodology you shared, in some folds not all states are present."
  }
}