{ "id": 614912, "title": "PCA trick to address target linear dependency", "url": "/competitions/csiro-biomass/discussion/614912", "author_name": "Ruslan Baynazarov", "post_date": "2025-11-07T12:03:10.075000", "votes": 33, "comment_count": 0, "views": 0, "content": "

As said by [LB 0.57]Infer + Model code, the 5 targets are linearly dependent. There are only 3 independent components.

\n

Approach 1: Predicting Key Components

\n

The [LB 0.57]Infer + Model code predicts only 3 components: Dry_Total_g, GDM_g and Dry_Green_g. The other two are calculated using subtraction. The reasoning to choose these specific components is that they have more weight in the metric.

\n

My approach: the PCA trick

\n

In SigLip+GradientBoosting+PCA trick I tried a different approach.

\n
    \n
  1. Fit a 3-component PCA on the target values
    2. \n
  2. Predict all 5 targets
    3. \n
  3. Apply the PCA to the prediction, and then apply the inverse transform
    4. \n
\n

This 5->3->5 bottleneck addresses for the inconsistency of model's predictions. It gives around 0.01 uplift to the CV and LB scores.

\n

I believe it doesn't hurt the Dry_Total_g (top weight in the score) because it is a sum of 3 other predictions. It makes sense to me to also correct the sum, once we predict all components independently.

\n

Notably, this trick does not affect the predictions of the DummyRegressor at all. Which makes sense I guess.

\n

The exact code:

\n
     i, (train_index, test_index)  (kf.split(X, groups=groups)):\n        \n        pca = PCA().fit(y_true[train_index])\n\n        \n         l  ((labels)):\n            m.fit(X[train_index], y_true[train_index, l])\n            y_pred[test_index, l] = m.predict(X[test_index]).clip()\n            y_pred_test[:, l] += m.predict(X_test).clip() / n_splits\n\n        \n        y_pred[test_index] = pca.inverse_transform(pca.transform(y_pred[test_index])).clip()\n        y_pred_test = pca.inverse_transform(pca.transform(y_pred_test)).clip()\n
\n

P.S. I don't claim that my approach is better, just wanted to share

", "messages": [ { "id": 3312536, "postDate": "2025-11-07T12:03:10.077Z", "content": "

As said by [LB 0.57]Infer + Model code, the 5 targets are linearly dependent. There are only 3 independent components.

\n

Approach 1: Predicting Key Components

\n

The [LB 0.57]Infer + Model code predicts only 3 components: Dry_Total_g, GDM_g and Dry_Green_g. The other two are calculated using subtraction. The reasoning to choose these specific components is that they have more weight in the metric.

\n

My approach: the PCA trick

\n

In SigLip+GradientBoosting+PCA trick I tried a different approach.

\n
    \n
  1. Fit a 3-component PCA on the target values
    2. \n
  2. Predict all 5 targets
    3. \n
  3. Apply the PCA to the prediction, and then apply the inverse transform
    4. \n
\n

This 5->3->5 bottleneck addresses for the inconsistency of model's predictions. It gives around 0.01 uplift to the CV and LB scores.

\n

I believe it doesn't hurt the Dry_Total_g (top weight in the score) because it is a sum of 3 other predictions. It makes sense to me to also correct the sum, once we predict all components independently.

\n

Notably, this trick does not affect the predictions of the DummyRegressor at all. Which makes sense I guess.

\n

The exact code:

\n
     i, (train_index, test_index)  (kf.split(X, groups=groups)):\n        \n        pca = PCA().fit(y_true[train_index])\n\n        \n         l  ((labels)):\n            m.fit(X[train_index], y_true[train_index, l])\n            y_pred[test_index, l] = m.predict(X[test_index]).clip()\n            y_pred_test[:, l] += m.predict(X_test).clip() / n_splits\n\n        \n        y_pred[test_index] = pca.inverse_transform(pca.transform(y_pred[test_index])).clip()\n        y_pred_test = pca.inverse_transform(pca.transform(y_pred_test)).clip()\n
\n

P.S. I don't claim that my approach is better, just wanted to share

", "rawMarkdown": "As said by [[LB 0.57]Infer + Model code](https://www.kaggle.com/code/none00000/lb-0-57-infer-model-code), the 5 targets are linearly dependent. There are only 3 independent components.\n\n## Approach 1: Predicting Key Components\nThe [[LB 0.57]Infer + Model code](https://www.kaggle.com/code/none00000/lb-0-57-infer-model-code) predicts only 3 components: Dry_Total_g, GDM_g and Dry_Green_g. The other two are calculated using subtraction. The reasoning to choose these specific components is that they have more weight in the metric.\n\n## My approach: the PCA trick\nIn [SigLip+GradientBoosting+PCA trick](https://www.kaggle.com/code/hocop1/siglip-gradientboosting-pca-trick) I tried a different approach.\n1. Fit a 3-component PCA on the target values\n2. Predict all 5 targets\n3. Apply the PCA to the prediction, and then apply the inverse transform\n\nThis 5->3->5 bottleneck addresses for the inconsistency of model's predictions. It gives around 0.01 uplift to the CV and LB scores.\n\nI believe it doesn't hurt the Dry_Total_g (top weight in the score) because it is a sum of 3 other predictions. It makes sense to me to also correct the sum, once we predict all components independently.\n\nNotably, this trick does not affect the predictions of the DummyRegressor at all. Which makes sense I guess.\n\nThe exact code:\n```python\n for i, (train_index, test_index) in enumerate(kf.split(X, groups=groups)):\n # Fit the PCA\n pca = PCA(3).fit(y_true[train_index])\n\n # Fit the model and predict all labels\n for l in range(len(labels)):\n m.fit(X[train_index], y_true[train_index, l])\n y_pred[test_index, l] = m.predict(X[test_index]).clip(0)\n y_pred_test[:, l] += m.predict(X_test).clip(0) / n_splits\n\n # Apply the PCA trick\n y_pred[test_index] = pca.inverse_transform(pca.transform(y_pred[test_index])).clip(0)\n y_pred_test = pca.inverse_transform(pca.transform(y_pred_test)).clip(0)\n```\n\nP.S. I don't claim that my approach is better, just wanted to share", "votes": 33 } ], "comments": [], "raw_markdown_by_id": { "3312536": "As said by [[LB 0.57]Infer + Model code](https://www.kaggle.com/code/none00000/lb-0-57-infer-model-code), the 5 targets are linearly dependent. There are only 3 independent components.\n\n## Approach 1: Predicting Key Components\nThe [[LB 0.57]Infer + Model code](https://www.kaggle.com/code/none00000/lb-0-57-infer-model-code) predicts only 3 components: Dry_Total_g, GDM_g and Dry_Green_g. The other two are calculated using subtraction. The reasoning to choose these specific components is that they have more weight in the metric.\n\n## My approach: the PCA trick\nIn [SigLip+GradientBoosting+PCA trick](https://www.kaggle.com/code/hocop1/siglip-gradientboosting-pca-trick) I tried a different approach.\n1. Fit a 3-component PCA on the target values\n2. Predict all 5 targets\n3. Apply the PCA to the prediction, and then apply the inverse transform\n\nThis 5->3->5 bottleneck addresses for the inconsistency of model's predictions. It gives around 0.01 uplift to the CV and LB scores.\n\nI believe it doesn't hurt the Dry_Total_g (top weight in the score) because it is a sum of 3 other predictions. It makes sense to me to also correct the sum, once we predict all components independently.\n\nNotably, this trick does not affect the predictions of the DummyRegressor at all. Which makes sense I guess.\n\nThe exact code:\n```python\n for i, (train_index, test_index) in enumerate(kf.split(X, groups=groups)):\n # Fit the PCA\n pca = PCA(3).fit(y_true[train_index])\n\n # Fit the model and predict all labels\n for l in range(len(labels)):\n m.fit(X[train_index], y_true[train_index, l])\n y_pred[test_index, l] = m.predict(X[test_index]).clip(0)\n y_pred_test[:, l] += m.predict(X_test).clip(0) / n_splits\n\n # Apply the PCA trick\n y_pred[test_index] = pca.inverse_transform(pca.transform(y_pred[test_index])).clip(0)\n y_pred_test = pca.inverse_transform(pca.transform(y_pred_test)).clip(0)\n```\n\nP.S. I don't claim that my approach is better, just wanted to share" } }