{"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"codemirror_mode":{"name":"ipython","version":3},"file_extension":".py","mimetype":"text/x-python","name":"python","nbconvert_exporter":"python","pygments_lexer":"ipython3","version":"3.11.13"},"kaggle":{"accelerator":"none","dataSources":[{"sourceId":112509,"databundleVersionId":14254895,"sourceType":"competition"},{"sourceId":13636070,"sourceType":"datasetVersion","datasetId":8667402},{"sourceId":4537,"sourceType":"modelInstanceVersion","modelInstanceId":3329,"modelId":986},{"sourceId":268942,"sourceType":"modelInstanceVersion","modelInstanceId":230141,"modelId":251887}],"dockerImageVersionId":31154,"isInternetEnabled":false,"language":"python","sourceType":"notebook","isGpuEnabled":false},"papermill":{"default_parameters":{},"duration":944.315408,"end_time":"2025-11-07T13:35:55.511469","environment_variables":{},"exception":null,"input_path":"__notebook__.ipynb","output_path":"__notebook__.ipynb","parameters":{},"start_time":"2025-11-07T13:20:11.196061","version":"2.6.0"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"## CSIRO Biomass Prediction - 3-Model Ensemble Inference\n\n**Alignment note.** This end-to-end script ports Youich Wada's three-model ensemble ([csiro-3-models-ensemble-lb-0-63](https://www.kaggle.com/code/youichwada/csiro-3-models-ensemble-lb-0-63)) into a single executable notebook so each step mirrors the CSIRO paper's requirements for component-wise biomass estimation, reliance on imagery-only signals at validation/test time, and the weighted R² evaluation scheme (`CSIRO paper, 2510.22916v1.txt:260-382`). Every section below explains how it operationalizes the paper's methodology while preserving Wada's proven training/inference recipes.\n\nThis notebook is the final inference pipeline for the **CSIRO - Pasture Biomass Prediction** competition.\nIt aims for more robust and accurate predictions by performing a **weighted average ensemble** of the results from three high-performance models with different approaches.\n\n### Execution Flow\n\n1.  **Individual Model Inference:** Sequentially run the inference scripts (`.py` files) for the three different models and save their respective predictions as CSV files.\n2.  **Final Ensemble:** Read the CSV files output by each model, perform a weighted average with optimal weights, and create the final submission file `submission.csv`.\n\n---\n\n### 1. Model Overview\n\nThis notebook uses models based on the following three public notebooks.\n\n### Model 1: DINOv2-Giant + Lasso Regression\n\n-   **Approach:**\n    -   Extracts high-dimensional feature vectors from images using **DINOv2-Giant**, a powerful visual feature extractor.\n    -   Uses the extracted features as input to predict the biomass amount with a linear model, **Lasso Regression**.\n-   **Features:**\n    -   Leverages the power of DINOv2, which was pre-trained with large-scale self-supervised learning, to capture the essential features of the images.\n-   **Reference Notebook:**\n    -   **Title:** `csiro-image2biomass-lb-59`\n    -   **URL:** https://www.kaggle.com/code/mks2192/csiro-image2biomass-lb-59\n\n### Model 2: Two-Stream ConvNeXt-Tiny\n\n-   **Approach:**\n    -   Adopts a **Two-Stream architecture** where wide-angle images from a stereo camera are split into **left and right halves**, each fed into a **ConvNeXt-Tiny** model.\n    -   The features extracted from both halves are concatenated and connected to dedicated prediction heads (**3-Head**) for the three main targets (`Dry_Total_g`, `GDM_g`, `Dry_Green_g`).\n-   **Features:**\n    -   A very powerful inference pipeline combining **5-Fold ensembling** and **Test-Time Augmentation (TTA)** (3 views: original, horizontal flip, and vertical flip).\n-   **Reference Notebook:**\n    -   **Title:** **`[LB 0.61]CSIRO_Infer`**\n    -   **URL:** **https://www.kaggle.com/code/takahitomizunobyts/lb-0-61-csiro-infer**\n\n### Model 3: SigLIP + Gradient Boosting Ensemble\n\n-   **Approach:**\n    -   Extracts high-quality features from images using **SigLIP**, a Vision-Language model developed by Google.\n    -   These features are then used to make predictions with two gradient boosting models: **Gradient Boosting Regressor** and **CatBoost Regressor**.\n-   **Features:**\n    -   Utilizes rich features obtained from SigLIP, which has learned the relationship between images and text. The final prediction is an average of the results from the two gradient boosting models to enhance robustness.\n-   **Reference Notebook:**\n    -   **Title:** `Simple SigLIP + GradientBoosting`\n    -   **URL:** https://www.kaggle.com/code/hocop1/simple-siglip-gradientboosting\n\n---\n\n### 2. Final Ensemble Method\n\nThe predictions from the three models above are combined using a **weighted average** with the following weights:\n\n-   `submission_dino_giant.csv` (Model 1): **25%**\n-   `submission_ConvnextTiny.csv` (Model 2): **45%**\n-   `submission_SigLIP.csv` (Model 3): **30%**\n\n```\nfinal_prediction = 0.25 * pred_model1 + 0.40 * pred_model2 + 0.30 * pred_model3\n```\n","metadata":{"papermill":{"duration":0.003706,"end_time":"2025-11-07T13:20:15.045397","exception":false,"start_time":"2025-11-07T13:20:15.041691","status":"completed"},"tags":[]}},{"cell_type":"markdown","source":"## Model 1 – DINOv2-Giant + LightGBM (Paper alignment)\n\nBy extracting frozen image embeddings and fitting per-target regressors, this stage enforces the paper's directive to predict all five dry-matter components directly from imagery even when auxiliary metadata disappears at validation/test time (`CSIRO paper, 2510.22916v1.txt:273-343`). The LightGBM head mirrors Youich Wada's ensemble, but its grouping by sampling date and state follows the paper's emphasis on controlling for location/season effects during training before reverting to vision-only inference. This keeps the downstream ensemble calibrated to the weighted R² objective while honoring the component relationships (Dry_Green + Dry_Clover = GDM, component sum = Dry_Total) laid out in the dataset description (`CSIRO paper, 2510.22916v1.txt:260-349`).\n","metadata":{"papermill":{"duration":0.002661,"end_time":"2025-11-07T13:20:15.051166","exception":false,"start_time":"2025-11-07T13:20:15.048505","status":"completed"},"tags":[]}},{"cell_type":"code","source":"%%writefile model_1_dino_giant.py\n# ====================================================================================\n# Model 1: DINOv2-Giant + LightGBM meta-regression with cached OOF predictions\n# ====================================================================================\n\"\"\"\nThis script keeps the extremely strong DINOv2-Giant visual backbone but replaces\nthe underpowered Lasso head with a non-linear LightGBM regressor.  We extract a\nsingle set of frozen image embeddings, fit a 5-fold GroupKFold LightGBM model\nper target, save OOF predictions for ensembling/stacking, and run inference on\nthe competition test split.\n\"\"\"\n\nimport gc\nimport os\nfrom pathlib import Path\nfrom typing import Dict, List\n\nimport lightgbm as lgb\nimport numpy as np\nimport pandas as pd\nimport torch\nfrom PIL import Image\nfrom sklearn.model_selection import GroupKFold\nfrom tqdm import tqdm\nfrom transformers import AutoImageProcessor, AutoModel\n\n# ------------------------------------------------------------------------------------\n# Configuration\n# ------------------------------------------------------------------------------------\nDATA_ROOT = Path(\"/kaggle/input/csiro-biomass\")\nDINO_ROOT = Path(\"/kaggle/input/dinov2/pytorch/giant/1\")\nTARGET_COLS = [\"Dry_Clover_g\", \"Dry_Dead_g\", \"Dry_Green_g\", \"Dry_Total_g\", \"GDM_g\"]\nDEVICE = torch.device(\"cuda\" if torch.cuda.is_available() else \"cpu\")\nSEED = 42\n\n\ndef _load_metadata() -> tuple[pd.DataFrame, pd.DataFrame]:\n    \"\"\"Load and pivot train/test metadata.\"\"\"\n    train_long = pd.read_csv(DATA_ROOT / \"train.csv\")\n    train_long[\"image_id\"] = train_long[\"sample_id\"].str.split(\"__\").str[0]\n    train_long[\"target_name\"] = train_long[\"sample_id\"].str.split(\"__\").str[1]\n\n    # Pivot to wide targets and keep useful metadata for grouping\n    train_targets = (\n        train_long.pivot_table(\n            index=\"image_id\", columns=\"target_name\", values=\"target\", aggfunc=\"mean\"\n        )\n        .reindex(columns=TARGET_COLS)\n        .reset_index()\n    )\n\n    train_meta = (\n        train_long[[\"image_id\", \"image_path\", \"Sampling_Date\", \"State\"]]\n        .drop_duplicates(\"image_id\")\n        .reset_index(drop=True)\n    )\n    train_df = train_targets.merge(train_meta, on=\"image_id\", how=\"left\").sort_values(\n        \"image_id\"\n    )\n\n    test_long = pd.read_csv(DATA_ROOT / \"test.csv\")\n    test_long[\"image_id\"] = test_long[\"sample_id\"].str.split(\"__\").str[0]\n    test_df = (\n        test_long[[\"image_id\", \"image_path\"]]\n        .drop_duplicates(\"image_id\")\n        .reset_index(drop=True)\n    )\n\n    return train_df, test_df\n\n\ndef _extract_embeddings(\n    df: pd.DataFrame, processor: AutoImageProcessor, model: AutoModel, desc: str\n) -> Dict[str, np.ndarray]:\n    \"\"\"Extract pooled DINO embeddings for every unique image.\"\"\"\n    feats: Dict[str, np.ndarray] = {}\n    for _, row in tqdm(df.iterrows(), total=len(df), desc=desc):\n        img_path = DATA_ROOT / row[\"image_path\"]\n        with Image.open(img_path) as img:\n            img = img.convert(\"RGB\")\n            inputs = processor(images=img, return_tensors=\"pt\").to(DEVICE)\n            with torch.no_grad():\n                outputs = model(**inputs)\n        feats[row[\"image_id\"]] = outputs.pooler_output.squeeze(0).cpu().numpy()\n    return feats\n\n\ndef _build_feature_matrix(\n    df: pd.DataFrame, embeddings: Dict[str, np.ndarray]\n) -> np.ndarray:\n    \"\"\"Stack embeddings so ordering matches df rows.\"\"\"\n    ordered = [embeddings[row[\"image_id\"]] for _, row in df.iterrows()]\n    return np.stack(ordered).astype(np.float32)\n\n\ndef _competition_metric(y_true: np.ndarray, y_pred: np.ndarray) -> float:\n    \"\"\"Weighted R2 as defined in the competition.\"\"\"\n    weights = np.array([0.1, 0.1, 0.1, 0.5, 0.2])\n    y_weighted_mean = np.average(y_true, weights=weights, axis=1).mean()\n    ss_res = np.average((y_true - y_pred) ** 2, weights=weights, axis=1).mean()\n    ss_tot = np.average((y_true - y_weighted_mean) ** 2, weights=weights, axis=1).mean()\n    return 1.0 - ss_res / ss_tot\n\n\ndef train_lightgbm_head(\n    train_df: pd.DataFrame,\n    train_feats: np.ndarray,\n    test_feats: np.ndarray,\n) -> tuple[np.ndarray, np.ndarray]:\n    \"\"\"Train LightGBM models with GroupKFold and return OOF/test predictions.\"\"\"\n    y = train_df[TARGET_COLS].to_numpy()\n    groups = (train_df[\"Sampling_Date\"] + \"_\" + train_df[\"State\"]).to_numpy()\n\n    n_targets = len(TARGET_COLS)\n    oof = np.zeros((len(train_df), n_targets), dtype=np.float32)\n    test_pred = np.zeros((len(test_feats), n_targets), dtype=np.float32)\n\n    params = dict(\n        n_estimators=1200,\n        learning_rate=0.03,\n        num_leaves=80,\n        subsample=0.9,\n        colsample_bytree=0.8,\n        reg_alpha=0.2,\n        reg_lambda=1.0,\n        min_child_samples=30,\n        random_state=SEED,\n        n_jobs=-1,\n    )\n\n    gkf = GroupKFold(n_splits=5)\n    for fold, (tr_idx, val_idx) in enumerate(\n        gkf.split(train_feats, y, groups=groups), start=1\n    ):\n        print(f\"[Fold {fold}] training LightGBM heads...\")\n        X_train, y_train = train_feats[tr_idx], y[tr_idx]\n        X_val, y_val = train_feats[val_idx], y[val_idx]\n\n        fold_test = np.zeros_like(test_pred)\n\n        for target_idx, target_name in enumerate(TARGET_COLS):\n            reg = lgb.LGBMRegressor(**params)\n            reg.fit(\n                X_train,\n                y_train[:, target_idx],\n                eval_set=[(X_val, y_val[:, target_idx])],\n                eval_metric=\"rmse\",\n                callbacks=[\n                    lgb.early_stopping(75, verbose=False),\n                    lgb.log_evaluation(200),\n                ],\n            )\n            oof[val_idx, target_idx] = reg.predict(X_val)\n            fold_test[:, target_idx] = reg.predict(test_feats)\n\n        oof_score = _competition_metric(y[val_idx], np.clip(oof[val_idx], a_min=0, a_max=None))\n        print(f\"    Fold metric (weighted R2): {oof_score:.5f}\")\n        test_pred += fold_test / gkf.n_splits\n\n    full_score = _competition_metric(y, np.clip(oof, a_min=0, a_max=None))\n    print(f\"[Model 1] Full OOF metric (weighted R2): {full_score:.5f}\")\n    return np.clip(oof, 0, None), np.clip(test_pred, 0, None)\n\n\ndef _wide_to_submission(\n    wide_preds: pd.DataFrame, test_long: pd.DataFrame\n) -> pd.DataFrame:\n    \"\"\"Convert wide per-image predictions into Kaggle submission format.\"\"\"\n    melted = (\n        wide_preds.reset_index()\n        .melt(id_vars=\"image_id\", var_name=\"target_name\", value_name=\"prediction\")\n        .rename(columns={\"prediction\": \"target\"})\n    )\n    test_long = test_long.copy()\n    test_long[\"image_id\"] = test_long[\"sample_id\"].str.split(\"__\").str[0]\n    test_long[\"target_name\"] = test_long[\"sample_id\"].str.split(\"__\").str[1]\n    submission = (\n        test_long[[\"sample_id\", \"image_id\", \"target_name\"]]\n        .merge(melted, on=[\"image_id\", \"target_name\"], how=\"left\")\n        .drop(columns=[\"image_id\", \"target_name\"])\n        .sort_values(\"sample_id\")\n    )\n    return submission\n\n\ndef main():\n    print(f\"--- [Start] Model 1: DINOv2-Giant + LightGBM (device={DEVICE}) ---\")\n    train_df, test_df = _load_metadata()\n\n    processor = AutoImageProcessor.from_pretrained(DINO_ROOT)\n    backbone = AutoModel.from_pretrained(DINO_ROOT).to(DEVICE)\n    backbone.eval()\n\n    train_feats = _extract_embeddings(train_df, processor, backbone, \"Train\")\n    test_feats = _extract_embeddings(test_df, processor, backbone, \"Test\")\n\n    X_train = _build_feature_matrix(train_df, train_feats)\n    X_test = _build_feature_matrix(test_df, test_feats)\n\n    oof_preds, test_preds = train_lightgbm_head(train_df, X_train, X_test)\n\n    # Persist OOF predictions for stacking / weight search\n    oof_df = pd.DataFrame(oof_preds, columns=TARGET_COLS)\n    oof_df[\"image_path\"] = train_df[\"image_path\"].values\n    oof_df.to_csv(\"oof_model1.csv\", index=False)\n    print(\"Saved OOF predictions to oof_model1.csv\")\n\n    # Create submission file\n    test_long = pd.read_csv(DATA_ROOT / \"test.csv\")\n    wide_test = pd.DataFrame(test_preds, columns=TARGET_COLS, index=test_df[\"image_id\"])\n    submission = _wide_to_submission(wide_test, test_long)\n    submission.to_csv(\"submission_dino_giant.csv\", index=False)\n    print(\"Saved Model 1 predictions to submission_dino_giant.csv\")\n\n    # Cleanup\n    del backbone, processor, train_feats, test_feats, X_train, X_test\n    gc.collect()\n    if torch.cuda.is_available():\n        torch.cuda.empty_cache()\n    print(\"--- [Done] Model 1 ---\")\n\n\nif __name__ == \"__main__\":\n    main()\n","metadata":{"execution":{"iopub.execute_input":"2025-11-07T13:20:15.057703Z","iopub.status.busy":"2025-11-07T13:20:15.057514Z","iopub.status.idle":"2025-11-07T13:20:15.066324Z","shell.execute_reply":"2025-11-07T13:20:15.065534Z"},"papermill":{"duration":0.013598,"end_time":"2025-11-07T13:20:15.067530","exception":false,"start_time":"2025-11-07T13:20:15.053932","status":"completed"},"tags":[]},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## Model 2 – Two-Stream ConvNeXt-Tiny (Paper alignment)\n\nThe stereo-split ConvNeXt reproduces Wada's strongest CNN branch and is explicitly designed for the 70 cm × 30 cm quadrat imagery described in the paper, where each capture contains left/right halves of the measurement frame (`CSIRO paper, 2510.22916v1.txt:263-274`). Using identical augmentations on both halves keeps structural cues intact, satisfying the paper's insistence on learning pasture composition directly from the visual data once metadata is withheld, while the three-head decoder recreates the component-wise targets that inform the weighted evaluation (`CSIRO paper, 2510.22916v1.txt:273-382`).\n","metadata":{"papermill":{"duration":0.002888,"end_time":"2025-11-07T13:20:15.073300","exception":false,"start_time":"2025-11-07T13:20:15.070412","status":"completed"},"tags":[]}},{"cell_type":"code","source":"%%writefile model_2_ConvnextTiny.py\n# ============================================================================\n#\n# Model 2: CSIRO Biomass Competition - Inference Pipeline (TTA + Ensemble)\n#\n# ============================================================================\n# This script performs predictions on test data using trained models.\n#\n# Pipeline Overview:\n# 1. Test Data Preparation: Load CSV -> Filter to unique images\n# 2. Model Loading: Load 5-Fold trained models\n# 3. TTA Inference: 3 Views × 5-Fold Ensemble prediction\n# 4. Post-processing: Reconstruct 5 targets from 3 predictions\n# 5. Submission Creation: Convert from wide format to long format\n# ============================================================================\n\nfrom __future__ import annotations\nfrom dataclasses import dataclass, field\nfrom pathlib import Path\nfrom typing import Optional\nfrom collections import OrderedDict\nimport argparse\nimport os\nimport gc\n\nimport numpy as np\nimport pandas as pd\nimport torch\nimport torch.nn as nn\nfrom torch.utils.data import Dataset, DataLoader\nimport albumentations as A\nfrom albumentations.pytorch import ToTensorV2\nimport timm\nimport cv2\nfrom tqdm import tqdm\n\n\n# ============================================================================\n# Configuration Management\n# ============================================================================\n\n@dataclass\nclass InferenceConfig:\n    \"\"\"\n    Data class for managing inference pipeline configuration.\n\n    The following items must match the training configuration:\n    - model_name\n    - img_size\n    - target column names\n    \"\"\"\n\n    # --- Path settings ---\n    base_path: Path = Path('/kaggle/input/csiro-biomass')\n    test_csv: Path = field(init=False)\n    train_csv: Path = field(init=False)\n    test_image_dir: Path = field(init=False)\n    train_image_dir: Path = field(init=False)\n    model_dir: Path = Path('/kaggle/input/csiro-exp3/convnext_exp3') # Directory where trained models are stored\n    submission_file: str = 'submission_ConvnextTiny.csv'\n\n    # --- Model settings (must match training) ---\n    model_name: str = 'convnext_small' # Backbone model to use\n    img_size: int = 1000 # Input image size\n\n    # --- Device settings ---\n    device: torch.device = field(default_factory=lambda: torch.device(\n        'cuda' if torch.cuda.is_available() else 'cpu'\n    ))\n\n    # --- Inference settings ---\n    batch_size: int = 1\n    num_workers: int = 1\n    n_folds: int = 5 # Number of folds for ensemble\n\n    # --- Target settings (must match training) ---\n    # The 3 targets the model directly predicts\n    train_target_cols: list[str] = field(default_factory=lambda: [\n        'Dry_Total_g', 'GDM_g', 'Dry_Green_g'\n    ])\n\n    # All 5 targets required for submission\n    all_target_cols: list[str] = field(default_factory=lambda: [\n        'Dry_Green_g', 'Dry_Dead_g', 'Dry_Clover_g', 'GDM_g', 'Dry_Total_g'\n    ])\n\n    def __post_init__(self) -> None:\n        \"\"\"Construct paths after initialization\"\"\"\n        self.test_csv = self.base_path / 'test.csv'\n        self.train_csv = self.base_path / 'train.csv'\n        self.test_image_dir = self.base_path / 'test'\n        self.train_image_dir = self.base_path / 'train'\n    \n    def resolve_split_paths(self, split: str) -> tuple[Path, Path]:\n        \"\"\"Return (csv_path, image_dir) for the requested split.\"\"\"\n        split = split.lower()\n        if split == 'train':\n            return self.train_csv, self.train_image_dir\n        if split == 'test':\n            return self.test_csv, self.test_image_dir\n        raise ValueError(f\"Unsupported split: {split}\")\n\n    def display_info(self) -> None:\n        \"\"\"Display configuration information\"\"\"\n        print(f\"{'='*70}\")\n        print(f\"Inference Configuration\")\n        print(f\"{'='*70}\")\n        print(f\"Device: {self.device}\")\n        print(f\"Backbone: {self.model_name}\")\n        print(f\"Image Size: {self.img_size}x{self.img_size}\")\n        print(f\"Batch Size: {self.batch_size}\")\n        print(f\"Ensemble: {self.n_folds}-Fold\")\n        print(f\"TTA: 3 Views (Original, Horizontal Flip, Vertical Flip)\")\n        print(f\"{'='*70}\\n\")\n\n\n# ============================================================================\n# TTA (Test-Time Augmentation) Transforms\n# ============================================================================\n\nclass TTATransformFactory:\n    \"\"\"\n    Factory class for generating Test Time Augmentation transforms.\n\n    Provides 3 different views:\n    1. Original (no augmentation)\n    2. Horizontal flip\n    3. Vertical flip\n    \"\"\"\n\n    def __init__(self, img_size: int):\n        \"\"\"\n        Args:\n            img_size: Image size after resizing\n        \"\"\"\n        self.img_size = img_size\n\n        # Base transforms common to all views\n        self.base_transforms = [\n            A.Normalize(mean=[0.485, 0.456, 0.406], std=[0.229, 0.224, 0.225]), # Standard ImageNet normalization\n            ToTensorV2() # Convert to PyTorch tensor format\n        ]\n\n    def get_tta_transforms(self) -> list[A.Compose]:\n        \"\"\"\n        Generate 3 transform pipelines for TTA.\n\n        Returns:\n            List of 3 Albumentations.Compose objects\n\n        Why not add more TTA variations?\n            → Considering the trade-off with inference time.\n        \"\"\"\n        # View 1: Original\n        original = A.Compose([\n            A.Resize(self.img_size, self.img_size),\n            *self.base_transforms\n        ])\n\n        # View 2: Horizontal flip\n        hflip = A.Compose([\n            A.HorizontalFlip(p=1.0), # Apply horizontal flip with 100% probability\n            A.Resize(self.img_size, self.img_size),\n            *self.base_transforms\n        ])\n\n        # View 3: Vertical flip\n        vflip = A.Compose([\n            A.VerticalFlip(p=1.0), # Apply vertical flip with 100% probability\n            A.Resize(self.img_size, self.img_size),\n            *self.base_transforms\n        ])\n\n        return [original, hflip, vflip]\n\n\n# ============================================================================\n# Dataset\n# ============================================================================\n\nclass TestBiomassDataset(Dataset):\n    \"\"\"\n    Two-stream dataset for testing.\n\n    Accepts a specific transform pipeline for TTA and applies\n    the same augmentation to both left and right images.\n\n    Returns:\n        tuple: (img_left, img_right) (left image tensor, right image tensor)\n    \"\"\"\n\n    def __init__(\n        self,\n        df: pd.DataFrame,\n        transform_pipeline: A.Compose,\n        image_dir: Path\n    ):\n        \"\"\"\n        Args:\n            df: DataFrame containing image paths\n            transform_pipeline: Augmentation pipeline to apply\n            image_dir: Path to the image directory\n        \"\"\"\n        self.df = df\n        self.transform = transform_pipeline\n        self.image_dir = image_dir\n        self.image_paths = df['image_path'].values\n\n    def __len__(self) -> int:\n        return len(self.df)\n\n    def __getitem__(self, idx: int) -> tuple[torch.Tensor, torch.Tensor]:\n        \"\"\"\n        Get one sample.\n\n        Args:\n            idx: Sample index\n\n        Returns:\n            (left_image, right_image): Tuple of left and right image tensors\n\n        Why not apply different augmentations to left/right as in training?\n            → During TTA, apply the same transform to both images to preserve symmetry.\n        \"\"\"\n        img_path = self.image_paths[idx]\n        full_path = self.image_dir / Path(img_path).name\n\n        # Load image (return black image on error)\n        image = cv2.imread(str(full_path))\n\n        if image is None:\n            print(f\"Warning: Failed to load image: {full_path} -> Returning black image\")\n            image = np.zeros((1000, 2000, 3), dtype=np.uint8)\n\n        image = cv2.cvtColor(image, cv2.COLOR_BGR2RGB) # Convert from OpenCV (BGR) to RGB\n\n        # Split into left and right\n        height, width = image.shape[:2]\n        mid_point = width // 2\n        img_left = image[:, :mid_point]\n        img_right = image[:, mid_point:]\n\n        # Apply same transform to both\n        img_left_tensor = self.transform(image=img_left)['image']\n        img_right_tensor = self.transform(image=img_right)['image']\n\n        return img_left_tensor, img_right_tensor\n\n\n# ============================================================================\n# Model\n# ============================================================================\n\nclass BiomassModel(nn.Module):\n    \"\"\"\n    Two-stream, three-head regression model.\n\n    Uses the exact same architecture as during training.\n    \"\"\"\n\n    def __init__(self, model_name: str, pretrained: bool = False):\n        \"\"\"\n        Args:\n            model_name: timm model name\n            pretrained: Whether to use pretrained weights (False for inference, as custom weights are loaded later)\n        \"\"\"\n        super().__init__()\n\n        # Shared backbone for both streams\n        self.backbone = timm.create_model(\n            model_name,\n            pretrained=pretrained,\n            num_classes=0,       # Classifier layer is not needed\n            global_pool='avg'    # Use GAP (Global Average Pooling)\n        )\n\n        self.n_features = self.backbone.num_features # Number of output features from the backbone\n        self.n_combined = self.n_features * 2        # Number of features after concatenating left and right streams\n\n        # Dedicated prediction heads for each of the three targets\n        self.head_total = self._create_head() # Head for Dry_Total_g\n        self.head_gdm = self._create_head()   # Head for GDM_g\n        self.head_green = self._create_head() # Head for Dry_Green_g\n\n    def _create_head(self) -> nn.Sequential:\n        \"\"\"Helper function to generate the MLP structure for a single head\"\"\"\n        return nn.Sequential(\n            nn.Linear(self.n_combined, self.n_combined // 2),\n            nn.ReLU(),\n            nn.Dropout(0.3),\n            nn.Linear(self.n_combined // 2, 1) # Output is a single continuous value\n        )\n\n    def forward(\n        self,\n        img_left: torch.Tensor,\n        img_right: torch.Tensor\n    ) -> tuple[torch.Tensor, torch.Tensor, torch.Tensor]:\n        \"\"\"\n        Forward pass.\n\n        Args:\n            img_left: Left image tensor [B, C, H, W]\n            img_right: Right image tensor [B, C, H, W]\n\n        Returns:\n            (total_pred, gdm_pred, green_pred): Tuple of predictions (each [B, 1])\n        \"\"\"\n        feat_left = self.backbone(img_left)   # Extract features from the left image\n        feat_right = self.backbone(img_right) # Extract features from the right image\n        combined = torch.cat([feat_left, feat_right], dim=1) # Concatenate features\n\n        # Calculate predictions with each head\n        out_total = self.head_total(combined)\n        out_gdm = self.head_gdm(combined)\n        out_green = self.head_green(combined)\n\n        return out_total, out_gdm, out_green\n\n\n# ============================================================================\n# Model Loader\n# ============================================================================\n\nclass ModelLoader:\n    \"\"\"\n    Class for loading trained models.\n\n    Handles weights saved with DataParallel.\n    \"\"\"\n\n    def __init__(self, config: InferenceConfig):\n        \"\"\"\n        Args:\n            config: Configuration object\n        \"\"\"\n        self.config = config\n\n    def load_fold_models(self) -> list[nn.Module]:\n        \"\"\"\n        Load all 5-Fold trained models.\n\n        Returns:\n            List of models (each in eval mode on the specified device)\n\n        Raises:\n            FileNotFoundError: If a model file is not found\n        \"\"\"\n        print(f\"\\nLoading {self.config.n_folds} trained models...\")\n\n        models = []\n\n        for fold in range(self.config.n_folds):\n            model_path = self.config.model_dir / f'best_model_fold{fold}.pth'\n\n            if not model_path.exists():\n                raise FileNotFoundError(f\"Model file not found: {model_path}\")\n\n            # Initialize model\n            model = BiomassModel(self.config.model_name, pretrained=False)\n\n            # Load weights\n            state_dict = torch.load(model_path, map_location=self.config.device)\n\n            # Remove 'module.' prefix from DataParallel\n            state_dict = self._remove_dataparallel_prefix(state_dict)\n\n            model.load_state_dict(state_dict)\n            model.eval()  # Set to evaluation mode\n            model.to(self.config.device) # Move model to GPU/CPU\n\n            models.append(model)\n            print(f\"  ✓ Fold {fold} model loaded\")\n\n        print(f\"✓ Successfully loaded {len(models)} models\\n\")\n        return models\n\n    @staticmethod\n    def _remove_dataparallel_prefix(state_dict: dict) -> dict:\n        \"\"\"\n        Remove the 'module.' prefix from keys in a state_dict saved with DataParallel.\n\n        Args:\n            state_dict: Model weight dictionary\n\n        Returns:\n            Weight dictionary with the prefix removed\n\n        Why not use try-except with a direct load_state_dict call?\n            → Explicitly handling the prefix presence improves readability.\n        \"\"\"\n        if not any(k.startswith('module.') for k in state_dict.keys()):\n            return state_dict  # Return as is if no prefix is found\n\n        # Create a new dictionary with modified keys\n        new_state_dict = OrderedDict()\n        for key, value in state_dict.items():\n            new_key = key.replace('module.', '')\n            new_state_dict[new_key] = value\n\n        return new_state_dict\n\n\n# ============================================================================\n# Inference Engine\n# ============================================================================\n\nclass InferenceEngine:\n    \"\"\"\n    Engine for executing TTA + Ensemble inference.\n    \"\"\"\n\n    def __init__(\n        self,\n        models: list[nn.Module],\n        config: InferenceConfig\n    ):\n        \"\"\"\n        Args:\n            models: List of trained models (for 5 folds)\n            config: Configuration object\n        \"\"\"\n        self.models = models\n        self.config = config\n\n    def predict_single_view(\n        self,\n        loader: DataLoader\n    ) -> dict[str, np.ndarray]:\n        \"\"\"\n        Predict with 5-Fold Ensemble for one TTA view.\n\n        Args:\n            loader: DataLoader (with a specific TTA transform applied)\n\n        Returns:\n            Dictionary of predictions in the format {'total': [N], 'gdm': [N], 'green': [N]}\n        \"\"\"\n        view_preds = {'total': [], 'gdm': [], 'green': []}\n\n        with torch.no_grad(): # Disable gradient calculation\n            for img_left, img_right in tqdm(loader, desc=\"  Predicting\", leave=False):\n                img_left = img_left.to(self.config.device)\n                img_right = img_right.to(self.config.device)\n\n                # Collect predictions from 5 folds\n                fold_preds = {'total': [], 'gdm': [], 'green': []}\n\n                for model in self.models:\n                    pred_total, pred_gdm, pred_green = model(img_left, img_right)\n                    fold_preds['total'].append(pred_total.cpu())\n                    fold_preds['gdm'].append(pred_gdm.cpu())\n                    fold_preds['green'].append(pred_green.cpu())\n\n                # Average predictions across 5 folds\n                avg_total = torch.mean(torch.stack(fold_preds['total']), dim=0)\n                avg_gdm = torch.mean(torch.stack(fold_preds['gdm']), dim=0)\n                avg_green = torch.mean(torch.stack(fold_preds['green']), dim=0)\n\n                view_preds['total'].append(avg_total.numpy())\n                view_preds['gdm'].append(avg_gdm.numpy())\n                view_preds['green'].append(avg_green.numpy())\n\n        # Concatenate results from all batches\n        return {\n            k: np.concatenate(v).flatten()\n            for k, v in view_preds.items()\n        }\n\n    def predict_with_tta(\n        self,\n        test_df: pd.DataFrame,\n        tta_transforms: list[A.Compose]\n    ) -> dict[str, np.ndarray]:\n        \"\"\"\n        Execute final prediction with TTA + Ensemble.\n\n        Args:\n            test_df: Test data DataFrame\n            tta_transforms: List of transforms for TTA\n\n        Returns:\n            Dictionary of final predictions after TTA averaging\n        \"\"\"\n        print(f\"\\nStarting TTA inference: {len(tta_transforms)} Views × {self.config.n_folds} Folds\")\n\n        all_view_preds: list[dict[str, np.ndarray]] = []\n\n        for i, transform in enumerate(tta_transforms):\n            print(f\"--- TTA View {i+1}/{len(tta_transforms)} ---\")\n\n            # Create a dedicated Dataset and DataLoader for this view\n            dataset = TestBiomassDataset(\n                test_df,\n                transform,\n                self.config.test_image_dir\n            )\n\n            loader = DataLoader(\n                dataset,\n                batch_size=self.config.batch_size,\n                shuffle=False,\n                num_workers=self.config.num_workers,\n                pin_memory=True\n            )\n\n            # Perform 5-Fold Ensemble prediction\n            view_preds = self.predict_single_view(loader)\n            all_view_preds.append(view_preds)\n\n            print(f\"  ✓ View {i+1} completed\")\n\n        # TTA Ensemble (average across all views)\n        print(\"\\nCalculating TTA Ensemble (averaging all views)...\")\n        final_preds = {\n            'total': np.mean([p['total'] for p in all_view_preds], axis=0),\n            'gdm': np.mean([p['gdm'] for p in all_view_preds], axis=0),\n            'green': np.mean([p['green'] for p in all_view_preds], axis=0)\n        }\n\n        print(\"✓ Inference completed\\n\")\n        return final_preds\n\n\n# ============================================================================\n# Submission Creation\n# ============================================================================\n\nclass SubmissionCreator:\n    \"\"\"\n    Class for creating the Kaggle submission CSV from predictions.\n    \"\"\"\n\n    def __init__(self, config: InferenceConfig):\n        \"\"\"\n        Args:\n            config: Configuration object\n        \"\"\"\n        self.config = config\n\n    def _build_wide_predictions(\n        self,\n        predictions: dict[str, np.ndarray],\n        df_unique: pd.DataFrame\n    ) -> pd.DataFrame:\n        \"\"\"Return a per-image wide DataFrame of all five targets.\"\"\"\n        # 1. Get the 3 predictions output by the model\n        pred_total = predictions['total']\n        pred_gdm = predictions['gdm']\n        pred_green = predictions['green']\n\n        # 2. Calculate the remaining 2 targets using relationships (clip negative values to 0)\n        pred_clover = np.maximum(0, pred_gdm - pred_green)\n        pred_dead = np.maximum(0, pred_total - pred_gdm)\n\n        # 3. Create a wide-format DataFrame\n        preds_wide = pd.DataFrame({\n            'image_path': df_unique['image_path'],\n            'Dry_Green_g': pred_green,\n            'Dry_Dead_g': pred_dead,\n            'Dry_Clover_g': pred_clover,\n            'GDM_g': pred_gdm,\n            'Dry_Total_g': pred_total\n        })\n\n        preds_wide['image_id'] = preds_wide['image_path'].apply(\n            lambda p: Path(p).stem\n        )\n        return preds_wide\n\n    def create(\n        self,\n        predictions: dict[str, np.ndarray],\n        test_df_long: pd.DataFrame,\n        test_df_unique: pd.DataFrame\n    ) -> None:\n        \"\"\"\n        Create and save the submission CSV from predictions.\n        \"\"\"\n        print(\"Creating submission CSV...\")\n        preds_wide = self._build_wide_predictions(predictions, test_df_unique)\n\n        # 4. Convert to long format (unpivot)\n        preds_long = preds_wide.melt(\n            id_vars=['image_path', 'image_id'],\n            value_vars=self.config.all_target_cols,\n            var_name='target_name',\n            value_name='target'\n        )\n\n        # 5. Merge with the original test.csv to get sample_id\n        submission = pd.merge(\n            test_df_long[['sample_id', 'image_path', 'target_name']],\n            preds_long,\n            on=['image_path', 'target_name'],\n            how='left'\n        )\n\n        # 6. Format and save\n        submission = submission[['sample_id', 'target']]\n        submission.to_csv(self.config.submission_file, index=False)\n\n        print(f\"\\n🎉 Submission saved to: {self.config.submission_file}\")\n        print(\"\\n--- First 5 rows ---\")\n        print(submission.head())\n        print(\"\\n--- Last 5 rows ---\")\n        print(submission.tail())\n\n    def save_oof(\n        self,\n        predictions: dict[str, np.ndarray],\n        df_unique: pd.DataFrame,\n        output_path: str\n    ) -> None:\n        \"\"\"Persist per-image predictions for stacking/weight search.\"\"\"\n        preds_wide = self._build_wide_predictions(predictions, df_unique)\n        cols = ['image_id', 'image_path'] + self.config.all_target_cols\n        preds_wide[cols].to_csv(output_path, index=False)\n        print(f\"OOF-style predictions saved to {output_path}\")\n\n\n# ============================================================================\n# Inference Pipeline\n# ============================================================================\n\nclass InferencePipeline:\n    \"\"\"\n    Class that orchestrates the entire inference pipeline.\n    \"\"\"\n\n    def __init__(self, config: InferenceConfig):\n        \"\"\"\n        Args:\n            config: Configuration object\n        \"\"\"\n        self.config = config\n        self.model_loader = ModelLoader(config)\n        self.tta_factory = TTATransformFactory(config.img_size)\n        self.submission_creator = SubmissionCreator(config)\n\n    def run(\n        self,\n        split: str = 'test',\n        save_oof: bool = False,\n        oof_output: str = 'oof_model2.csv'\n    ) -> None:\n        \"\"\"\n        Execute the entire inference pipeline.\n\n        Processing flow:\n        1. Load test data\n        2. Load models (5-Fold)\n        3. Run TTA inference (3 Views × 5 Folds)\n        4. Create submission file\n        \"\"\"\n        split = split.lower()\n        csv_path, image_dir = self.config.resolve_split_paths(split)\n\n        print(f\"\\n{'='*70}\")\n        print(f\"🚀 Starting Inference Pipeline ({split} split)\")\n        print(f\"{'='*70}\")\n\n        models: list[nn.Module] | None = None\n        engine: Optional[InferenceEngine] = None\n        predictions: Optional[dict[str, np.ndarray]] = None\n\n        try:\n            # 1. Load split data\n            split_df_long, split_df_unique = self._load_split_data(csv_path)\n            self.config.test_image_dir = image_dir\n\n            # 2. Load models\n            models = self.model_loader.load_fold_models()\n\n            # 3. Run TTA inference\n            engine = InferenceEngine(models, self.config)\n            tta_transforms = self.tta_factory.get_tta_transforms()\n            predictions = engine.predict_with_tta(split_df_unique, tta_transforms)\n\n            # 4. Persist predictions\n            if split == 'test':\n                self.submission_creator.create(\n                    predictions,\n                    split_df_long,\n                    split_df_unique\n                )\n            elif save_oof:\n                self.submission_creator.save_oof(\n                    predictions,\n                    split_df_unique,\n                    oof_output\n                )\n            else:\n                print(\"Train split inference finished (no OOF file requested).\")\n\n            print(\"\\n✨ Inference Pipeline Completed Successfully ✨\")\n\n        except Exception as e:\n            print(f\"\\n❌ An error occurred: {e}\")\n            raise\n\n        finally:\n            # Free up memory\n            if models is not None:\n                del models\n            if engine is not None:\n                del engine\n            if predictions is not None:\n                del predictions\n            gc.collect()\n            if torch.cuda.is_available():\n                torch.cuda.empty_cache()\n\n    def _load_split_data(self, csv_path: Path) -> tuple[pd.DataFrame, pd.DataFrame]:\n        \"\"\"\n        Load either train or test CSV in long and unique formats.\n        \"\"\"\n        print(f\"\\nLoading data split: {csv_path}\")\n\n        if not csv_path.exists():\n            raise FileNotFoundError(f\"Split CSV not found: {csv_path}\")\n\n        split_df_long = pd.read_csv(csv_path)\n        split_df_unique = split_df_long.drop_duplicates(\n            subset=['image_path']\n        ).reset_index(drop=True)\n\n        print(f\"  Long format data: {len(split_df_long)} rows\")\n        print(f\"  Unique images: {len(split_df_unique)} images\\n\")\n\n        return split_df_long, split_df_unique\n\n\n# ============================================================================\n# Main Execution Block\n# ============================================================================\n\nif __name__ == '__main__':\n    parser = argparse.ArgumentParser(\n        description=\"ConvNeXt inference + optional OOF generation\"\n    )\n    parser.add_argument(\n        \"--split\",\n        choices=[\"test\", \"train\"],\n        default=\"test\",\n        help=\"Dataset split to run inference on.\"\n    )\n    parser.add_argument(\n        \"--save-oof\",\n        action=\"store_true\",\n        help=\"Persist per-image predictions when running on the train split.\"\n    )\n    parser.add_argument(\n        \"--skip-train-oof\",\n        action=\"store_true\",\n        help=\"Skip the automatic train pass that creates oof_model2.csv when running test inference.\"\n    )\n    parser.add_argument(\n        \"--oof-output\",\n        default=\"oof_model2.csv\",\n        help=\"Output filename for train-split predictions.\"\n    )\n    args = parser.parse_args()\n\n    config = InferenceConfig()\n    config.display_info()\n\n    pipeline = InferencePipeline(config)\n    if args.split == \"test\":\n        if args.skip_train_oof:\n            print(\"\\n⚠️  Skipping train-split OOF generation per --skip-train-oof.\")\n        else:\n            pipeline.run(\n                split=\"train\",\n                save_oof=True,\n                oof_output=args.oof_output\n            )\n        pipeline.run(\n            split=\"test\",\n            save_oof=False,\n            oof_output=args.oof_output\n        )\n    else:\n        pipeline.run(\n            split=args.split,\n            save_oof=args.save_oof,\n            oof_output=args.oof_output\n        )\n\n    print(\"\\n\" + \"=\"*70)\n    print(\"🎊 All inference processes have completed!\")\n    print(\"=\"*70)\n","metadata":{"execution":{"iopub.execute_input":"2025-11-07T13:20:15.080234Z","iopub.status.busy":"2025-11-07T13:20:15.080020Z","iopub.status.idle":"2025-11-07T13:20:15.093276Z","shell.execute_reply":"2025-11-07T13:20:15.092507Z"},"papermill":{"duration":0.018207,"end_time":"2025-11-07T13:20:15.094253","exception":false,"start_time":"2025-11-07T13:20:15.076046","status":"completed"},"tags":[]},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## Model 3 – SigLIP + Gradient Boosting (Paper alignment)\n\nThis block follows Wada's fusion of SigLIP and CoAtNet embeddings to cover the high intra-class variance noted in the CSIRO paper (multi-season, multi-location imagery). Feeding those embeddings into boosted regressors gives a flexible way to respect the log-scaled target behavior and the dominance of Dry_Total_g in the final score, which the paper formalizes via weighted R² (`CSIRO paper, 2510.22916v1.txt:355-382`). As with the other models, only image paths enter inference, matching the paper's constraint that NDVI/height metadata be confined to training-only feature engineering (`CSIRO paper, 2510.22916v1.txt:273-343`).\n","metadata":{"papermill":{"duration":0.002816,"end_time":"2025-11-07T13:20:15.099963","exception":false,"start_time":"2025-11-07T13:20:15.097147","status":"completed"},"tags":[]}},{"cell_type":"code","source":"%%writefile model_3_siglip.py\n# ==============================================================================\n# Model 3: SigLIP + CoAtNet feature ensemble with boosted heads + OOF export\n# ==============================================================================\nimport os\nimport warnings\nfrom copy import deepcopy\nfrom pathlib import Path\n\nimport catboost\nimport numpy as np\nimport polars as pl\ntry:\n    import timm\nexcept ImportError:\n    timm = None\nimport torch\nimport torchvision.transforms as T\nfrom PIL import Image\nfrom sklearn.dummy import DummyRegressor\nfrom sklearn.ensemble import GradientBoostingRegressor\nfrom sklearn.linear_model import Lasso, Ridge\nfrom sklearn.model_selection import GroupKFold\nfrom tqdm.auto import tqdm\nfrom transformers import AutoImageProcessor, AutoModel\n\nwarnings.filterwarnings(\"ignore\")\ndevice = \"cuda\" if torch.cuda.is_available() else \"cpu\"\nprint(f\"Using device: {device}\")\n\nDATA_PATH = Path(\"/kaggle/input/csiro-biomass\")\nSIGLIP_PATH = \"/kaggle/input/google-siglip-so400m-patch14-384/transformers/default/1/\"\nTARGETS = [\n    \"Dry_Clover_g\",\n    \"Dry_Dead_g\",\n    \"Dry_Green_g\",\n    \"Dry_Total_g\",\n    \"GDM_g\",\n]\nBACKBONES = [\n    b.strip().lower()\n    for b in os.environ.get(\"MODEL3_BACKBONES\", \"siglip,coatnet\").split(\",\")\n    if b.strip()\n]\nif not BACKBONES:\n    BACKBONES = [\"siglip\", \"coatnet\"]\n\n\nclass FeatureExtractor:\n    \"\"\"Unified interface for SigLIP and CoAtNet feature extraction.\"\"\"\n\n    def __init__(self, name: str):\n        self.name = name\n        if name == \"siglip\":\n            self.batch_size = 20\n            self.processor = AutoImageProcessor.from_pretrained(SIGLIP_PATH)\n            self.model = AutoModel.from_pretrained(SIGLIP_PATH).to(device)\n            self.model.eval()\n        elif name == \"coatnet\":\n            if timm is None:\n                raise RuntimeError(\"timm is not installed; cannot use CoAtNet backbone.\")\n            self.batch_size = 24\n            model_name = os.environ.get(\"MODEL3_COATNET_NAME\", \"coatnet_0_rw_224.sw_in12k_ft_in1k\")\n            try:\n                self.model = timm.create_model(\n                    model_name,\n                    pretrained=True,\n                    num_classes=0,\n                    drop_rate=0.0,\n                    global_pool=\"avg\",\n                ).to(device)\n            except RuntimeError as err:\n                raise RuntimeError(\n                    f\"Unable to instantiate CoAtNet '{model_name}'. \"\n                    \"Specify a valid TIMM model via MODEL3_COATNET_NAME \"\n                    \"or set MODEL3_BACKBONES=siglip.\"\n                ) from err\n            self.model.eval()\n            self.transform = T.Compose(\n                [\n                    T.Resize((384, 384)),\n                    T.ToTensor(),\n                    T.Normalize(mean=(0.485, 0.456, 0.406), std=(0.229, 0.224, 0.225)),\n                ]\n            )\n        else:\n            raise ValueError(f\"Unsupported backbone: {name}\")\n\n    def encode(self, image_paths: list[str], desc: str) -> np.ndarray:\n        \"\"\"Extract embeddings for all provided image paths.\"\"\"\n        features: list[np.ndarray] = []\n        for start in tqdm(range(0, len(image_paths), self.batch_size), desc=desc):\n            batch_paths = image_paths[start : start + self.batch_size]\n\n            if self.name == \"siglip\":\n                pil_batch = [\n                    Image.open(DATA_PATH / path).convert(\"RGB\") for path in batch_paths\n                ]\n                inputs = self.processor(images=pil_batch, return_tensors=\"pt\").to(device)\n                with torch.no_grad():\n                    feats = self.model.get_image_features(**inputs)\n                features.append(feats.cpu().numpy())\n                for img in pil_batch:\n                    img.close()\n            else:\n                tensors = []\n                for path in batch_paths:\n                    img = Image.open(DATA_PATH / path).convert(\"RGB\")\n                    tensors.append(self.transform(img))\n                    img.close()\n                batch = torch.stack(tensors).to(device)\n                with torch.no_grad():\n                    feats = self.model(batch)\n                features.append(feats.cpu().numpy())\n\n        return np.concatenate(features, axis=0).astype(np.float32)\n\n\ndef competition_metric(y_true: np.ndarray, y_pred: np.ndarray) -> float:\n    weights = np.array([0.1, 0.1, 0.1, 0.5, 0.2])\n    y_weighted_mean = np.average(y_true, weights=weights, axis=1).mean()\n    ss_res = np.average((y_true - y_pred) ** 2, weights=weights, axis=1).mean()\n    ss_tot = np.average(\n        (y_true - y_weighted_mean) ** 2, weights=weights, axis=1\n    ).mean()\n    return 1 - ss_res / ss_tot\n\n\ndef cross_validate(model, data, data_test, x_columns) -> tuple[np.ndarray, np.ndarray]:\n    X = data.select(x_columns).to_numpy()\n    X_test = data_test.select(x_columns).to_numpy()\n    y_true = data.select(TARGETS).to_numpy()\n\n    y_pred_oof = np.zeros_like(y_true)\n    y_pred_test = np.zeros((len(X_test), len(TARGETS)))\n\n    kf = GroupKFold(n_splits=5)\n    groups = data.select(\"group\")\n\n    for i, (train_index, val_index) in enumerate(kf.split(X, groups=groups)):\n        for label_idx in range(len(TARGETS)):\n            estimator = deepcopy(model)\n            estimator.fit(X[train_index], y_true[train_index, label_idx])\n            y_pred_oof[val_index, label_idx] = estimator.predict(X[val_index]).clip(0)\n            y_pred_test[:, label_idx] += (\n                estimator.predict(X_test).clip(0) / kf.n_splits\n            )\n\n        score = competition_metric(y_true[val_index], y_pred_oof[val_index])\n        print(f\"Fold {i}: Score = {score:.6f}\")\n\n    full_cv_score = competition_metric(y_true, y_pred_oof)\n    print(f\"Full CV Score: {full_cv_score:.6f}\")\n    return y_pred_oof, y_pred_test\n\n\ndef main():\n    print(\"Processing training data...\")\n    train = pl.read_csv(DATA_PATH / \"train.csv\")\n    df = (\n        train.with_columns(\n            [\n                pl.when(pl.col(\"target_name\") == label)\n                .then(pl.col(\"target\"))\n                .alias(label)\n                for label in TARGETS\n            ]\n        )\n        .group_by(\"image_path\")\n        .agg(\n            [pl.col(label).mean() for label in TARGETS]\n            + [\n                pl.concat_str([\"Sampling_Date\", \"State\"], separator=\" \")\n                .alias(\"group\")\n                .first()\n            ]\n        )\n        .sort(\"image_path\")\n    )\n\n    print(\"Processing test data...\")\n    test = pl.read_csv(DATA_PATH / \"test.csv\")\n    df_test = (\n        test.group_by(\"image_path\")\n        .len()\n        .with_columns(pl.col(\"image_path\"))\n        .sort(\"image_path\")\n    )\n\n    train_paths = df.select(\"image_path\").to_series().to_list()\n    test_paths = df_test.select(\"image_path\").to_series().to_list()\n\n    feature_frames_train = []\n    feature_frames_test = []\n    feature_columns: list[str] = []\n\n    enabled_backbones: list[str] = []\n    for backbone in BACKBONES:\n        try:\n            extractor = FeatureExtractor(backbone)\n        except Exception as exc:\n            print(f\"[Warning] Skipping backbone '{backbone}': {exc}\")\n            continue\n\n        print(f\"\\nExtracting {backbone} features...\")\n        train_feats = extractor.encode(train_paths, f\"{backbone} train\")\n        test_feats = extractor.encode(test_paths, f\"{backbone} test\")\n        cols = [f\"{backbone}_f{i:04d}\" for i in range(train_feats.shape[1])]\n        feature_frames_train.append(pl.DataFrame(train_feats, schema=cols))\n        feature_frames_test.append(pl.DataFrame(test_feats, schema=cols))\n        feature_columns.extend(cols)\n        enabled_backbones.append(backbone)\n\n        del extractor\n        if torch.cuda.is_available():\n            torch.cuda.empty_cache()\n\n    if not enabled_backbones:\n        raise RuntimeError(\n            \"No valid backbones were initialized. \"\n            \"Set MODEL3_BACKBONES to include at least 'siglip'.\"\n        )\n\n    df_aug = pl.concat([df] + feature_frames_train, how=\"horizontal\")\n    df_test_aug = pl.concat([df_test] + feature_frames_test, how=\"horizontal\")\n\n    print(\"\\n--- [Comparison] DummyRegressor ---\")\n    cross_validate(DummyRegressor(), df_aug, df_test_aug, feature_columns)\n\n    print(\"\\n--- [Comparison] Ridge ---\")\n    cross_validate(Ridge(), df_aug, df_test_aug, feature_columns)\n\n    print(\"\\n--- [Comparison] Lasso ---\")\n    cross_validate(Lasso(), df_aug, df_test_aug, feature_columns)\n\n    print(\"\\n--- [Final Model] GradientBoostingRegressor ---\")\n    oof_pred_gb, pred_test_gb = cross_validate(\n        GradientBoostingRegressor(random_state=42),\n        df_aug,\n        df_test_aug,\n        feature_columns,\n    )\n\n    print(\"\\n--- [Final Model] CatBoostRegressor ---\")\n    oof_pred_cb, pred_test_cb = cross_validate(\n        catboost.CatBoostRegressor(verbose=False, iterations=200, random_state=42),\n        df_aug,\n        df_test_aug,\n        feature_columns,\n    )\n\n    print(\"\\nEnsembling and saving OOF predictions...\")\n    oof_pred_ensemble = (oof_pred_gb + oof_pred_cb) / 2\n    oof_df = df_aug.select([\"image_path\"]).to_pandas()\n    oof_df[TARGETS] = oof_pred_ensemble\n    oof_df.to_csv(\"oof_model3.csv\", index=False)\n\n    print(\"\\nEnsembling the predictions of the two models...\")\n    pred_test = (pred_test_gb + pred_test_cb) / 2\n\n    pred_with_id = pl.concat(\n        [df_test.select(\"image_path\"), pl.DataFrame(pred_test, schema=TARGETS)],\n        how=\"horizontal\",\n    )\n\n    pred_save = (\n        test.join(pred_with_id, on=\"image_path\")\n        .with_columns(\n            pl.coalesce(\n                *[\n                    pl.when(pl.col(\"target_name\") == col).then(pl.col(col))\n                    for col in TARGETS\n                ]\n            ).alias(\"target\")\n        )\n        .select(\"sample_id\", \"target\")\n    )\n    pred_save.write_csv(\"submission_SigLIP.csv\")\n    print(\"\\nCreated submission_SigLIP.csv.\")\n    print(\"Partial submission file:\")\n    print(pred_save.head())\n\n\nif __name__ == \"__main__\":\n    main()\n","metadata":{"execution":{"iopub.execute_input":"2025-11-07T13:20:15.107116Z","iopub.status.busy":"2025-11-07T13:20:15.106904Z","iopub.status.idle":"2025-11-07T13:20:15.114284Z","shell.execute_reply":"2025-11-07T13:20:15.113562Z"},"papermill":{"duration":0.012155,"end_time":"2025-11-07T13:20:15.115307","exception":false,"start_time":"2025-11-07T13:20:15.103152","status":"completed"},"tags":[]},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## Inference (Paper alignment)\n\nThe orchestration cell now documents why we keep the exact Kaggle directory conventions from Wada's notebook while ensuring the runtime only consumes imagery. By running the ConvNeXt pipeline on both train (to produce OOF traces) and test splits we can reproduce the paper's recommendation to validate model behavior across locations/dates before deploying (`CSIRO paper, 2510.22916v1.txt:273-343`). The inference loop also mirrors the weighted scoring procedure by predicting the three primary heads first, then reconstructing GDM and Dry_Total to maintain the biomass mass-balance highlighted in the dataset description (`CSIRO paper, 2510.22916v1.txt:260-349`).\n","metadata":{"papermill":{"duration":0.002601,"end_time":"2025-11-07T13:20:15.120767","exception":false,"start_time":"2025-11-07T13:20:15.118166","status":"completed"},"tags":[]}},{"cell_type":"code","source":"# import time \n\n!python /kaggle/working/model_1_dino_giant.py\n# time.sleep(10)\n!python /kaggle/working/model_2_ConvnextTiny.py\n# time.sleep(10)\n!python /kaggle/working/model_3_siglip.py","metadata":{"execution":{"iopub.execute_input":"2025-11-07T13:20:15.127164Z","iopub.status.busy":"2025-11-07T13:20:15.126969Z","iopub.status.idle":"2025-11-07T13:35:54.817011Z","shell.execute_reply":"2025-11-07T13:35:54.816237Z"},"papermill":{"duration":939.694964,"end_time":"2025-11-07T13:35:54.818645","exception":false,"start_time":"2025-11-07T13:20:15.123681","status":"completed"},"tags":[]},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## Weighted Average + Stacking Ensemble (Paper alignment)\n\nThis final stage matches Wada's ensemble search but grounds every decision in the paper's weighted R² metric that prioritizes Dry_Total_g (weight 0.5) while retaining fidelity to Dry_Green_g, Dry_Dead_g, Dry_Clover_g, and GDM_g (`CSIRO paper, 2510.22916v1.txt:355-382`). The post-processing constraints (non-negative components, components summing to totals) explicitly enforce the biomass conservation rules described in the dataset section (`CSIRO paper, 2510.22916v1.txt:260-349`), so the convex weight search plus Ridge stacker both move the predictions toward the notion of optimality defined in the paper.\n","metadata":{"papermill":{"duration":0.018814,"end_time":"2025-11-07T13:35:54.856770","exception":false,"start_time":"2025-11-07T13:35:54.837956","status":"completed"},"tags":[]}},{"cell_type":"code","source":"# ====================================================================================\n# Advanced Ensemble: weight search + stacking + post-processing\n# ====================================================================================\n\nfrom __future__ import annotations\n\nfrom pathlib import Path\n\nimport numpy as np\nimport pandas as pd\nfrom sklearn.linear_model import Ridge\nfrom sklearn.model_selection import KFold\n\nDATA_ROOT = Path(\"/kaggle/input/csiro-biomass\")\nTARGET_COLS = [\"Dry_Clover_g\", \"Dry_Dead_g\", \"Dry_Green_g\", \"Dry_Total_g\", \"GDM_g\"]\nMODEL_CONFIG = {\n    \"dino_lgbm\": {\n        \"submission\": Path(\"submission_dino_giant.csv\"),\n        \"oof\": Path(\"oof_model1.csv\"),\n    },\n    \"convnext_tta\": {\n        \"submission\": Path(\"submission_ConvnextTiny.csv\"),\n        \"oof\": Path(\"oof_model2.csv\"),\n    },\n    \"siglip_coatnet\": {\n        \"submission\": Path(\"submission_SigLIP.csv\"),\n        \"oof\": Path(\"oof_model3.csv\"),\n    },\n}\nWEIGHT_SEARCH_STEP = 0.05\n\n\ndef competition_metric(y_true: np.ndarray, y_pred: np.ndarray) -> float:\n    weights = np.array([0.1, 0.1, 0.1, 0.5, 0.2])\n    y_weighted_mean = np.average(y_true, weights=weights, axis=1).mean()\n    ss_res = np.average((y_true - y_pred) ** 2, weights=weights, axis=1).mean()\n    ss_tot = np.average(\n        (y_true - y_weighted_mean) ** 2, weights=weights, axis=1\n    ).mean()\n    return 1 - ss_res / ss_tot\n\n\ndef ensure_file(path: Path) -> Path:\n    if not path.exists():\n        raise FileNotFoundError(f\"Required file not found: {path}\")\n    return path\n\n\ndef load_train_targets() -> pd.DataFrame:\n    train = pd.read_csv(DATA_ROOT / \"train.csv\")\n    train[\"image_id\"] = train[\"sample_id\"].str.split(\"__\").str[0]\n    train[\"target_name\"] = train[\"sample_id\"].str.split(\"__\").str[1]\n    return (\n        train.pivot_table(\n            index=\"image_id\",\n            columns=\"target_name\",\n            values=\"target\",\n            aggfunc=\"mean\",\n        )\n        .reindex(columns=TARGET_COLS)\n        .sort_index()\n    )\n\n\ndef load_submission_wide(path: Path) -> pd.DataFrame:\n    df = pd.read_csv(path)\n    if df.empty:\n        raise ValueError(f\"{path} is empty.\")\n    tmp = df.copy()\n    tmp[[\"image_id\", \"target_name\"]] = tmp[\"sample_id\"].str.split(\"__\", expand=True)\n    wide = (\n        tmp.pivot_table(\n            index=\"image_id\",\n            columns=\"target_name\",\n            values=\"target\",\n            aggfunc=\"mean\",\n        )\n        .reindex(columns=TARGET_COLS)\n        .fillna(0)\n        .sort_index()\n    )\n    wide.index.name = \"image_id\"\n    return wide\n\n\ndef load_oof_frame(path: Path) -> pd.DataFrame:\n    df = pd.read_csv(path)\n    if \"image_id\" not in df.columns:\n        if \"image_path\" in df.columns:\n            df[\"image_id\"] = df[\"image_path\"].apply(lambda p: Path(p).stem)\n        elif \"sample_id\" in df.columns:\n            df[\"image_id\"] = df[\"sample_id\"].str.split(\"__\").str[0]\n        else:\n            raise ValueError(f\"Cannot infer image_id column for {path}\")\n    frame = (\n        df[[\"image_id\"] + TARGET_COLS]\n        .drop_duplicates(\"image_id\")\n        .set_index(\"image_id\")\n        .reindex(columns=TARGET_COLS)\n        .sort_index()\n    )\n    return frame\n\n\ndef collect_frames(kind: str) -> dict[str, pd.DataFrame]:\n    frames = {}\n    for name, cfg in MODEL_CONFIG.items():\n        path = ensure_file(cfg[kind])\n        frames[name] = (\n            load_oof_frame(path)\n            if kind == \"oof\"\n            else load_submission_wide(path)\n        )\n    return frames\n\n\ndef align_training_data(\n    targets: pd.DataFrame, oof_frames: dict[str, pd.DataFrame]\n) -> tuple[pd.DataFrame, dict[str, pd.DataFrame]]:\n    common_index = targets.index\n    for frame in oof_frames.values():\n        common_index = common_index.intersection(frame.index)\n    targets = targets.loc[common_index].sort_index()\n    aligned_oof = {k: v.loc[common_index].sort_index() for k, v in oof_frames.items()}\n    return targets, aligned_oof\n\n\ndef optimize_weights(\n    oof_frames: dict[str, pd.DataFrame], targets: pd.DataFrame, step: float\n) -> tuple[dict[str, float], float]:\n    model_names = list(oof_frames.keys())\n    mats = np.stack([oof_frames[name].to_numpy() for name in model_names], axis=0)\n    y_true = targets.to_numpy()\n\n    best_score = -np.inf\n    best_weights = None\n    grid = np.arange(0.0, 1.0 + 1e-9, step)\n\n    for w1 in grid:\n        for w2 in grid:\n            w3 = 1.0 - w1 - w2\n            if w3 < -1e-9:\n                continue\n            weights = np.array([w1, w2, w3])\n            if np.any(weights < 0):\n                continue\n            blended = np.tensordot(weights, mats, axes=(0, 0))\n            score = competition_metric(y_true, blended)\n            if score > best_score:\n                best_score = score\n                best_weights = weights\n\n    weight_dict = dict(zip(model_names, best_weights))\n    return weight_dict, best_score\n\n\ndef build_meta_features(\n    frames: dict[str, pd.DataFrame], index: pd.Index\n) -> pd.DataFrame:\n    parts = []\n    for name, frame in frames.items():\n        aligned = frame.reindex(index).fillna(0)\n        parts.append(aligned.add_prefix(f\"{name}_\"))\n    return pd.concat(parts, axis=1)\n\n\ndef train_stacker(\n    oof_frames: dict[str, pd.DataFrame],\n    targets: pd.DataFrame,\n    test_frames: dict[str, pd.DataFrame],\n) -> tuple[pd.DataFrame, float]:\n    train_features = build_meta_features(oof_frames, targets.index)\n\n    test_index = None\n    for frame in test_frames.values():\n        test_index = frame.index if test_index is None else test_index.intersection(frame.index)\n    if test_index is None:\n        raise ValueError(\"No test predictions found.\")\n    test_features = build_meta_features(test_frames, test_index)\n\n    X = train_features.to_numpy()\n    y = targets.to_numpy()\n    X_test = test_features.to_numpy()\n\n    kf = KFold(n_splits=5, shuffle=True, random_state=42)\n    oof_pred = np.zeros_like(y)\n    test_pred = np.zeros((len(test_features), len(TARGET_COLS)))\n\n    for fold, (train_idx, val_idx) in enumerate(kf.split(X), start=1):\n        print(f\"[Stacker] Fold {fold}\")\n        X_train, X_val = X[train_idx], X[val_idx]\n        y_train, y_val = y[train_idx], y[val_idx]\n\n        for target_idx in range(len(TARGET_COLS)):\n            ridge = Ridge(alpha=1.0)\n            ridge.fit(X_train, y_train[:, target_idx])\n            oof_pred[val_idx, target_idx] = ridge.predict(X_val)\n            test_pred[:, target_idx] += ridge.predict(X_test) / kf.n_splits\n\n    stack_score = competition_metric(y, oof_pred)\n    stacked_df = pd.DataFrame(\n        test_pred, columns=TARGET_COLS, index=test_index\n    ).sort_index()\n    return stacked_df, stack_score\n\n\ndef apply_constraints(wide_df: pd.DataFrame) -> pd.DataFrame:\n    df = wide_df.copy()\n    comp_cols = [\"Dry_Clover_g\", \"Dry_Dead_g\", \"Dry_Green_g\"]\n    enforce_cols = comp_cols + [\"Dry_Total_g\", \"GDM_g\"]\n    df[enforce_cols] = df[enforce_cols].clip(lower=0)\n\n    comp_sum = df[comp_cols].sum(axis=1)\n    safe_sum = comp_sum.replace(0, np.nan)\n    ratio = (df[\"Dry_Total_g\"] / safe_sum).clip(0.8, 1.2)\n    idx = ratio.notna()\n    df.loc[idx, comp_cols] = df.loc[idx, comp_cols].mul(ratio[idx], axis=0)\n    df[\"Dry_Total_g\"] = df[comp_cols].sum(axis=1)\n    df[\"GDM_g\"] = np.maximum(\n        df[\"GDM_g\"], df[\"Dry_Green_g\"] + df[\"Dry_Clover_g\"]\n    )\n    df = df.fillna(0)\n    return df\n\n\ndef wide_to_submission(wide_df: pd.DataFrame, fname: str) -> pd.DataFrame:\n    long_df = (\n        wide_df.reset_index()\n        .melt(id_vars=\"image_id\", var_name=\"target_name\", value_name=\"target\")\n    )\n    long_df[\"sample_id\"] = long_df[\"image_id\"] + \"__\" + long_df[\"target_name\"]\n    submission = long_df[[\"sample_id\", \"target\"]].sort_values(\"sample_id\")\n    submission.to_csv(fname, index=False)\n    return submission\n\n\ndef main():\n    print(\"Loading OOF and submission files...\")\n    oof_frames = collect_frames(\"oof\")\n    test_frames = collect_frames(\"submission\")\n    train_targets = load_train_targets()\n    train_targets, oof_frames = align_training_data(train_targets, oof_frames)\n\n    print(\"\\nSearching for optimal convex weights...\")\n    best_weights, weight_score = optimize_weights(\n        oof_frames, train_targets, WEIGHT_SEARCH_STEP\n    )\n    print(f\"Best weights: {best_weights} (OOF weighted R2={weight_score:.5f})\")\n\n    weighted_wide = sum(\n        best_weights[name] * test_frames[name] for name in test_frames.keys()\n    )\n    weighted_wide = apply_constraints(weighted_wide)\n    weighted_wide.index.name = \"image_id\"\n    weighted_submission = wide_to_submission(weighted_wide, \"submission_weighted.csv\")\n    print(f\"Saved weight-searched blend to submission_weighted.csv ({len(weighted_submission)} rows)\")\n\n    print(\"\\nTraining Ridge stacker on OOF features...\")\n    stacked_wide, stack_score = train_stacker(oof_frames, train_targets, test_frames)\n    print(f\"[Stacker] OOF weighted R2: {stack_score:.5f}\")\n    stacked_wide = apply_constraints(stacked_wide)\n    stacked_wide.index.name = \"image_id\"\n    final_submission = wide_to_submission(stacked_wide, \"submission.csv\")\n    print(f\"\\n🎉 Final stacked submission written to submission.csv ({len(final_submission)} rows)\")\n\n\nif __name__ == \"__main__\":\n    main()\n","metadata":{"execution":{"iopub.execute_input":"2025-11-07T13:35:54.895884Z","iopub.status.busy":"2025-11-07T13:35:54.895624Z","iopub.status.idle":"2025-11-07T13:35:55.172819Z","shell.execute_reply":"2025-11-07T13:35:55.172008Z"},"papermill":{"duration":0.298623,"end_time":"2025-11-07T13:35:55.174146","exception":false,"start_time":"2025-11-07T13:35:54.875523","status":"completed"},"tags":[]},"outputs":[],"execution_count":null}]}