{"cells":[{"cell_type":"markdown","metadata":{},"source":"## Config"},{"cell_type":"code","execution_count":1,"metadata":{},"outputs":[],"source":"import os"},{"cell_type":"code","execution_count":2,"metadata":{},"outputs":[],"source":"# 环境设置\n\nTRAIN = True  # submission时只用跑推理，设为 False\n\nDEBUG = os.environ.get('KAGGLE_KERNEL_RUN_TYPE', '') == 'Interactive'  # 交互式环境下会少跑一些epoch，用于快速跑通流程，方便调试\nLOCAL = os.environ.get('KAGGLE_KERNEL_RUN_TYPE', '') == ''  # 用于在本地开发，觉得代码ok后通过 `kaggle k push` 命令提交到 kaggle 平台\n\nif LOCAL:\n    os.environ['HF_ENDPOINT'] = 'https://hf-mirror.com'"},{"cell_type":"code","execution_count":3,"metadata":{},"outputs":[],"source":"# 训练设置\n\n## CV\nCV_STRATEGY = 'groupby_Sampling_Date'  # groupby_Sampling_Date\nNFOLD = 5\nKFOLD_SEED = 42\n\n## Model\nMODEL_NAME = 'efficientnet_b2'\n\n## Training Hyper Params\nLR = 1e-2"},{"cell_type":"markdown","metadata":{"execution":{"iopub.execute_input":"2025-11-21T05:07:31.762158Z","iopub.status.busy":"2025-11-21T05:07:31.760963Z","iopub.status.idle":"2025-11-21T05:07:31.766349Z","shell.execute_reply":"2025-11-21T05:07:31.765388Z","shell.execute_reply.started":"2025-11-21T05:07:31.762125Z"}},"source":"## Load Data"},{"cell_type":"markdown","metadata":{},"source":"### 数据解读\n\n#### 干物质（Dry Matter）\n牧草含有大量水分，其含水量会因天气、时间、生长阶段等因素而剧烈变化。为了得到一个稳定、可比较的衡量标准来评估牧草的真实营养价值和数量，农学家和农民会先将牧草样本烘干，去除所有水分，然后称量剩余部分的重量。这个重量就是“干物质”或“干燥生物量”。比赛中的所有指标都是基于这个“干”重来计算的。\n\n#### 1. Dry green vegetation (excluding clover) / 干燥的绿色植被（不包括三叶草）\n\n*   **这是什么？**\n    这主要是指牧场中正在生长的、绿色的禾本科牧草（如黑麦草、羊茅等）的干重。它代表了牧场中主要的、有活力的非豆科植物部分。\n*   **为什么重要？**\n    这是牲畜（如牛、羊）**最主要的能量来源**。这些绿色的草富含碳水化合物，为动物提供了日常活动和生长所需的基本能量。它的数量直接决定了牧场能养活多少动物。将其与三叶草分开测量，是因为它们的营养成分（特别是蛋白质含量）有显著差异。\n\n#### 2. Dry dead material / 干燥的枯死物质\n\n*   **这是什么？**\n    这是指牧场中已经枯黄、死亡的植物部分的干重。它可能是前一个季节留下的老草，或是因干旱、过度成熟而死亡的植物。\n*   **为什么重要？**\n    这部分是**低质量的饲料**。它的营养价值非常低（蛋白质和能量含量都很低），消化率差，而且口感不好，牲畜通常会尽量避免采食。\n    *   **指示作用**：如果这部分占比较高，说明牧场管理可能存在问题（例如，放牧不及时导致牧草长老、枯死），或者牧场健康状况不佳。它会“稀释”优质饲料的比例，降低牲畜的采食效率。\n    *   **生态影响**：过多的枯死物质会覆盖在地面，阻碍阳光照射，抑制新草的生长。\n\n#### 3. Dry clover biomass / 干燥的三叶草生物量\n\n*   **这是什么？**\n    这是指牧场中所有三叶草（Clover）或其他豆科植物（如苜蓿）的干重。\n*   **为什么重要？**\n    三叶草是牧场中的“**超级食物**”。\n    *   **高蛋白质**：与禾本科牧草相比，三叶草的蛋白质含量要高得多。蛋白质是动物增重、产奶和维持健康的关键。因此，三叶草的含量直接关系到饲料的“质量”。\n    *   **天然肥料**：三叶草具有“固氮”能力，能将空气中的氮气转化为土壤中的氮肥，为周围的禾本科牧草提供天然养分，从而提高整个牧场的生产力并减少对化肥的依赖。\n\n#### 4. Green dry matter (GDM) / 绿色干物质\n\n*   **这是什么？**\n    这是 **(干燥的绿色植被) + (干燥的三叶草生物量)** 的总和。简单来说，它代表了牧场中所有**有生命的、绿色的植物**的总干重。\n*   **为什么重要？**\n    这是评估牧场**当前可用优质饲料总量**的核心指标。当农民决定一个牧区（paddock）可以放养多少头牛、能放养多少天时，他们最关心的就是GDM。它直接反映了牧场的“承载能力”（Carrying Capacity）。这个数值越高，意味着可供牲畜采食的优质饲料越多。\n\n#### 5. Total dry biomass / 总干燥生物量\n\n*   **这是什么？**\n    这是牧场中所有地上部分生物量的总和，即 **(绿色植被) + (三叶草) + (枯死物质)** 的总干重。\n*   **为什么重要？**\n    这个指标反映了牧场上**所有植物物质的总量**。通过比较“总干燥生物量”和“绿色干物质（GDM）”，农民可以快速了解牧场的健康状况。\n    *   **健康指标**：如果“总干燥生物量”很高，但“绿色干物质”占比很低，说明牧场里堆积了大量无用的枯死物质，需要进行管理（如通过短期重度放牧清理，或用机械割除）。\n    *   **长期规划**：这个数据有助于了解牧场的季节性生长周期和整体生产力。"},{"cell_type":"code","execution_count":4,"metadata":{"execution":{"iopub.execute_input":"2025-11-26T05:13:24.197503Z","iopub.status.busy":"2025-11-26T05:13:24.197001Z","iopub.status.idle":"2025-11-26T05:13:24.888291Z","shell.execute_reply":"2025-11-26T05:13:24.887702Z","shell.execute_reply.started":"2025-11-26T05:13:24.19748Z"}},"outputs":[],"source":"from PIL import Image\nimport pandas as pd\nimport numpy as np\nimport matplotlib.pyplot as plt\n%matplotlib inline\n\nfrom sklearn.model_selection import KFold, GroupKFold, StratifiedGroupKFold\nfrom tqdm.auto import tqdm\ntqdm.pandas()"},{"cell_type":"code","execution_count":5,"metadata":{"execution":{"iopub.execute_input":"2025-11-26T05:13:24.889917Z","iopub.status.busy":"2025-11-26T05:13:24.889209Z","iopub.status.idle":"2025-11-26T05:13:24.893298Z","shell.execute_reply":"2025-11-26T05:13:24.892718Z","shell.execute_reply.started":"2025-11-26T05:13:24.889897Z"}},"outputs":[],"source":"DATA_ROOT = '../input/' if LOCAL else '/kaggle/input/csiro-biomass/'"},{"cell_type":"code","execution_count":6,"metadata":{"execution":{"iopub.execute_input":"2025-11-26T05:13:27.347148Z","iopub.status.busy":"2025-11-26T05:13:27.346841Z","iopub.status.idle":"2025-11-26T05:13:27.386982Z","shell.execute_reply":"2025-11-26T05:13:27.386384Z","shell.execute_reply.started":"2025-11-26T05:13:27.347127Z"}},"outputs":[],"source":"train_df = pd.read_csv(f'{DATA_ROOT}/train.csv')\ntrain_df.head()"},{"cell_type":"code","execution_count":7,"metadata":{"execution":{"iopub.execute_input":"2025-11-26T05:13:27.631939Z","iopub.status.busy":"2025-11-26T05:13:27.631688Z","iopub.status.idle":"2025-11-26T05:13:27.641364Z","shell.execute_reply":"2025-11-26T05:13:27.64078Z","shell.execute_reply.started":"2025-11-26T05:13:27.631919Z"}},"outputs":[],"source":"train_df[['sample_id_prefix', 'sample_id_suffix']] = train_df.sample_id.str.split('__', expand=True)"},{"cell_type":"code","execution_count":8,"metadata":{"execution":{"iopub.execute_input":"2025-11-26T05:13:28.169403Z","iopub.status.busy":"2025-11-26T05:13:28.168916Z","iopub.status.idle":"2025-11-26T05:13:28.174843Z","shell.execute_reply":"2025-11-26T05:13:28.174144Z","shell.execute_reply.started":"2025-11-26T05:13:28.169378Z"}},"outputs":[],"source":"(train_df.sample_id_suffix == train_df.target_name).all()"},{"cell_type":"code","execution_count":9,"metadata":{"execution":{"iopub.execute_input":"2025-11-26T05:13:29.483361Z","iopub.status.busy":"2025-11-26T05:13:29.483071Z","iopub.status.idle":"2025-11-26T05:13:51.252318Z","shell.execute_reply":"2025-11-26T05:13:51.251717Z","shell.execute_reply.started":"2025-11-26T05:13:29.483336Z"}},"outputs":[],"source":"cols = ['sample_id_prefix', 'image_path', 'Sampling_Date', 'State', 'Species', 'Pre_GSHH_NDVI', 'Height_Ave_cm']\nagg_train_df = train_df.groupby(cols).apply(lambda df: df.set_index('target_name').target)\nagg_train_df.reset_index(inplace=True)\nagg_train_df.columns.name = None\n\nagg_train_df['image'] = agg_train_df.image_path.progress_apply(\n    lambda path: Image.open(DATA_ROOT + path).convert('RGB')\n)\n\nagg_train_df.head()"},{"cell_type":"code","execution_count":10,"metadata":{"execution":{"iopub.execute_input":"2025-11-26T05:13:51.253733Z","iopub.status.busy":"2025-11-26T05:13:51.253494Z","iopub.status.idle":"2025-11-26T05:13:51.261665Z","shell.execute_reply":"2025-11-26T05:13:51.261045Z","shell.execute_reply.started":"2025-11-26T05:13:51.253715Z"}},"outputs":[],"source":"agg_train_df['image_size'] = agg_train_df.image.apply(lambda x: x.size)\nagg_train_df['image_size'].value_counts()"},{"cell_type":"code","execution_count":11,"metadata":{"execution":{"iopub.execute_input":"2025-11-26T05:13:51.262599Z","iopub.status.busy":"2025-11-26T05:13:51.262345Z","iopub.status.idle":"2025-11-26T05:13:51.292319Z","shell.execute_reply":"2025-11-26T05:13:51.29177Z","shell.execute_reply.started":"2025-11-26T05:13:51.262578Z"}},"outputs":[],"source":"np.isclose(\n    agg_train_df[['Dry_Green_g', 'Dry_Clover_g']].sum(axis=1),\n    agg_train_df['GDM_g'],\n    atol=1e-04\n).mean()"},{"cell_type":"code","execution_count":12,"metadata":{"execution":{"iopub.execute_input":"2025-11-26T05:13:51.294185Z","iopub.status.busy":"2025-11-26T05:13:51.293933Z","iopub.status.idle":"2025-11-26T05:13:51.30743Z","shell.execute_reply":"2025-11-26T05:13:51.306726Z","shell.execute_reply.started":"2025-11-26T05:13:51.29417Z"}},"outputs":[],"source":"np.isclose(\n    agg_train_df[['GDM_g', 'Dry_Dead_g']].sum(axis=1),\n    agg_train_df['Dry_Total_g'],\n    atol=1e-04\n).mean()"},{"cell_type":"code","execution_count":13,"metadata":{"execution":{"iopub.execute_input":"2025-11-26T05:13:51.30832Z","iopub.status.busy":"2025-11-26T05:13:51.308155Z","iopub.status.idle":"2025-11-26T05:13:51.888601Z","shell.execute_reply":"2025-11-26T05:13:51.887775Z","shell.execute_reply.started":"2025-11-26T05:13:51.308306Z"}},"outputs":[],"source":"plt.figure(figsize=(16, 4))\nplt.subplot(1, 3, 1)\nagg_train_df.Dry_Green_g.plot(kind='hist')\n_ = plt.title('Dry_Green_g')\n\nplt.subplot(1, 3, 2)\nagg_train_df.Dry_Clover_g.plot(kind='hist')\n_ = plt.title('Dry_Clover_g')\n\nplt.subplot(1, 3, 3)\nagg_train_df.Dry_Dead_g.plot(kind='hist')\n_ = plt.title('Dry_Dead_g')"},{"cell_type":"markdown","metadata":{},"source":"## CV Strategy"},{"cell_type":"markdown","metadata":{},"source":"It appears there is a significant gap between the local CV score and the public LB score. We need to figure out the distributional differences between the test set and training set in order to select an appropriate CV strategy. Here are some discussions regarding the distribution of the test set.\n\n1. [Is there data drift between the training data and the test data?](https://www.kaggle.com/competitions/csiro-biomass/discussion/613724)\n\n    From **Competition Host**:\n> The data split between the public and private sets is not completely random.\n> \n> We used data from various seasons and states throughout the year as the training set. For validation and testing, we included some data from overlapping time periods and locations, **while also incorporating data from non-overlapping time periods to evaluate the model's generalization ability**.\n\n2. [State and Species](https://www.kaggle.com/competitions/csiro-biomass/discussion/615003)\n\n    From **Competition Host**:\n> Hi there, images in the testing set contain the same State and Species\n\n3. [Are There Any New Species in the Test Dataset?](https://www.kaggle.com/competitions/csiro-biomass/discussion/614083)\n    From **Competition Host**:\n> Great question. The direct answer is that all the dominant species in the test set are the same as those in the training set.\n> \n> Please note that the \"Species\" column refers to the visually annotated dominant species in the images and does not include every species present (which would be practically impossible). Additionally, the \"Species\" column is only provided for the training set and is not available for the testing set (in case you didn't know it).\n\nIt appears the distribution differences between the test and training sets are **primarily reflected in the date**. Therefore, we could attempt using `Sampling_Date` as groups for K-fold cross-validation."},{"cell_type":"code","execution_count":14,"metadata":{},"outputs":[],"source":"agg_train_df['Sampling_Date_Month'] = agg_train_df.Sampling_Date.apply(lambda x: x.split('/')[1].strip())"},{"cell_type":"code","execution_count":15,"metadata":{},"outputs":[],"source":"agg_train_df = agg_train_df.sort_index().sample(frac=1.0, random_state=31).copy()  # shuffle"},{"cell_type":"code","execution_count":16,"metadata":{},"outputs":[],"source":"agg_train_df['idx'] = agg_train_df.index"},{"cell_type":"code","execution_count":17,"metadata":{"scrolled":true},"outputs":[],"source":"half_num = agg_train_df.shape[0] // 6\nhalf_num"},{"cell_type":"code","execution_count":18,"metadata":{},"outputs":[],"source":"head_df = agg_train_df.iloc[:half_num].reset_index(drop=True)\ntail_df = agg_train_df.iloc[half_num:].reset_index(drop=True)\nhead_df.shape[0], tail_df.shape[0], len(set(head_df.idx) | set(tail_df.idx))"},{"cell_type":"code","execution_count":19,"metadata":{},"outputs":[],"source":"agg_train_df['fold'] = None"},{"cell_type":"code","execution_count":20,"metadata":{"scrolled":true},"outputs":[],"source":"kfold = KFold(n_splits=NFOLD, shuffle=True, random_state=KFOLD_SEED)\nfor i, (trn_idx, val_idx) in enumerate(kfold.split(head_df.index, y=head_df.State, groups=head_df.Sampling_Date)):\n    ori_val_idx = head_df.loc[val_idx, 'idx']\n    agg_train_df.loc[ori_val_idx, 'fold'] = i"},{"cell_type":"code","execution_count":22,"metadata":{"scrolled":true},"outputs":[],"source":"kfold = StratifiedGroupKFold(n_splits=NFOLD, shuffle=True, random_state=KFOLD_SEED)\nfor i, (trn_idx, val_idx) in enumerate(kfold.split(tail_df.index, y=tail_df.State, groups=tail_df.Sampling_Date)):\n    ori_val_idx = tail_df.loc[val_idx, 'idx']\n    agg_train_df.loc[ori_val_idx, 'fold'] = i"},{"cell_type":"code","execution_count":23,"metadata":{},"outputs":[],"source":"agg_train_df.sort_index(inplace=True)"},{"cell_type":"code","execution_count":24,"metadata":{},"outputs":[],"source":"agg_train_df.fold.value_counts(dropna=False).sort_index()"},{"cell_type":"code","execution_count":25,"metadata":{},"outputs":[],"source":"for i in range(NFOLD):\n    trn_df = agg_train_df[agg_train_df.fold != i]\n    val_df = agg_train_df[agg_train_df.fold == i]\n    \n    flag = val_df.Sampling_Date.isin(trn_df.Sampling_Date)\n    print(f'trn({trn_df.shape[0]}) -> val({val_df.shape[0]}): {flag.mean()}')"},{"cell_type":"markdown","metadata":{},"source":"## DataLoader"},{"cell_type":"code","execution_count":26,"metadata":{"execution":{"iopub.execute_input":"2025-11-26T05:13:51.899647Z","iopub.status.busy":"2025-11-26T05:13:51.899382Z","iopub.status.idle":"2025-11-26T05:13:58.321093Z","shell.execute_reply":"2025-11-26T05:13:58.320344Z","shell.execute_reply.started":"2025-11-26T05:13:51.899623Z"}},"outputs":[],"source":"from torch.utils.data import Dataset, DataLoader\nfrom torchvision import transforms"},{"cell_type":"code","execution_count":27,"metadata":{"execution":{"iopub.execute_input":"2025-11-26T05:13:58.322393Z","iopub.status.busy":"2025-11-26T05:13:58.321963Z","iopub.status.idle":"2025-11-26T05:13:58.329794Z","shell.execute_reply":"2025-11-26T05:13:58.329054Z","shell.execute_reply.started":"2025-11-26T05:13:58.322368Z"}},"outputs":[],"source":"class RegressionDataset(Dataset):\n    def __init__(self, data, transform=None, vertical_split=True):\n        self.data = data\n        self.transform = transform\n        self.vertical_split = vertical_split\n\n    def __len__(self):\n        return self.data.shape[0]\n\n    def __getitem__(self, idx):\n        item = self.data.iloc[idx]\n        image = item.image\n        targets = [item['Dry_Green_g'], item['Dry_Clover_g'], item['Dry_Dead_g']]\n        \n        if self.vertical_split:\n            # 垂直均分成左右两张图片\n            width, height = image.size\n            mid_point = width // 2\n            left_image = image.crop((0, 0, mid_point, height))\n            right_image = image.crop((mid_point, 0, width, height))\n            \n            if self.transform:\n                left_image = self.transform(left_image)\n                right_image = self.transform(right_image)\n            \n            return left_image, right_image, targets\n        \n        else:\n            if self.transform:\n                image = self.transform(image)\n\n            return image, targets\n\n\ndef create_dataloader(data, target_image_size=(256, 256), batch_size=32, shuffle=True, aug=True, tta_transform=None):    \n    if aug:\n        transform = transforms.Compose([\n            transforms.Resize(target_image_size),\n            transforms.RandomHorizontalFlip(p=0.5),\n            transforms.RandomVerticalFlip(p=0.5),\n            transforms.RandomApply([transforms.RandomRotation([90, 90])], p=0.5),\n            transforms.ColorJitter(brightness=0.1, contrast=0.1, saturation=0.1, hue=0.05),\n            transforms.ToTensor(),\n            transforms.Normalize(mean=[0.485, 0.456, 0.406], std=[0.229, 0.224, 0.225])\n        ])\n    \n    else:\n        if tta_transform:\n            transform = transforms.Compose([\n                transforms.Resize(target_image_size),\n                tta_transform,\n                transforms.ToTensor(),\n                transforms.Normalize(mean=[0.485, 0.456, 0.406], std=[0.229, 0.224, 0.225])\n            ])\n        else:\n            transform = transforms.Compose([\n                transforms.Resize(target_image_size),\n\n                transforms.ToTensor(),\n                transforms.Normalize(mean=[0.485, 0.456, 0.406], std=[0.229, 0.224, 0.225])\n            ])\n\n    dataset = RegressionDataset(data, transform=transform)\n    print('dataset size:', len(dataset))\n    dataloader = DataLoader(dataset, batch_size=batch_size, shuffle=shuffle, num_workers=4)\n    return dataloader"},{"cell_type":"code","execution_count":28,"metadata":{},"outputs":[],"source":"def get_tta_dataloaders(data, target_image_size, batch_size):\n    res = []\n    for transform in [None, transforms.RandomHorizontalFlip(p=1.0), transforms.RandomVerticalFlip(p=1.0), transforms.RandomRotation([90, 90])]:\n        res.append(\n            create_dataloader(data, target_image_size, batch_size, shuffle=False, aug=False, tta_transform=transform)\n        )\n    return res"},{"cell_type":"markdown","metadata":{},"source":"## Model"},{"cell_type":"code","execution_count":29,"metadata":{"execution":{"iopub.execute_input":"2025-11-26T05:13:58.330975Z","iopub.status.busy":"2025-11-26T05:13:58.330716Z","iopub.status.idle":"2025-11-26T05:14:01.753392Z","shell.execute_reply":"2025-11-26T05:14:01.752597Z","shell.execute_reply.started":"2025-11-26T05:13:58.330953Z"}},"outputs":[],"source":"import timm\nimport torch\nimport torch.nn as nn"},{"cell_type":"code","execution_count":30,"metadata":{"execution":{"iopub.execute_input":"2025-11-26T05:14:01.760763Z","iopub.status.busy":"2025-11-26T05:14:01.760506Z","iopub.status.idle":"2025-11-26T05:14:03.816872Z","shell.execute_reply":"2025-11-26T05:14:03.816344Z","shell.execute_reply.started":"2025-11-26T05:14:01.760741Z"}},"outputs":[],"source":"model = timm.create_model(MODEL_NAME, pretrained=False, num_classes=3)"},{"cell_type":"code","execution_count":31,"metadata":{"execution":{"iopub.execute_input":"2025-11-26T05:14:03.817519Z","iopub.status.busy":"2025-11-26T05:14:03.817348Z","iopub.status.idle":"2025-11-26T05:14:03.822464Z","shell.execute_reply":"2025-11-26T05:14:03.821865Z","shell.execute_reply.started":"2025-11-26T05:14:03.817503Z"}},"outputs":[],"source":"model.pretrained_cfg"},{"cell_type":"code","execution_count":32,"metadata":{},"outputs":[],"source":"TARGET_IMAGE_SIZE = model.pretrained_cfg['input_size'][1:]"},{"cell_type":"code","execution_count":33,"metadata":{},"outputs":[],"source":"class FiLM(nn.Module):\n    def __init__(self, feat_dim):\n        super().__init__()\n        hidden = max(32, feat_dim // 2)\n        self.mlp = nn.Sequential(\n            nn.Linear(feat_dim, hidden), nn.ReLU(inplace=True), nn.Linear(hidden, feat_dim * 2)\n        )\n\n    def forward(self, context):\n        gamma_beta = self.mlp(context)\n        return torch.chunk(gamma_beta, 2, dim=1)"},{"cell_type":"code","execution_count":34,"metadata":{},"outputs":[],"source":"class MultiTargetRegressor(nn.Module):\n    def __init__(self, model_name, pretrained=True, num_classes=3, dropout=0.0, freeze_backbone=False):\n        super().__init__()\n        self.backbone = timm.create_model(model_name, pretrained=pretrained, num_classes=0, global_pool='avg')\n        \n        self.film = FiLM(self.backbone.num_features)\n        \n        self.dropout = nn.Dropout(dropout)\n        \n        def make_head():\n            return nn.Sequential(\n                nn.Linear(self.backbone.num_features * 2, 8),\n                nn.ReLU(inplace=True),\n                nn.Dropout(dropout),\n                nn.Linear(8, 1),\n            )\n\n        self.head_green = make_head()\n        self.head_clover = make_head()\n        self.head_dead = make_head()\n        \n        self.softplus = nn.Softplus(beta=1.0)\n    \n        if freeze_backbone:\n            for p in self.backbone.parameters():\n                p.requires_grad = False\n    \n    \n    def forward(self, left_img, right_img):\n        left_feat = self.backbone(left_img)\n        right_feat = self.backbone(right_img)\n        \n        context = (left_feat + right_feat) / 2\n        gamma, beta = self.film(context)\n        \n        left_feat_modulated = left_feat * (1 + gamma) + beta\n        right_feat_modulated = right_feat * (1 + gamma) + beta\n        \n        combined = torch.cat([left_feat_modulated, right_feat_modulated], dim=1)\n        \n        green = self.softplus(self.head_green(combined))    # Bx1\n        clover = self.softplus(self.head_clover(combined))  # Bx1\n        dead = self.softplus(self.head_dead(combined))   # Bx1\n    \n        logits = torch.cat([green, clover, dead], dim=1)  # Bx3 \n\n        return logits"},{"cell_type":"markdown","metadata":{},"source":"## Train"},{"cell_type":"code","execution_count":35,"metadata":{"execution":{"iopub.execute_input":"2025-11-26T05:14:03.82358Z","iopub.status.busy":"2025-11-26T05:14:03.823383Z","iopub.status.idle":"2025-11-26T05:14:03.844755Z","shell.execute_reply":"2025-11-26T05:14:03.844054Z","shell.execute_reply.started":"2025-11-26T05:14:03.823562Z"}},"outputs":[],"source":"import torch\nimport torch.nn as nn\nimport torch.optim as optim\nfrom torch.optim import AdamW\nfrom transformers import get_cosine_schedule_with_warmup"},{"cell_type":"code","execution_count":36,"metadata":{"execution":{"iopub.execute_input":"2025-11-26T05:14:03.861197Z","iopub.status.busy":"2025-11-26T05:14:03.86099Z","iopub.status.idle":"2025-11-26T05:14:03.875253Z","shell.execute_reply":"2025-11-26T05:14:03.874606Z","shell.execute_reply.started":"2025-11-26T05:14:03.861173Z"}},"outputs":[],"source":"# ======== Weighted R² ========\ndef weighted_r2_score(y_true: np.ndarray, y_pred: np.ndarray):\n    \"\"\"\n    y_true, y_pred: shape (N, 5): Green/Clover/Dead/GDM/Total\n    \"\"\"\n    weights = np.array([0.1, 0.1, 0.1, 0.2, 0.5])\n    r2_scores = []\n    for i in range(5):\n        y_t = y_true[:, i]\n        y_p = y_pred[:, i]\n        ss_res = np.sum((y_t - y_p) ** 2)\n        ss_tot = np.sum((y_t - np.mean(y_t)) ** 2)\n        r2 = 1 - ss_res / ss_tot if ss_tot > 0 else 0.0\n        r2_scores.append(r2)\n    r2_scores = np.array(r2_scores)\n    weighted_r2 = np.sum(r2_scores * weights) / np.sum(weights)\n    return weighted_r2, r2_scores"},{"cell_type":"code","execution_count":37,"metadata":{"execution":{"iopub.execute_input":"2025-11-26T05:14:03.876167Z","iopub.status.busy":"2025-11-26T05:14:03.875973Z","iopub.status.idle":"2025-11-26T05:14:03.893652Z","shell.execute_reply":"2025-11-26T05:14:03.892829Z","shell.execute_reply.started":"2025-11-26T05:14:03.876144Z"}},"outputs":[],"source":"def calc_metric(outputs, targets):\n    '''\n        outputs/targets: shape (N, 3): Green/Clover/Dead\n    '''\n    y_true = np.column_stack((\n        targets,\n        targets[:, :2].sum(axis=1),\n        targets.sum(axis=1),\n    ))\n    \n    y_pred = np.column_stack((\n        outputs,\n        outputs[:, :2].sum(axis=1),\n        outputs.sum(axis=1),\n    ))\n    \n    weighted_r2, r2_scores = weighted_r2_score(y_true, y_pred)\n    return weighted_r2, r2_scores"},{"cell_type":"code","execution_count":38,"metadata":{"execution":{"iopub.execute_input":"2025-11-26T05:14:03.894694Z","iopub.status.busy":"2025-11-26T05:14:03.894431Z","iopub.status.idle":"2025-11-26T05:14:03.908794Z","shell.execute_reply":"2025-11-26T05:14:03.908076Z","shell.execute_reply.started":"2025-11-26T05:14:03.894667Z"}},"outputs":[],"source":"def train_epoch(model, dataloader, criterion, optimizer, scheduler, device):\n    model.train()\n    total_loss = 0\n\n    for left_images, right_images, targets in dataloader:\n        left_images = left_images.to(device)\n        right_images = right_images.to(device)\n        targets = torch.stack(targets).T.float().to(device)\n\n        optimizer.zero_grad()\n        outputs = model(left_images, right_images)\n        loss = criterion(outputs, targets)\n        loss.backward()\n        optimizer.step()\n        scheduler.step()\n\n        total_loss += loss.item()\n\n    return total_loss / len(dataloader)\n\n\ndef validate(model, dataloader, criterion, device):\n    model.eval()\n    total_loss = 0\n    all_outputs = []\n    all_targets = []\n\n    with torch.no_grad():\n        for left_images, right_images, targets in dataloader:\n            left_images = left_images.to(device)\n            right_images = right_images.to(device)\n            targets = torch.stack(targets).T.float().to(device)\n\n            outputs = model(left_images, right_images)\n            loss = criterion(outputs, targets)\n            total_loss += loss.item()\n\n            all_outputs.append(outputs.detach().cpu())\n            all_targets.append(targets.detach().cpu())\n\n    outputs = torch.cat(all_outputs).numpy()\n    targets = torch.cat(all_targets).numpy()\n    \n    return total_loss / len(dataloader), outputs, targets\n    \n    # weighted_r2, r2_scores = calc_metric(outputs, targets)\n    # return total_loss / len(dataloader), weighted_r2, r2_scores\n\n\ndef tta_validate(model, dataloaders, criterion, device):\n    if not isinstance(dataloaders, list):\n        dataloaders = [dataloaders]\n    \n    all_loss = []\n    all_outputs = []\n    all_targets = []\n    for dataloader in dataloaders:\n        loss, outputs, targets = validate(model, dataloader, criterion, device)\n        all_loss.append(loss)\n        all_outputs.append(outputs)\n        all_targets.append(targets)\n    \n    avg_loss = np.mean(all_loss)\n    avg_outputs = np.mean(all_outputs, axis=0)\n    avg_targets = np.mean(all_targets, axis=0)\n    \n    weighted_r2, r2_scores = calc_metric(avg_outputs, avg_targets)\n    \n    return avg_loss, weighted_r2, r2_scores"},{"cell_type":"code","execution_count":39,"metadata":{"execution":{"iopub.execute_input":"2025-11-26T05:14:03.909865Z","iopub.status.busy":"2025-11-26T05:14:03.909607Z","iopub.status.idle":"2025-11-26T05:14:03.928121Z","shell.execute_reply":"2025-11-26T05:14:03.927467Z","shell.execute_reply.started":"2025-11-26T05:14:03.909844Z"}},"outputs":[],"source":"def train_fold(data, fold, batch_size=32, continue_training=False):\n    device = torch.device('cuda:0' if torch.cuda.is_available() else 'cpu')\n    \n    # Hyperparameters\n    # batch_size = 8\n    lr = LR\n    patience = 10\n    num_epochs = 5 if DEBUG else 100\n    warmup_ratio = 0.05\n\n    # data\n    train_loader = create_dataloader(data[data.fold != fold], TARGET_IMAGE_SIZE, batch_size, shuffle=True, aug=True)\n    val_loader = create_dataloader(data[data.fold == fold], TARGET_IMAGE_SIZE, batch_size, shuffle=False, aug=False)\n    # val_loaders = get_tta_dataloaders(data[data.fold == fold], TARGET_IMAGE_SIZE, batch_size)\n\n    # model, loss, optimizer\n    # model = timm.create_model(MODEL_NAME, pretrained=True, num_classes=3)\n    if continue_training:\n        model = MultiTargetRegressor(MODEL_NAME, pretrained=False, num_classes=3, freeze_backbone=False)\n        model_file = f'{OUTPUT_DIR}/best_mode_fold{fold}.pth'\n        model.load_state_dict(torch.load(model_file))\n        \n    else:\n        model = MultiTargetRegressor(MODEL_NAME, pretrained=True, num_classes=3, freeze_backbone=True)\n        \n    model.to(device)\n\n    criterion = nn.SmoothL1Loss()  # nn.MSELoss()\n    optimizer = AdamW(model.parameters(), lr=lr / 500 if continue_training else lr)\n\n    num_training_steps = num_epochs * len(train_loader)\n    warmup_steps = int(warmup_ratio * num_training_steps)\n    \n    # scheduler = optim.lr_scheduler.ReduceLROnPlateau(optimizer, mode='max', factor=0.5, patience=patience//2)\n    scheduler = get_cosine_schedule_with_warmup(optimizer, num_warmup_steps=warmup_steps, num_training_steps=num_training_steps)\n\n    # Training loop\n    history = []\n    best_score = -float('inf')\n    # best_loss = float('inf')\n    epochs_without_improvement = 0\n    \n    for epoch in range(num_epochs):\n        train_loss = train_epoch(model, train_loader, criterion, optimizer, scheduler, device)\n        # val_loss, weighted_r2, r2_scores = validate(model, val_loader, criterion, device)\n        val_loss, weighted_r2, r2_scores = tta_validate(model, val_loader, criterion, device)\n\n        if epoch % 10 == 0:\n            print(f\"Epoch [{epoch}/{num_epochs}]: Train Loss: {train_loss:.4f}, Val Loss: {val_loss:.4f}, weighted_r2: {weighted_r2:.4f}, lr: {optimizer.param_groups[0]['lr']}\")\n        \n        history.append({\n            'train_loss': train_loss,\n            'val_loss': val_loss,\n            'lr': optimizer.param_groups[0]['lr'],\n            'weighted_r2': weighted_r2,\n            'r2_scores': r2_scores,\n        })\n        \n        # 早停\n        if weighted_r2 > best_score:\n        # if val_loss < best_loss:\n            best_loss = val_loss\n            best_score = weighted_r2\n            epochs_without_improvement = 0\n            # 保存最佳模型\n            torch.save(model.state_dict(), f'{OUTPUT_DIR}/best_mode_fold{fold}.pth')\n        \n        else:\n            epochs_without_improvement += 1\n        \n        if epochs_without_improvement >= patience:\n            print(f\"早停: epoch={epoch}, {patience} 个 epoch 无改善\")\n            break\n\n    print(f\"\\nTraining completed. Best weighted_r2: {best_score:.4f}\")\n    \n    return history, best_score"},{"cell_type":"code","execution_count":40,"metadata":{"execution":{"iopub.execute_input":"2025-11-26T05:14:03.845773Z","iopub.status.busy":"2025-11-26T05:14:03.845478Z","iopub.status.idle":"2025-11-26T05:14:03.860248Z","shell.execute_reply":"2025-11-26T05:14:03.859597Z","shell.execute_reply.started":"2025-11-26T05:14:03.845756Z"}},"outputs":[],"source":"OUTPUT_DIR = 'trained_models/'\nif not os.path.exists(OUTPUT_DIR):\n    os.mkdir(OUTPUT_DIR)"},{"cell_type":"markdown","metadata":{},"source":"### Stage 1"},{"cell_type":"code","execution_count":41,"metadata":{"execution":{"iopub.execute_input":"2025-11-26T05:14:09.475734Z","iopub.status.busy":"2025-11-26T05:14:09.47535Z"},"scrolled":true},"outputs":[],"source":"if TRAIN:\n    all_best_score = []\n\n    for i in range(2 if DEBUG else NFOLD):\n        print(f'### fold={i}')\n        history, best_score = train_fold(agg_train_df, fold=i, batch_size=32, continue_training=False)\n        all_best_score.append(best_score)\n        history = pd.DataFrame(history)\n\n        history.to_json(\n            f'{OUTPUT_DIR}/history_fold{i}.jsonl',\n            orient='records',\n            lines=True,\n            force_ascii=False,\n        )\n\n        # plot\n        plt.figure(figsize=(16, 4))\n\n        plt.subplot(1, 3, 1)\n        plt.title('LR')\n        plt.plot(history.lr)\n\n        plt.subplot(1, 3, 2)\n        plt.title('Loss')\n        plt.plot(history.train_loss, label='train')\n        plt.plot(history.val_loss, label='val')\n        plt.legend()\n\n        plt.subplot(1, 3, 3)\n        plt.title('weighted_r2')\n        plt.plot(history.weighted_r2)\n        plt.show()\n    \n    print('Avg CV:', np.mean(all_best_score))"},{"cell_type":"markdown","metadata":{},"source":"### Stage2"},{"cell_type":"code","execution_count":42,"metadata":{"execution":{"iopub.execute_input":"2025-11-26T05:14:09.475734Z","iopub.status.busy":"2025-11-26T05:14:09.47535Z"},"scrolled":true},"outputs":[],"source":"if TRAIN:\n    all_best_score = []\n\n    for i in range(2 if DEBUG else NFOLD):\n        print(f'### fold={i}')\n        history, best_score = train_fold(agg_train_df, fold=i, batch_size=8, continue_training=True)\n        all_best_score.append(best_score)\n        history = pd.DataFrame(history)\n\n        history.to_json(\n            f'{OUTPUT_DIR}/history_fold{i}.jsonl',\n            orient='records',\n            lines=True,\n            force_ascii=False,\n        )\n\n        # plot\n        plt.figure(figsize=(16, 4))\n\n        plt.subplot(1, 3, 1)\n        plt.title('LR')\n        plt.plot(history.lr)\n\n        plt.subplot(1, 3, 2)\n        plt.title('Loss')\n        plt.plot(history.train_loss, label='train')\n        plt.plot(history.val_loss, label='val')\n        plt.legend()\n\n        plt.subplot(1, 3, 3)\n        plt.title('weighted_r2')\n        plt.plot(history.weighted_r2)\n        plt.show()\n    \n    print('Avg CV:', np.mean(all_best_score))"},{"cell_type":"markdown","metadata":{},"source":"## Inference"},{"cell_type":"code","execution_count":45,"metadata":{},"outputs":[],"source":"import os\nfrom pathlib import Path"},{"cell_type":"code","execution_count":46,"metadata":{},"outputs":[],"source":"def get_lastest_saved_models():\n    model_root = '/kaggle/input/csiro-simple-output/pytorch/default/'\n    \n    latest = 1\n    for version in os.listdir(model_root):\n        try:\n            version = int(version)\n        except:\n            continue\n\n        if version > latest:\n            latest = version\n\n    return f'{model_root}/{latest}/trained_models/'"},{"cell_type":"code","execution_count":47,"metadata":{},"outputs":[],"source":"SAVED_MODELS = './trained_models/' if TRAIN else get_lastest_saved_models()"},{"cell_type":"code","execution_count":50,"metadata":{},"outputs":[],"source":"def predict(model, dataloader, device):\n    model.to(device)\n    model.eval()\n\n    all_outputs = []\n    with torch.no_grad():\n        for left_images, right_images, targets in dataloader:\n            left_images = left_images.to(device)\n            right_images = right_images.to(device)\n    \n            outputs = model(left_images, right_images)\n            all_outputs.append(outputs.detach().cpu())\n    \n    outputs = torch.cat(all_outputs).numpy()\n    return outputs"},{"cell_type":"code","execution_count":51,"metadata":{},"outputs":[],"source":"def tta_predict(model, dataloaders, device):\n    all_outputs = []\n    for dataloader in dataloaders:\n        outputs = predict(model, dataloader, device)\n        all_outputs.append(outputs)\n    avg_outputs = np.mean(all_outputs, axis=0)\n    return avg_outputs"},{"cell_type":"code","execution_count":57,"metadata":{},"outputs":[],"source":"def kfold_predict(dataloaders):\n    device = torch.device('cuda:0' if torch.cuda.is_available() else 'cpu')\n    \n    all_preds = []\n    for model_file in Path(SAVED_MODELS).glob('*.pth'):\n        # model = timm.create_model(MODEL_NAME, pretrained=False, num_classes=3)\n        fold_idx = int(model_file.name.split('.')[0].split('fold')[1])\n        if fold_idx >= NFOLD: continue\n        print(model_file.name)\n        model = MultiTargetRegressor(MODEL_NAME, pretrained=False, num_classes=3)\n        model.load_state_dict(torch.load(model_file))\n\n        preds = tta_predict(model, dataloaders, device)\n        all_preds.append(preds)\n\n    avg_preds = np.mean(all_preds, axis=0)\n    return avg_preds"},{"cell_type":"code","execution_count":53,"metadata":{},"outputs":[],"source":"test_df = pd.read_csv(DATA_ROOT + 'test.csv')\n\ntest_df['target'] = 0.0\ntest_df[['sample_id_prefix', 'sample_id_suffix']] = test_df.sample_id.str.split('__', expand=True)"},{"cell_type":"code","execution_count":54,"metadata":{},"outputs":[],"source":"cols = ['sample_id_prefix', 'image_path']\nagg_test_df = test_df.groupby(cols).apply(lambda df: df.set_index('target_name').target)\nagg_test_df.reset_index(inplace=True)\nagg_test_df.columns.name = None\n\nagg_test_df['image'] = agg_test_df.image_path.progress_apply(\n    lambda path: Image.open(DATA_ROOT + path).convert('RGB')\n)\n\nagg_test_df.head()"},{"cell_type":"code","execution_count":55,"metadata":{},"outputs":[],"source":"test_loader = get_tta_dataloaders(agg_test_df, TARGET_IMAGE_SIZE, 64)"},{"cell_type":"code","execution_count":58,"metadata":{},"outputs":[],"source":"preds = kfold_predict(test_loader)"},{"cell_type":"code","execution_count":59,"metadata":{},"outputs":[],"source":"agg_test_df[['Dry_Green_g', 'Dry_Clover_g', 'Dry_Dead_g']] = preds\nagg_test_df['GDM_g'] = agg_test_df.Dry_Green_g + agg_test_df.Dry_Clover_g\nagg_test_df['Dry_Total_g'] = agg_test_df.GDM_g + agg_test_df.Dry_Dead_g"},{"cell_type":"code","execution_count":60,"metadata":{},"outputs":[],"source":"agg_test_df.head()"},{"cell_type":"code","execution_count":61,"metadata":{},"outputs":[],"source":"cols = ['Dry_Clover_g', 'Dry_Dead_g', 'Dry_Green_g', 'Dry_Total_g', 'GDM_g']\nsub_df = agg_test_df.set_index('sample_id_prefix')[cols].stack()\nsub_df = sub_df.reset_index()\nsub_df.columns = ['sample_id_prefix', 'target_name', 'target']\n\nsub_df['sample_id'] = sub_df.sample_id_prefix + '__' + sub_df.target_name"},{"cell_type":"code","execution_count":62,"metadata":{},"outputs":[],"source":"cols = ['sample_id', 'target']\nsub_df[cols].to_csv('submission.csv', index=False)"},{"cell_type":"code","execution_count":63,"metadata":{},"outputs":[],"source":"!head submission.csv"},{"cell_type":"code","execution_count":null,"metadata":{},"outputs":[],"source":""}],"metadata":{"kaggle":{"accelerator":"gpu","dataSources":[{"databundleVersionId":14254895,"sourceId":112509,"sourceType":"competition"}],"dockerImageVersionId":31193,"isGpuEnabled":true,"isInternetEnabled":true,"language":"python","sourceType":"notebook"},"kernelspec":{"display_name":"Python 3 (ipykernel)","language":"python","name":"python3"},"language_info":{"codemirror_mode":{"name":"ipython","version":3},"file_extension":".py","mimetype":"text/x-python","name":"python","nbconvert_exporter":"python","pygments_lexer":"ipython3","version":"3.8.15"}},"nbformat":4,"nbformat_minor":4}