{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"name":"python","version":"3.12.12","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"},"kaggle":{"accelerator":"nvidiaTeslaT4","dataSources":[{"sourceId":112509,"databundleVersionId":14254895,"sourceType":"competition"}],"dockerImageVersionId":31234,"isInternetEnabled":false,"language":"python","sourceType":"notebook","isGpuEnabled":true}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# CSIRO - Image2Biomass Prediciton\n\nHi! My name is Kayden Schwartz and this is my notebook working through the CSIRO - Image2Biomass Prediciton problem. I am a ML/software engineer with a background in environmental science, so this challenge is right up my alley.\n\nThrough this Notebook I will start with some EDA and data cleaning if necessary. This will allow me to understand data limitations and possible best handling for data transformations. Then I'll build a baseline model on the metadata, potentially using XGBoost or LightGBM. Next I'll use an image based model strategy, likely using EfficientNet, ResNet, or ConvNeXt to measure it's effectiveness. This way I can start to understand the relationships between the image information, tabular information and prediction power. Once I have a baseline, I'll think about ideal architecture for the model. Next, I'll do some data strategy considerations: image rotations/flips, color jittering, and random crops. Then, I'll attempt some feature engineering to see what kinds of improvements we can get. Lastly, I'll predict the test data and ship the model!","metadata":{}},{"cell_type":"markdown","source":"# EDA\nFirst I'll load in the packages I think I will want to have for EDA and load in the dataset.","metadata":{}},{"cell_type":"code","source":"# This Python 3 environment comes with many helpful analytics libraries installed\n# It is defined by the kaggle/python Docker image: https://github.com/kaggle/docker-python\n# Load in packages\n\nimport numpy as np\nimport pandas as pd\nimport matplotlib.pyplot as plt\nimport seaborn as sns\nfrom PIL import Image\nimport os\n\n# Running this will list all files under the input directory\n# import os\n# for dirname, _, filenames in os.walk('/kaggle/input'):\n#     for filename in filenames:\n#         print(os.path.join(dirname, filename))\n\n# You can write up to 20GB to the current directory (/kaggle/working/) that gets preserved as output when you create a version using \"Save & Run All\" \n# You can also write temporary files to /kaggle/temp/, but they won't be saved outside of the current session","metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true,"execution":{"iopub.status.busy":"2026-01-11T01:37:30.721903Z","iopub.execute_input":"2026-01-11T01:37:30.722122Z","iopub.status.idle":"2026-01-11T01:37:33.625588Z","shell.execute_reply.started":"2026-01-11T01:37:30.722096Z","shell.execute_reply":"2026-01-11T01:37:33.624911Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"df = pd.read_csv('/kaggle/input/csiro-biomass/train.csv')\nprint(\"Shape:\", df.shape)\nprint(\"\\nUnique images:\", df['image_path'].nunique())\nprint(\"Total rows:\", len(df))\nprint(\"Rows per image:\", len(df) / df['image_path'].nunique())\nprint(\"\\nTarget names:\", df['target_name'].unique())\nprint(\"\\nMissing values:\\n\", df.isnull().sum())","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2026-01-10T21:38:50.855788Z","iopub.execute_input":"2026-01-10T21:38:50.856301Z","iopub.status.idle":"2026-01-10T21:38:50.895343Z","shell.execute_reply.started":"2026-01-10T21:38:50.856274Z","shell.execute_reply":"2026-01-10T21:38:50.894483Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"Alright, let's break down the analysis here. We can tell our tabular data is 1785 rows by 9 columns. We have 357 unique images each with 5 rows, one for each target ('Dry_Clover_g' 'Dry_Dead_g' 'Dry_Green_g' 'Dry_Total_g' 'GDM_g'). Luckily we can also see there are NO missing values :)\n\nNext up, I'll take a quick look as the distribution of each target value separately.","metadata":{}},{"cell_type":"code","source":"# Separate targets and print summary stats\nfor target in df['target_name'].unique():\n    target_data = df[df['target_name'] == target]['target']\n\n    print(f\"\\n{target}:\")\n    print(f\" Mean: {target_data.mean():.2f}\")\n    print(f\" Median: {target_data.median():.2f}\")\n    print(f\" Std: {target_data.median():.2f}\")\n    print(f\" Min: {target_data.min():.2f}, Max: {target_data.max():.2f}\")\n    print(f\" Zeros: {(target_data == 0).sum()} ({(target_data == 0).sum()/len(target_data)*100:.1f}%)\")\n\n# Visualize distributions by target\nfig, axes = plt.subplots(2, 3, figsize=(15,8))\naxes = axes.flatten()\n\nfor idx, target in enumerate(df['target_name'].unique()):\n    target_data = df[df['target_name'] == target]['target']\n    axes[idx].hist(target_data, bins=50, edgecolor='black')\n    axes[idx].set_title(f'{target}\\nMean: {target_data.mean():.1f}')\n    axes[idx].set_xlabel('Biomass (g)')\n\nplt.tight_layout()\nplt.show()","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2026-01-10T21:38:53.254343Z","iopub.execute_input":"2026-01-10T21:38:53.254632Z","iopub.status.idle":"2026-01-10T21:38:54.224207Z","shell.execute_reply.started":"2026-01-10T21:38:53.254612Z","shell.execute_reply":"2026-01-10T21:38:54.223213Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"Let's breakdown what we see here. Some significant skewing for Dry_Clover_g, Dry_Dead_g, and Dry_Green_g. Dry_total however is less skewed then the component targets. GDM_g show more skewing than Dry_Total_g, but less than the components. We also see sighnificant proportion of zero values for Dry_Clover_g, Dry_Dead_g, and Dry_Green_g.\n\nNow I want to investigate target relationships as I have a hunch I want to confirm with data: Dry_Total_g is always equal to the sum of Dry_Clover_g, Dry_Dead_g, and Dry_Green_g.","metadata":{}},{"cell_type":"code","source":"# Creating wide format for correlation analysis\ndf_wide = df.pivot_table(\n    index='image_path',\n    columns='target_name',\n    values='target'\n).reset_index()\n\n# Now confirming my hunch - is Dry_Total_g = Dry_Clover_g + Dry_Dead_g + Dry_Green_g ?\ndf_wide['calculated_total'] = (\n    df_wide['Dry_Clover_g'] +\n    df_wide['Dry_Dead_g'] +\n    df_wide['Dry_Green_g']\n)\ndf_wide['total_difference'] = df_wide['Dry_Total_g'] - df_wide['calculated_total']\nprint(\"\\nDoes Dry_Total = Clover + Dead + Green?\")\nprint(f\"Mean Difference: {df_wide['total_difference'].mean():.4f}\")\nprint(f\"Max Difference: {df_wide['total_difference'].abs().max():.4f}\")\n\n# Now making a correlation matrix\ntarget_cols = ['Dry_Clover_g', 'Dry_Dead_g', 'Dry_Green_g', 'Dry_Total_g', 'GDM_g']\ncorr = df_wide[target_cols].corr()\n\nplt.figure(figsize=(8,6))\nsns.heatmap(corr, annot=True, fmt='.2f', cmap='coolwarm', center=0)\nplt.title('Target Correlations')\nplt.show()","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2026-01-10T21:44:52.003222Z","iopub.execute_input":"2026-01-10T21:44:52.003534Z","iopub.status.idle":"2026-01-10T21:44:52.177051Z","shell.execute_reply.started":"2026-01-10T21:44:52.003509Z","shell.execute_reply":"2026-01-10T21:44:52.176191Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"So from the above it looks like at least one image Dry_Total does not equal Clover + Dead + Green. This will require further investigation to see how many values this is true for, and it also looks as it the difference is not always positive (the calculated total could be greater than dry_total)\n\nWe are seeing some strong correlations here that make a lot of sense between targets. Dry Total and Dry Green are strongly correlated. Dry Clover and Dry green are weakly correlated negatively, suggesting the presence of Dry clover implies less Dry Green.","metadata":{}},{"cell_type":"code","source":"# Count how many entries have a non-zero value for total_difference\nprint(df_wide[df_wide['total_difference'] != 0].count())\n\n# See if its usually negative (calculated total greater than Dry Total) or positive (vice versa)\nprint(df_wide[df_wide['total_difference'] < 0].count())\nprint(df_wide[df_wide['total_difference'] > 0].count())","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2026-01-10T21:49:40.054528Z","iopub.execute_input":"2026-01-10T21:49:40.055201Z","iopub.status.idle":"2026-01-10T21:49:40.064033Z","shell.execute_reply.started":"2026-01-10T21:49:40.055169Z","shell.execute_reply":"2026-01-10T21:49:40.063314Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"Looks like about 103 total entries where dry total is not equal to the sum of the clover dead and green, split about 50/50.\n\nNext, I'll start to dive into some of the metadata","metadata":{}},{"cell_type":"code","source":"# Now pivot the full metadata dataframe\ndf_wide = df.pivot_table(\n    index=['image_path', 'Sampling_Date', 'State', 'Species', 'Pre_GSHH_NDVI', 'Height_Ave_cm'], \n    columns='target_name', \n    values='target'\n).reset_index()\n\n# Use the wide format dataframe\nprint(\"\\nSamples per State:\")\nprint(df_wide['State'].value_counts())\n\nprint(\"\\nSamples per Species:\")\nprint(df_wide['Species'].value_counts())\n\nprint(\"\\nNDVI statistics:\")\nprint(df_wide['Pre_GSHH_NDVI'].describe())\n\nprint(\"\\nHeight statistics:\")\nprint(df_wide['Height_Ave_cm'].describe())\n\n# Visualize metadata vs targets\nfig, axes = plt.subplots(2, 2, figsize=(14, 10))\n\n# NDVI vs Dry_Total\naxes[0, 0].scatter(df_wide['Pre_GSHH_NDVI'], df_wide['Dry_Total_g'], alpha=0.5)\naxes[0, 0].set_xlabel('NDVI')\naxes[0, 0].set_ylabel('Dry_Total_g')\naxes[0, 0].set_title('NDVI vs Total Biomass')\n\n# Height vs Dry_Total\naxes[0, 1].scatter(df_wide['Height_Ave_cm'], df_wide['Dry_Total_g'], alpha=0.5)\naxes[0, 1].set_xlabel('Height (cm)')\naxes[0, 1].set_ylabel('Dry_Total_g')\naxes[0, 1].set_title('Height vs Total Biomass')\n\n# Species boxplot\ndf_wide.boxplot(column='Dry_Total_g', by='Species', ax=axes[1, 0])\naxes[1, 0].set_title('Total Biomass by Species')\n\n# State boxplot\ndf_wide.boxplot(column='Dry_Total_g', by='State', ax=axes[1, 1])\naxes[1, 1].set_title('Total Biomass by State')\n\nplt.tight_layout()\nplt.show()","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2026-01-10T21:58:34.596653Z","iopub.execute_input":"2026-01-10T21:58:34.596950Z","iopub.status.idle":"2026-01-10T21:58:35.354319Z","shell.execute_reply.started":"2026-01-10T21:58:34.596925Z","shell.execute_reply":"2026-01-10T21:58:35.353325Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"These are some very interesting results. We can see most common state for samples are Tasmania with 138 and Victoria with 112. Most common species in images are Ryegrass_Clover and Ryegrass. We can see from out plots there is likely and NDVI correlation as well as a Height correlation. Species also may be helpful. State will likely not be very useful in this case.\n\nNow we're going to start some image investigation.","metadata":{}},{"cell_type":"code","source":"# Look at sample images\nsample_images = df_wide.sample(9, random_state=42)\n\nfig, axes = plt.subplots(3, 3, figsize=(15, 15))\naxes = axes.flatten()\n\nfor idx, (_, row) in enumerate(sample_images.iterrows()):\n    img_path = row['image_path']\n    img = Image.open('/kaggle/input/csiro-biomass/' + img_path)\n    \n    axes[idx].imshow(img)\n    axes[idx].axis('off')\n    axes[idx].set_title(\n        f\"Total: {row['Dry_Total_g']:.1f}g\\n\"\n        f\"NDVI: {row['Pre_GSHH_NDVI']:.2f}\\n\"\n        f\"{row['Species']}\"\n    )\n\nplt.tight_layout()\nplt.show()\n\n# Check image properties\nprint(\"\\nImage properties:\")\nsample_img = Image.open('/kaggle/input/csiro-biomass/' + df_wide.iloc[0]['image_path'])\nprint(f\"Size: {sample_img.size}\")\nprint(f\"Mode: {sample_img.mode}\")","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2026-01-10T22:01:02.178361Z","iopub.execute_input":"2026-01-10T22:01:02.178670Z","iopub.status.idle":"2026-01-10T22:01:04.315995Z","shell.execute_reply.started":"2026-01-10T22:01:02.178648Z","shell.execute_reply":"2026-01-10T22:01:04.315042Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"Alright we can see some sample images above here with some of their metadata. We can see the images are taken pretty close to the ground (as the average height metadata suggests), and there is wide variation in NDVI. The images appear relatively consistent however and I think there's a good chance they will greatly help predict biomass.\n\nLastly before attempting a baseline build, I'm going to look at some temporal statistics/ indicators about our dataset.","metadata":{}},{"cell_type":"code","source":"# Convert date to datetime\ndf_wide['date'] = pd.to_datetime(df_wide['Sampling_Date'])\ndf_wide['month'] = df_wide['date'].dt.month\ndf_wide['year'] = df_wide['date'].dt.year\n\nprint(\"\\nSamples per month:\")\nprint(df_wide['month'].value_counts().sort_index())\n\nprint(\"\\nSamples per year:\")\nprint(df_wide['year'].value_counts().sort_index())\n\n# Biomass by month\ndf_wide.boxplot(column='Dry_Total_g', by='month', figsize=(12, 6))\nplt.title('Total Biomass by Month')\nplt.suptitle('')\nplt.xlabel('Month')\nplt.show()","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2026-01-10T22:06:56.845136Z","iopub.execute_input":"2026-01-10T22:06:56.845457Z","iopub.status.idle":"2026-01-10T22:06:57.016808Z","shell.execute_reply.started":"2026-01-10T22:06:56.845430Z","shell.execute_reply":"2026-01-10T22:06:57.016124Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"Hmm no clear patterns off the bat. I am surprised to see that biomass appears generally lower in the summer months, however (June, July, August).","metadata":{}},{"cell_type":"markdown","source":"# Develop Baseline Model for Metadata","metadata":{}},{"cell_type":"markdown","source":"Ok, now that we have done some data exploration, I think we are ready to start playing around with a baseline model for the tabular data.","metadata":{}},{"cell_type":"code","source":"# from sklearn.model_selection import KFold\n# from sklearn.preprocessing import LabelEncoder\n# from sklearn.metrics import mean_squared_error, mean_absolute_error, r2_score\n# import xgboost as xgb\n\n# # Load and prepare data (assuming you have df already loaded)\n# df = pd.read_csv('/kaggle/input/csiro-biomass/train.csv')\n\n# # Create wide format with all metadata\n# df_wide = df.pivot_table(\n#     index=['image_path', 'Sampling_Date', 'State', 'Species', 'Pre_GSHH_NDVI', 'Height_Ave_cm'], \n#     columns='target_name', \n#     values='target'\n# ).reset_index()\n\n# # Feature engineering\n# df_wide['date'] = pd.to_datetime(df_wide['Sampling_Date'])\n# df_wide['year'] = df_wide['date'].dt.year\n# df_wide['month'] = df_wide['date'].dt.month\n# df_wide['day_of_year'] = df_wide['date'].dt.dayofyear\n\n# # Encode categorical features\n# le_state = LabelEncoder()\n# le_species = LabelEncoder()\n# df_wide['State_encoded'] = le_state.fit_transform(df_wide['State'])\n# df_wide['Species_encoded'] = le_species.fit_transform(df_wide['Species'])\n\n# # Create interaction features\n# df_wide['NDVI_x_Height'] = df_wide['Pre_GSHH_NDVI'] * df_wide['Height_Ave_cm']\n# df_wide['NDVI_squared'] = df_wide['Pre_GSHH_NDVI'] ** 2\n\n# # Define features\n# feature_cols = [\n#     'Pre_GSHH_NDVI', 'Height_Ave_cm', 'State_encoded', 'Species_encoded',\n#     'year', 'month', 'day_of_year', 'NDVI_x_Height', 'NDVI_squared'\n# ]\n\n# target_cols = ['Dry_Clover_g', 'Dry_Dead_g', 'Dry_Green_g', 'Dry_Total_g', 'GDM_g']\n\n# X = df_wide[feature_cols].values\n# y = df_wide[target_cols].values\n\n# print(\"Feature matrix shape:\", X.shape)\n# print(\"Target matrix shape:\", y.shape)\n\n# # 5-Fold Cross-Validation\n# n_splits = 5\n# kf = KFold(n_splits=n_splits, shuffle=True, random_state=42)\n\n# # Store results\n# cv_results = {target: {'rmse': [], 'mae': [], 'r2': []} for target in target_cols}\n# models = {target: [] for target in target_cols}\n\n# # Train separate model for each target\n# for fold, (train_idx, val_idx) in enumerate(kf.split(X)):\n#     print(f\"\\n{'='*50}\")\n#     print(f\"Fold {fold + 1}/{n_splits}\")\n#     print(f\"{'='*50}\")\n    \n#     X_train, X_val = X[train_idx], X[val_idx]\n#     y_train, y_val = y[train_idx], y[val_idx]\n    \n#     for i, target in enumerate(target_cols):\n#         # XGBoost parameters\n#         params = {\n#             'objective': 'reg:squarederror',\n#             'max_depth': 6,\n#             'learning_rate': 0.05,\n#             'subsample': 0.8,\n#             'colsample_bytree': 0.8,\n#             'min_child_weight': 3,\n#             'random_state': 42,\n#             'tree_method': 'hist'  # Fast for CPU\n#         }\n        \n#         # Train model\n#         dtrain = xgb.DMatrix(X_train, label=y_train[:, i])\n#         dval = xgb.DMatrix(X_val, label=y_val[:, i])\n        \n#         model = xgb.train(\n#             params,\n#             dtrain,\n#             num_boost_round=200,\n#             evals=[(dval, 'val')],\n#             early_stopping_rounds=20,\n#             verbose_eval=False\n#         )\n        \n#         # Predict\n#         y_pred = model.predict(dval)\n        \n#         # Calculate metrics\n#         rmse = np.sqrt(mean_squared_error(y_val[:, i], y_pred))\n#         mae = mean_absolute_error(y_val[:, i], y_pred)\n#         r2 = r2_score(y_val[:, i], y_pred)\n        \n#         cv_results[target]['rmse'].append(rmse)\n#         cv_results[target]['mae'].append(mae)\n#         cv_results[target]['r2'].append(r2)\n#         models[target].append(model)\n        \n#         print(f\"{target:15} - RMSE: {rmse:.3f}, MAE: {mae:.3f}, R2: {r2:.3f}\")\n\n# # Print overall results\n# print(f\"\\n{'='*60}\")\n# print(\"OVERALL CV RESULTS (Mean ± Std)\")\n# print(f\"{'='*60}\")\n\n# for target in target_cols:\n#     rmse_mean = np.mean(cv_results[target]['rmse'])\n#     rmse_std = np.std(cv_results[target]['rmse'])\n#     mae_mean = np.mean(cv_results[target]['mae'])\n#     r2_mean = np.mean(cv_results[target]['r2'])\n    \n#     print(f\"{target:15} - RMSE: {rmse_mean:.3f}±{rmse_std:.3f}, MAE: {mae_mean:.3f}, R2: {r2_mean:.3f}\")\n\n# # Feature importance (from first fold, Dry_Total_g model)\n# model_total = models['Dry_Total_g'][0]\n# importance = model_total.get_score(importance_type='gain')\n\n# # Plot feature importance\n# fig, ax = plt.subplots(figsize=(10, 6))\n# sorted_idx = sorted(importance.items(), key=lambda x: x[1], reverse=True)\n# features = [feature_cols[int(k[1:])] for k, v in sorted_idx[:10]]\n# values = [v for k, v in sorted_idx[:10]]\n\n# ax.barh(features, values)\n# ax.set_xlabel('Gain')\n# ax.set_title('Top 10 Feature Importance (Dry_Total_g)')\n# plt.tight_layout()\n# plt.show()\n\n# print(\"\\n✅ Baseline 1 Complete! This is metadata-only performance.\")","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2026-01-10T22:32:49.607177Z","iopub.execute_input":"2026-01-10T22:32:49.607462Z","iopub.status.idle":"2026-01-10T22:32:51.541451Z","shell.execute_reply.started":"2026-01-10T22:32:49.607439Z","shell.execute_reply":"2026-01-10T22:32:51.540714Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"Ok, some very interesting results here. Now for the CNN baseline using EfficientNet-B0","metadata":{}},{"cell_type":"code","source":"# \"\"\"\n# OPTION 1: Pure Image-Only Model\n# FASTEST PATH TO SUBMISSION - Run this first!\n\n# Expected time: 2-3 hours GPU for 5-fold CV\n# Expected score: 0.42-0.50 Competition R²\n# \"\"\"\n\n# import torch\n# import torch.nn as nn\n# import torch.optim as optim\n# from torch.utils.data import Dataset, DataLoader\n# from torchvision import models, transforms\n# from PIL import Image\n# import pandas as pd\n# import numpy as np\n# from sklearn.model_selection import KFold\n# from sklearn.metrics import mean_squared_error, mean_absolute_error, r2_score\n# from tqdm import tqdm\n\n# device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')\n# print(f\"Using device: {device}\")\n\n# # ==================== COMPETITION METRIC (SAME AS BEFORE) ====================\n\n# def competition_r2(y_true, y_pred, target_names):\n#     weights_dict = {\n#         'Dry_Green_g': 0.1,\n#         'Dry_Dead_g': 0.1,\n#         'Dry_Clover_g': 0.1,\n#         'GDM_g': 0.2,\n#         'Dry_Total_g': 0.5\n#     }\n#     weights = np.array([weights_dict[name] for name in target_names])\n#     y_true_flat = y_true.flatten()\n#     y_pred_flat = y_pred.flatten()\n#     weights_flat = np.tile(weights, len(y_true))\n#     weighted_mean = np.average(y_true_flat, weights=weights_flat)\n#     ss_tot = np.sum(weights_flat * (y_true_flat - weighted_mean) ** 2)\n#     ss_res = np.sum(weights_flat * (y_true_flat - y_pred_flat) ** 2)\n#     r2 = 1 - (ss_res / ss_tot)\n#     return r2\n\n# # ==================== DATA PREPARATION ====================\n\n# # Load training data (same as before)\n# df = pd.read_csv('/kaggle/input/csiro-biomass/train.csv')\n# df['image_path'] = '/kaggle/input/csiro-biomass/' + df['image_path']\n\n# # Create wide format - BUT we won't use metadata for training!\n# df_wide = df.pivot_table(\n#     index=['image_path'], \n#     columns='target_name', \n#     values='target'\n# ).reset_index()\n\n# target_cols = ['Dry_Clover_g', 'Dry_Dead_g', 'Dry_Green_g', 'Dry_Total_g', 'GDM_g']\n# image_paths = df_wide['image_path'].values\n# targets = df_wide[target_cols].values\n\n# print(f\"Training data: {len(image_paths)} images\")\n# print(f\"Targets shape: {targets.shape}\")\n\n# # ==================== DATASET (NO METADATA!) ====================\n\n# class ImageOnlyDataset(Dataset):\n#     def __init__(self, image_paths, targets, transform=None):\n#         self.image_paths = image_paths\n#         self.targets = torch.FloatTensor(targets) if targets is not None else None\n#         self.transform = transform\n    \n#     def __len__(self):\n#         return len(self.image_paths)\n    \n#     def __getitem__(self, idx):\n#         img = Image.open(self.image_paths[idx]).convert('RGB')\n#         if self.transform:\n#             img = self.transform(img)\n        \n#         if self.targets is not None:\n#             return img, self.targets[idx]\n#         else:\n#             return img\n\n# # ==================== AGGRESSIVE AUGMENTATION ====================\n\n# # Since we have NO metadata, we need VERY strong augmentation!\n# train_transform = transforms.Compose([\n#     transforms.Resize((288, 288)),\n#     transforms.RandomCrop(224),\n    \n#     # Geometric augmentations\n#     transforms.RandomHorizontalFlip(p=0.5),\n#     transforms.RandomVerticalFlip(p=0.5),\n#     transforms.RandomRotation(degrees=30),\n#     transforms.RandomAffine(degrees=0, translate=(0.1, 0.1), scale=(0.9, 1.1)),\n    \n#     # Color augmentations (simulate different conditions)\n#     transforms.ColorJitter(brightness=0.4, contrast=0.4, saturation=0.3, hue=0.15),\n#     transforms.RandomGrayscale(p=0.1),\n    \n#     # Convert to tensor\n#     transforms.ToTensor(),\n#     transforms.Normalize([0.485, 0.456, 0.406], [0.229, 0.224, 0.225])\n# ])\n\n# val_transform = transforms.Compose([\n#     transforms.Resize((224, 224)),\n#     transforms.ToTensor(),\n#     transforms.Normalize([0.485, 0.456, 0.406], [0.229, 0.224, 0.225])\n# ])\n\n# # ==================== MODEL ====================\n\n# class ConstrainedImageOnlyModel(nn.Module):\n#     \"\"\"\n#     Image-only model with constraint: Dry_Total = Clover + Dead + Green\n#     Uses EfficientNet-B1 (bigger than B0 since we only have images)\n#     \"\"\"\n#     def __init__(self):\n#         super(ConstrainedImageOnlyModel, self).__init__()\n        \n#         # EfficientNet-B1 backbone\n#         self.backbone = models.efficientnet_b1(weights='IMAGENET1K_V1')\n#         n_features = self.backbone.classifier[1].in_features  # 1280\n#         self.backbone.classifier = nn.Identity()\n        \n#         # Prediction head - predict 4 components\n#         self.head = nn.Sequential(\n#             nn.Linear(n_features, 512),\n#             nn.ReLU(),\n#             nn.Dropout(0.5),  # High dropout since small dataset\n#             nn.Linear(512, 256),\n#             nn.ReLU(),\n#             nn.Dropout(0.4),\n#             nn.Linear(256, 128),\n#             nn.ReLU(),\n#             nn.Dropout(0.3),\n#             nn.Linear(128, 4)  # Clover, Dead, Green, GDM\n#         )\n    \n#     def forward(self, img):\n#         # Extract image features\n#         img_features = self.backbone(img)\n        \n#         # Predict components\n#         components = self.head(img_features)\n        \n#         clover = components[:, 0:1]\n#         dead = components[:, 1:2]\n#         green = components[:, 2:3]\n#         gdm = components[:, 3:4]\n        \n#         # Sum to get total\n#         total = clover + dead + green\n        \n#         # Return [Clover, Dead, Green, Total, GDM]\n#         output = torch.cat([clover, dead, green, total, gdm], dim=1)\n        \n#         return output\n\n# # ==================== WEIGHTED LOSS ====================\n\n# class WeightedMSELoss(nn.Module):\n#     def __init__(self, target_names):\n#         super().__init__()\n#         weights_dict = {\n#             'Dry_Green_g': 0.1,\n#             'Dry_Dead_g': 0.1,\n#             'Dry_Clover_g': 0.1,\n#             'GDM_g': 0.2,\n#             'Dry_Total_g': 0.5\n#         }\n#         weights = [weights_dict[name] for name in target_names]\n#         self.weights = nn.Parameter(torch.tensor(weights), requires_grad=False)\n    \n#     def forward(self, pred, target):\n#         mse_per_target = (pred - target) ** 2\n#         weighted_mse = mse_per_target * self.weights\n#         return weighted_mse.mean()\n\n# # ==================== TRAINING FUNCTIONS ====================\n\n# def train_epoch(model, loader, criterion, optimizer, device):\n#     model.train()\n#     running_loss = 0.0\n    \n#     for img, targets in tqdm(loader, desc='Training', leave=False):\n#         img, targets = img.to(device), targets.to(device)\n        \n#         optimizer.zero_grad()\n#         outputs = model(img)\n#         loss = criterion(outputs, targets)\n#         loss.backward()\n#         optimizer.step()\n        \n#         running_loss += loss.item() * img.size(0)\n    \n#     return running_loss / len(loader.dataset)\n\n# def validate(model, loader, criterion, device):\n#     model.eval()\n#     running_loss = 0.0\n#     all_preds = []\n#     all_targets = []\n    \n#     with torch.no_grad():\n#         for img, targets in tqdm(loader, desc='Validating', leave=False):\n#             img, targets = img.to(device), targets.to(device)\n#             outputs = model(img)\n#             loss = criterion(outputs, targets)\n            \n#             running_loss += loss.item() * img.size(0)\n#             all_preds.append(outputs.cpu().numpy())\n#             all_targets.append(targets.cpu().numpy())\n    \n#     all_preds = np.vstack(all_preds)\n#     all_targets = np.vstack(all_targets)\n    \n#     return running_loss / len(loader.dataset), all_preds, all_targets\n\n# # ==================== CROSS-VALIDATION ====================\n\n# n_splits = 5\n# n_epochs = 25  # Longer training since harder task\n# batch_size = 12  # Smaller batch for bigger model\n# learning_rate = 5e-5  # Lower LR for EfficientNet-B1\n\n# kf = KFold(n_splits=n_splits, shuffle=True, random_state=42)\n# cv_results = {\n#     'competition_r2': [],\n#     'targets': {target: {'rmse': [], 'mae': [], 'r2': []} for target in target_cols}\n# }\n\n# fold_models = []  # Save models for ensemble\n\n# for fold, (train_idx, val_idx) in enumerate(kf.split(image_paths)):\n#     print(f\"\\n{'='*60}\")\n#     print(f\"Fold {fold + 1}/{n_splits}\")\n#     print(f\"{'='*60}\")\n    \n#     # Create datasets\n#     train_dataset = ImageOnlyDataset(\n#         image_paths[train_idx],\n#         targets[train_idx],\n#         transform=train_transform\n#     )\n    \n#     val_dataset = ImageOnlyDataset(\n#         image_paths[val_idx],\n#         targets[val_idx],\n#         transform=val_transform\n#     )\n    \n#     train_loader = DataLoader(train_dataset, batch_size=batch_size, shuffle=True, num_workers=2)\n#     val_loader = DataLoader(val_dataset, batch_size=batch_size, shuffle=False, num_workers=2)\n    \n#     # Initialize model\n#     model = ConstrainedImageOnlyModel().to(device)\n#     criterion = WeightedMSELoss(target_cols).to(device)\n#     optimizer = optim.AdamW(model.parameters(), lr=learning_rate, weight_decay=1e-4)\n#     scheduler = optim.lr_scheduler.ReduceLROnPlateau(optimizer, mode='min', factor=0.5, patience=3)\n    \n#     # Training loop\n#     best_comp_r2 = -float('inf')\n#     patience_counter = 0\n    \n#     for epoch in range(n_epochs):\n#         train_loss = train_epoch(model, train_loader, criterion, optimizer, device)\n#         val_loss, val_preds, val_targets = validate(model, val_loader, criterion, device)\n#         comp_r2 = competition_r2(val_targets, val_preds, target_cols)\n        \n#         scheduler.step(val_loss)\n        \n#         print(f\"Epoch {epoch+1:2d}/{n_epochs} - \"\n#               f\"Train Loss: {train_loss:.4f}, Val Loss: {val_loss:.4f}, \"\n#               f\"Competition R²: {comp_r2:.4f}\")\n        \n#         if comp_r2 > best_comp_r2:\n#             best_comp_r2 = comp_r2\n#             best_preds = val_preds\n#             patience_counter = 0\n#             # Save best model for this fold\n#             best_model_state = model.state_dict().copy()\n#         else:\n#             patience_counter += 1\n#             if patience_counter >= 5:\n#                 print(\"Early stopping\")\n#                 break\n    \n#     # Load best model state and save for ensemble\n#     model.load_state_dict(best_model_state)\n#     fold_models.append(model)\n    \n#     # Calculate metrics\n#     print(f\"\\nFold {fold + 1} Final Results:\")\n#     print(f\"{'='*60}\")\n    \n#     weights_dict = {\n#         'Dry_Green_g': 0.1, 'Dry_Dead_g': 0.1, 'Dry_Clover_g': 0.1,\n#         'GDM_g': 0.2, 'Dry_Total_g': 0.5\n#     }\n    \n#     for i, target in enumerate(target_cols):\n#         rmse = np.sqrt(mean_squared_error(val_targets[:, i], best_preds[:, i]))\n#         mae = mean_absolute_error(val_targets[:, i], best_preds[:, i])\n#         r2 = r2_score(val_targets[:, i], best_preds[:, i])\n        \n#         cv_results['targets'][target]['rmse'].append(rmse)\n#         cv_results['targets'][target]['mae'].append(mae)\n#         cv_results['targets'][target]['r2'].append(r2)\n        \n#         weight = weights_dict[target]\n#         print(f\"{target:15} (w={weight:.1f}) - RMSE: {rmse:6.3f}, MAE: {mae:6.3f}, R²: {r2:6.3f}\")\n    \n#     cv_results['competition_r2'].append(best_comp_r2)\n#     print(f\"\\n📊 COMPETITION R² (Fold {fold + 1}): {best_comp_r2:.4f}\")\n\n# # ==================== FINAL RESULTS ====================\n\n# print(f\"\\n{'='*70}\")\n# print(\"OVERALL CROSS-VALIDATION RESULTS\")\n# print(f\"{'='*70}\")\n\n# comp_r2_mean = np.mean(cv_results['competition_r2'])\n# comp_r2_std = np.std(cv_results['competition_r2'])\n\n# print(f\"\\n🏆 COMPETITION WEIGHTED R² (Image-Only): {comp_r2_mean:.4f} ± {comp_r2_std:.4f}\")\n# print(f\"   (Your baseline with metadata was: 0.5290)\")\n\n# print(f\"\\n{'='*70}\")\n# print(\"Per-Target Metrics:\")\n# print(f\"{'='*70}\")\n\n# for target in target_cols:\n#     rmse_mean = np.mean(cv_results['targets'][target]['rmse'])\n#     rmse_std = np.std(cv_results['targets'][target]['rmse'])\n#     mae_mean = np.mean(cv_results['targets'][target]['mae'])\n#     r2_mean = np.mean(cv_results['targets'][target]['r2'])\n#     weight = weights_dict[target]\n    \n#     print(f\"{target:15} (w={weight:.1f}) - \"\n#           f\"RMSE: {rmse_mean:6.3f}±{rmse_std:.3f}, \"\n#           f\"MAE: {mae_mean:6.3f}, R²: {r2_mean:6.3f}\")\n\n# print(f\"\\n{'='*70}\")\n# print(\"✅ Option 1 Complete - Image-Only Model\")\n# print(f\"{'='*70}\")\n# print(f\"\\nYour image-only score: {comp_r2_mean:.4f}\")\n# print(\"This is your first submission!\")\n# print(\"\\nNext: Try Option 3 (Multi-Task Learning) to potentially improve to 0.50-0.56\")","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2026-01-11T00:49:13.022510Z","iopub.execute_input":"2026-01-11T00:49:13.022749Z","iopub.status.idle":"2026-01-11T01:11:56.989968Z","shell.execute_reply.started":"2026-01-11T00:49:13.022727Z","shell.execute_reply":"2026-01-11T01:11:56.989159Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"Image-only model performed ok. 0.19 competition R squared is not that great. Going to try a multi-task model which will use the image meta data for training along with the images","metadata":{}},{"cell_type":"code","source":"# \"\"\"\n# COMPLETE MULTI-TASK LEARNING MODEL\n# Run this entire cell with GPU enabled - it's fully self-contained!\n\n# Expected time: 3-4 hours for 5-fold CV\n# Expected score: 0.35-0.45 Competition R²\n# \"\"\"\n\n# # ==================== IMPORTS ====================\n\n# import torch\n# import torch.nn as nn\n# import torch.optim as optim\n# from torch.utils.data import Dataset, DataLoader\n# from torchvision import models, transforms\n# from PIL import Image\n# import pandas as pd\n# import numpy as np\n# from sklearn.model_selection import KFold\n# from sklearn.preprocessing import LabelEncoder\n# from sklearn.metrics import mean_squared_error, mean_absolute_error, r2_score\n# from tqdm import tqdm\n# import warnings\n# warnings.filterwarnings('ignore')\n\n# # Check GPU\n# device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')\n# print(f\"Using device: {device}\")\n# if device.type == 'cuda':\n#     print(f\"GPU: {torch.cuda.get_device_name(0)}\")\n\n# # ==================== DATA LOADING ====================\n\n# # Load training data - ADJUST PATHS AS NEEDED!\n# df = pd.read_csv('/kaggle/input/csiro-biomass/train.csv')\n# df['image_path'] = '/kaggle/input/csiro-biomass/' + df['image_path']\n\n# print(f\"Loaded {len(df)} rows from training data\")\n# print(f\"Columns: {df.columns.tolist()}\")\n\n# # ==================== DATA PREPARATION ====================\n\n# # Pivot to wide format but KEEP metadata columns\n# df_wide = df.pivot_table(\n#     index=['image_path', 'Sampling_Date', 'State', 'Species', 'Pre_GSHH_NDVI', 'Height_Ave_cm'], \n#     columns='target_name', \n#     values='target'\n# ).reset_index()\n\n# print(f\"\\nWide format: {df_wide.shape}\")\n# print(f\"Unique images: {df_wide['image_path'].nunique()}\")\n\n# # Encode Species to integers (needed for classification task)\n# le_species = LabelEncoder()\n# df_wide['Species_encoded'] = le_species.fit_transform(df_wide['Species'])\n# n_species = len(le_species.classes_)\n\n# print(f\"Number of species: {n_species}\")\n# print(f\"Species: {le_species.classes_}\")\n\n# # Define target columns\n# target_cols = ['Dry_Clover_g', 'Dry_Dead_g', 'Dry_Green_g', 'Dry_Total_g', 'GDM_g']\n\n# # ==================== COMPETITION METRIC ====================\n\n# def competition_r2(y_true, y_pred, target_names):\n#     \"\"\"Calculate weighted R² matching Kaggle competition scoring\"\"\"\n#     weights_dict = {\n#         'Dry_Green_g': 0.1,\n#         'Dry_Dead_g': 0.1,\n#         'Dry_Clover_g': 0.1,\n#         'GDM_g': 0.2,\n#         'Dry_Total_g': 0.5\n#     }\n#     weights = np.array([weights_dict[name] for name in target_names])\n#     y_true_flat = y_true.flatten()\n#     y_pred_flat = y_pred.flatten()\n#     weights_flat = np.tile(weights, len(y_true))\n#     weighted_mean = np.average(y_true_flat, weights=weights_flat)\n#     ss_tot = np.sum(weights_flat * (y_true_flat - weighted_mean) ** 2)\n#     ss_res = np.sum(weights_flat * (y_true_flat - y_pred_flat) ** 2)\n#     r2 = 1 - (ss_res / ss_tot)\n#     return r2\n\n# # ==================== DATA TRANSFORMS ====================\n\n# # Aggressive training augmentation\n# train_transform = transforms.Compose([\n#     transforms.Resize((288, 288)),\n#     transforms.RandomCrop(224),\n#     transforms.RandomHorizontalFlip(p=0.5),\n#     transforms.RandomVerticalFlip(p=0.5),\n#     transforms.RandomRotation(degrees=30),\n#     transforms.RandomAffine(degrees=0, translate=(0.1, 0.1), scale=(0.9, 1.1)),\n#     transforms.ColorJitter(brightness=0.4, contrast=0.4, saturation=0.3, hue=0.15),\n#     transforms.RandomGrayscale(p=0.1),\n#     transforms.ToTensor(),\n#     transforms.Normalize([0.485, 0.456, 0.406], [0.229, 0.224, 0.225])\n# ])\n\n# # Validation transform (no augmentation)\n# val_transform = transforms.Compose([\n#     transforms.Resize((224, 224)),\n#     transforms.ToTensor(),\n#     transforms.Normalize([0.485, 0.456, 0.406], [0.229, 0.224, 0.225])\n# ])\n\n# # ==================== DATASET ====================\n\n# class MultiTaskDataset(Dataset):\n#     \"\"\"\n#     Dataset that returns:\n#     - Image\n#     - Biomass targets (5 values)\n#     - NDVI (1 value)\n#     - Height (1 value)\n#     - Species (integer class)\n#     \"\"\"\n#     def __init__(self, df, transform=None):\n#         self.image_paths = df['image_path'].values\n#         self.transform = transform\n        \n#         # Biomass targets\n#         self.biomass = torch.FloatTensor(df[target_cols].values)\n        \n#         # Metadata targets\n#         self.ndvi = torch.FloatTensor(df['Pre_GSHH_NDVI'].values).unsqueeze(1)\n#         self.height = torch.FloatTensor(df['Height_Ave_cm'].values).unsqueeze(1)\n#         self.species = torch.LongTensor(df['Species_encoded'].values)\n    \n#     def __len__(self):\n#         return len(self.image_paths)\n    \n#     def __getitem__(self, idx):\n#         # Load image\n#         img = Image.open(self.image_paths[idx]).convert('RGB')\n#         if self.transform:\n#             img = self.transform(img)\n        \n#         # Return image and all targets as dict\n#         targets = {\n#             'biomass': self.biomass[idx],\n#             'ndvi': self.ndvi[idx],\n#             'height': self.height[idx],\n#             'species': self.species[idx]\n#         }\n        \n#         return img, targets\n\n# # ==================== MULTI-TASK MODEL ====================\n\n# class MultiTaskBiomassModel(nn.Module):\n#     \"\"\"\n#     One model that predicts:\n#     1. Biomass (primary task)\n#     2. NDVI (auxiliary - helps learn vegetation features)\n#     3. Height (auxiliary - helps learn scale/density)\n#     4. Species (auxiliary - helps learn textures/patterns)\n#     \"\"\"\n#     def __init__(self, n_species):\n#         super(MultiTaskBiomassModel, self).__init__()\n        \n#         # Shared backbone - EfficientNet-B1\n#         self.backbone = models.efficientnet_b1(weights='IMAGENET1K_V1')\n#         n_features = self.backbone.classifier[1].in_features  # 1280\n#         self.backbone.classifier = nn.Identity()\n        \n#         # Shared feature processing\n#         self.shared_features = nn.Sequential(\n#             nn.Linear(n_features, 512),\n#             nn.BatchNorm1d(512),\n#             nn.ReLU(),\n#             nn.Dropout(0.4)\n#         )\n        \n#         # PRIMARY TASK: Biomass prediction head\n#         self.biomass_head = nn.Sequential(\n#             nn.Linear(512, 256),\n#             nn.BatchNorm1d(256),\n#             nn.ReLU(),\n#             nn.Dropout(0.3),\n#             nn.Linear(256, 128),\n#             nn.ReLU(),\n#             nn.Dropout(0.2),\n#             nn.Linear(128, 4)  # Clover, Dead, Green, GDM\n#         )\n        \n#         # AUXILIARY TASK 1: NDVI prediction\n#         self.ndvi_head = nn.Sequential(\n#             nn.Linear(512, 64),\n#             nn.ReLU(),\n#             nn.Linear(64, 1),\n#             nn.Sigmoid()  # NDVI is between 0-1\n#         )\n        \n#         # AUXILIARY TASK 2: Height prediction\n#         self.height_head = nn.Sequential(\n#             nn.Linear(512, 64),\n#             nn.ReLU(),\n#             nn.Linear(64, 1),\n#             nn.ReLU()  # Height is positive\n#         )\n        \n#         # AUXILIARY TASK 3: Species classification\n#         self.species_head = nn.Sequential(\n#             nn.Linear(512, 128),\n#             nn.ReLU(),\n#             nn.Dropout(0.3),\n#             nn.Linear(128, n_species)\n#         )\n    \n#     def forward(self, img):\n#         # Extract features from image\n#         img_features = self.backbone(img)\n#         shared = self.shared_features(img_features)\n        \n#         # Primary task: Predict biomass components\n#         components = self.biomass_head(shared)\n        \n#         # Apply ReLU to ensure non-negative\n#         clover = torch.relu(components[:, 0:1])\n#         dead = torch.relu(components[:, 1:2])\n#         green = torch.relu(components[:, 2:3])\n#         gdm = torch.relu(components[:, 3:4])\n        \n#         # Calculate total as sum\n#         total = clover + dead + green\n        \n#         # Combine all biomass predictions\n#         biomass = torch.cat([clover, dead, green, total, gdm], dim=1)\n        \n#         # Auxiliary tasks\n#         ndvi = self.ndvi_head(shared)\n#         height = self.height_head(shared)\n#         species = self.species_head(shared)\n        \n#         return {\n#             'biomass': biomass,\n#             'ndvi': ndvi,\n#             'height': height,\n#             'species': species\n#         }\n\n# # ==================== MULTI-TASK LOSS ====================\n\n# class MultiTaskLoss(nn.Module):\n#     \"\"\"\n#     Combined loss function for all tasks.\n    \n#     Total Loss = α * biomass_loss + β * ndvi_loss + γ * height_loss + δ * species_loss\n    \n#     Weights are tuned to balance the tasks.\n#     \"\"\"\n#     def __init__(self, target_names, n_species):\n#         super().__init__()\n        \n#         # Weighted MSE for biomass (competition weights)\n#         weights_dict = {\n#             'Dry_Green_g': 0.1,\n#             'Dry_Dead_g': 0.1,\n#             'Dry_Clover_g': 0.1,\n#             'GDM_g': 0.2,\n#             'Dry_Total_g': 0.5\n#         }\n#         weights = [weights_dict[name] for name in target_names]\n#         self.biomass_weights = nn.Parameter(torch.tensor(weights), requires_grad=False)\n        \n#         # Loss functions\n#         self.mse = nn.MSELoss()\n#         self.ce = nn.CrossEntropyLoss()\n        \n#         # Task weights (how much each task contributes to total loss)\n#         self.task_weights = {\n#             'biomass': 10.0,   # MUCH higher - biomass is PRIMARY task\n#             'ndvi': 0.1,       # Reduced\n#             'height': 0.1,     # Reduced\n#             'species': 0.2     # Reduced\n#         }\n    \n#     def forward(self, predictions, targets):\n#         # Biomass loss (weighted by competition rules)\n#         biomass_mse = (predictions['biomass'] - targets['biomass']) ** 2\n#         biomass_loss = (biomass_mse * self.biomass_weights).mean()\n        \n#         # NDVI loss\n#         ndvi_loss = self.mse(predictions['ndvi'], targets['ndvi'])\n        \n#         # Height loss\n#         height_loss = self.mse(predictions['height'], targets['height'])\n        \n#         # Species loss\n#         species_loss = self.ce(predictions['species'], targets['species'])\n        \n#         # Combined weighted loss\n#         total_loss = (\n#             self.task_weights['biomass'] * biomass_loss +\n#             self.task_weights['ndvi'] * ndvi_loss +\n#             self.task_weights['height'] * height_loss +\n#             self.task_weights['species'] * species_loss\n#         )\n        \n#         return {\n#             'total': total_loss,\n#             'biomass': biomass_loss.item(),\n#             'ndvi': ndvi_loss.item(),\n#             'height': height_loss.item(),\n#             'species': species_loss.item()\n#         }\n\n# # ==================== TRAINING FUNCTIONS ====================\n\n# def train_epoch(model, loader, criterion, optimizer, device):\n#     \"\"\"Train for one epoch\"\"\"\n#     model.train()\n#     total_loss = 0.0\n#     task_losses = {'biomass': 0.0, 'ndvi': 0.0, 'height': 0.0, 'species': 0.0}\n    \n#     for img, targets in tqdm(loader, desc='Training', leave=False):\n#         img = img.to(device)\n#         targets = {k: v.to(device) for k, v in targets.items()}\n        \n#         optimizer.zero_grad()\n#         predictions = model(img)\n#         losses = criterion(predictions, targets)\n        \n#         losses['total'].backward()\n        \n#         # Gradient clipping for stability\n#         torch.nn.utils.clip_grad_norm_(model.parameters(), max_norm=1.0)\n        \n#         optimizer.step()\n        \n#         total_loss += losses['total'].item() * img.size(0)\n#         for task in ['biomass', 'ndvi', 'height', 'species']:\n#             task_losses[task] += losses[task] * img.size(0)\n    \n#     n = len(loader.dataset)\n#     return total_loss / n, {k: v / n for k, v in task_losses.items()}\n\n# def validate(model, loader, criterion, device):\n#     \"\"\"Validate the model\"\"\"\n#     model.eval()\n#     total_loss = 0.0\n#     all_biomass_preds = []\n#     all_biomass_targets = []\n    \n#     with torch.no_grad():\n#         for img, targets in tqdm(loader, desc='Validating', leave=False):\n#             img = img.to(device)\n#             targets = {k: v.to(device) for k, v in targets.items()}\n            \n#             predictions = model(img)\n#             losses = criterion(predictions, targets)\n            \n#             total_loss += losses['total'].item() * img.size(0)\n#             all_biomass_preds.append(predictions['biomass'].cpu().numpy())\n#             all_biomass_targets.append(targets['biomass'].cpu().numpy())\n    \n#     all_biomass_preds = np.vstack(all_biomass_preds)\n#     all_biomass_targets = np.vstack(all_biomass_targets)\n    \n#     return total_loss / len(loader.dataset), all_biomass_preds, all_biomass_targets\n\n# # ==================== CROSS-VALIDATION ====================\n\n# n_splits = 5\n# n_epochs = 30\n# batch_size = 10\n# learning_rate = 1e-4  # Higher learning rate\n\n# kf = KFold(n_splits=n_splits, shuffle=True, random_state=42)\n\n# cv_results = {\n#     'competition_r2': [],\n#     'targets': {target: {'rmse': [], 'mae': [], 'r2': []} for target in target_cols}\n# }\n\n# print(f\"\\n{'='*70}\")\n# print(\"STARTING 5-FOLD CROSS-VALIDATION\")\n# print(f\"{'='*70}\\n\")\n\n# for fold, (train_idx, val_idx) in enumerate(kf.split(df_wide)):\n#     print(f\"\\n{'='*70}\")\n#     print(f\"FOLD {fold + 1}/{n_splits}\")\n#     print(f\"{'='*70}\")\n    \n#     # Create fold datasets\n#     train_df = df_wide.iloc[train_idx].reset_index(drop=True)\n#     val_df = df_wide.iloc[val_idx].reset_index(drop=True)\n    \n#     train_dataset = MultiTaskDataset(train_df, transform=train_transform)\n#     val_dataset = MultiTaskDataset(val_df, transform=val_transform)\n    \n#     train_loader = DataLoader(train_dataset, batch_size=batch_size, shuffle=True, num_workers=2, pin_memory=True)\n#     val_loader = DataLoader(val_dataset, batch_size=batch_size, shuffle=False, num_workers=2, pin_memory=True)\n    \n#     # Initialize model, loss, optimizer\n#     model = MultiTaskBiomassModel(n_species=n_species).to(device)\n#     criterion = MultiTaskLoss(target_cols, n_species).to(device)\n#     optimizer = optim.AdamW(model.parameters(), lr=learning_rate, weight_decay=1e-4)\n    \n#     # Learning rate scheduler\n#     scheduler = optim.lr_scheduler.ReduceLROnPlateau(\n#         optimizer, mode='max', factor=0.5, patience=3\n#     )\n    \n#     # Training loop\n#     best_comp_r2 = -float('inf')\n#     best_preds = None\n#     patience_counter = 0\n    \n#     for epoch in range(n_epochs):\n#         train_loss, task_losses = train_epoch(model, train_loader, criterion, optimizer, device)\n#         val_loss, val_preds, val_targets = validate(model, val_loader, criterion, device)\n        \n#         # Calculate competition R²\n#         comp_r2 = competition_r2(val_targets, val_preds, target_cols)\n        \n#         # Update scheduler\n#         scheduler.step(comp_r2)\n        \n#         print(f\"Epoch {epoch+1:2d}/{n_epochs} | \"\n#               f\"Val Loss: {val_loss:.4f} | \"\n#               f\"Comp R²: {comp_r2:.4f} | \"\n#               f\"Bio: {task_losses['biomass']:.3f} | \"\n#               f\"NDVI: {task_losses['ndvi']:.3f} | \"\n#               f\"Ht: {task_losses['height']:.3f} | \"\n#               f\"Spc: {task_losses['species']:.3f}\")\n        \n#         # Save best model\n#         if comp_r2 > best_comp_r2:\n#             best_comp_r2 = comp_r2\n#             best_preds = val_preds\n#             best_targets = val_targets\n#             patience_counter = 0\n#         else:\n#             patience_counter += 1\n#             if patience_counter >= 5:\n#                 print(f\"Early stopping at epoch {epoch+1}\")\n#                 break\n    \n#     # Calculate per-target metrics for best epoch\n#     print(f\"\\n{'='*70}\")\n#     print(f\"FOLD {fold + 1} FINAL RESULTS (Best Epoch)\")\n#     print(f\"{'='*70}\")\n    \n#     weights_dict = {\n#         'Dry_Green_g': 0.1, 'Dry_Dead_g': 0.1, 'Dry_Clover_g': 0.1,\n#         'GDM_g': 0.2, 'Dry_Total_g': 0.5\n#     }\n    \n#     for i, target in enumerate(target_cols):\n#         rmse = np.sqrt(mean_squared_error(best_targets[:, i], best_preds[:, i]))\n#         mae = mean_absolute_error(best_targets[:, i], best_preds[:, i])\n#         r2 = r2_score(best_targets[:, i], best_preds[:, i])\n        \n#         cv_results['targets'][target]['rmse'].append(rmse)\n#         cv_results['targets'][target]['mae'].append(mae)\n#         cv_results['targets'][target]['r2'].append(r2)\n        \n#         weight = weights_dict[target]\n#         print(f\"{target:15} (w={weight:.1f}) | RMSE: {rmse:6.3f} | MAE: {mae:6.3f} | R²: {r2:6.3f}\")\n    \n#     cv_results['competition_r2'].append(best_comp_r2)\n#     print(f\"\\n🏆 Competition R² (Fold {fold + 1}): {best_comp_r2:.4f}\")\n\n# # ==================== FINAL RESULTS ====================\n\n# print(f\"\\n{'='*70}\")\n# print(\"OVERALL CROSS-VALIDATION RESULTS\")\n# print(f\"{'='*70}\")\n\n# comp_r2_mean = np.mean(cv_results['competition_r2'])\n# comp_r2_std = np.std(cv_results['competition_r2'])\n\n# print(f\"\\n🏆 MULTI-TASK COMPETITION R²: {comp_r2_mean:.4f} ± {comp_r2_std:.4f}\")\n# print(f\"\\nComparison:\")\n# print(f\"  Image-only (Option 1):     0.19 ± 0.17\")\n# print(f\"  Multi-task (Option 3):     {comp_r2_mean:.4f} ± {comp_r2_std:.4f}\")\n# print(f\"  Image+Metadata (original): 0.53 ± 0.05\")\n\n# if comp_r2_mean > 0.19:\n#     improvement = comp_r2_mean - 0.19\n#     print(f\"\\n✅ Improvement over image-only: +{improvement:.4f}\")\n# else:\n#     print(f\"\\n⚠️ Score worse than image-only baseline\")\n\n# print(f\"\\n{'='*70}\")\n# print(\"Per-Target Metrics (Mean ± Std):\")\n# print(f\"{'='*70}\")\n\n# for target in target_cols:\n#     rmse_mean = np.mean(cv_results['targets'][target]['rmse'])\n#     rmse_std = np.std(cv_results['targets'][target]['rmse'])\n#     mae_mean = np.mean(cv_results['targets'][target]['mae'])\n#     r2_mean = np.mean(cv_results['targets'][target]['r2'])\n#     weight = weights_dict[target]\n    \n#     print(f\"{target:15} (w={weight:.1f}) | \"\n#           f\"RMSE: {rmse_mean:6.3f}±{rmse_std:.3f} | \"\n#           f\"MAE: {mae_mean:6.3f} | \"\n#           f\"R²: {r2_mean:6.3f}\")\n\n# print(f\"\\n{'='*70}\")\n# print(\"✅ MULTI-TASK MODEL TRAINING COMPLETE!\")\n# print(f\"{'='*70}\")\n# print(f\"\\nYour final competition score estimate: {comp_r2_mean:.4f}\")\n# print(\"\\nNext steps:\")\n# print(\"1. If score > 0.35: Make submission!\")\n# print(\"2. If score < 0.35: Try ensemble with image-only model\")\n# print(\"3. Advanced: Add Test-Time Augmentation (TTA)\")","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2026-01-11T20:49:50.780556Z","iopub.execute_input":"2026-01-11T20:49:50.781149Z","iopub.status.idle":"2026-01-11T21:01:15.563825Z","shell.execute_reply.started":"2026-01-11T20:49:50.781115Z","shell.execute_reply":"2026-01-11T21:01:15.562260Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"After a few iterations, it appears the multi-task model is not performing as well as the baseline image only model. Let's switch back and try an improved image only model.","metadata":{}},{"cell_type":"code","source":"\"\"\"\nIMPROVED CUSTOM CNN FOR BIOMASS PREDICTION\nNo pretrained weights needed - works fully offline\n\nImprovements over baseline:\n- Huber loss (robust to outliers)\n- 4x TTA at inference\n- Stronger augmentation\n- 40 epochs\n- CosineAnnealingWarmRestarts scheduler\n\nExpected runtime: 2-3 hours\n\"\"\"\n\nimport torch\nimport torch.nn as nn\nimport torch.optim as optim\nfrom torch.utils.data import Dataset, DataLoader\nfrom torchvision import transforms\nfrom PIL import Image\nimport pandas as pd\nimport numpy as np\nfrom sklearn.model_selection import KFold\nimport warnings\nwarnings.filterwarnings('ignore')\n\ndevice = torch.device('cuda' if torch.cuda.is_available() else 'cpu')\nprint(f\"Device: {device}\")\n\n# ==================== DATA ====================\n\ntrain_df = pd.read_csv('/kaggle/input/csiro-biomass/train.csv')\ntrain_df['image_path'] = '/kaggle/input/csiro-biomass/' + train_df['image_path']\n\ndf_wide = train_df.pivot_table(\n    index=['image_path'], \n    columns='target_name', \n    values='target'\n).reset_index()\n\ntarget_cols = ['Dry_Clover_g', 'Dry_Dead_g', 'Dry_Green_g', 'Dry_Total_g', 'GDM_g']\nimage_paths = df_wide['image_path'].values\ntargets = df_wide[target_cols].values\n\ntarget_means = targets.mean(axis=0)\ntarget_stds = targets.std(axis=0)\n\ndef normalize_targets(t):\n    return (t - target_means) / target_stds\n\ndef denormalize_targets(t):\n    return t * target_stds + target_means\n\nprint(f\"Training: {len(image_paths)} images\")\nprint(f\"Target means: {target_means}\")\nprint(f\"Target stds: {target_stds}\")\n\n# ==================== IMPROVED CNN ====================\n\nclass ImprovedCNN(nn.Module):\n    \"\"\"Deeper CNN with residual-style connections\"\"\"\n    def __init__(self):\n        super(ImprovedCNN, self).__init__()\n        \n        self.features = nn.Sequential(\n            # Block 1: 224 -> 112\n            nn.Conv2d(3, 32, 3, padding=1),\n            nn.BatchNorm2d(32),\n            nn.ReLU(),\n            nn.Conv2d(32, 32, 3, padding=1),\n            nn.BatchNorm2d(32),\n            nn.ReLU(),\n            nn.MaxPool2d(2),\n            \n            # Block 2: 112 -> 56\n            nn.Conv2d(32, 64, 3, padding=1),\n            nn.BatchNorm2d(64),\n            nn.ReLU(),\n            nn.Conv2d(64, 64, 3, padding=1),\n            nn.BatchNorm2d(64),\n            nn.ReLU(),\n            nn.MaxPool2d(2),\n            \n            # Block 3: 56 -> 28\n            nn.Conv2d(64, 128, 3, padding=1),\n            nn.BatchNorm2d(128),\n            nn.ReLU(),\n            nn.Conv2d(128, 128, 3, padding=1),\n            nn.BatchNorm2d(128),\n            nn.ReLU(),\n            nn.MaxPool2d(2),\n            \n            # Block 4: 28 -> 14\n            nn.Conv2d(128, 256, 3, padding=1),\n            nn.BatchNorm2d(256),\n            nn.ReLU(),\n            nn.Conv2d(256, 256, 3, padding=1),\n            nn.BatchNorm2d(256),\n            nn.ReLU(),\n            nn.MaxPool2d(2),\n            \n            # Block 5: 14 -> 7\n            nn.Conv2d(256, 512, 3, padding=1),\n            nn.BatchNorm2d(512),\n            nn.ReLU(),\n            nn.Conv2d(512, 512, 3, padding=1),\n            nn.BatchNorm2d(512),\n            nn.ReLU(),\n            nn.MaxPool2d(2),\n            \n            # Block 6: 7 -> 1 (global pool)\n            nn.Conv2d(512, 512, 3, padding=1),\n            nn.BatchNorm2d(512),\n            nn.ReLU(),\n            nn.AdaptiveAvgPool2d(1)\n        )\n        \n        # Regression head\n        self.head = nn.Sequential(\n            nn.Flatten(),\n            nn.Linear(512, 256),\n            nn.ReLU(),\n            nn.Dropout(0.4),\n            nn.Linear(256, 128),\n            nn.ReLU(),\n            nn.Dropout(0.3),\n            nn.Linear(128, 4)  # clover, dead, green, gdm\n        )\n    \n    def forward(self, x):\n        x = self.features(x)\n        components = self.head(x)\n        \n        clover = components[:, 0:1]\n        dead = components[:, 1:2]\n        green = components[:, 2:3]\n        gdm = components[:, 3:4]\n        \n        # Physical constraint: total = clover + dead + green\n        total = clover + dead + green\n        \n        return torch.cat([clover, dead, green, total, gdm], dim=1)\n\n# ==================== DATASET ====================\n\nclass NormalizedDataset(Dataset):\n    def __init__(self, image_paths, targets, transform=None):\n        self.image_paths = image_paths\n        self.targets = torch.FloatTensor(normalize_targets(targets))\n        self.transform = transform\n    \n    def __len__(self):\n        return len(self.image_paths)\n    \n    def __getitem__(self, idx):\n        img = Image.open(self.image_paths[idx]).convert('RGB')\n        if self.transform:\n            img = self.transform(img)\n        return img, self.targets[idx]\n\n# ==================== TRANSFORMS ====================\n\n# Stronger augmentation\ntrain_transform = transforms.Compose([\n    transforms.Resize((240, 240)),\n    transforms.RandomCrop(224),\n    transforms.RandomHorizontalFlip(p=0.5),\n    transforms.RandomVerticalFlip(p=0.5),\n    transforms.RandomRotation(20),\n    transforms.ColorJitter(brightness=0.3, contrast=0.3, saturation=0.3, hue=0.1),\n    transforms.RandomAffine(degrees=0, translate=(0.1, 0.1), scale=(0.9, 1.1)),\n    transforms.ToTensor(),\n    transforms.Normalize([0.485, 0.456, 0.406], [0.229, 0.224, 0.225])\n])\n\nval_transform = transforms.Compose([\n    transforms.Resize((224, 224)),\n    transforms.ToTensor(),\n    transforms.Normalize([0.485, 0.456, 0.406], [0.229, 0.224, 0.225])\n])\n\n# TTA transforms for test time\ntta_transforms = [\n    # Original\n    transforms.Compose([\n        transforms.Resize((224, 224)),\n        transforms.ToTensor(),\n        transforms.Normalize([0.485, 0.456, 0.406], [0.229, 0.224, 0.225])\n    ]),\n    # Horizontal flip\n    transforms.Compose([\n        transforms.Resize((224, 224)),\n        transforms.RandomHorizontalFlip(p=1.0),\n        transforms.ToTensor(),\n        transforms.Normalize([0.485, 0.456, 0.406], [0.229, 0.224, 0.225])\n    ]),\n    # Vertical flip\n    transforms.Compose([\n        transforms.Resize((224, 224)),\n        transforms.RandomVerticalFlip(p=1.0),\n        transforms.ToTensor(),\n        transforms.Normalize([0.485, 0.456, 0.406], [0.229, 0.224, 0.225])\n    ]),\n    # Both flips\n    transforms.Compose([\n        transforms.Resize((224, 224)),\n        transforms.RandomHorizontalFlip(p=1.0),\n        transforms.RandomVerticalFlip(p=1.0),\n        transforms.ToTensor(),\n        transforms.Normalize([0.485, 0.456, 0.406], [0.229, 0.224, 0.225])\n    ]),\n]\n\n# ==================== TRAINING FUNCTIONS ====================\n\ndef train_epoch(model, loader, criterion, optimizer, device):\n    model.train()\n    total_loss = 0\n    for img, targets in loader:\n        img, targets = img.to(device), targets.to(device)\n        optimizer.zero_grad()\n        outputs = model(img)\n        loss = criterion(outputs, targets)\n        loss.backward()\n        torch.nn.utils.clip_grad_norm_(model.parameters(), 1.0)\n        optimizer.step()\n        total_loss += loss.item() * img.size(0)\n    return total_loss / len(loader.dataset)\n\ndef validate(model, loader, device):\n    model.eval()\n    all_preds, all_targets = [], []\n    with torch.no_grad():\n        for img, targets in loader:\n            img, targets = img.to(device), targets.to(device)\n            outputs = model(img)\n            all_preds.append(denormalize_targets(outputs.cpu().numpy()))\n            all_targets.append(denormalize_targets(targets.cpu().numpy()))\n    return np.vstack(all_preds), np.vstack(all_targets)\n\ndef competition_r2(y_true, y_pred):\n    \"\"\"Weighted R² as per competition metric\"\"\"\n    weights = np.array([0.1, 0.1, 0.1, 0.5, 0.2])\n    y_true_flat = y_true.flatten()\n    y_pred_flat = y_pred.flatten()\n    weights_flat = np.tile(weights, len(y_true))\n    weighted_mean = np.average(y_true_flat, weights=weights_flat)\n    ss_tot = np.sum(weights_flat * (y_true_flat - weighted_mean) ** 2)\n    ss_res = np.sum(weights_flat * (y_true_flat - y_pred_flat) ** 2)\n    return 1 - (ss_res / ss_tot)\n\n# ==================== TRAIN 5-FOLD ENSEMBLE ====================\n\nprint(\"\\n\" + \"=\"*50)\nprint(\"Training 5-fold ensemble with Improved CNN\")\nprint(\"=\"*50)\n\nkf = KFold(n_splits=5, shuffle=True, random_state=42)\nmodels = []\nfold_scores = []\n\nfor fold, (train_idx, val_idx) in enumerate(kf.split(image_paths)):\n    print(f\"\\n{'='*20} Fold {fold+1}/5 {'='*20}\")\n    \n    train_dataset = NormalizedDataset(image_paths[train_idx], targets[train_idx], train_transform)\n    val_dataset = NormalizedDataset(image_paths[val_idx], targets[val_idx], val_transform)\n    \n    train_loader = DataLoader(train_dataset, batch_size=16, shuffle=True, num_workers=2)\n    val_loader = DataLoader(val_dataset, batch_size=16, shuffle=False, num_workers=2)\n    \n    model = ImprovedCNN().to(device)\n    \n    # Huber loss - more robust to outliers than MSE\n    criterion = nn.HuberLoss(delta=1.0)\n    \n    optimizer = optim.AdamW(model.parameters(), lr=1e-3, weight_decay=1e-4)\n    \n    # Cosine annealing with warm restarts\n    scheduler = optim.lr_scheduler.CosineAnnealingWarmRestarts(optimizer, T_0=10, T_mult=2)\n    \n    best_r2 = -999\n    best_model_state = None\n    epochs = 40\n    \n    for epoch in range(epochs):\n        train_loss = train_epoch(model, train_loader, criterion, optimizer, device)\n        val_preds, val_targets = validate(model, val_loader, device)\n        r2 = competition_r2(val_targets, val_preds)\n        scheduler.step()\n        \n        if r2 > best_r2:\n            best_r2 = r2\n            best_model_state = {k: v.cpu().clone() for k, v in model.state_dict().items()}\n        \n        if (epoch + 1) % 10 == 0:\n            current_lr = optimizer.param_groups[0]['lr']\n            print(f\"  Epoch {epoch+1:2d}: Loss={train_loss:.4f}, R²={r2:.4f}, Best={best_r2:.4f}, LR={current_lr:.6f}\")\n    \n    # Load best model for this fold\n    model.load_state_dict(best_model_state)\n    model = model.to(device)\n    models.append(model)\n    fold_scores.append(best_r2)\n    print(f\"  ✓ Fold {fold+1} Best R²: {best_r2:.4f}\")\n\nprint(\"\\n\" + \"=\"*50)\nprint(\"Training Complete!\")\nprint(f\"Fold scores: {[f'{s:.4f}' for s in fold_scores]}\")\nprint(f\"Mean CV R²: {np.mean(fold_scores):.4f} (+/- {np.std(fold_scores):.4f})\")\nprint(\"=\"*50)\n\n# ==================== TEST PREDICTIONS WITH TTA ====================\n\nprint(\"\\nGenerating test predictions with TTA...\")\n\ntest_df = pd.read_csv('/kaggle/input/csiro-biomass/test.csv')\ntest_df['image_path'] = '/kaggle/input/csiro-biomass/' + test_df['image_path']\n\nunique_images = test_df.drop_duplicates(subset='image_path')['image_path'].values\nprint(f\"Unique test images: {len(unique_images)}\")\n\nclass TestDatasetTTA(Dataset):\n    def __init__(self, paths, transform):\n        self.paths = paths\n        self.transform = transform\n    \n    def __len__(self):\n        return len(self.paths)\n    \n    def __getitem__(self, idx):\n        img = Image.open(self.paths[idx]).convert('RGB')\n        return self.transform(img)\n\n# Collect predictions from all models and all TTA transforms\nall_predictions = []\n\nfor model_idx, model in enumerate(models):\n    model.eval()\n    print(f\"  Model {model_idx + 1}/5 predicting...\")\n    \n    for tta_idx, tta_transform in enumerate(tta_transforms):\n        test_dataset = TestDatasetTTA(unique_images, tta_transform)\n        test_loader = DataLoader(test_dataset, batch_size=16, shuffle=False, num_workers=2)\n        \n        preds = []\n        with torch.no_grad():\n            for imgs in test_loader:\n                imgs = imgs.to(device)\n                outputs = model(imgs)\n                preds.append(denormalize_targets(outputs.cpu().numpy()))\n        \n        all_predictions.append(np.vstack(preds))\n\n# Average all predictions (5 models x 4 TTA = 20 predictions per image)\nfinal_preds = np.mean(all_predictions, axis=0)\nprint(f\"  Averaged {len(all_predictions)} prediction sets\")\n\n# Map predictions to image paths\nimage_to_preds = {img_path: final_preds[i] for i, img_path in enumerate(unique_images)}\n\n# ==================== SUBMISSION ====================\n\ntarget_to_idx = {\n    'Dry_Clover_g': 0,\n    'Dry_Dead_g': 1,\n    'Dry_Green_g': 2,\n    'Dry_Total_g': 3,\n    'GDM_g': 4\n}\n\nsubmission = []\n\nfor _, row in test_df.iterrows():\n    img_path = row['image_path']\n    target_name = row['target_name']\n    preds = image_to_preds[img_path]\n    target_idx = target_to_idx[target_name]\n    pred_value = preds[target_idx]\n    \n    submission.append({\n        'sample_id': row['sample_id'],\n        'target': pred_value\n    })\n\nsubmission_df = pd.DataFrame(submission)\nsubmission_df.to_csv('submission.csv', index=False)\n\nprint(f\"\\n{'='*50}\")\nprint(\"✅ Submission saved: submission.csv\")\nprint(f\"Rows: {len(submission_df)}\")\nprint(\"=\"*50)\n\nprint(\"\\nModel summary:\")\nprint(\"  - Custom CNN (no pretrained weights needed)\")\nprint(\"  - 224x224 input images\")\nprint(\"  - 5-fold ensemble\")\nprint(\"  - 4x TTA (original + flips)\")\nprint(\"  - 40 epochs per fold\")\nprint(\"  - Huber loss (robust to outliers)\")\nprint(\"  - CosineAnnealingWarmRestarts scheduler\")\nprint(\"  - Strong augmentation (crop, flip, rotate, color, affine)\")\nprint(f\"\\nMean CV R²: {np.mean(fold_scores):.4f}\")","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2026-01-16T23:56:58.196900Z","iopub.execute_input":"2026-01-16T23:56:58.197548Z","iopub.status.idle":"2026-01-16T23:57:24.300925Z","shell.execute_reply.started":"2026-01-16T23:56:58.197523Z","shell.execute_reply":"2026-01-16T23:57:24.299494Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"# Submit to competition","metadata":{}}]}