{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"name":"python","version":"3.11.13","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"},"kaggle":{"accelerator":"tpuV5e8","dataSources":[{"sourceId":112509,"databundleVersionId":14254895,"sourceType":"competition"},{"sourceId":14484911,"sourceType":"datasetVersion","datasetId":8767956},{"sourceId":290747047,"sourceType":"kernelVersion"}],"dockerImageVersionId":31089,"isInternetEnabled":true,"language":"python","sourceType":"notebook","isGpuEnabled":false}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# Overview and main features","metadata":{}},{"cell_type":"markdown","source":"**1. Import packages and setup environment (CPU/GPU/TPU):**\n* Automatic recognition of TPU environment and installation of necessary packages (as for TPU accelerator currently no dependency installation code is available).\n\n**2. Load and preprocess data into datasets:**\n* Preprocess images with concatenating (CONCAT_IMAGE == True) or cropping (to get square image by preserving aspect ratio) and resizing to target size (image_size).\n* Split data into train and validation sets in a stratified manner by discretization of a single or multiple labels (*EXTENDED_STRAT == True*).\n* Samples with possibly wrong labels can be removed from the training dataset (*REMOVE_SUSPECIOUS_SAMPLES == True*).\n* 5-Fold training strategy can be activated (*FOLD_TRAINING == True*) resulting in 5 independent models for inference\n\n**3. Explore data:**\n* Show some samples including labels and metadata from validation dataset.\n* Show distribution of labels in train and validation sets for each fold to check split \"quality\".\n\n**4. Build and compile neural network:**\n* Custom loss and metric based on competition specific R2 score.\n* Custom augmentation layer with random horizontal/vertical flipping, brightness, contrast and transposing functionality (Info: Tensorflow RandomTranslation layer was not working on TPU).\n* Neural network with pretrained ConvNeXtXLarge (https://arxiv.org/abs/2201.03545) for feature extraction + DNN layers for regression. Optionally, 2 of 5 labels can be determined based on the other 3 labels (*REDUNDANT_TARGETS_CALC == True*) instead of using the last fully connected layer for each label independently.\n* Selected layers of base model can be re-trained. Current configuration finetunes the last 9 layer of base feature extraction model. The finetuned feature extraction model can be then re-used as feature extraction model for classical ML methods (see linked notebook for reference).\n* During submission (*SUBMISSION == True*) no training occurs only prediction with loaded model. Owing to an unsolved problem with the custom loss/metric, the validation batch size must be set to 4 during inference on GPU and to 72 during training on TPU.\n\n**5. Training:**\n* One can choose between normal training (*TRAINING == True*), tuning (*TUNING > 0*) and fine-tuning of an already trained network (*FINETUNING == True*).\n* Training/tuning is only possible on TPU kernel owing to OOM issues on GPU. TPU is working properly only with the original environment (do not update to latest one).\n* 3 possible tuner configurations (GridSearch/RandomSearch/Hyperband) for hyperparameter search.\n\n**6. Evaluation:**\n* Show training curves (metric/loss/learning rate).\n* Compare validation and predicted labels.\n* Show the worse predictions on validation data including labels and predictions.\n\n**7. Submission:**\n* Predict labels of test images and create submission.csv file.\n* Same augmentation being used for training images can be used also during test time (*TEST_AUG_FAC > 0*).","metadata":{}},{"cell_type":"markdown","source":"# 1. Import packages and setup environment (CPU/GPU/TPU)","metadata":{}},{"cell_type":"code","source":"## For TPU environment (install missing packages / reinstall tensorflow to solve NaN topic during training / restart kernel)\n\nimport IPython\nimport tensorflow as tf\nIPython.display.clear_output() # Workaround for error messages leading to Failed notebook\nprint('Tensorflow version: '+ tf.__version__)\n\nif len(tf.config.experimental.list_logical_devices('TPU')) > 0:\n    !pip install -q tensorflow-tpu -f https://storage.googleapis.com/libtpu-tf-releases/index.html --force-reinstall\n    !pip install -q pydot\n    !pip install -q -U keras-tuner\n    !pip install -q protobuf==5.29.5 # to solve tuner compatibility issue\n    IPython.Application.instance().kernel.do_shutdown(True)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2026-01-08T07:53:43.046160Z","iopub.execute_input":"2026-01-08T07:53:43.046589Z","iopub.status.idle":"2026-01-08T07:53:43.056804Z","shell.execute_reply.started":"2026-01-08T07:53:43.046552Z","shell.execute_reply":"2026-01-08T07:53:43.056049Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"## Import packages\n\n# General purpose modules\nimport shutil\nimport re\nimport gc\nimport math\nfrom tqdm import tqdm\nfrom PIL import Image\nimport warnings\n\n# Data handling and visualization modules\nimport numpy as np\nimport pandas as pd\nimport matplotlib.pyplot as plt\nfrom matplotlib.pyplot import imshow\nimport seaborn as sns\n\n# Skikit-learn preprocessing modules\nfrom sklearn.model_selection import StratifiedShuffleSplit\nfrom sklearn.model_selection import StratifiedKFold\nfrom sklearn.model_selection import StratifiedGroupKFold\nfrom sklearn.model_selection import GridSearchCV\nfrom sklearn.pipeline import Pipeline\nfrom sklearn.preprocessing import KBinsDiscretizer\nfrom sklearn.preprocessing import LabelEncoder\nfrom sklearn.metrics import r2_score\n\n# Tensorflow modules\nimport tensorflow as tf\nimport keras_tuner as kt\nprint('Tensorflow version: '+ tf.__version__)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2026-01-13T17:56:39.504979Z","iopub.execute_input":"2026-01-13T17:56:39.506055Z","iopub.status.idle":"2026-01-13T17:57:01.485366Z","shell.execute_reply.started":"2026-01-13T17:56:39.506019Z","shell.execute_reply":"2026-01-13T17:57:01.484411Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"## Detect hardware (CPU/GPU/TPU), setup environment and return appropriate distribution strategy\n\ntry:\n    tpu = tf.distribute.cluster_resolver.TPUClusterResolver.connect(tpu='local') # set tpu is local as it should be available in the VM\n    print('✅ Running on TPU ', tpu.master())\nexcept:\n    print('❌ Using CPU/GPU')\n    tpu = None\n\nif tpu:\n    strategy = tf.distribute.TPUStrategy(tpu)\nelse:\n    strategy = tf.distribute.get_strategy() # default distribution strategy in Tensorflow. Works on CPU and single GPU.\n\nprint(\"REPLICAS: \", strategy.num_replicas_in_sync)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2026-01-08T07:53:43.071745Z","iopub.execute_input":"2026-01-08T07:53:43.072032Z","iopub.status.idle":"2026-01-08T07:53:43.423362Z","shell.execute_reply.started":"2026-01-08T07:53:43.071992Z","shell.execute_reply":"2026-01-08T07:53:43.422664Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"# 2. Load and preprocess data into datasets","metadata":{}},{"cell_type":"code","source":"## Read csv files and merge them into a single dataframe\n\npath = '/kaggle/input/csiro-biomass/'\ntrainval = pd.read_csv(path + \"train.csv\")","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2026-01-08T07:53:43.425375Z","iopub.execute_input":"2026-01-08T07:53:43.425716Z","iopub.status.idle":"2026-01-08T07:53:43.440763Z","shell.execute_reply.started":"2026-01-08T07:53:43.425676Z","shell.execute_reply":"2026-01-08T07:53:43.440045Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"## Preprocessing functions\n\nimage_size = 512 # input image size for neural network model\nCONCAT_IMAGE = False # choose concat and resize or crop and resize as image preprocessing\n\n# Pad and resize images (while retaining aspect ratio)\ndef pad_and_resize(image):\n    image_size_rows, image_size_cols, _ = image.shape\n    pad_size = max(image_size_rows, image_size_cols)\n    if CONCAT_IMAGE:\n        image = tf.concat([image, image], axis=0)\n    image_padded = tf.image.resize_with_crop_or_pad(image, pad_size, pad_size)\n    image_resized = tf.image.resize(image_padded, [image_size, image_size])\n    return image_resized\n\n# Flip/translate images\ndef image_augmentation(image, flip_horizontal=False, flip_vertical=False, roll=False):\n    if flip_horizontal:\n        image = tf.image.flip_up_down(image)\n    if flip_vertical:\n        image = tf.image.flip_left_right(image)\n    if roll:\n        roll_y = np.random.randint(-1000, 1000)\n        image = tf.roll(image, roll_y, axis=-2)\n        if CONCAT_IMAGE:\n            roll_x = np.random.randint(-1000, 1000)\n            image = tf.roll(image, roll_x, axis=-3)\n    return image\n    \n# Preprocess image (Padding and resizing to image_size x image_size x 3)\ndef preprocess_images(image, flip_horizontal=False, flip_vertical=False, roll=False):\n    image_scaled = image.astype(dtype=np.float32)/255\n    image_aug = image_augmentation(image_scaled, flip_horizontal=flip_horizontal, flip_vertical=flip_vertical, roll=roll)\n    image_resized = pad_and_resize(image_aug)\n    image_resized = tf.cast(image_resized*255, dtype=tf.uint8)\n    return image_resized","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2026-01-08T07:53:43.441600Z","iopub.execute_input":"2026-01-08T07:53:43.441890Z","iopub.status.idle":"2026-01-08T07:53:43.451217Z","shell.execute_reply.started":"2026-01-08T07:53:43.441867Z","shell.execute_reply":"2026-01-08T07:53:43.450324Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"## Load data and preprocess images and labels\n\nREMOVE_SUSPECIOUS_SAMPLES = False # Remove samples with possibly wrong labels\nAUG_FAC = 0 # Multiplicative augmentation factor\ntarget_columns = ['Dry_Clover_g', 'Dry_Dead_g', 'Dry_Green_g', 'Dry_Total_g', 'GDM_g']\n\ndef load_data(data):\n    info_columns = ['Sampling_Date', 'State', 'Species', 'Pre_GSHH_NDVI', 'Height_Ave_cm', 'image_path']\n    images = []\n    labels = []\n    infos = []\n    \n    # Load images and corresponding labels/infos image by image\n    for image_path in tqdm(data['image_path'].unique()):\n        \n        # Read image and corresponding labels/metadata \n        if REMOVE_SUSPECIOUS_SAMPLES: # Skip suspecious images with possibly wrong labels\n            if any(sub in image_path for sub in ['ID230058600', 'ID1403107574', 'ID1337107565', 'ID1761544403', 'ID681680726']):\n                continue\n        image = np.array(Image.open(path+image_path))\n        data_slice = data[data['image_path'] == image_path]\n        info = data_slice.iloc[0][info_columns].values\n        slice_labels = []\n        for col in target_columns:\n            slice_label = data_slice['target'][data_slice['target_name'] == col].values[0]\n            slice_labels.append(slice_label)\n\n        # Augment image and append resulting images/labels/metadata in lists\n        if AUG_FAC > 0:\n            for i in range(AUG_FAC):\n                for flip_horizontal in [False, True]:\n                    for flip_vertical in [False, True]:\n                        roll = flip_horizontal or flip_vertical or i>0 # first image smaple shall not be augmented\n                        image_resized = preprocess_images(image, flip_horizontal=flip_horizontal, flip_vertical=flip_vertical, roll=roll)\n                        images.append(image_resized)\n                        label = np.stack(slice_labels, axis=0)\n                        labels.append(label)\n                        infos.append(info)\n        else:\n            image_resized = preprocess_images(image)\n            images.append(image_resized)\n            label = np.stack(slice_labels, axis=0)\n            labels.append(label)\n            infos.append(info)\n\n    # Stack images/labels/infos lists into images array/labels dataframe/infos dataframe\n    images = np.stack(images, axis=0)\n    labels = pd.DataFrame(labels, columns=target_columns)\n    infos = pd.DataFrame(infos, columns=info_columns)\n    return images, labels, infos, \n\ntrainval_images, trainval_labels, trainval_info = load_data(trainval)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2026-01-08T07:53:43.452383Z","iopub.execute_input":"2026-01-08T07:53:43.452673Z","iopub.status.idle":"2026-01-08T07:54:17.390237Z","shell.execute_reply.started":"2026-01-08T07:53:43.452652Z","shell.execute_reply":"2026-01-08T07:54:17.389506Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"## Split train/validation data in a stratified manner by discretization of a single or multiple labels \n\nEXTENDED_STRAT = False # discretization with multiple labels \nFOLD_TRAINING = False # activate 5-Fold training strategy resulting in 5 independent models for inference\n\n# Determine stratification classes by discretization of a single or multiple labels \nif EXTENDED_STRAT: # discretization with multiple labels \n    kbins_multi = KBinsDiscretizer(n_bins=[3,3,6], encode='ordinal', strategy='quantile', random_state=42)\n    strat_bins = pd.DataFrame(kbins_multi.fit_transform(trainval_labels[['Dry_Clover_g', 'Dry_Dead_g', 'Dry_Green_g']]))\n    strat_classes = LabelEncoder().fit_transform(strat_bins.astype(str).agg('_'.join, axis=1))\nelse: # discretization with a single label\n    kbins = KBinsDiscretizer(n_bins=72, encode='ordinal', strategy='quantile', random_state=42)\n    strat_classes = kbins.fit_transform(trainval_labels[['Dry_Total_g']])\n\n# Define split generators\nif AUG_FAC > 0:\n    skf = StratifiedGroupKFold(n_splits=5, shuffle=True, random_state=42).split(trainval_images, strat_classes,\n                                                                                trainval_info['image_path'])\nelse:\n    sss = StratifiedShuffleSplit(n_splits=5, test_size=0.200, random_state=42).split(trainval_images, strat_classes)\n    skf = StratifiedKFold(n_splits=5, shuffle=True, random_state=42).split(trainval_images, strat_classes)\n\n# Split train/validation data\nfor i, (train_idx, val_idx) in enumerate(sss):\n    globals()[f'train_images{i+1}'] = trainval_images[train_idx]\n    globals()[f'val_images{i+1}'] = trainval_images[val_idx]\n    globals()[f'train_labels{i+1}'] = trainval_labels.iloc[train_idx].reset_index(drop=True)\n    globals()[f'val_labels{i+1}'] = trainval_labels.iloc[val_idx].reset_index(drop=True)\n    globals()[f'train_info{i+1}'] = trainval_info.iloc[train_idx].reset_index(drop=True)\n    globals()[f'val_info{i+1}'] = trainval_info.iloc[val_idx].reset_index(drop=True)\n    if not FOLD_TRAINING:\n        break\n\n# Verify sample sizes\nprint(f\"Total samples:      {int(len(trainval_images))}\")\nprint(f\"Dev train samples:  {int(len(train_images1))} ({len(train_images1)/len(trainval_images):.2%})\")\nprint(f\"Dev valid samples:  {int(len(val_images1))} ({len(val_images1)/len(trainval_images):.2%})\")","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2026-01-08T07:54:17.391390Z","iopub.execute_input":"2026-01-08T07:54:17.391695Z","iopub.status.idle":"2026-01-08T07:54:17.494601Z","shell.execute_reply.started":"2026-01-08T07:54:17.391672Z","shell.execute_reply":"2026-01-08T07:54:17.493757Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"## Create train and validation datasets for each fold\n\nSUBMISSIONING = False # during submission only inference with loaded model(s)\nSEED = 42\nbatch_size = 32\nbatch_size_val = 4 if SUBMISSIONING else 72 # shall be set to 4 during inference on GPU and to 72 during training on TPU\nsteps_per_epoch = len(train_labels1)//batch_size\n\nfor i in range(5 if FOLD_TRAINING else 1):\n    train_ds = tf.data.Dataset.from_tensor_slices((globals()[f'train_images{i+1}'], globals()[f'train_labels{i+1}']))\n    globals()[f'train_ds{i+1}'] = train_ds.shuffle(len(train_labels1), seed=SEED).repeat().batch(\n        batch_size, drop_remainder=True).prefetch(tf.data.AUTOTUNE)\n    val_ds = tf.data.Dataset.from_tensor_slices((globals()[f'val_images{i+1}'], globals()[f'val_labels{i+1}']))\n    globals()[f'val_ds{i+1}'] = val_ds.batch(batch_size_val, drop_remainder=True).prefetch(tf.data.AUTOTUNE)\n\n# Verify dataset sizes\nprint('Size of train dataset: '+ str(len(train_labels1)))\nprint('Number of batches in train dataset: '+ f'{len(train_labels1)//batch_size}')\nprint('Size of validation dataset: '+ str(len(val_labels1)))\nprint('Number of batches in val dataset: '+ f'{len(val_labels1)//batch_size_val}')\n\n# Check validation dataset batch dimensions\nfor X, y in val_ds1.take(1):\n    print(f'Image tensor dimensions of validation batches: {X.shape}')\n    print(f'Label tensor dimensions of validation batches: {y.shape}')","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2026-01-08T07:54:17.495503Z","iopub.execute_input":"2026-01-08T07:54:17.495735Z","iopub.status.idle":"2026-01-08T07:54:18.201176Z","shell.execute_reply.started":"2026-01-08T07:54:17.495715Z","shell.execute_reply":"2026-01-08T07:54:18.200344Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"# 3. Explore data","metadata":{}},{"cell_type":"code","source":"## Show some samples including labels and metadata from validation dataset\n\nnum_examples = 16\nnum_columns = 4\nnum_rows = math.ceil(num_examples/num_columns)\ngroup_size = max(4*AUG_FAC, 1)\nplt.figure(figsize=(16, 16))\nfor i, image in enumerate(val_images1[::group_size][:num_examples]):\n    plt.subplot(num_rows, num_columns, i + 1)\n    plt.imshow(image)\n    plt.text(3, 100, str(val_labels1.iloc[::group_size].iloc[i]), fontsize=8, color='green')\n    plt.text(3, 500, str(val_info1.iloc[::group_size].iloc[i]), fontsize=6, color='white')\n    plt.suptitle(\"Examples from train dataset\")\n    plt.xticks([])\n    plt.yticks([])\nplt.tight_layout()","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2026-01-08T07:54:18.202125Z","iopub.execute_input":"2026-01-08T07:54:18.202682Z","iopub.status.idle":"2026-01-08T07:54:21.036393Z","shell.execute_reply.started":"2026-01-08T07:54:18.202655Z","shell.execute_reply":"2026-01-08T07:54:21.035532Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"## Show distribution of labels in train and validation sets for each fold to check split \"quality\"\n\nif FOLD_TRAINING:\n    df_plot = pd.concat([globals()[f'{set}_labels{i+1}'].assign(Source=f'{set}_fold{i+1}') for i in range(5) for set in ['train', 'val']])\nelse:\n    df_plot = pd.concat([train_labels1.assign(Source='Train'), val_labels1.assign(Source='Validation')])\nwarnings.filterwarnings(\"ignore\", category=FutureWarning, module=\"seaborn\") # Suppress the specific FutureWarning\n\nn_cols = 3\nn_rows = (len(target_columns) + n_cols - 1) // n_cols\nfig, axes = plt.subplots(n_rows, n_cols, figsize=(15, 4 * n_rows))\nfig.suptitle('Label distributions in train and validation sets for each fold ')\naxes = axes.flatten()\nfor i in range(len(target_columns), len(axes)):\n    axes[i].axis('off')\nfor i, col in enumerate(target_columns):\n    sns.kdeplot(data=df_plot, x=col, ax=axes[i], hue='Source', common_norm=False, fill=False)\nplt.tight_layout()\nplt.show()\ndel df_plot","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2026-01-08T07:54:21.039218Z","iopub.execute_input":"2026-01-08T07:54:21.039649Z","iopub.status.idle":"2026-01-08T07:54:22.086347Z","shell.execute_reply.started":"2026-01-08T07:54:21.039613Z","shell.execute_reply":"2026-01-08T07:54:22.085331Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"# 4. Build and compile neural network","metadata":{}},{"cell_type":"code","source":"## Weighted R2 function\n\n# Label weights for loss and metric calculation ['Dry_Clover_g', 'Dry_Dead_g', 'Dry_Green_g', 'Dry_Total_g', 'GDM_g']\nloss_column_weights = [0.1, 0.1, 0.1, 0.5, 0.2]\nmetric_column_weights = [0.1, 0.1, 0.1, 0.5, 0.2]\n\ndef r2_score(y_true, y_pred):\n    y_true = tf.cast(y_true, dtype=tf.float32)\n    y_pred = tf.cast(y_pred, dtype=tf.float32)\n    sample_weights = tf.constant([metric_column_weights for i in range(y_true.shape[0])])\n    weighted_mean = tf.reduce_sum(tf.multiply(y_true, sample_weights)) / tf.reduce_sum(sample_weights)\n    ss_res = tf.reduce_sum(tf.multiply((y_true - y_pred)**2, sample_weights))\n    ss_tot = tf.reduce_sum(tf.multiply((y_true - weighted_mean)**2, sample_weights))\n    r2_weighted = 1 - (ss_res/ss_tot)\n    return r2_weighted","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2026-01-08T07:54:22.087352Z","iopub.execute_input":"2026-01-08T07:54:22.087638Z","iopub.status.idle":"2026-01-08T07:54:22.094674Z","shell.execute_reply.started":"2026-01-08T07:54:22.087616Z","shell.execute_reply":"2026-01-08T07:54:22.093816Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"## Custom weighted MSE loss functions\n\n@tf.keras.utils.register_keras_serializable()\ndef weighted_mse(y_true, y_pred):\n    y_true = tf.cast(y_true, dtype=tf.float32)\n    y_pred = tf.cast(y_pred, dtype=tf.float32)\n    sample_weights = tf.constant([loss_column_weights for i in range(y_true.shape[0])])\n    # Compute binary cross-entropy\n    w_mse = tf.reduce_mean(tf.multiply(tf.square(y_true - y_pred), sample_weights))\n    return w_mse","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2026-01-08T07:54:22.095658Z","iopub.execute_input":"2026-01-08T07:54:22.095936Z","iopub.status.idle":"2026-01-08T07:54:22.116164Z","shell.execute_reply.started":"2026-01-08T07:54:22.095916Z","shell.execute_reply":"2026-01-08T07:54:22.115272Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"## Custom weighted R2 metric class\n\n@tf.keras.utils.register_keras_serializable()\nclass R2_score(tf.keras.metrics.Metric):\n    def __init__(self, name='r2_score', **kwargs):\n        super().__init__(name=name, **kwargs)\n        self.r2_score_fn = r2_score\n        self.total = self.add_weight(shape=(), name=\"total\", initializer=\"zeros\")\n        self.count = self.add_weight(shape=(), name=\"count\", initializer=\"zeros\")\n    def update_state(self, y_true, y_pred, sample_weight=None):\n        metric = self.r2_score_fn(y_true, y_pred)\n        self.total.assign_add(metric)\n        self.count.assign_add(tf.cast(1, tf.float32))\n    def result(self):\n        return self.total / self.count","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2026-01-08T07:54:22.117120Z","iopub.execute_input":"2026-01-08T07:54:22.117470Z","iopub.status.idle":"2026-01-08T07:54:22.133692Z","shell.execute_reply.started":"2026-01-08T07:54:22.117448Z","shell.execute_reply":"2026-01-08T07:54:22.132828Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"## Preprocessing and image augmentation (horizontal/vertical flipping, brightness, contrast and transposing) layers\n\n@tf.keras.utils.register_keras_serializable()\nclass PreProcess(tf.keras.layers.Layer):\n    def __init__(self, name, **kwargs):\n        super(PreProcess, self).__init__(**kwargs)\n        self.name = name\n        self.preprocess_input = lambda x: x # Only place holder, ConvNeXtLarge has a build-in preprocessing layer\n    def call(self, inputs):\n        return self.preprocess_input(inputs)\n\n@tf.keras.utils.register_keras_serializable()\nclass Augmentation(tf.keras.layers.Layer):\n    def __init__(self, name, max_roll_x=100, max_roll_y=100, **kwargs):\n        super(Augmentation, self).__init__(**kwargs)\n        self.name = name\n        self.max_roll_x = max_roll_x\n        self.max_roll_y = max_roll_y\n        self.random_flip_left_right = tf.image.random_flip_left_right\n        self.random_flip_up_down = tf.image.random_flip_up_down\n        self.roll_fn = tf.roll\n        self.roll_x = tf.Variable(0, name='roll_x', dtype=tf.int64, trainable=False)\n        self.roll_y = tf.Variable(0, name='roll_y', dtype=tf.int64, trainable=False)\n    def call(self, inputs, training=True):\n        x = inputs\n        if training:\n            if AUG_FAC == 0:\n                x = self.random_flip_left_right(x)\n                x = self.random_flip_up_down(x)\n            #x = tf.image.random_brightness(x, max_delta=0.2)\n            #x = tf.image.random_contrast(x, 0.8, 1.2)\n            self.roll_x.assign(tf.random.uniform(shape=(), minval=-self.max_roll_x, maxval=self.max_roll_x, dtype=tf.int64))\n            self.roll_y.assign(tf.random.uniform(shape=(), minval=-self.max_roll_y, maxval=self.max_roll_y, dtype=tf.int64))\n            x = self.roll_fn(x, self.roll_y, axis=-2)\n            x = self.roll_fn(x, self.roll_x, axis=-3)\n        return x","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2026-01-08T07:54:22.134575Z","iopub.execute_input":"2026-01-08T07:54:22.134851Z","iopub.status.idle":"2026-01-08T07:54:22.152131Z","shell.execute_reply.started":"2026-01-08T07:54:22.134828Z","shell.execute_reply":"2026-01-08T07:54:22.151280Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"## Build architecture with pre-trained ConvNeXtXLarge\n\nREDUNDANT_TARGETS_CALC = False\nN_MODELS = 2\n\ndef build_network(hp):\n    # Define the sets of inputs\n    input_img = tf.keras.Input(shape=(image_size, image_size, 3), name='input_img')\n\n    # Augmentation layer\n    max_roll_y = 100 #hp.Choice(name='max_roll_y', values=[0, 50, 100, 150, 200, 250])\n    x = Augmentation(name='image_augmentation', max_roll_y=max_roll_y)(input_img)\n\n    # Cast, rescale and preprocess image tensors (if necessary)\n    x = PreProcess(name='preprocess_images')(x)\n\n    # Load a pretrained ConvNeXtBase as base model\n    base_model = tf.keras.applications.ConvNeXtXLarge(include_top=False, include_preprocessing=True,  input_shape=[image_size, image_size, 3],\n                                                      pooling='avg', weights='imagenet')\n    \n    # Choose base model layers to be trained\n    base_model.trainable = False # freeze base model layers\n    layer_id = -9 #hp.Choice(name='layer_id', values=[-1, -5, -9, -12, -16]) # layer number from the network shall be trained\n    print('Unfreeze base model layers from layer ' + str(base_model.layers[layer_id]))\n    for layer in base_model.layers[layer_id:]: # unfreeze choosen layers\n        layer.trainable = True\n\n    # Feature extraction\n    x = base_model(x)\n    \n    # Define DNN layer(s) for regression\n    x_reg = x\n    l1 = 0.07 #hp.Float(name='l1_dnn', min_value=0.07, max_value=0.09, step=0.02, default=None)\n    l2 = 0.15 #hp.Float(name='l2_dnn', min_value=0.10, max_value=0.20, step=0.05, default=None)\n    reg_dnn = tf.keras.regularizers.L1L2(l1=l1, l2=l2)\n    ki_dnn = 'he_normal' #hp.Choice(name='ki_dnn', values=['he_normal', 'he_uniform', 'glorot_normal', 'glorot_uniform', 'lecun_normal', 'lecun_uniform'])\n    act_dnn = hp.Choice(name='act_dnn', values=['relu', 'elu', 'gelu', 'selu', 'leaky_relu'])\n    do_dnn = 0.1 #hp.Float(name='do_dnn', min_value=0.0, max_value=0.5, step=0.1, default=0.1)\n    layers_final = 2 #hp.Int(name='layers_final_reg', min_value=1, max_value=2, step=1, default=2)\n    units_final = 384 #hp.Int(name='units_final_reg', min_value=128, max_value=384, step=128, default=384)\n    for i, units in enumerate([units_final for j in range(layers_final)]):\n        x_reg = tf.keras.layers.Dense(units, activation=act_dnn, kernel_regularizer=reg_dnn,\n                                      kernel_initializer=ki_dnn, name=f'final_fc{i+1}')(x_reg)\n        x_reg = tf.keras.layers.Dropout(do_dnn, name=f'final_do{i+1}')(x_reg)\n\n    # Define final layer (2 of 5 labels can be determined based on the other 3 labels or each label can be determined independently)\n    if REDUNDANT_TARGETS_CALC:\n        out = tf.keras.layers.Dense(3, activation='relu', kernel_regularizer=reg_dnn,\n                                    kernel_initializer=ki_dnn, name='final_reg')(x_reg)\n        out3 = tf.keras.layers.Add(name='add_total')([out[:,0:1], out[:,1:2], out[:,2:3]])\n        out4 = tf.keras.layers.Add(name='add_GDM')([out[:,0:1], out[:,2:3]])\n        out = tf.keras.layers.Concatenate(name='concat_final')([out, out3, out4])\n    else:\n        out = tf.keras.layers.Dense(5, activation='relu', kernel_regularizer=reg_dnn,\n                                    kernel_initializer=ki_dnn, name='final_reg')(x_reg)\n\n    # Define model\n    model = tf.keras.Model(inputs=input_img, outputs=out, name='CSIRO_Reg')\n\n    # Define optimizer/loss and compile model\n    lr_tune = 5e-3 #hp.Choice(name='learning_rate', values=[5e-4, 1e-3, 3e-3, 5e-3, 1e-2, 3e-2])\n    optimizer = tf.keras.optimizers.Adam(lr_tune)\n    loss = weighted_mse\n    metrics = R2_score()\n    model.compile(optimizer=optimizer, loss=loss, metrics=[metrics])\n    return model\n\nif not SUBMISSIONING:\n    with strategy.scope():\n        # Build models for training\n        for i in range(5 if FOLD_TRAINING else N_MODELS):\n            globals()[f'model{i+1}'] = build_network(kt.HyperParameters())\nelse:\n    # Load pre-trained model(s) for submission\n    for i in range(5 if FOLD_TRAINING else N_MODELS):\n        globals()[f'model{i+1}'] = tf.keras.models.load_model(f'/kaggle/input/csiro-1xx/csiro_1_106_{i+1}.keras')\n        print(f'Model{i+1} have been loaded!')","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2026-01-08T07:54:22.153034Z","iopub.execute_input":"2026-01-08T07:54:22.153325Z","iopub.status.idle":"2026-01-08T07:55:02.132497Z","shell.execute_reply.started":"2026-01-08T07:54:22.153296Z","shell.execute_reply":"2026-01-08T07:55:02.131744Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"## Explore model architecture\n\nmodel1.summary(line_length=110)\n# tf.keras.utils.plot_model(model1, to_file='model_architecture.png', show_shapes=True, show_dtype=False,\n#                           show_layer_names=True, show_layer_activations=True, show_trainable=False)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2026-01-08T07:55:02.133880Z","iopub.execute_input":"2026-01-08T07:55:02.134157Z","iopub.status.idle":"2026-01-08T07:55:02.161120Z","shell.execute_reply.started":"2026-01-08T07:55:02.134137Z","shell.execute_reply":"2026-01-08T07:55:02.160294Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"# 5. Training","metadata":{}},{"cell_type":"code","source":"## Training parameters\n\nepochs = 100\nTUNING = True and not SUBMISSIONING # only supported with single fold strategy (FOLD_TRAINING == False)\nTRAINING = False and not SUBMISSIONING\nFINETUNING = False and not SUBMISSIONING","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2026-01-08T07:55:02.162053Z","iopub.execute_input":"2026-01-08T07:55:02.162842Z","iopub.status.idle":"2026-01-08T07:55:02.170521Z","shell.execute_reply.started":"2026-01-08T07:55:02.162821Z","shell.execute_reply":"2026-01-08T07:55:02.169692Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"## Tuner configurations\n\nif TUNING:\n    i_TunerTyp = 1 # Choose desired tuner type: {1: 'grid', 2: 'random', 3: 'hyper'}\n    TunerStr = {1: 'grid', 2: 'random', 3: 'hyper'}\n    tuner_params = {'hypermodel': build_network, 'objective': kt.Objective(\"val_r2_score\", direction=\"max\"),\n                    'overwrite': True, 'directory': 'tuner', 'project_name': 'CSIRO', 'distribution_strategy': strategy}\n    \n    tuner_grid = kt.GridSearch(**tuner_params, max_trials=15, max_consecutive_failed_trials=1)\n    tuner_random = kt.RandomSearch(**tuner_params, max_trials=15, executions_per_trial=1)\n    tuner_hyper = kt.Hyperband(**tuner_params, max_epochs=60, factor=4, hyperband_iterations=1)\n    \n    tuner = globals()[f'tuner_{TunerStr[i_TunerTyp]}']\n    tuner.search_space_summary()","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2026-01-08T07:55:02.171218Z","iopub.execute_input":"2026-01-08T07:55:02.171476Z","iopub.status.idle":"2026-01-08T07:55:02.190683Z","shell.execute_reply.started":"2026-01-08T07:55:02.171446Z","shell.execute_reply":"2026-01-08T07:55:02.189877Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"## Train or tune model\n\n# Callback functions\nlr_scheduler = tf.keras.callbacks.ReduceLROnPlateau(factor=0.2, patience=8, verbose=1, monitor='val_r2_score', mode='max')\nearly_stopping_cb = tf.keras.callbacks.EarlyStopping(patience=16, verbose=1, monitor='val_r2_score', mode='max', restore_best_weights=True)\nlr_schedule = tf.keras.callbacks.LearningRateScheduler(lambda epoch: 1e-5 * 10**(epoch / 10)) # Find starting learning\n\n# Training\nif TRAINING or FINETUNING:\n    # Train models for each fold\n    for i in range(5 if FOLD_TRAINING else N_MODELS):\n        globals()[f'history{i+1}'] = globals()[f'model{i+1}'].fit(\n            globals()[f'train_ds{i+1}'], validation_data=globals()[f'val_ds{i+1}'],\n            epochs=epochs, steps_per_epoch=steps_per_epoch*5, callbacks=[lr_scheduler, early_stopping_cb])\n\n# Tuning\nif TUNING:  \n    tuner.search(train_ds1, validation_data=val_ds1, epochs=epochs, steps_per_epoch=steps_per_epoch*5,\n                 callbacks=[lr_scheduler, early_stopping_cb])\n    best_models = tuner.get_best_models(num_models=N_MODELS)\n    for i in range(N_MODELS):\n        globals()[f'model{i+1}'] = best_models[i]\n    model1.summary()\n    tuner.results_summary()\n    shutil.rmtree('/kaggle/working/tuner')\n    gc.collect()","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2026-01-08T07:55:02.191756Z","iopub.execute_input":"2026-01-08T07:55:02.192065Z","iopub.status.idle":"2026-01-08T07:55:02.209617Z","shell.execute_reply.started":"2026-01-08T07:55:02.192039Z","shell.execute_reply":"2026-01-08T07:55:02.208706Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"## Save model(s) after training/finetuning/tuning\n\nif TRAINING or FINETUNING or TUNING:\n    for i in range(5 if FOLD_TRAINING else N_MODELS):\n        globals()[f'model{i+1}'].save(f'csiro_1_107_{i+1}.keras', include_optimizer=False)\n        print(f'Model{i+1} have been saved!')","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2026-01-08T07:55:02.210637Z","iopub.execute_input":"2026-01-08T07:55:02.210926Z","iopub.status.idle":"2026-01-08T07:55:02.232445Z","shell.execute_reply.started":"2026-01-08T07:55:02.210906Z","shell.execute_reply":"2026-01-08T07:55:02.231757Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"# 6. Evaluation","metadata":{}},{"cell_type":"code","source":"## Plot learning curves for each model\n\nif TRAINING or FINETUNING:\n    fig2, axes2 = plt.subplots(3, 1, figsize=(8, 16))\n    for i in range(5 if FOLD_TRAINING else 1):\n        history_fil = {f'{key} (fold{i+1})': globals()[f'history{i+1}'].history[key] for key in ['r2_score', 'val_r2_score']}\n        history_fil2 = {f'{key} (fold{i+1})': globals()[f'history{i+1}'].history[key] for key in ['loss', 'val_loss']}\n        history_fil3 = {f'{key} (fold{i+1})': globals()[f'history{i+1}'].history[key] for key in ['learning_rate']}\n        \n        pd.DataFrame(history_fil).plot(ax=axes2[0])\n        axes2[0].set_ylabel(\"R2_score\")\n        axes2[0].set_xlabel(\"epochs\")\n        pd.DataFrame(history_fil2).plot(ax=axes2[1])\n        axes2[1].set_ylabel(\"Loss(wMSE)\")\n        axes2[1].set_xlabel(\"epochs\")\n        pd.DataFrame(history_fil3).plot(ax=axes2[2])\n        axes2[2].set_ylabel(\"Learning rate\")\n        axes2[2].set_xlabel(\"epochs\")","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2026-01-08T07:55:02.233283Z","iopub.execute_input":"2026-01-08T07:55:02.233569Z","iopub.status.idle":"2026-01-08T07:55:02.251768Z","shell.execute_reply.started":"2026-01-08T07:55:02.233538Z","shell.execute_reply":"2026-01-08T07:55:02.250900Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"## Calculate predictions\n\nfold_pred_labels = []\nfold_val_labels = []\nfold_val_info = []\nfold_val_images = []\n\n# Predict and concat predictions/labels/info for each fold\nfor i in range(5 if FOLD_TRAINING else N_MODELS):\n    pred_labels_np = globals()[f'model{i+1}'].predict(globals()[f'val_ds{i+1 if FOLD_TRAINING else 1}'], verbose=0)\n    fold_pred_labels.append(pred_labels_np)\n    fold_val_labels.append(np.array(globals()[f'val_labels{i+1 if FOLD_TRAINING else 1}']))\n    fold_val_info.append(globals()[f'val_info{i+1 if FOLD_TRAINING else 1}'])\n    fold_val_images.append(globals()[f'val_images{i+1 if FOLD_TRAINING else 1}'])\n    # Custom score of single folds/models\n    print(f'Custom specific R2 score on validation data for fold/model {i+1} is: {r2_score(globals()[f\"val_labels{i+1 if FOLD_TRAINING else 1}\"], pred_labels_np)}')\nfold_pred_labels = np.concatenate(fold_pred_labels)\nfold_val_labels = np.concatenate(fold_val_labels)\nfold_val_info = pd.concat(fold_val_info).reset_index(drop=True)\nfold_val_images = np.concatenate(fold_val_images)\n\n# Custom score\nprint(f'Custom specific R2 score on validation data is: {r2_score(fold_val_labels, fold_pred_labels)}')","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2026-01-08T07:57:34.120226Z","iopub.execute_input":"2026-01-08T07:57:34.120662Z","iopub.status.idle":"2026-01-08T07:58:33.923969Z","shell.execute_reply.started":"2026-01-08T07:57:34.120627Z","shell.execute_reply":"2026-01-08T07:58:33.922719Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"## Show predicted/ground true values\n\neval_df = pd.DataFrame()\nfor i, col in enumerate(target_columns):\n    eval_df[col] = fold_val_labels[:, i]\n    eval_df[f'p_{col}'] = fold_pred_labels[:, i]\n    # Calculate relative error\n    eval_df[f're_{col}'] = np.round(((eval_df[col]-eval_df[f'p_{col}'])/(eval_df[col]+0.0001))*100, 1).abs()\neval_df['add_Dry_Total_g'] = eval_df['p_Dry_Clover_g'] + eval_df['p_Dry_Dead_g'] + eval_df['p_Dry_Green_g']\neval_df['add_GDM_g'] = eval_df['p_Dry_Clover_g'] + eval_df['p_Dry_Green_g']\neval_df","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2026-01-08T07:58:54.720997Z","iopub.execute_input":"2026-01-08T07:58:54.721868Z","iopub.status.idle":"2026-01-08T07:58:54.762726Z","shell.execute_reply.started":"2026-01-08T07:58:54.721837Z","shell.execute_reply":"2026-01-08T07:58:54.761687Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"## Show worse predictions from validation dataset\n\nworse_index = eval_df.sort_values(by='re_Dry_Total_g', ascending=False).index\npred_columns = [f'p_{col}' for col in target_columns]\n\n# Visualize data\nnum_examples = 16\nnum_columns = 4\nnum_rows = math.ceil(num_examples/num_columns)\nplt.figure(figsize=(16, 16))\nfor i, index in enumerate(worse_index[:num_examples]):\n    image = fold_val_images[index]\n    plt.subplot(num_rows, num_columns, i + 1)\n    plt.imshow(image)\n    plt.text(3, 110, str(eval_df[target_columns].loc[index]), fontsize=8, color='green')\n    plt.text(250, 110, str(eval_df[pred_columns].loc[index]), fontsize=8, color='red')\n    plt.text(3, 500, str(fold_val_info.loc[index]), fontsize=6, color='white')\n    plt.suptitle(\"Worse predictions from val dataset\")\n    plt.xticks([])\n    plt.yticks([])\nplt.tight_layout()","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2026-01-08T07:58:57.227046Z","iopub.execute_input":"2026-01-08T07:58:57.227383Z","iopub.status.idle":"2026-01-08T07:59:00.234642Z","shell.execute_reply.started":"2026-01-08T07:58:57.227359Z","shell.execute_reply":"2026-01-08T07:59:00.232876Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"# 7. Submission","metadata":{}},{"cell_type":"code","source":"## Test prediction & submission \n\nTEST_AUG_FAC = 1 # test time image augmentation factor\ntest = pd.read_csv(path + \"test.csv\")\ntest_pred = {}\n\ntest_images = []\ntest_paths = test['image_path'].unique()\nfor image_path in test_paths:\n    # Prepare tensor for each test image w/wo augmentation\n    image_id = re.split('/|.j', image_path)[1]\n    image = np.array(Image.open(path+image_path))\n    if TEST_AUG_FAC > 0:\n        test_images = []\n        for i in range(TEST_AUG_FAC):\n            for flip_horizontal in [False, True]:\n                for flip_vertical in [False, True]:\n                    roll = flip_horizontal or flip_vertical or i>0 # first image smaple shall not be augmented\n                    image_resized = preprocess_images(image, flip_horizontal=flip_horizontal, flip_vertical=flip_vertical, roll=roll)\n                    test_images.append(image_resized)\n        image_tensor = tf.stack(test_images, axis=0)\n    else:\n        image_resized = preprocess_images(image)\n        image_tensor = tf.expand_dims(image_resized, axis=0)\n\n    # Perform predictions for each fold\n    model_preds = []\n    for i in range(5 if FOLD_TRAINING else N_MODELS):\n        preds = globals()[f'model{i+1}'].predict(image_tensor, verbose=0)\n        model_preds.append(preds)\n    preds = np.concatenate(model_preds)\n\n    # Rearrange predictions in dictionary format of submission file\n    for i, col in enumerate(target_columns):\n        col_res = float(np.median(preds[:, i]))\n        test_pred[f'{image_id}__{col}'] = col_res\n\n# Create submission dataframe and .csv file\nsubmission_df = pd.DataFrame([{\"sample_id\": k, \"target\": v} for k, v in test_pred.items()])\nsubmission_df.to_csv(\"submission.csv\", index=False)\nprint(\"✅ submission.csv saved!\")\n\n# Show submission file\nsubmission_df","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2026-01-08T07:59:04.532388Z","iopub.execute_input":"2026-01-08T07:59:04.533181Z","iopub.status.idle":"2026-01-08T07:59:26.603532Z","shell.execute_reply.started":"2026-01-08T07:59:04.533156Z","shell.execute_reply":"2026-01-08T07:59:26.602394Z"}},"outputs":[],"execution_count":null}]}