{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"name":"python","version":"3.11.13","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"},"kaggle":{"accelerator":"gpu","dataSources":[{"sourceId":112509,"databundleVersionId":14254895,"sourceType":"competition"},{"sourceId":14434741,"sourceType":"datasetVersion","datasetId":8736719,"isSourceIdPinned":true},{"sourceId":291713943,"sourceType":"kernelVersion"}],"dockerImageVersionId":31192,"isInternetEnabled":false,"language":"python","sourceType":"notebook","isGpuEnabled":true}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# Overview and main features","metadata":{}},{"cell_type":"markdown","source":"**1. Import packages**\n\n**2. Load, preprocess and explore data:**\n* Preprocess images with concatenating (CONCAT_IMAGE == True) or cropping (to get square image by preserving aspect ratio) and resizing to target size (image_size).\n* Image augmentation with horizontal/vertical flipping and transposing can be activated (AUG_FAC > 0).\n* Split data into train and validation sets in a stratified manner by discretization of a single or multiple labels (EXTENDED_STRAT == True). Further we used grouping to keep augmented versions of same image together and prevent data leakage.\n* Samples with possibly wrong labels can be removed from the training dataset (REMOVE_SUSPECIOUS_SAMPLES == True).\n* 5-Fold training strategy can be activated (FOLD_TRAINING == True) resulting in 5 independent models for inference.\n* Show some samples including labels and metadata from validation dataset.\n* Show distribution of labels in train and validation sets to check split \"quality\".\n\n**3. Feature extraction:**\n* ConvNeXtXLarge, a pretrained neural network (https://arxiv.org/abs/2201.03545) without top layers is used for feature extraction. This model takes the preprocessed images as input and gives multidimensional features as output. Further, the last 9 layers of the model are retrained based on CSIRO dataset (see linked notebook for reference).\n* Scaling of generated features can be made additionally (*SCALING == True*). ConvNeXtXLarge has a normed output but some of the ML methods (e.g.: SVR) shows some improvement after additional normalization.\n\n**4. Define model space for ML methods:**\n* Definition of model space with a large number of possible ML methods:\n    * Linear models: Lasso / ElasticNet / Ridge / LassoLars\n    * Support vector machine model: SVR (linear)\n    * Ensemble models: RandomForestRegressor / GradientBoostingRegressor / AdaBoostRegressor / HistGradientBoostingRegressor / XGBRegressor / LGBMRegressor / CatBoostRegressor\n    * Other models: KNeighborsRegressor\n\n**5. Training:**\n* Definition of model parameters for single fitting and tuner interfaces.\n* Simple fitting or tuning (*TUNING_ML == True*) choosen models (EST_IDS) for each 5 labels on whole dataset.\n* PCA can be added into the pipeline to reduce number of features (*PCA_active == True*).\n* Print train and validation score for each estimator and each fold.\n\n**6. Evaluation:**\n* Compare validation and predicted labels.\n* Calculate competition specific weighted R2 score.\n* Show the worse predictions on validation data including labels and predictions.\n\n**7. Submission:**\n* Predict labels of test images and create submission.csv file.\n* If more than 1 model have been choosen in EST_IDS (Ensemble solution) then the prediction will be an average of the predictions of the single models.\n* Same augmentation being used for training images can be used also during test time (TEST_AUG_FAC > 0).\n* Optionally, 2 of 5 labels can be determined based on the other 3 labels (*TARGET_CALCULATED == True*) instead of using a model for each label independently.\n\n**8. Experimental code:**\n* Inactive draft features for experimental purposes.","metadata":{}},{"cell_type":"markdown","source":"# 1. Import packages","metadata":{}},{"cell_type":"code","source":"## Import packages\n\n# General purpose modules\nimport IPython\nimport os\nimport re\nimport math\nimport warnings\nimport gc\nfrom copy import deepcopy\nfrom tqdm import tqdm\nfrom PIL import Image\n\n# Data handling and visualization modules\nimport numpy as np\nimport pandas as pd\nimport matplotlib.pyplot as plt\nfrom matplotlib.pyplot import imshow\nimport seaborn as sns\n\n# Skikit-learn modules\nfrom sklearn.model_selection import StratifiedGroupKFold\nfrom sklearn.model_selection import StratifiedShuffleSplit\nfrom sklearn.model_selection import GridSearchCV\nfrom sklearn.pipeline import Pipeline\nfrom sklearn.pipeline import make_pipeline\nfrom sklearn.impute import SimpleImputer\nfrom sklearn.compose import ColumnTransformer\nfrom sklearn.preprocessing import KBinsDiscretizer\nfrom sklearn.preprocessing import StandardScaler\nfrom sklearn.preprocessing import RobustScaler\nfrom sklearn.preprocessing import MinMaxScaler\nfrom sklearn.preprocessing import OneHotEncoder\nfrom sklearn.preprocessing import LabelEncoder\nfrom sklearn.metrics import r2_score\n\nfrom sklearn.decomposition import PCA\nfrom sklearn.svm import SVR\nfrom sklearn.svm import LinearSVR\nfrom sklearn.neighbors import KNeighborsRegressor\nfrom sklearn.ensemble import RandomForestRegressor\nfrom sklearn.ensemble import GradientBoostingRegressor\nfrom sklearn.ensemble import HistGradientBoostingRegressor\nfrom sklearn.ensemble import AdaBoostRegressor\nfrom sklearn.tree import DecisionTreeRegressor\nfrom sklearn.linear_model import SGDRegressor\nfrom sklearn.linear_model import Lasso\nfrom sklearn.linear_model import ElasticNet\nfrom sklearn.linear_model import Ridge\nfrom sklearn.linear_model import LassoLars\n\n# Further ML modules\nimport xgboost as xgboost\nimport lightgbm as lightgbm\nfrom catboost import CatBoostRegressor\n\n# Tensorflow modules\nimport tensorflow as tf\nIPython.display.clear_output() # Workaround for AttributeError messages leading to Failed notebook\nprint('Tensorflow version: '+ tf.__version__)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2026-01-28T07:14:43.574776Z","iopub.execute_input":"2026-01-28T07:14:43.575210Z","iopub.status.idle":"2026-01-28T07:15:12.235015Z","shell.execute_reply.started":"2026-01-28T07:14:43.575172Z","shell.execute_reply":"2026-01-28T07:15:12.234370Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"# 2. Load, preprocess and explore data","metadata":{}},{"cell_type":"code","source":"## Read csv files and merge them into a single dataframe\n\npath = '/kaggle/input/csiro-biomass/'\ntrainval = pd.read_csv(path + \"train.csv\")","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2026-01-28T07:15:12.236229Z","iopub.execute_input":"2026-01-28T07:15:12.236826Z","iopub.status.idle":"2026-01-28T07:15:12.256843Z","shell.execute_reply.started":"2026-01-28T07:15:12.236784Z","shell.execute_reply":"2026-01-28T07:15:12.256021Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"## Preprocessing functions\n\nimage_size = 512 # input image size fo neural network model\nCONCAT_IMAGE = False # choose concat and resize or crop and resize as image preprocessing\n\n# Pad and resize images (while retaining aspect ratio)\ndef pad_and_resize(image):\n    image_size_rows, image_size_cols, _ = image.shape\n    pad_size = max(image_size_rows, image_size_cols)\n    if CONCAT_IMAGE:\n        image = tf.concat([image, image], axis=0)\n    image_padded = tf.image.resize_with_crop_or_pad(image, pad_size, pad_size)\n    image_resized = tf.image.resize(image_padded, [image_size, image_size])\n    return image_resized\n\n# Flip/translate images\ndef image_augmentation(image, flip_horizontal=False, flip_vertical=False, roll=False):\n    if flip_horizontal:\n        image = tf.image.flip_up_down(image)\n    if flip_vertical:\n        image = tf.image.flip_left_right(image)\n    if roll:\n        roll_y = np.random.randint(-1000, 1000)\n        image = tf.roll(image, roll_y, axis=-2)\n    return image\n    \n# Preprocess image (Padding and resizing to image_size x image_size x 3)\ndef preprocess_images(image, flip_horizontal=False, flip_vertical=False, roll=False):\n    image_scaled = image.astype(dtype=np.float32)/255\n    image_aug = image_augmentation(image_scaled, flip_horizontal=flip_horizontal, flip_vertical=flip_vertical, roll=roll)\n    image_resized = pad_and_resize(image_aug)\n    image_resized = tf.cast(image_resized*255, dtype=tf.uint8)\n    return image_resized","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2026-01-28T07:15:12.257727Z","iopub.execute_input":"2026-01-28T07:15:12.258204Z","iopub.status.idle":"2026-01-28T07:15:12.272471Z","shell.execute_reply.started":"2026-01-28T07:15:12.258176Z","shell.execute_reply":"2026-01-28T07:15:12.271834Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"## Load data and preprocess images and labels\n\nREMOVE_SUSPECIOUS_SAMPLES = False # Remove samples with possibly wrong labels\nAUG_FAC = 1 # Multiplicative augmentation factor\ntarget_columns = ['Dry_Clover_g', 'Dry_Dead_g', 'Dry_Green_g', 'Dry_Total_g', 'GDM_g']\n\ndef load_data(data):\n    info_columns = ['Sampling_Date', 'State', 'Species', 'Pre_GSHH_NDVI', 'Height_Ave_cm', 'image_path']\n    images = []\n    labels = []\n    infos = []\n\n    # Load images and corresponding labels/infos image by image\n    for image_path in tqdm(data['image_path'].unique()):\n\n        # Read image and corresponding labels and metadata \n        if REMOVE_SUSPECIOUS_SAMPLES: # Skip suspecious images with possibly wrong labels\n            if any(sub in image_path for sub in ['ID230058600', 'ID1403107574', 'ID1337107565', 'ID1761544403', 'ID681680726']):\n                continue\n        image = np.array(Image.open(path+image_path))\n        data_slice = data[data['image_path'] == image_path]\n        info = data_slice.iloc[0][info_columns].values\n        slice_labels = []\n        for col in target_columns:\n            slice_label = data_slice['target'][data_slice['target_name'] == col].values[0]\n            slice_labels.append(slice_label)\n\n        # Augment image and append resulting images/labels/metadata in lists\n        if AUG_FAC > 0:\n            for i in range(AUG_FAC):\n                for flip_horizontal in [False, True]:\n                    for flip_vertical in [False, True]:\n                        roll = flip_horizontal or flip_vertical or i>0 # first image smaple shall not be augmented\n                        image_resized = preprocess_images(image, flip_horizontal=flip_horizontal, flip_vertical=flip_vertical, roll=roll)\n                        images.append(image_resized)\n                        label = np.stack(slice_labels, axis=0)\n                        labels.append(label)\n                        infos.append(info)\n        else:\n            image_resized = preprocess_images(image)\n            images.append(image_resized)\n            label = np.stack(slice_labels, axis=0)\n            labels.append(label)\n            infos.append(info)\n\n    # Stack images/labels/infos lists into images array/labels dataframe/infos dataframe\n    images = np.stack(images, axis=0)\n    labels = pd.DataFrame(labels, columns=data_slice['target_name'].values)\n    infos = pd.DataFrame(infos, columns=info_columns)\n    return images, labels, infos\n\ntrainval_images, trainval_labels, trainval_info = load_data(trainval)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2026-01-28T07:15:12.273963Z","iopub.execute_input":"2026-01-28T07:15:12.274262Z","iopub.status.idle":"2026-01-28T07:16:42.405889Z","shell.execute_reply.started":"2026-01-28T07:15:12.274244Z","shell.execute_reply":"2026-01-28T07:16:42.405231Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"## Split train/validation data in a stratified manner by discretization of a single or multiple labels \n\nEXTENDED_STRAT = False # discretization with multiple labels\nFOLD_TRAINING = True # activate 5-Fold training strategy resulting in 5 independent models for inference\n\n# Determine stratification classes by discretization of a single or multiple labels \nif EXTENDED_STRAT: # discretization with multiple labels \n    kbins_multi = KBinsDiscretizer(n_bins=[2,3,5], encode='ordinal', strategy='quantile', random_state=42)\n    strat_bins = pd.DataFrame(kbins_multi.fit_transform(trainval_labels[['Dry_Clover_g', 'Dry_Dead_g', 'Dry_Green_g']]))\n    strat_classes = LabelEncoder().fit_transform(strat_bins.astype(str).agg('_'.join, axis=1))\nelse: # discretization with a single label\n    kbins = KBinsDiscretizer(n_bins=72, encode='ordinal', strategy='quantile', random_state=42)\n    strat_classes = kbins.fit_transform(trainval_labels[['Dry_Total_g']])\ntrainval_info['strat_classes'] = strat_classes\n\nif AUG_FAC > 0:\n    sgkf = StratifiedGroupKFold(n_splits=5, shuffle=True, random_state=0).split(trainval_images, trainval_info['strat_classes'],\n                                                                                 groups=trainval_info['image_path'])\nelse:\n    sgkf = StratifiedShuffleSplit(n_splits=5, test_size=0.200, random_state=42).split(trainval_images, trainval_info['strat_classes'])\n\n# Split train/validation data\nfor i, (train_idx, val_idx) in enumerate(sgkf):\n    globals()[f'train_idx{i+1}'] = train_idx\n    globals()[f'val_idx{i+1}'] = val_idx\n    globals()[f'train_images{i+1}'] = trainval_images[train_idx]\n    globals()[f'val_images{i+1}'] = trainval_images[val_idx]\n    globals()[f'train_labels{i+1}'] = trainval_labels.iloc[train_idx].reset_index(drop=True)\n    globals()[f'val_labels{i+1}'] = trainval_labels.iloc[val_idx].reset_index(drop=True)\n    globals()[f'train_info{i+1}'] = trainval_info.iloc[train_idx].reset_index(drop=True)\n    globals()[f'val_info{i+1}'] = trainval_info.iloc[val_idx].reset_index(drop=True)\n    if not FOLD_TRAINING:\n        break\n\n# Verify sample sizes\nprint(f\"Total samples:      {int(len(trainval_images))}\")\nprint(f\"Dev train samples:  {int(len(train_images1))} ({len(train_images1)/len(trainval_images):.2%})\")\nprint(f\"Dev valid samples:  {int(len(val_images1))} ({len(val_images1)/len(trainval_images):.2%})\")","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2026-01-28T07:16:42.406668Z","iopub.execute_input":"2026-01-28T07:16:42.406886Z","iopub.status.idle":"2026-01-28T07:16:44.139986Z","shell.execute_reply.started":"2026-01-28T07:16:42.406868Z","shell.execute_reply":"2026-01-28T07:16:44.139178Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"## Show some samples including labels and metadata from validation dataset\n\nnum_examples = 16\nnum_columns = 4\nnum_rows = math.ceil(num_examples/num_columns)\nplt.figure(figsize=(16, 16))\nfor i, image in enumerate(val_images1[::max(4*AUG_FAC,1)][:num_examples]):\n    plt.subplot(num_rows, num_columns, i + 1)\n    plt.imshow(image)\n    plt.text(3, 100, str(val_labels1.iloc[::max(4*AUG_FAC,1)].iloc[i]), fontsize=8, color='green')\n    plt.text(3, 500, str(val_info1.iloc[::max(4*AUG_FAC,1)].iloc[i]), fontsize=6, color='white')\n    plt.suptitle(\"Examples from train dataset\")\n    plt.xticks([])\n    plt.yticks([])\nplt.tight_layout()","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2026-01-28T07:16:44.141004Z","iopub.execute_input":"2026-01-28T07:16:44.141682Z","iopub.status.idle":"2026-01-28T07:16:46.661837Z","shell.execute_reply.started":"2026-01-28T07:16:44.141660Z","shell.execute_reply":"2026-01-28T07:16:46.660996Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"## Show distribution of labels in train and validation sets for each fold to check split \"quality\"\n\nif FOLD_TRAINING:\n    df_plot = pd.concat([globals()[f'{set}_labels{i+1}'].assign(Dataset=f'{set}_fold{i+1}') for i in range(5) for set in ['train', 'val']])\nelse:\n    df_plot = pd.concat([train_labels1.assign(Dataset='Train'), val_labels1.assign(Dataset='Validation')])\nwarnings.filterwarnings(\"ignore\", category=FutureWarning, module=\"seaborn\") # Suppress the specific FutureWarning\n\nn_cols = 3\nn_rows = (len(target_columns) + n_cols - 1) // n_cols\nfig, axes = plt.subplots(n_rows, n_cols, figsize=(15, 4 * n_rows))\nfig.suptitle('Label distributions in train and validation sets for each fold ')\naxes = axes.flatten()\nfor i in range(len(target_columns), len(axes)):\n    axes[i].axis('off')\nfor i, col in enumerate(target_columns):\n    sns.kdeplot(data=df_plot, x=col, ax=axes[i], hue='Dataset', common_norm=False, fill=False)\nplt.tight_layout()\nplt.show()\ndel df_plot","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2026-01-28T07:16:46.662919Z","iopub.execute_input":"2026-01-28T07:16:46.663264Z","iopub.status.idle":"2026-01-28T07:16:48.199836Z","shell.execute_reply.started":"2026-01-28T07:16:46.663233Z","shell.execute_reply":"2026-01-28T07:16:48.199046Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"# 3. Feature extraction","metadata":{}},{"cell_type":"code","source":"## Image feature extraction\n\nINTERNET = False # Before submission, pretrained model must be saved first\n\npreprocess = lambda x: x # Only place holder, ConvNeXtLarge has a build-in preprocessing layer\n\n# Load and show feature extraction model\nif INTERNET:\n    model = tf.keras.applications.ConvNeXtLarge(include_top=False, include_preprocessing=True,  input_shape=[image_size, image_size, 3],\n                                                pooling='avg', weights='imagenet')\n    model.save('ConvNeXtXLargePoolStage2Avg.keras', include_optimizer=False)\nelse:\n    model = tf.keras.models.load_model('/kaggle/input/csiro-featext/ConvNeXtXLargePoolAvgTuned_1_100_1.keras')\n    layer_ids = [102, 137, 172, 207, 243, 257] #102, 137, 172, 207, 243, 257\n    for layer in layer_ids:\n        globals()[f'out_layer{layer}'] = tf.keras.layers.GlobalAveragePooling2D()(model.layers[layer].output)\n    concat = tf.keras.layers.Concatenate()([globals()[f'out_layer{layer}'] for layer in layer_ids])\n    model = tf.keras.Model(inputs=model.input, outputs=concat)\n#model.summary()\n\n# Extract image features and split it to train and validation sets for each fold\ntrainval_image_features = model.predict(preprocess(trainval_images), verbose=0)\nfor i in range(5 if FOLD_TRAINING else 1):\n    globals()[f'train_image_features{i+1}'] = trainval_image_features[globals()[f'train_idx{i+1}']]\n    globals()[f'val_image_features{i+1}'] = trainval_image_features[globals()[f'val_idx{i+1}']]","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2026-01-28T07:16:48.200651Z","iopub.execute_input":"2026-01-28T07:16:48.200966Z","iopub.status.idle":"2026-01-28T07:19:55.401275Z","shell.execute_reply.started":"2026-01-28T07:16:48.200924Z","shell.execute_reply":"2026-01-28T07:19:55.400593Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"## Image feature scaling\n\nSCALING = True # Activate additional normalization of extracted image features\n\nif SCALING:\n    scaler = MinMaxScaler() #StandardScaler MinMaxScaler\n    trainval_prepared = scaler.fit_transform(trainval_image_features)\n    for i in range(5 if FOLD_TRAINING else 1):\n        globals()[f'train_prepared{i+1}'] = scaler.transform(globals()[f'train_image_features{i+1}'])\n        globals()[f'val_prepared{i+1}'] = scaler.transform(globals()[f'val_image_features{i+1}'])\nelse:\n    trainval_prepared = trainval_image_features\n    for i in range(5 if FOLD_TRAINING else 1):\n        globals()[f'train_prepared{i+1}'] = globals()[f'train_image_features{i+1}']\n        globals()[f'val_prepared{i+1}'] = globals()[f'val_image_features{i+1}']","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2026-01-28T07:19:55.402348Z","iopub.execute_input":"2026-01-28T07:19:55.402571Z","iopub.status.idle":"2026-01-28T07:19:55.689831Z","shell.execute_reply.started":"2026-01-28T07:19:55.402553Z","shell.execute_reply":"2026-01-28T07:19:55.689197Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"# 4. Define model space for ML methods","metadata":{}},{"cell_type":"code","source":"## Models and model space\n\nsvr = SVR(kernel='poly')\nrfr = RandomForestRegressor(random_state=42)\nkneigh = KNeighborsRegressor()\ngbr = GradientBoostingRegressor(random_state=42)\nxgb = xgboost.XGBRegressor(device='cpu', seed=0) #eval_metric='rmse', early_stopping_rounds=10\nada = AdaBoostRegressor(random_state=42)\nhgbr = HistGradientBoostingRegressor(random_state=42)\nlgbm = lightgbm.LGBMRegressor(device_type ='cpu', random_state=42, verbosity=-1)\ncatb = CatBoostRegressor(random_state=42, silent=True, task_type='CPU')\nlasso = Lasso(random_state=42)\nenet = ElasticNet(random_state=42)\nridge = Ridge(random_state=42)\nlars = LassoLars()\n\nEstimatorStr = {1: 'svr', 2: 'rfr', 3: 'kneigh', 4: 'gbr', 5: 'xgb', 6: 'ada', 7: 'hgbr', 8: 'lgbm', 9: 'catb', 10: 'lasso', 11: 'enet', 12: 'ridge', 13: 'lars'}\nEstimatorMdl = {1: svr, 2: rfr, 3: kneigh, 4: gbr, 5: xgb, 6: ada, 7: hgbr, 8: lgbm, 9: catb, 10: lasso, 11: enet, 12: ridge, 13: lars}","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2026-01-28T07:19:55.691752Z","iopub.execute_input":"2026-01-28T07:19:55.692065Z","iopub.status.idle":"2026-01-28T07:19:55.700035Z","shell.execute_reply.started":"2026-01-28T07:19:55.692047Z","shell.execute_reply":"2026-01-28T07:19:55.699259Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"# 5. Training","metadata":{}},{"cell_type":"code","source":"## Helping function to create parameter grids\n\ndef make_param(param_dict, model):\n    for elem in param_dict.copy():\n        if elem == 'n_components':\n            param_dict['pca'+'__'+elem] = param_dict.pop(elem)\n        else:\n            param_dict[model+'__'+elem] = param_dict.pop(elem)\n    return param_dict","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2026-01-28T07:19:55.700777Z","iopub.execute_input":"2026-01-28T07:19:55.701016Z","iopub.status.idle":"2026-01-28T07:19:55.713159Z","shell.execute_reply.started":"2026-01-28T07:19:55.701001Z","shell.execute_reply":"2026-01-28T07:19:55.712453Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"## Training parameters\n\n# Tuned hyperparameter sets\n# Svr parameter\nparam_single_svr = make_param({'C': 50, 'kernel': 'poly', 'degree': 1,\n                               'gamma': 0.01,\n                               #'n_components': 500,\n                               }, 'svr') # \n# Rfr parameter\nparam_single_rfr = make_param({#'n_estimators': 400, 'max_depth': 12, 'max_leaf_nodes': 100, 'min_samples_split': 3,\n                               }, 'randomforestregressor') # 0.9344/0.4833/0.5647\n# Kneight parameter\nparam_single_kneigh = make_param({#'n_estimators': 400, 'max_depth': 12, 'max_leaf_nodes': 100, 'min_samples_split': 3,\n                                  }, 'kneighborsregressor') # 0.9344/0.4833/0.5647\n# Gbr parameter\nparam_single_gbr = make_param({#'n_estimators': 300, 'max_depth': 4, 'learning_rate': 0.05, 'loss': 'huber', 'min_samples_split': 3,\n                              }, 'gradientboostingregressor') # 0.9800/0.4897/0.5282\n# xgb parameter\nparam_single_xgb = make_param({#'n_estimators': 200, 'tree_method': 'hist', 'max_depth': 2, 'reg_lambda': 0.75, 'learning_rate': 0.07,\n                              }, 'xgbregressor') # 0.8861/0.4939/0.5262\n# Ada parameter\nparam_single_ada = make_param({#'estimator': DecisionTreeRegressor(max_depth=180, criterion='squared_error', max_leaf_nodes=150, min_samples_split=4),\n                               #'n_estimators': 300, 'learning_rate': 0.75, 'loss': 'square',\n                               }, 'adaboostregressor') # 0.9967/0.5029/0.5859\n# Hgbr parameter\nparam_single_hgbr = make_param({#'max_depth':10, 'learning_rate': 0.05\n                                }, 'histgradientboostingregressor') # 0.9481/0.5836/0.5938\n# lgbm parameter\nparam_single_lgbm = make_param({#'max_depth': 16, 'learning_rate': 0.1, 'n_estimators': 200,\n                                }, 'lgbmregressor') #0.9900/0.5696/0.5933\n# catb parameter\nparam_single_catb = make_param({#'learning_rate': 0.06, 'depth': 2, 'l2_leaf_reg': 0.3\n                                }, 'catboostregressor')\n# lasso parameter\nparam_single_lasso = make_param({#'learning_rate': 0.06, 'depth': 2, 'l2_leaf_reg': 0.3\n                                }, 'lasso')\n# enet parameter\nparam_single_enet = make_param({#'alpha': 1, 'l1_ratio': 0.5,\n                                }, 'elasticnet')\n# Ridge parameter\nparam_single_ridge = make_param({#'alpha': 1, 'l1_ratio': 0.5,\n                                 }, 'ridge')\n# Lars parameter\nparam_single_lars = make_param({#'alpha': 1, 'l1_ratio': 0.5,\n                                }, 'lars')","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2026-01-28T07:26:51.676787Z","iopub.execute_input":"2026-01-28T07:26:51.677100Z","iopub.status.idle":"2026-01-28T07:26:51.683277Z","shell.execute_reply.started":"2026-01-28T07:26:51.677080Z","shell.execute_reply":"2026-01-28T07:26:51.682362Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"## Hyperparameter sets for parameter tuning\n\nparam_grid_svr = make_param({'C': [40,45,50],\n                             #'loss': ['epsilon_insensitive', 'squared_epsilon_insensitive'],\n                             #'gamma': [0.0003, 0.001, 0.003, 0.01],\n                             'kernel': ['poly'],\n                             'epsilon': [0, 0.1, 0.25, 0.5, 1, 2, 3, 4],\n                             'degree': [1],\n                               }, 'svr')\nparam_grid_enet = make_param({'alpha': [0.1, 0.3, 0.5, 0.7, 0.9, 1],\n                              'l1_ratio' : [0.1, 0.3, 0.5, 0.7, 0.9],\n                              #'max_iter' : [20000],\n                              #'n_components': [50,100,150,200,250,300, 350, 400, 450, 500],\n                              }, 'elasticnet')\n\nparam_grid_kneigh = make_param({#'n_neighbors': [3, 5, 7, 9], 'weights': ['uniform', 'distance'],\n                                }, 'kneighborsregressor')\n\nparam_grid_ridge = make_param({'alpha': [10, 30, 50, 60, 70, 80, 90, 100, 110, 120, 140, 160, 180, 200],\n                               #'fit_intercept': [False, True],\n                               #'positive': [False, True]\n                              #'max_iter' : [10000]\n                              #'n_components': [50,60,70,80,90,100,110,120,130,140,150],\n                              }, 'ridge')","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2026-01-28T07:31:47.393171Z","iopub.execute_input":"2026-01-28T07:31:47.393967Z","iopub.status.idle":"2026-01-28T07:31:47.399160Z","shell.execute_reply.started":"2026-01-28T07:31:47.393933Z","shell.execute_reply":"2026-01-28T07:31:47.398463Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"# 6. Evaluation","metadata":{}},{"cell_type":"code","source":"## Single fitting with tuned parameters or grid search for machine learning methods\n\nTUNING_ML = True # Choose between single fitting or parameter tuning\nPCA_active = True # Activate PCA\ncolumn_weights = [0.1, 0.1, 0.1, 0.5, 0.2] # ['Dry_Clover_g', 'Dry_Dead_g', 'Dry_Green_g', 'Dry_Total_g', 'GDM_g']\nEST_IDS = [[1],[1],[1],[1],[1]] # Choose model(s) to tune {1: svr, 2: rfr, 3: kneigh, 4: gbr, 5: xgb, 6: ada, 7: hgbr, 8: lgbm, 9: catb, 10: lasso, 11: enet, 12: ridge, 13: lars}\n\nw_train_scores = []\nw_val_scores = []\nfor i, col in enumerate(target_columns):\n    print(f'Training for {col} target:')\n    for est_id in EST_IDS[i]:\n        param = globals()[f'param_grid_{EstimatorStr[est_id]}' if TUNING_ML else f'param_single_{EstimatorStr[est_id]}'] \n        \n        # PCA\n        if PCA_active:\n            pipeline = make_pipeline(PCA(0.999, random_state=42), EstimatorMdl[est_id])\n        else:\n            pipeline = make_pipeline(EstimatorMdl[est_id])\n        \n        # Fitting or tuning on whole trainval dataset with 5 fold cross-validation\n        for j in range(5 if FOLD_TRAINING else 1):\n            if TUNING_ML:\n                # Cross-validation configuration\n                cv_gen = StratifiedGroupKFold(n_splits=5, shuffle=True, random_state=42).split(globals()[f'train_prepared{j+1}'],\n                            globals()[f'train_info{j+1}']['strat_classes'], groups=globals()[f'train_info{j+1}']['image_path'])\n\n                grid = GridSearchCV(pipeline, param, scoring='r2', verbose=1, cv=cv_gen, n_jobs=-1)\n                grid.fit(globals()[f'train_prepared{j+1}'], np.array(globals()[f'train_labels{j+1}'])[:, i])\n                print(grid.best_params_)\n                pipeline_train = grid.best_estimator_\n            else:\n                pipeline_train = deepcopy(pipeline)\n                pipeline_train.set_params(**param)\n                pipeline_train.fit(globals()[f'train_prepared{j+1}'], np.array(globals()[f'train_labels{j+1}'])[:, i])\n            globals()[f'model{j+1}_{EstimatorStr[est_id]}_{col}'] = pipeline_train\n\n            # Print train and validation score for each estimator and each fold\n            train_score = r2_score(pipeline_train.predict(globals()[f'train_prepared{j+1}']), np.array(globals()[f'train_labels{j+1}'])[:, i])\n            val_score = r2_score(pipeline_train.predict(globals()[f'val_prepared{j+1}']), np.array(globals()[f'val_labels{j+1}'])[:, i])\n            w_train_scores.append(train_score*column_weights[i])\n            w_val_scores.append(val_score*column_weights[i])\n            print(f'Train R2 score of {EstimatorStr[est_id]} estimator for fold {j+1}: {train_score}')\n            print(f'Valid R2 score of {EstimatorStr[est_id]} estimator for fold {j+1}: {val_score}')\n    print('-'*40)\n        \nprint(f'Average weighted train R2 score: {sum(w_train_scores)/len(EST_IDS[0])/(5 if FOLD_TRAINING else 1)}')\nprint(f'Average weighted valid R2 score: {sum(w_val_scores)/len(EST_IDS[0])/(5 if FOLD_TRAINING else 1)}')","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2026-01-28T07:31:50.773238Z","iopub.execute_input":"2026-01-28T07:31:50.773980Z","iopub.status.idle":"2026-01-28T07:39:41.087180Z","shell.execute_reply.started":"2026-01-28T07:31:50.773952Z","shell.execute_reply":"2026-01-28T07:39:41.086075Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"## Calculate predictions and relative error of predictions\n\nTARGET_CALCULATED = False\n\neval_df = pd.DataFrame()\npred_labels = []\nfor i, col in enumerate(target_columns):\n    col_ress = []\n    for est_id in EST_IDS[i]:\n        col_res = []\n        for j in range(5 if FOLD_TRAINING else 1):\n            model_val = globals()[f'model{j+1}_{EstimatorStr[est_id]}_{col}']\n            col_res.append(model_val.predict(globals()[f'val_prepared{j+1}']))\n        col_ress.append(np.concatenate(col_res))\n    pred = np.maximum(np.mean(np.stack(col_ress), axis=0), 0)\n    val_labels = pd.concat([globals()[f'val_labels{j+1}'] for j in range(5 if FOLD_TRAINING else 1)], ignore_index=True)\n    val_info = pd.concat([globals()[f'val_info{j+1}'] for j in range(5 if FOLD_TRAINING else 1)], ignore_index=True)\n    eval_df[col] = val_labels[[col]].reset_index(drop=True)\n    eval_df[f'p_{col}'] = pred\n    # Calculate relative error\n    eval_df[f're_{col}'] = np.round(((eval_df[col]-eval_df[f'p_{col}'])/(eval_df[col] + 0.0001))*100, 1).abs()\n    pred_labels.append(pred)\neval_df['add_Dry_Total_g'] = eval_df['p_Dry_Clover_g'] + eval_df['p_Dry_Dead_g'] + eval_df['p_Dry_Green_g']\neval_df['add_GDM_g'] = eval_df['p_Dry_Clover_g'] + eval_df['p_Dry_Green_g']\npred_labels = np.stack((pred_labels), axis=-1)\n\nif TARGET_CALCULATED:\n    #pred_labels[:, 1] = np.maximum(0, pred_labels[:, 3]-pred_labels[:, 2]-pred_labels[:, 0])\n    pred_labels[:, 3] = np.maximum(0, pred_labels[:, 0]+pred_labels[:, 1]+pred_labels[:, 2])\n    #pred_labels[:, 1] = np.maximum(0, pred_labels[:, 3]-pred_labels[:, 4])\n\n# Custom score\nsample_weights = np.stack([np.array(column_weights)]*len(pred_labels))\n\ndef weighted_r2_score(y_true, y_pred, sample_weights):\n    y_true = y_true.to_numpy()\n    weighted_mean = (y_true*sample_weights).sum()/sample_weights.sum()\n    ss_res = ((y_true - y_pred)**2*sample_weights).sum()\n    ss_tot = ((y_true - weighted_mean)**2*sample_weights).sum()\n    r2_weighted = 1 - (ss_res/ss_tot)\n    return r2_weighted\n\nprint(f'Custom specific score on validation data is: {weighted_r2_score(val_labels, pred_labels, sample_weights)}')","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2026-01-18T17:50:13.530207Z","iopub.execute_input":"2026-01-18T17:50:13.530800Z","iopub.status.idle":"2026-01-18T17:50:15.547216Z","shell.execute_reply.started":"2026-01-18T17:50:13.530772Z","shell.execute_reply":"2026-01-18T17:50:15.546443Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"## Show predicted/ground true values\n\neval_df","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2026-01-18T17:51:03.467357Z","iopub.execute_input":"2026-01-18T17:51:03.467915Z","iopub.status.idle":"2026-01-18T17:51:03.496138Z","shell.execute_reply.started":"2026-01-18T17:51:03.467893Z","shell.execute_reply":"2026-01-18T17:51:03.495588Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"## Show worse predictions from validation dataset\n\nworse_index = eval_df.sort_values(by='re_Dry_Total_g', ascending=False).index\npred_labels = ['p_Dry_Clover_g', 'p_Dry_Dead_g', 'p_Dry_Green_g', 'p_Dry_Total_g', 'p_GDM_g']\nval_images = np.concatenate([globals()[f'val_images{j+1}'] for j in range(5 if FOLD_TRAINING else 1)])\n\n# Visualize data\nnum_examples = 16\nnum_columns = 4\nnum_rows = math.ceil(num_examples/num_columns)\nplt.figure(figsize=(16, 16))\nfor i, index in enumerate(worse_index[:num_examples]):\n    image = val_images[index]\n    plt.subplot(num_rows, num_columns, i + 1)\n    plt.imshow(image)\n    plt.text(3, 110, str(eval_df[target_columns].loc[index]), fontsize=8, color='green')\n    plt.text(250, 110, str(eval_df[pred_labels].loc[index]), fontsize=8, color='red')\n    plt.text(3, 500, str(val_info.loc[index]), fontsize=6, color='white')\n    plt.suptitle(\"Worse predictions from val dataset\")\n    plt.xticks([])\n    plt.yticks([])\nplt.tight_layout()","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2026-01-18T13:36:45.647899Z","iopub.execute_input":"2026-01-18T13:36:45.648655Z","iopub.status.idle":"2026-01-18T13:36:48.567455Z","shell.execute_reply.started":"2026-01-18T13:36:45.648629Z","shell.execute_reply":"2026-01-18T13:36:48.566161Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"# 7. Submission","metadata":{}},{"cell_type":"code","source":"## Test prediction & submission \n\nTEST_AUG_FAC = 1\ntest = pd.read_csv(path + \"test.csv\")\ntest_pred = {}\n\ntest_images = []\ntest_paths = test['image_path'].unique()\nfor image_path in test_paths:\n    # Prepare tensor for each test image w/wo augmentation\n    image_id = re.split('/|.j', image_path)[1]\n    image = np.array(Image.open(path+image_path))\n    if TEST_AUG_FAC > 0:\n        test_images = []\n        for i in range(TEST_AUG_FAC):\n            for flip_horizontal in [False, True]:\n                for flip_vertical in [False, True]:\n                    roll = flip_horizontal or flip_vertical or i>0 # first image smaple shall not be augmented\n                    image_resized = preprocess_images(image, flip_horizontal=flip_horizontal, flip_vertical=flip_vertical, roll=roll)\n                    test_images.append(image_resized)\n        image_tensor = np.stack(test_images, axis=0)\n    else:\n        image_resized = preprocess_images(image)\n        image_tensor = np.expand_dims(image_resized, axis=0)\n\n    # Feature extraction with ConvNeXtXLarge\n    test_image_features = model.predict(preprocess(image_tensor), verbose=0)\n\n    # Normalize test features\n    if SCALING:\n        test_prepared = scaler.transform(test_image_features)\n    else:\n        test_prepared = test_image_features\n\n    # For each label predict with each choosen estimator for all augmented versions of the image and calculate the mean of the predictions\n    for i, col in enumerate(target_columns):\n        col_ress = []\n        for j in [0]: #range(5 if FOLD_TRAINING else 1)\n            for est_id in EST_IDS[i]:\n                model_val = globals()[f'model{j+1}_{EstimatorStr[est_id]}_{col}']\n                col_ress.append(model_val.predict(test_prepared))\n        test_pred[f'{image_id}__{col}'] = np.maximum(np.mean(col_ress), 0)\n\n    # Some of the labels can be optionally determined by other labels as well\n    if TARGET_CALCULATED:\n        # test_pred[f'{image_id}__Dry_Dead_g'] = test_pred[f'{image_id}__Dry_Total_g'] - (\n        #     test_pred[f'{image_id}__Dry_Green_g'] + test_pred[f'{image_id}__Dry_Clover_g'])\n        test_pred[f'{image_id}__Dry_Total_g'] = test_pred[f'{image_id}__Dry_Clover_g'] + (\n            test_pred[f'{image_id}__Dry_Dead_g'] + test_pred[f'{image_id}__Dry_Green_g'])\n        #test_pred[f'{image_id}__Dry_Dead_g'] = test_pred[f'{image_id}__Dry_Total_g'] - (\n        #    test_pred[f'{image_id}__GDM_g'])\n\n# Create submission dataframe and .csv file\nsubmission_df = pd.DataFrame([{\"sample_id\": k, \"target\": v} for k, v in test_pred.items()])\nsubmission_df.to_csv(\"submission.csv\", index=False)\nprint(\"✅ submission.csv saved!\")\nsubmission_df","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2026-01-18T13:36:48.569310Z","iopub.execute_input":"2026-01-18T13:36:48.569689Z","iopub.status.idle":"2026-01-18T13:36:53.844385Z","shell.execute_reply.started":"2026-01-18T13:36:48.569657Z","shell.execute_reply":"2026-01-18T13:36:53.843751Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"# 8. Experimental code","metadata":{}},{"cell_type":"code","source":"# ## Preprocess features (currently unused)\n\n# minmax_columns = ['Pre_GSHH_NDVI', 'Height_Ave_cm']\n# minmax_pipeline = Pipeline([('imputer', SimpleImputer(strategy=\"median\")),\n#                             ('standard_scaling', MinMaxScaler())])\n# onehot_columns = ['State', 'Species']\n# onehot_pipeline = Pipeline([('imputer', SimpleImputer(strategy=\"most_frequent\")),\n#                             ('onehot_encoding', OneHotEncoder(sparse_output=False, handle_unknown='ignore'))])\n\n# preprocessing_features = ColumnTransformer([(\"minmax_scl\", minmax_pipeline, minmax_columns),\n#                                             (\"onehot_enc\", onehot_pipeline, onehot_columns)])\n\n# train_features_scaled = preprocessing_features.fit_transform(train_features)\n# val_features_scaled = preprocessing_features.transform(val_features)\n\n# #train_prepared = np.concatenate((train_image_features, train_features_scaled), axis=-1)\n# #val_prepared = np.concatenate((val_image_features, val_features_scaled), axis=-1)","metadata":{"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"code","source":"# iii = 9\n# layer_id = [9,16,23,31,38,45,53,60,67,74,81,88,95,102,109,116,123,130,137,144,151,158,165,172,179,186,193,200,207,214,221,228,235,243,250,257][iii]\n# print(str(layer_id) +' '+ str(iii))\n# iii += -1","metadata":{"trusted":true},"outputs":[],"execution_count":null}]}