{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"name":"python","version":"3.12.12","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"},"kaggle":{"accelerator":"nvidiaTeslaT4","dataSources":[{"sourceId":112509,"databundleVersionId":14254895,"sourceType":"competition"}],"dockerImageVersionId":31259,"isInternetEnabled":true,"language":"python","sourceType":"notebook","isGpuEnabled":true}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# Prologue to the Research\n\n## Abstract \nThe present notebook introduces a nice, modular solution to the multilabel image regression task at hand. It utilizes **ConvNeXtV2**, a transformer-like CNN architecture made by Meta AI, to predict 5 target values for every image in the testset.\n\n## From the author\nThe first real research notebook. It took me a while to assemble, considering the fact that I've tried to make it comfortable for tuning.\n\nThere were multiple instances of completely changing the model build. I've originally thought that rawdogging the competition with a SwinV2 would do it, but the validation loss said otherwise. The problem seemed to be in the inefficiency of transformers on small datasets. Then I found out about ConvNeXtV2, a CNN architecture that took inspiration from transformers. It **outperformed** the latter model by 2 times (1000 vs 400 MSE). The PyTorch guys managed to make an outstanding CV model.\n\nOn the topic of ConvNeXt, it has this silly feature that the validation loss sometimes catches up and eventually overtakes the train loss. I spent most of my time working with transformers, so I've never seen such an occurrence.\n\nThe notebook isn't going to be as verbose as my other works since there isn't really anything new to describe. I preprocess the dataframes, turning the images and labels into HuggingFace datasets, applying the image processor to them afterwards. Training is as understandable.\n\nHave a nice read, lone traveler.","metadata":{}},{"cell_type":"markdown","source":"# Configuration","metadata":{}},{"cell_type":"code","source":"!pip install -U transformers peft datasets --quiet\n\nimport numpy as np\nimport pandas as pd\n\nfrom PIL import Image\n\nimport torch\nfrom torch import nn\n\nfrom transformers import (\n    AutoImageProcessor,\n    AutoModelForImageClassification as AutoModel,\n    TrainingArguments,\n    Trainer\n)\nfrom peft import LoraConfig, get_peft_model\nimport datasets\n\nimport gc\nimport random\nimport warnings\nwarnings.filterwarnings('ignore')\nfrom dataclasses import dataclass, field\nfrom pathlib import Path\nfrom tqdm.notebook import tqdm","metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true,"execution":{"iopub.status.busy":"2026-01-31T18:04:29.381029Z","iopub.execute_input":"2026-01-31T18:04:29.381309Z","iopub.status.idle":"2026-01-31T18:05:04.911360Z","shell.execute_reply.started":"2026-01-31T18:04:29.381278Z","shell.execute_reply":"2026-01-31T18:05:04.910717Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"SEED = 31*31*31\nROOT = Path('/kaggle/input/csiro-biomass')\ntargets = ['Dry_Clover_g', 'Dry_Dead_g', 'Dry_Green_g', 'Dry_Total_g', 'GDM_g']\n\n@dataclass\nclass Config:\n    model_id: str = 'facebook/convnextv2-base-1k-224'\n    device: str = torch.device('cuda' if torch.cuda.is_available() else 'cpu')\n    use_lora: bool = False\n    \n    lr: float = 5e-5\n    epochs: int = 5\n    batch_size: int = 128\n    \n    r: int = 6\n    alpha: int = field(init=False)\n    dropout: float = 2e-1\n    target_modules: list = field(\n        default_factory=lambda:['query', 'value']\n    )\n\n    def __post_init__(self):\n        self.alpha = 2*self.r","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2026-01-31T18:05:04.913239Z","iopub.execute_input":"2026-01-31T18:05:04.913783Z","iopub.status.idle":"2026-01-31T18:05:04.919961Z","shell.execute_reply.started":"2026-01-31T18:05:04.913745Z","shell.execute_reply":"2026-01-31T18:05:04.919261Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"# Functions","metadata":{}},{"cell_type":"code","source":"def set_seed(seed=SEED):\n    random.seed(SEED)\n    np.random.seed(SEED)\n    torch.manual_seed(SEED)\n    torch.cuda.manual_seed(SEED)\n    torch.cuda.manual_seed_all(SEED)\n    torch.backends.cudnn.deteministic = True\n    torch.backends.cudnn.benchmark = False\n\ndef preprocess_df(df: pd.DataFrame, is_test: bool = False) -> pd.DataFrame:\n        df = df.copy()\n        if not is_test:\n            df = df.pivot(\n                index='image_path',\n                columns='target_name',\n                values='target'\n            ).reset_index()\n            df.columns.name = None\n        else:\n            df = df[['image_path']].drop_duplicates().reset_index(drop=True)\n    \n        df.image_path = df.image_path.apply(lambda x: ROOT/x).astype(str)\n        return df\n\ndef create_dataset(df: pd.DataFrame, is_test: bool = False) -> datasets.Dataset:\n    df = df.rename(columns={'image_path': 'image'})\n    ds = datasets.Dataset.from_pandas(df.copy())\n    ds = ds.cast_column('image', datasets.Image())\n    return ds\n\ndef preprocess_dataset(ds: datasets.Dataset, processor) -> datasets.Dataset:\n    inputs = processor(\n        [image.convert('RGB') for image in ds['image']],\n        return_tensors='pt'\n    )\n\n    if targets[0] in ds:\n        labels = []\n        for i in range(len(ds['image'])):\n            labels.append([ds[target][i] for target in targets])\n        inputs['labels'] = torch.tensor(labels, dtype=torch.float32)\n\n    return inputs\n\ndef train(cfg: Config, train_dataset, eval_dataset, use_lora: bool = False) -> Trainer:\n    model = AutoModel.from_pretrained(\n        cfg.model_id,\n        num_labels=5,\n        ignore_mismatched_sizes=True,\n        problem_type='regression'\n    )\n    \n    if use_lora:\n        lora = LoraConfig(\n            r=cfg.r,\n            lora_alpha=cfg.alpha,\n            lora_dropout=cfg.dropout,\n            target_modules=cfg.target_modules\n        )\n        model = get_peft_model(model, lora)\n        \n    model = model.to(cfg.device)\n\n    args = TrainingArguments(\n        report_to='none',\n        push_to_hub=False,\n        output_dir=None,\n\n        per_device_train_batch_size=cfg.batch_size,\n        per_device_eval_batch_size=cfg.batch_size,\n        learning_rate=cfg.lr,\n        num_train_epochs=cfg.epochs,\n\n        torch_empty_cache_steps=50,\n        warmup_steps=50,\n        # gradient_accumulation_steps=4,\n        # gradient_checkpointing=True,\n        logging_strategy='epoch',\n        eval_strategy='epoch',\n        save_strategy='no',\n        fp16=torch.cuda.is_available()\n    )\n\n    trainer = Trainer(\n        model=model,\n        args=args,\n        train_dataset=train_dataset,\n        eval_dataset=eval_dataset\n    )\n    trainer.train()\n    torch.cuda.empty_cache()\n    gc.collect()\n    \n    return trainer\n\ndef create_submission(predictions, test_df):\n    submission = test_df.copy()\n    submission[targets] = predictions\n    submission = submission.melt(\n        id_vars=['image_path'],\n        value_vars=targets,\n        var_name='target_name',\n        value_name='target',\n    ).drop(columns='target_name')\n    submission.to_csv('submission.csv', index=False)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2026-01-31T18:05:04.920742Z","iopub.execute_input":"2026-01-31T18:05:04.921043Z","iopub.status.idle":"2026-01-31T18:05:04.962723Z","shell.execute_reply.started":"2026-01-31T18:05:04.921011Z","shell.execute_reply":"2026-01-31T18:05:04.962089Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"# Main","metadata":{}},{"cell_type":"code","source":"def main(cfg):\n    set_seed()\n    processor = AutoImageProcessor.from_pretrained(cfg.model_id)\n\n    train_raw = pd.read_csv(ROOT/'train.csv')\n    test_raw = pd.read_csv(ROOT/'test.csv')\n\n    print('Preprocessing dataframes...', end=' ')\n    train_df = preprocess_df(train_raw)\n    test_df = preprocess_df(test_raw, is_test=True)\n    print('Done!')\n\n    print('Creating datasets...', end=' ')\n    labelset_ = create_dataset(train_df)\n    testset_ = create_dataset(test_df)\n    print('Done!')\n\n    print('Processing images...', end=' ')\n    labelset = labelset_.map(\n        preprocess_dataset,\n        batched=True,\n        batch_size=128,\n        remove_columns=labelset_.column_names,\n        fn_kwargs={'processor': processor}\n    )\n    labelset.set_format('torch')\n    \n    testset = testset_.map(\n        preprocess_dataset,\n        batched=True,\n        batch_size=128,\n        remove_columns=testset_.column_names,\n        fn_kwargs={'processor': processor}\n    )\n    testset.set_format('torch')\n    print('Done!')\n    \n    del labelset_, testset_\n    labelset = labelset.train_test_split(test_size=0.2)\n    trainset, evalset = labelset['train'], labelset['test']\n\n    print('Training the model...', end=' ')\n    trainer = train(\n        cfg=cfg,\n        train_dataset=trainset,\n        eval_dataset=evalset,\n        use_lora=cfg.use_lora\n    )\n    print('Done!')\n\n    predictions = trainer.predict(testset).predictions\n    create_submission(predictions, test_df)\n    print('Submission created.')","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2026-01-31T18:05:04.963720Z","iopub.execute_input":"2026-01-31T18:05:04.964027Z","iopub.status.idle":"2026-01-31T18:05:04.976819Z","shell.execute_reply.started":"2026-01-31T18:05:04.964008Z","shell.execute_reply":"2026-01-31T18:05:04.976182Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"# Training & Inference","metadata":{}},{"cell_type":"code","source":"# facebook/convnextv2-tiny-1k-224\nmodels = {\n    'convnext-base-224': 'facebook/convnextv2-base-1k-224',\n    'convnext-base-384': 'facebook/convnextv2-base-22k-384',\n    'convnext-large-384': 'facebook/convnextv2-large-22k-384',\n    'convnext-huge-512': 'facebook/convnextv2-huge-22k-512',\n    'vit': 'google/vit-base-patch16-224'\n}\n\ncfg = Config(\n    batch_size=6,\n    lr=65e-4,\n    epochs=8,\n    model_id=models['convnext-base-384']\n)\n\n# print(f'Used device: {cfg.device}\\n')\nprint(\n    f\"\"\"\n┌ Configuration: \n└───┬ Model checkpoint          {cfg.model_id}\n    ├ Per device batch size     {cfg.batch_size}\n    ├ Learning rate             {cfg.lr}\n    ├ Number of epochs          {cfg.epochs}\n    └ LoRA in use               {cfg.use_lora}\n    \"\"\"\n)\n\nmain(cfg)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2026-01-31T18:14:56.525395Z","iopub.execute_input":"2026-01-31T18:14:56.526027Z","iopub.status.idle":"2026-01-31T18:26:38.724071Z","shell.execute_reply.started":"2026-01-31T18:14:56.526000Z","shell.execute_reply":"2026-01-31T18:26:38.723127Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"","metadata":{"trusted":true},"outputs":[],"execution_count":null}]}