{"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"codemirror_mode":{"name":"ipython","version":3},"file_extension":".py","mimetype":"text/x-python","name":"python","nbconvert_exporter":"python","pygments_lexer":"ipython3","version":"3.10.12"},"kaggle":{"accelerator":"nvidiaTeslaT4","dataSources":[{"sourceId":91451,"databundleVersionId":11223220,"sourceType":"competition"}],"dockerImageVersionId":30918,"isInternetEnabled":true,"language":"python","sourceType":"notebook","isGpuEnabled":true}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# 🐾 AnimalCLEF2025 Competition: Official Starter notebook + Metrics Implementation & Baseline evaluation\n\nThe **Goal of the** [AnimalCLEF2025](https://www.kaggle.com/competitions/animal-clef-2025/) competition is to identify individual animal (lynxes, salamanders and sea turtles) in photos. This notebook visualize the provided dataset and propose a baseline solution, based on the state-of-the-art re-identification model [MegaDescriptor](https://huggingface.co/BVRA/MegaDescriptor-L-384). The dataset is split into the database and query sets. For each image from the query set, the goal is to:\n\n- Predict whether the depicted individual is in the database.\n- If no, the prediction is `new_individual`.\n- If yes, the prediction should be the same as the individual in the database.","metadata":{}},{"cell_type":"markdown","source":"## Dependencies instalation\nFor the competition we provide two Python packages for loading and preprocessing of available datasets ([wildlife-datasets](https://github.com/WildlifeDatasets/wildlife-datasets)) and tools / method for animal re-identification ([wildlife-tools](https://github.com/WildlifeDatasets/wildlife-tools)).","metadata":{"execution":{"iopub.status.busy":"2025-02-28T16:18:13.120141Z","iopub.execute_input":"2025-02-28T16:18:13.120497Z","iopub.status.idle":"2025-02-28T16:18:13.1244Z","shell.execute_reply.started":"2025-02-28T16:18:13.120467Z","shell.execute_reply":"2025-02-28T16:18:13.12338Z"}}},{"cell_type":"code","source":"!pip install git+https://github.com/WildlifeDatasets/wildlife-datasets@develop\n!pip install git+https://github.com/WildlifeDatasets/wildlife-tools","metadata":{"_kg_hide-output":true,"execution":{"iopub.status.busy":"2025-03-25T16:43:13.663748Z","iopub.execute_input":"2025-03-25T16:43:13.664112Z","iopub.status.idle":"2025-03-25T16:43:56.231497Z","shell.execute_reply.started":"2025-03-25T16:43:13.664087Z","shell.execute_reply":"2025-03-25T16:43:56.230379Z"},"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"# Metrics Definition","metadata":{}},{"cell_type":"code","source":"import numpy as np\nfrom typing import List, Union\n\ndef baks_compute(\n        y_true: Union[List, np.ndarray],\n        y_pred: Union[List, np.ndarray],\n        identity_test_only: Union[List, np.ndarray]\n    ) -> float:\n    \"\"\"Computes BAKS (balanced accuracy on known samples).\n    \n    Focuses only on samples with known identities (not in identity_test_only).\n    \n    Args:\n        y_true: True labels\n        y_pred: Predicted labels\n        identity_test_only: Labels of unknown identities (only in test set)\n        \n    Returns:\n        Balanced accuracy score for known samples\n    \"\"\"\n    # Convert inputs to numpy arrays with object dtype to handle mixed types\n    y_true = np.array(y_true, dtype=object)\n    y_pred = np.array(y_pred, dtype=object)\n    identity_test_only = np.array(identity_test_only, dtype=object)\n    \n    # Filter out unknown samples\n    mask = ~np.isin(y_true, identity_test_only)\n    y_true_known = y_true[mask]\n    y_pred_known = y_pred[mask]\n    \n    if len(y_true_known) == 0:\n        return 0.0\n    \n    # Get unique classes in the filtered true labels\n    unique_classes = np.unique(y_true_known)\n    n_classes = len(unique_classes)\n    \n    # Compute per-class accuracy and average\n    class_accuracies = []\n    for cls in unique_classes:\n        cls_mask = (y_true_known == cls)\n        if np.sum(cls_mask) > 0:\n            cls_acc = np.mean(y_pred_known[cls_mask] == cls)\n            class_accuracies.append(cls_acc)\n    \n    # Return the balanced accuracy (mean of per-class accuracies)\n    return np.mean(class_accuracies) if class_accuracies else 0.0\n\ndef baus_compute(\n        y_true: Union[List, np.ndarray],\n        y_pred: Union[List, np.ndarray],\n        identity_test_only: Union[List, np.ndarray],\n        new_class: Union[int, str]\n    ) -> float:\n    \"\"\"Computes BAUS (balanced accuracy on unknown samples).\n    \n    Focuses only on samples with unknown identities (in identity_test_only).\n    \n    Args:\n        y_true: True labels\n        y_pred: Predicted labels\n        identity_test_only: Labels of unknown identities (only in test set)\n        new_class: Label used for identifying unknown samples\n        \n    Returns:\n        Balanced accuracy score for unknown samples\n    \"\"\"\n    # Convert inputs to numpy arrays with object dtype to handle mixed types\n    y_true = np.array(y_true, dtype=object)\n    y_pred = np.array(y_pred, dtype=object)\n    identity_test_only = np.array(identity_test_only, dtype=object)\n    \n    # Filter to include only unknown samples\n    mask = np.isin(y_true, identity_test_only)\n    y_true_unknown = y_true[mask]\n    y_pred_unknown = y_pred[mask]\n    \n    if len(y_true_unknown) == 0:\n        return 0.0\n    \n    # Get unique unknown classes\n    unique_unknown_classes = np.unique(y_true_unknown)\n    \n    # For each unknown class, check if they were correctly predicted as new_class\n    class_accuracies = []\n    for cls in unique_unknown_classes:\n        cls_mask = (y_true_unknown == cls)\n        if np.sum(cls_mask) > 0:\n            # For unknown samples, correct prediction is new_class\n            cls_acc = np.mean(y_pred_unknown[cls_mask] == new_class)\n            class_accuracies.append(cls_acc)\n    \n    # Return the balanced accuracy (mean of per-class accuracies)\n    return np.mean(class_accuracies) if class_accuracies else 0.0\n\n\ndef compute_geometric_mean(baks, baus):\n    return np.sqrt(baks * baus)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-03-25T16:43:56.233147Z","iopub.execute_input":"2025-03-25T16:43:56.233515Z","iopub.status.idle":"2025-03-25T16:43:56.242978Z","shell.execute_reply.started":"2025-03-25T16:43:56.233481Z","shell.execute_reply":"2025-03-25T16:43:56.242121Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## Dependencies import\nWe load all the required packages and then define the function `create_sample_submission`, which converts provided predictions and a submission file for the competition.","metadata":{}},{"cell_type":"code","source":"import os\nimport numpy as np\nimport pandas as pd\nimport timm\nimport torchvision.transforms as T\nfrom wildlife_datasets.datasets import AnimalCLEF2025\nfrom wildlife_tools.features import DeepFeatures\nfrom wildlife_tools.similarity import CosineSimilarity\ndef create_sample_submission(dataset_query, predictions, file_name='sample_submission.csv'):\n    df = pd.DataFrame({\n        'image_id': dataset_query.metadata['image_id'],\n        'identity': predictions\n    })\n    df.to_csv(file_name, index=False)","metadata":{"execution":{"iopub.status.busy":"2025-03-25T16:43:56.244921Z","iopub.execute_input":"2025-03-25T16:43:56.245195Z","iopub.status.idle":"2025-03-25T16:44:25.664140Z","shell.execute_reply.started":"2025-03-25T16:43:56.245173Z","shell.execute_reply":"2025-03-25T16:44:25.663197Z"},"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"We need to specify the `root`, where the data are stored and then two image transformations. \n1. The first transform only resizes the images and is used for visualization.\n2. The second transform also converts it to torch tensor and is used for operations on neural networks.","metadata":{}},{"cell_type":"code","source":"root = '/kaggle/input/animal-clef-2025'\ntransform_display = T.Compose([\n    T.Resize([384, 384]),\n    ])\ntransform = T.Compose([\n    *transform_display.transforms,\n    T.ToTensor(),\n    T.Normalize(mean=(0.485, 0.456, 0.406), std=(0.229, 0.224, 0.225))\n    ])","metadata":{"execution":{"iopub.status.busy":"2025-03-25T16:44:25.665367Z","iopub.execute_input":"2025-03-25T16:44:25.666075Z","iopub.status.idle":"2025-03-25T16:44:25.670952Z","shell.execute_reply.started":"2025-03-25T16:44:25.666046Z","shell.execute_reply":"2025-03-25T16:44:25.670029Z"},"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"# Import the dataset\n\nWhen importing the database and filtering it, we can see that the salamander species are missing.","metadata":{}},{"cell_type":"code","source":"from wildlife_datasets import datasets, splits\ndataset = AnimalCLEF2025(root, transform=transform_display, load_label=True)\ndf = dataset.df \ndf = df[df['split'] == 'database']\ndf.species.value_counts()","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-03-25T16:44:25.671851Z","iopub.execute_input":"2025-03-25T16:44:25.672105Z","iopub.status.idle":"2025-03-25T16:45:09.111846Z","shell.execute_reply.started":"2025-03-25T16:44:25.672084Z","shell.execute_reply":"2025-03-25T16:45:09.111068Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"df.isna().value_counts()","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-03-25T16:45:09.112676Z","iopub.execute_input":"2025-03-25T16:45:09.112919Z","iopub.status.idle":"2025-03-25T16:45:09.133118Z","shell.execute_reply.started":"2025-03-25T16:45:09.112899Z","shell.execute_reply":"2025-03-25T16:45:09.132416Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"df.fillna('salamander', inplace=True)\ndf.species.value_counts()","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-03-25T16:45:09.134068Z","iopub.execute_input":"2025-03-25T16:45:09.134308Z","iopub.status.idle":"2025-03-25T16:45:09.149863Z","shell.execute_reply.started":"2025-03-25T16:45:09.134287Z","shell.execute_reply":"2025-03-25T16:45:09.149082Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"# Splitting the database into train and validation(Known identities)\nFor splitting we can use wildlife tools, that provide direct implementations of open-set data split and many more.","metadata":{}},{"cell_type":"code","source":"splitter = splits.OpenSetSplit(0.8, 0.1)\nfor idx_train, idx_test in splitter.split(df):\n    splits.analyze_split(df, idx_train, idx_test)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-03-25T16:45:09.152095Z","iopub.execute_input":"2025-03-25T16:45:09.152318Z","iopub.status.idle":"2025-03-25T16:45:09.806481Z","shell.execute_reply.started":"2025-03-25T16:45:09.152298Z","shell.execute_reply":"2025-03-25T16:45:09.805513Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"df_train, df_test = df.loc[idx_train], df.loc[idx_test]\nlen(df_train)\nlen(df_test)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-03-25T16:45:09.807647Z","iopub.execute_input":"2025-03-25T16:45:09.807922Z","iopub.status.idle":"2025-03-25T16:45:09.816235Z","shell.execute_reply.started":"2025-03-25T16:45:09.807885Z","shell.execute_reply":"2025-03-25T16:45:09.815285Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"df_train","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-03-25T16:45:09.817052Z","iopub.execute_input":"2025-03-25T16:45:09.817320Z","iopub.status.idle":"2025-03-25T16:45:09.851480Z","shell.execute_reply.started":"2025-03-25T16:45:09.817299Z","shell.execute_reply":"2025-03-25T16:45:09.850478Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"training_dataloader = dataset.get_subset(df_train.index)\nval_dataloader = dataset.get_subset(df_test.index)\n\n\ntraining_dataloader.transform = T.Compose([\n    *transform_display.transforms,\n    T.ToTensor(),\n    T.Normalize(mean=(0.485, 0.456, 0.406), std=(0.229, 0.224, 0.225))\n    ])\ntraining_dataloader.transform\n\nval_dataloader.transform = T.Compose([\n    *transform_display.transforms,\n    T.ToTensor(),\n    T.Normalize(mean=(0.485, 0.456, 0.406), std=(0.229, 0.224, 0.225))\n    ])\nval_dataloader.transform","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-03-25T16:45:09.852337Z","iopub.execute_input":"2025-03-25T16:45:09.852659Z","iopub.status.idle":"2025-03-25T16:45:09.867911Z","shell.execute_reply.started":"2025-03-25T16:45:09.852624Z","shell.execute_reply":"2025-03-25T16:45:09.866988Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"# Inference with MegaDescriptor\n\nHere we perform the provided inference but with the freshly splitted dataset.","metadata":{}},{"cell_type":"code","source":"name = 'hf-hub:BVRA/MegaDescriptor-L-384'\ndevice = 'cuda'\nmodel = timm.create_model(name, num_classes=0, pretrained=True)\nextractor = DeepFeatures(model, device=device, batch_size=32, num_workers=0)\nfeatures_database = extractor(training_dataloader)\nfeatures_query = extractor(val_dataloader)\nn_query = len(val_dataloader)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-03-25T16:45:09.868734Z","iopub.execute_input":"2025-03-25T16:45:09.868957Z","iopub.status.idle":"2025-03-25T17:07:26.996804Z","shell.execute_reply.started":"2025-03-25T16:45:09.868939Z","shell.execute_reply":"2025-03-25T17:07:26.995702Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"Here we extract the similarity between the images. The similarity equals to the cosine similarity between the corresponding feature vectors. Since the cosine similarity reflects the angle between the feature vectors, high similarity means that the feature vectors are close to each other and should depict the same individual.","metadata":{}},{"cell_type":"code","source":"similarity = CosineSimilarity()(features_query, features_database)","metadata":{"execution":{"iopub.status.busy":"2025-03-25T17:07:26.997968Z","iopub.execute_input":"2025-03-25T17:07:26.998235Z","iopub.status.idle":"2025-03-25T17:07:27.520765Z","shell.execute_reply.started":"2025-03-25T17:07:26.998214Z","shell.execute_reply":"2025-03-25T17:07:27.519854Z"},"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"For each query image we extract the closest image as predicted by MegaDescriptor. Their indices are stored in `pred_idx` and the similarity score of the top match in `pred_scores`.","metadata":{}},{"cell_type":"code","source":"pred_idx = similarity.argsort(axis=1)[:,-1]\npred_scores = similarity[range(n_query), pred_idx]","metadata":{"execution":{"iopub.status.busy":"2025-03-25T17:07:27.521683Z","iopub.execute_input":"2025-03-25T17:07:27.521983Z","iopub.status.idle":"2025-03-25T17:07:28.383946Z","shell.execute_reply.started":"2025-03-25T17:07:27.521954Z","shell.execute_reply":"2025-03-25T17:07:28.383017Z"},"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"This is sufficient to perform predictions as shown below in `predictions`. However, we also need to predict whether the individual is new. We utilize the simplest idea and predict that the individual is new whenever the similarity score of the top prediction is below a certain threshold which we arbirtarily selected as 0.6. We create a submission file, which scores 30.0% on the public leaderboard.","metadata":{}},{"cell_type":"code","source":"new_individual = 'new_individual'\nthreshold = 0.6\nlabels = training_dataloader.labels_string\npredictions = labels[pred_idx]\npredictions[pred_scores < threshold] = new_individual","metadata":{"execution":{"iopub.status.busy":"2025-03-25T17:08:32.676813Z","iopub.execute_input":"2025-03-25T17:08:32.677169Z","iopub.status.idle":"2025-03-25T17:08:32.682254Z","shell.execute_reply.started":"2025-03-25T17:08:32.677139Z","shell.execute_reply":"2025-03-25T17:08:32.681316Z"},"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"# Evaluation","metadata":{}},{"cell_type":"markdown","source":"## Find unseen values","metadata":{}},{"cell_type":"code","source":"unseen_ids = []\nall_ids = df['identity'].unique()\n\nfor i in all_ids:\n    if i in val_dataloader.metadata['identity'].tolist():\n        if i not in training_dataloader.metadata['identity'].tolist():\n            unseen_ids.append(i)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-03-25T17:08:36.183166Z","iopub.execute_input":"2025-03-25T17:08:36.183490Z","iopub.status.idle":"2025-03-25T17:08:36.367207Z","shell.execute_reply.started":"2025-03-25T17:08:36.183465Z","shell.execute_reply":"2025-03-25T17:08:36.366406Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## Calculate BAUS, BAKS and their mean","metadata":{}},{"cell_type":"code","source":"val_true_labels = val_dataloader.labels_string\nbaks_score = baks_compute(val_true_labels, predictions, unseen_ids)\nbaus_score = baus_compute(val_true_labels, predictions, unseen_ids, \"new_individual\")\ngeo_mean = compute_geometric_mean(baks_score, baus_score)\n\nprint(f\"Balanced Accuracy Known Samples (BAKS): {baks_score:.4f}\")\nprint(f\"Balanced Accuracy Unknown Samples (BAUS): {baus_score:.4f}\")\nprint(f\"Geometric Mean (BAKS & BAUS): {geo_mean:.4f}\")","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-03-25T17:11:07.484950Z","iopub.execute_input":"2025-03-25T17:11:07.485268Z","iopub.status.idle":"2025-03-25T17:11:07.551039Z","shell.execute_reply.started":"2025-03-25T17:11:07.485244Z","shell.execute_reply":"2025-03-25T17:11:07.550394Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## Visualizing the metrics by threshold","metadata":{}},{"cell_type":"code","source":"thresholds = np.arange(0.1, 0.9, 0.05)\nresults = []\nlabels = np.array(training_dataloader.labels_string)\n\n\nfor threshold in thresholds:\n    predictions = labels[pred_idx].copy()\n    predictions[pred_scores < threshold] = 'new_individual'\n\n    baks = baks_compute(val_true_labels, predictions, unseen_ids)\n    baus = baus_compute(val_true_labels, predictions, unseen_ids, 'new_individual')\n    geo_mean = compute_geometric_mean(baks, baus)\n    results.append((threshold, baks, baus, geo_mean))\n\n    print(f\"Threshold: {threshold:.2f} | BAKS: {baks:.4f} | BAUS: {baus:.4f} | GEO_MEAN: {geo_mean:.4f}\")","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-03-25T17:22:27.322321Z","iopub.execute_input":"2025-03-25T17:22:27.322642Z","iopub.status.idle":"2025-03-25T17:22:28.161254Z","shell.execute_reply.started":"2025-03-25T17:22:27.322615Z","shell.execute_reply":"2025-03-25T17:22:28.160350Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"import matplotlib.pyplot as plt\n\nthresholds, baks_scores, baus_scores, geo = zip(*results)\nplt.plot(thresholds, baks_scores, label='BAKS (Known)')\nplt.plot(thresholds, baus_scores, label='BAUS (Unknown)')\nplt.plot(thresholds, geo, label='Geometrical Mean')\nplt.xlabel('Cosine Threshold')\nplt.ylabel('Balanced Accuracy')\nplt.legend()\nplt.show()","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-03-25T17:22:35.437667Z","iopub.execute_input":"2025-03-25T17:22:35.437971Z","iopub.status.idle":"2025-03-25T17:22:35.615222Z","shell.execute_reply.started":"2025-03-25T17:22:35.437947Z","shell.execute_reply":"2025-03-25T17:22:35.614217Z"}},"outputs":[],"execution_count":null}]}