{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"name":"python","version":"3.12.12","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"},"kaggle":{"accelerator":"gpu","dataSources":[{"sourceType":"competition","sourceId":127282,"databundleVersionId":15520105}],"dockerImageVersionId":31260,"isInternetEnabled":true,"language":"python","sourceType":"notebook","isGpuEnabled":true}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# 🐾 AnimalCLEF2026: UMAP\n\nThe **Goal of the** [AnimalCLEF2026](https://www.kaggle.com/competitions/animal-clef-2026/) competition is to cluster individual animal (loggerhead sea turtles and Texas horned lizards) in photos. \n\n\nThe dataset is split into the training and test sets. For each image from the test set, the goal is to assing in a cluster. Each cluster should correspond to one individual animal. The cluster name must start with `cluster_SeaTurtleID2022_` or `cluster_TexasHornedLizards_`.\n\n\nWe start with the official starter notebook to load the required data, and proceed to apply UMAP to form clusters. ","metadata":{}},{"cell_type":"markdown","source":"## Dependencies instalation\nFor the competition we provide two Python packages for loading and preprocessing of available datasets ([wildlife-datasets](https://github.com/WildlifeDatasets/wildlife-datasets)) and tools / method for animal re-identification ([wildlife-tools](https://github.com/WildlifeDatasets/wildlife-tools)).","metadata":{}},{"cell_type":"code","source":"! pip install umap-learn","metadata":{"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"code","source":"!pip install wildlife-datasets git+https://github.com/WildlifeDatasets/wildlife-tools --quiet --upgrade-strategy only-if-needed","metadata":{"trusted":true,"_kg_hide-input":false,"_kg_hide-output":true,"execution":{"iopub.status.busy":"2026-03-18T23:57:34.436793Z","iopub.execute_input":"2026-03-18T23:57:34.437541Z","iopub.status.idle":"2026-03-18T23:57:53.385861Z","shell.execute_reply.started":"2026-03-18T23:57:34.437513Z","shell.execute_reply":"2026-03-18T23:57:53.385100Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## Dependencies import\n\nWe load all the required packages.","metadata":{}},{"cell_type":"code","source":"import warnings\nwarnings.filterwarnings(\"ignore\")\nimport numpy as np\nimport pandas as pd\nimport matplotlib.pyplot as plt\nimport torchvision.transforms as T\nimport timm\nfrom transformers import AutoModel\nfrom sklearn.cluster import HDBSCAN\nfrom wildlife_datasets.datasets import AnimalCLEF2026\nfrom wildlife_tools.features import DeepFeatures\nfrom wildlife_tools.similarity import CosineSimilarity","metadata":{"trusted":true,"_kg_hide-output":true,"execution":{"iopub.status.busy":"2026-03-18T23:57:54.932755Z","iopub.execute_input":"2026-03-18T23:57:54.933610Z","iopub.status.idle":"2026-03-18T23:58:06.337922Z","shell.execute_reply.started":"2026-03-18T23:57:54.933573Z","shell.execute_reply":"2026-03-18T23:58:06.337258Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"import umap\nfrom sklearn.metrics import adjusted_rand_score","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2026-03-19T05:17:48.104960Z","iopub.execute_input":"2026-03-19T05:17:48.105761Z","iopub.status.idle":"2026-03-19T05:17:48.109029Z","shell.execute_reply.started":"2026-03-19T05:17:48.105730Z","shell.execute_reply":"2026-03-19T05:17:48.108303Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"root = '/kaggle/input/animal-clef-2026'\ndataset_full = AnimalCLEF2026(\n    root,\n    transform=None,\n    load_label=True,\n    factorize_label=True,\n    check_files=False\n)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2026-03-18T23:58:06.454945Z","iopub.execute_input":"2026-03-18T23:58:06.455217Z","iopub.status.idle":"2026-03-18T23:58:06.506137Z","shell.execute_reply.started":"2026-03-18T23:58:06.455193Z","shell.execute_reply":"2026-03-18T23:58:06.505564Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"dataset_full.df","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2026-03-19T03:38:17.285433Z","iopub.execute_input":"2026-03-19T03:38:17.286140Z","iopub.status.idle":"2026-03-19T03:38:17.304665Z","shell.execute_reply.started":"2026-03-19T03:38:17.286109Z","shell.execute_reply":"2026-03-19T03:38:17.304119Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"dataset_test = dataset_full.get_subset(dataset_full.df['split'] == 'test')\n\ntest_datasets = {}\nfor name in dataset_test.metadata['dataset'].unique():\n    test_datasets[name] = dataset_test.get_subset(dataset_test.df['dataset'] == name)\ntest_datasets","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2026-03-19T06:40:54.023766Z","iopub.execute_input":"2026-03-19T06:40:54.024056Z","iopub.status.idle":"2026-03-19T06:40:54.042728Z","shell.execute_reply.started":"2026-03-19T06:40:54.024032Z","shell.execute_reply":"2026-03-19T06:40:54.042143Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"📌 **Plotting a sample grid** of the data. We see that there is a large variability in sea turtles, which were taken underwater in their natural habitat. On the other hands, the images of lizards were taken in captivity.","metadata":{}},{"cell_type":"markdown","source":"## Finetuning\n\nTo utilize the training dataset, we finetune the feature extractor.","metadata":{}},{"cell_type":"code","source":"import itertools\nfrom torch.optim import Adam\nfrom wildlife_tools.train import ArcFaceLoss, BasicTrainer\nfrom wildlife_tools.train import set_seed","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2026-03-18T23:58:16.733177Z","iopub.execute_input":"2026-03-18T23:58:16.734031Z","iopub.status.idle":"2026-03-18T23:58:16.737550Z","shell.execute_reply.started":"2026-03-18T23:58:16.734003Z","shell.execute_reply":"2026-03-18T23:58:16.736719Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"dataset_train = dataset_full.get_subset(dataset_full.df['split'] == 'train')","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2026-03-18T23:58:19.353030Z","iopub.execute_input":"2026-03-18T23:58:19.353816Z","iopub.status.idle":"2026-03-18T23:58:19.365625Z","shell.execute_reply.started":"2026-03-18T23:58:19.353787Z","shell.execute_reply":"2026-03-18T23:58:19.364814Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"backbone1 = timm.create_model(\"hf-hub:BVRA/MegaDescriptor-L-384\", pretrained=True)\nbackbone2 = AutoModel.from_pretrained('conservationxlabs/miewid-msv3', trust_remote_code=True)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2026-03-19T01:32:14.214826Z","iopub.execute_input":"2026-03-19T01:32:14.215290Z","iopub.status.idle":"2026-03-19T01:32:21.845221Z","shell.execute_reply.started":"2026-03-19T01:32:14.215255Z","shell.execute_reply":"2026-03-19T01:32:21.844424Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"train_groups = {}\ntrain_groups[\"group1\"] = dataset_train.get_subset(dataset_train.df['dataset'].isin(\n                                                ['SalamanderID2025', 'SeaTurtleID2022']))\ntrain_groups[\"group2\"] = dataset_train.get_subset(dataset_train.df['dataset'].isin(\n                                                  ['LynxID2025', 'TexasHornedLizards']))\ntrain_groups","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2026-03-19T00:59:29.932697Z","iopub.execute_input":"2026-03-19T00:59:29.933000Z","iopub.status.idle":"2026-03-19T00:59:29.952232Z","shell.execute_reply.started":"2026-03-19T00:59:29.932974Z","shell.execute_reply":"2026-03-19T00:59:29.951683Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"def get_transform(group):\n    \n    if group == \"group1\":\n        size = 384\n    else:\n        size = 512\n        \n    return T.Compose([\n            T.Resize(size=(size, size)),\n            T.ToTensor(),\n            T.Normalize(mean=(0.485, 0.456, 0.406), std=(0.229, 0.224, 0.225)),\n        ])","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2026-03-19T01:00:29.068681Z","iopub.execute_input":"2026-03-19T01:00:29.068960Z","iopub.status.idle":"2026-03-19T01:00:29.074259Z","shell.execute_reply.started":"2026-03-19T01:00:29.068939Z","shell.execute_reply":"2026-03-19T01:00:29.073636Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"tranform1 = get_transform(\"group1\")\ntrain_groups[\"group1\"].set_transform(tranform1)\nembedding_size1 = backbone1(train_groups[\"group1\"][0][0].unsqueeze(0)).size(1)\nobj1 = ArcFaceLoss(\n    num_classes=train_groups[\"group1\"].num_classes,\n    embedding_size=embedding_size1,\n    margin=0.5,\n    scale=64\n)\nparams1 = itertools.chain(backbone1.parameters(), obj1.parameters())\nopt1 = Adam(params=params1, lr=0.0001)\n\ntrainer1 = BasicTrainer(\n    dataset=train_groups[\"group1\"],\n    model=backbone1,\n    objective=obj1,\n    optimizer=opt1,\n    epochs=2,\n    batch_size=8,\n    device='cuda',\n)\n\ntrainer1.train()","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2026-03-19T01:47:34.342850Z","iopub.execute_input":"2026-03-19T01:47:34.343745Z","iopub.status.idle":"2026-03-19T02:29:21.404025Z","shell.execute_reply.started":"2026-03-19T01:47:34.343709Z","shell.execute_reply":"2026-03-19T02:29:21.403055Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"tranform2 = get_transform(\"group2\")\ntrain_groups[\"group2\"].set_transform(tranform2)\nembedding_size2 = backbone2(train_groups[\"group2\"][0][0].unsqueeze(0)).size(1)\nobj2 = ArcFaceLoss(\n    num_classes=train_groups[\"group2\"].num_classes,\n    embedding_size=embedding_size2,\n    margin=0.5,\n    scale=64\n)\nparams2 = itertools.chain(backbone2.parameters(), obj2.parameters())\nopt2 = Adam(params=params2, lr=0.0001)\n\ntrainer2 = BasicTrainer(\n    dataset=train_groups[\"group2\"],\n    model=backbone2,\n    objective=obj2,\n    optimizer=opt2,\n    epochs=1,\n    batch_size=8,\n    device='cuda',\n)\ntrainer2.train()","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2026-03-19T01:35:43.154953Z","iopub.execute_input":"2026-03-19T01:35:43.155714Z","iopub.status.idle":"2026-03-19T01:39:00.476714Z","shell.execute_reply.started":"2026-03-19T01:35:43.155686Z","shell.execute_reply":"2026-03-19T01:39:00.475714Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## Extract features","metadata":{}},{"cell_type":"code","source":"tuned_extractor1 = DeepFeatures(model=trainer1.model, device=\"cuda\", batch_size=32)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2026-03-19T03:41:09.712482Z","iopub.execute_input":"2026-03-19T03:41:09.712991Z","iopub.status.idle":"2026-03-19T03:41:09.716430Z","shell.execute_reply.started":"2026-03-19T03:41:09.712965Z","shell.execute_reply":"2026-03-19T03:41:09.715836Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"tuned_extractor2 = DeepFeatures(model=trainer2.model, device=\"cuda\", batch_size=32)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2026-03-19T02:31:54.090211Z","iopub.execute_input":"2026-03-19T02:31:54.090814Z","iopub.status.idle":"2026-03-19T02:31:54.094859Z","shell.execute_reply.started":"2026-03-19T02:31:54.090778Z","shell.execute_reply":"2026-03-19T02:31:54.094094Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"Test clustering in the training set; if the direct clustering is poor, we use UMAP.","metadata":{}},{"cell_type":"code","source":"train_datasets = {}\nfor name in dataset_train.metadata['dataset'].unique():\n    train_datasets[name] = dataset_train.get_subset(dataset_train.df['dataset'] == name)\ntrain_datasets","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2026-03-19T02:58:23.075844Z","iopub.execute_input":"2026-03-19T02:58:23.076555Z","iopub.status.idle":"2026-03-19T02:58:23.099904Z","shell.execute_reply.started":"2026-03-19T02:58:23.076525Z","shell.execute_reply":"2026-03-19T02:58:23.099380Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"features = {}\nfor name, dataset in train_datasets.items():\n    if name == 'LynxID2025':\n        dataset.set_transform(get_transform(\"group2\"))\n        features[name] = tuned_extractor2(dataset)\n    else:\n        dataset.set_transform(get_transform(\"group1\"))\n        features[name] = tuned_extractor1(dataset)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2026-03-19T04:48:30.444906Z","iopub.execute_input":"2026-03-19T04:48:30.445784Z","iopub.status.idle":"2026-03-19T04:57:13.713969Z","shell.execute_reply.started":"2026-03-19T04:48:30.445755Z","shell.execute_reply":"2026-03-19T04:57:13.713300Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"def eval_hdbscan(features):\n    matcher = CosineSimilarity()\n    similarity = matcher(features, features)\n    clusters = run_DBSCAN(similarity)\n    \n    return adjusted_rand_score(clusters, features.labels)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2026-03-19T05:02:12.728949Z","iopub.execute_input":"2026-03-19T05:02:12.729711Z","iopub.status.idle":"2026-03-19T05:02:12.733391Z","shell.execute_reply.started":"2026-03-19T05:02:12.729681Z","shell.execute_reply":"2026-03-19T05:02:12.732657Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"for name, feature in features.items():\n\n    print(name, eval_hdbscan(feature))    ","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2026-03-19T05:03:23.432842Z","iopub.execute_input":"2026-03-19T05:03:23.433136Z","iopub.status.idle":"2026-03-19T05:03:28.251701Z","shell.execute_reply.started":"2026-03-19T05:03:23.433110Z","shell.execute_reply":"2026-03-19T05:03:28.251063Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"Try first two datasets with UMAP.","metadata":{}},{"cell_type":"code","source":"def umap_cluster(features, n_components, n_neighbors, densemap = False, min_samples = 2):\n    \n    reducer = umap.UMAP(\n        densmap=densemap,\n        n_components=n_components, n_neighbors=n_neighbors, random_state=42, transform_seed=42,\n        metric = \"cosine\", verbose=False\n    )\n    reducer.fit(features.features, features.labels)\n    umap_supervised = reducer.transform(features.features)\n    \n    clustering = HDBSCAN(min_samples = min_samples)\n    clustering.fit(umap_supervised)\n    \n    return relabel_negatives(clustering.labels_)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2026-03-19T05:35:20.654820Z","iopub.execute_input":"2026-03-19T05:35:20.655549Z","iopub.status.idle":"2026-03-19T05:35:20.660243Z","shell.execute_reply.started":"2026-03-19T05:35:20.655520Z","shell.execute_reply":"2026-03-19T05:35:20.659498Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"def umap_cluster_test(train_features, test_features, n_components, n_neighbors, densemap = False, min_samples = 2):\n    \n    reducer = umap.UMAP(\n        densmap=densemap,\n        n_components=n_components, n_neighbors=n_neighbors, random_state=42, transform_seed=42,\n        metric = \"cosine\", verbose=False\n    )\n    # reducer.fit(train_features.features, train_features.labels)\n    umap_supervised = reducer.fit_transform(test_features.features)\n    \n    clustering = HDBSCAN(min_samples = min_samples)\n    clustering.fit(umap_supervised)\n    \n    return relabel_negatives(clustering.labels_)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2026-03-19T07:16:48.145907Z","iopub.execute_input":"2026-03-19T07:16:48.146636Z","iopub.status.idle":"2026-03-19T07:16:48.150860Z","shell.execute_reply.started":"2026-03-19T07:16:48.146607Z","shell.execute_reply":"2026-03-19T07:16:48.150248Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"def eval_hdbscan(features):\n    matcher = CosineSimilarity()\n    similarity = matcher(features, features)\n    clusters = run_DBSCAN(similarity)\n    \n    return adjusted_rand_score(clusters, features.labels)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2026-03-19T05:01:39.300068Z","iopub.execute_input":"2026-03-19T05:01:39.300518Z","iopub.status.idle":"2026-03-19T05:01:39.304729Z","shell.execute_reply.started":"2026-03-19T05:01:39.300491Z","shell.execute_reply":"2026-03-19T05:01:39.304022Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"def eval_umap_params(features, n_components, n_neighbors, densemap = False, min_samples = 2):\n\n    clusters_labels = umap_cluster(features, n_components, n_neighbors, densemap = densemap, min_samples = min_samples)\n    \n    return adjusted_rand_score(clusters_labels, features.labels)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2026-03-19T05:35:39.367678Z","iopub.execute_input":"2026-03-19T05:35:39.368378Z","iopub.status.idle":"2026-03-19T05:35:39.372064Z","shell.execute_reply.started":"2026-03-19T05:35:39.368350Z","shell.execute_reply":"2026-03-19T05:35:39.371394Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"eval_umap_params(features[\"LynxID2025\"], 150, 45, min_samples = 10)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2026-03-19T05:35:41.912054Z","iopub.execute_input":"2026-03-19T05:35:41.912704Z","iopub.status.idle":"2026-03-19T05:36:24.092034Z","shell.execute_reply.started":"2026-03-19T05:35:41.912674Z","shell.execute_reply":"2026-03-19T05:36:24.091385Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"eval_umap_params(features[\"SalamanderID2025\"], 1000, 10, densemap = False)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2026-03-19T05:14:40.901778Z","iopub.execute_input":"2026-03-19T05:14:40.902425Z","iopub.status.idle":"2026-03-19T05:15:10.237538Z","shell.execute_reply.started":"2026-03-19T05:14:40.902397Z","shell.execute_reply":"2026-03-19T05:15:10.236716Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"test_datasets['LynxID2025'].df","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2026-03-19T06:38:47.443383Z","iopub.execute_input":"2026-03-19T06:38:47.443928Z","iopub.status.idle":"2026-03-19T06:38:47.460081Z","shell.execute_reply.started":"2026-03-19T06:38:47.443903Z","shell.execute_reply":"2026-03-19T06:38:47.459380Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## Make predictions","metadata":{}},{"cell_type":"code","source":"test_features = {}\nfor name, dataset in test_datasets.items():\n    if name in ['LynxID2025', 'TexasHornedLizards']:\n        dataset.set_transform(get_transform(\"group2\"))\n        test_features[name] = tuned_extractor2(dataset)\n    else:\n        dataset.set_transform(get_transform(\"group1\"))\n        test_features[name] = tuned_extractor1(dataset)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2026-03-19T06:59:50.781745Z","iopub.execute_input":"2026-03-19T06:59:50.782286Z","iopub.status.idle":"2026-03-19T07:01:35.958162Z","shell.execute_reply.started":"2026-03-19T06:59:50.782258Z","shell.execute_reply":"2026-03-19T07:01:35.957397Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"test_datasets[\"TexasHornedLizards\"].set_transform(get_transform(\"group1\"))\ntest_features['TexasHornedLizards1'] = tuned_extractor1(test_datasets[\"TexasHornedLizards\"])","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2026-03-19T06:12:53.484839Z","iopub.execute_input":"2026-03-19T06:12:53.485539Z","iopub.status.idle":"2026-03-19T06:13:09.823307Z","shell.execute_reply.started":"2026-03-19T06:12:53.485505Z","shell.execute_reply":"2026-03-19T06:13:09.822394Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"\ntest_datasets[\"TexasHornedLizards\"].set_transform(get_transform(\"group2\"))\ntest_features['TexasHornedLizards2'] = tuned_extractor2(test_datasets[\"TexasHornedLizards\"])","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2026-03-19T06:13:09.825025Z","iopub.execute_input":"2026-03-19T06:13:09.825310Z","iopub.status.idle":"2026-03-19T06:13:25.704024Z","shell.execute_reply.started":"2026-03-19T06:13:09.825282Z","shell.execute_reply":"2026-03-19T06:13:25.703049Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"combined_feature = np.concatenate([test_features['TexasHornedLizards1'].features,  \n                          test_features['TexasHornedLizards2'].features], axis = 1)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2026-03-19T06:17:31.467689Z","iopub.execute_input":"2026-03-19T06:17:31.467972Z","iopub.status.idle":"2026-03-19T06:17:31.472405Z","shell.execute_reply.started":"2026-03-19T06:17:31.467947Z","shell.execute_reply":"2026-03-19T06:17:31.471778Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"from sklearn.metrics.pairwise import cosine_similarity ","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2026-03-19T06:25:22.742357Z","iopub.execute_input":"2026-03-19T06:25:22.742956Z","iopub.status.idle":"2026-03-19T06:25:22.746340Z","shell.execute_reply.started":"2026-03-19T06:25:22.742929Z","shell.execute_reply":"2026-03-19T06:25:22.745569Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"matcher = CosineSimilarity()\nsimilarity = cosine_similarity(combined_feature, combined_feature)\nclusters = run_DBSCAN(similarity)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2026-03-19T06:25:38.335345Z","iopub.execute_input":"2026-03-19T06:25:38.335874Z","iopub.status.idle":"2026-03-19T06:25:38.356373Z","shell.execute_reply.started":"2026-03-19T06:25:38.335848Z","shell.execute_reply":"2026-03-19T06:25:38.355773Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"reducer = umap.UMAP(\n    n_components=150, n_neighbors= 45, random_state=42, transform_seed=42,\n    metric = \"cosine\", verbose=False\n)\nreducer.fit(features[\"LynxID2025\"].features, features[\"LynxID2025\"].labels)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2026-03-19T06:54:19.128737Z","iopub.execute_input":"2026-03-19T06:54:19.129477Z","iopub.status.idle":"2026-03-19T06:55:00.299882Z","shell.execute_reply.started":"2026-03-19T06:54:19.129448Z","shell.execute_reply":"2026-03-19T06:55:00.299236Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"umap_supervised = reducer.transform(test_features[\"LynxID2025\"].features)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2026-03-19T06:55:17.149378Z","iopub.execute_input":"2026-03-19T06:55:17.149951Z","iopub.status.idle":"2026-03-19T06:56:15.702145Z","shell.execute_reply.started":"2026-03-19T06:55:17.149924Z","shell.execute_reply":"2026-03-19T06:56:15.701256Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"results = []\n\nfor name, tf in test_features.items():\n\n    if name in ['SeaTurtleID2022', 'TexasHornedLizards', 'SalamanderID2025']:\n        \n        matcher = CosineSimilarity()\n        similarity = matcher(tf, tf)\n        clusters = run_DBSCAN(similarity)\n        \n        results.append(pd.DataFrame({\n            'image_id': test_datasets[name].metadata['image_id'],\n            'cluster': [f'cluster_{name}_{c}' for c in clusters]\n        }))\n    elif name == \"LynxID2025\":\n\n        clusters = umap_cluster_test(features[name], tf, 150, 45, min_samples = 10)\n\n        results.append(pd.DataFrame({\n            'image_id': test_datasets[name].metadata['image_id'],\n            'cluster': [f'cluster_{name}_{c}' for c in clusters]\n        }))\n    else:\n        clusters = umap_cluster_test(features[name], tf, 1000, 10)\n        \n        results.append(pd.DataFrame({\n            'image_id': test_datasets[name].metadata['image_id'],\n            'cluster': [f'cluster_{name}_{c}' for c in clusters]\n        }))        ","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2026-03-19T07:19:00.530991Z","iopub.execute_input":"2026-03-19T07:19:00.531296Z","iopub.status.idle":"2026-03-19T07:19:10.162091Z","shell.execute_reply.started":"2026-03-19T07:19:00.531270Z","shell.execute_reply":"2026-03-19T07:19:10.161302Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"def relabel_negatives(labels):\n    labels = np.array(labels)\n    neg_indices = np.where(labels == -1)[0]\n    new_labels = np.arange(labels.max()+1, labels.max()+1+len(neg_indices))\n    labels[neg_indices] = new_labels\n    return labels\n\ndef run_DBSCAN(similarity):\n    # Convert similarity (high is good) to distance (small is good)\n    distance = (np.max(similarity) - np.maximum(similarity, 0)) / np.max(similarity)\n    # Obtain predictions\n    clustering = HDBSCAN(metric='precomputed', min_samples=2)\n    clusters = clustering.fit(distance)\n    # Relabel -1 clusters into separate clusters\n    return relabel_negatives(clusters.labels_)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2026-03-19T03:02:48.490877Z","iopub.execute_input":"2026-03-19T03:02:48.491163Z","iopub.status.idle":"2026-03-19T03:02:48.496545Z","shell.execute_reply.started":"2026-03-19T03:02:48.491138Z","shell.execute_reply":"2026-03-19T03:02:48.495897Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"res_df = pd.concat(results, axis=0)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2026-03-19T07:19:10.163389Z","iopub.execute_input":"2026-03-19T07:19:10.163653Z","iopub.status.idle":"2026-03-19T07:19:10.168826Z","shell.execute_reply.started":"2026-03-19T07:19:10.163630Z","shell.execute_reply":"2026-03-19T07:19:10.168006Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"res_df.to_csv('submission.csv', index=False)\n","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2026-03-19T07:19:10.169873Z","iopub.execute_input":"2026-03-19T07:19:10.170168Z","iopub.status.idle":"2026-03-19T07:19:10.184137Z","shell.execute_reply.started":"2026-03-19T07:19:10.170148Z","shell.execute_reply":"2026-03-19T07:19:10.183596Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"pd.concat(results, axis=0).to_csv('submission.csv', index=False)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2026-03-19T05:53:51.811429Z","iopub.execute_input":"2026-03-19T05:53:51.811982Z","iopub.status.idle":"2026-03-19T05:53:51.838414Z","shell.execute_reply.started":"2026-03-19T05:53:51.811956Z","shell.execute_reply":"2026-03-19T05:53:51.837872Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"","metadata":{"trusted":true},"outputs":[],"execution_count":null}]}