๐พ LynxID2025: Neural Feature Mapping & Re-ID Optimization ๐
\n\n---\n\n## ๐ Project Overview\nThis project is dedicated to solving the **LynxID2025** challenge. The primary goal is to build an advanced AI system capable of distinguishing between individual Lynx through their unique fur patterns. By leveraging deep learning, we aim to support wildlife preservation efforts through automated, high-precision individual re-identification.\n\n---\n\n## ๐ฏ Main Objectives\n\n* **Deep Feature Extraction:** Utilizing state-of-the-art `MegaDescriptor` models to transform Lynx imagery into high-dimensional numerical embeddings.\n* **Similarity Analysis:** Constructing precision-grade **Similarity Matrices** to quantify the relationship between different captures of the same individual.\n* **Smart Clustering:** Implementing the **DBSCAN** algorithm to partition images into distinct clusters, where each cluster represents a unique individual.\n* **Precision Benchmark:** Achieving a robust **Confidence Rate (>99%)** to ensure reliable identification and minimize False Positives.\n\n---\n\n## ๐ ๏ธ Tech Stack\n\n| Component | Tool / Library |\n| :--- | :--- |\n| **Model Architecture** | `Timm` (PyTorch) - MegaDescriptor Series |\n| **Domain Framework** | `Wildlife Tools` & `Wildlife Datasets` |\n| **Clustering Engine** | `Scikit-Learn` (DBSCAN) |\n| **Visualization** | `Matplotlib` & `Seaborn` |\n\n---\n\nWe implemented a carefully tuned Similarity Threshold to strike a balance between discovering new individuals (outliers) and accurately merging known identities. This strategic tuning is what drives our high confidence scores and ensures the model remains robust across different environmental conditions.
\n๐ ๏ธ Environment Setup & Dependency Management
\n\nThis section initializes the technical workspace by importing essential libraries for **Deep Learning**, **Computer Vision**, and **Clustering Analysis**.\n\n### ๐ฆ Key Modules:\n* **Data Processing:** `NumPy` & `Pandas` for efficient matrix operations and metadata management.\n* **Visualization:** `Matplotlib` & `Seaborn` for analyzing feature distributions and similarity heatmaps.\n* **Computer Vision:** `OpenCV`, `PIL`, and `Torchvision` for image preprocessing and transformations.\n* **Deep Learning:** `Timm` and `HuggingFace Transformers` for loading state-of-the-art **Re-ID** backbones.\n* **Clustering & Similarity:** `DBSCAN` for identity grouping and `CosineSimilarity` to measure feature proximity.\n* **Domain Tools:** `Wildlife Tools` for specialized animal re-identification workflows.\n\n---","metadata":{}},{"cell_type":"code","source":"# 1. Standard Library Imports\nimport warnings\nwarnings.filterwarnings('ignore') # Suppress warnings for a clean output\n\n# 2. Data Science & Matrix Operations\nimport numpy as np \nimport pandas as pd\n\n# 3. Visualization Tools\nimport matplotlib.pyplot as plt\nimport seaborn as sns\nfrom IPython.display import display, HTML\n\n# 4. Image Processing\nimport cv2\nfrom PIL import Image\nimport random\nimport os\n\n# 5. Deep Learning & Computer Vision Backbones\nimport torchvision.transforms as T\nimport timm\nfrom transformers import AutoModel\n\n# 6. Machine Learning (Clustering & Normalization)\nfrom sklearn.cluster import DBSCAN\nfrom sklearn.preprocessing import normalize\n\n# 7. Specialized Wildlife Re-ID Frameworks\nfrom wildlife_datasets.datasets import AnimalCLEF2026\nfrom wildlife_tools.features import DeepFeatures\nfrom wildlife_tools.similarity import CosineSimilarity\n\nprint(\"โ All dependencies loaded successfully.\")","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2026-03-11T12:33:28.729936Z","iopub.execute_input":"2026-03-11T12:33:28.730198Z","iopub.status.idle":"2026-03-11T12:33:55.777428Z","shell.execute_reply.started":"2026-03-11T12:33:28.730171Z","shell.execute_reply":"2026-03-11T12:33:55.776789Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"# Data Initialization & Exploratory Analysis\n\nIn this stage, we establish the connection to the competition dataset and perform an initial inspection of the metadata. Understanding the distribution of images across different species is critical for tuning our Re-ID pipeline.\n\n### ๐ Key Activities:\n* **Dataset Loading:** Accessing the `AnimalCLEF2026` directory via the specified root path.\n* **Metadata Extraction:** Converting raw dataset information into a structured `Pandas DataFrame`.\n* **Statistical Overview:** Calculating total image counts and inspecting the initial entries.\n* **Species Distribution:** Visualizing the frequency of each species to identify potential data imbalances.\n\n---","metadata":{}},{"cell_type":"code","source":"# --- Dataset Configuration ---\n# Setting the root directory for the AnimalCLEF2026 competition data\nroot = '/kaggle/input/competitions/animal-clef-2026'\n\n# Initializing the dataset and extracting metadata\nfull_dataset_1 = AnimalCLEF2026(root=root)\ndf = full_dataset_1.metadata\n\n# --- Summary Statistics ---\nprint(\"-\" * 50)\nprint(f\"๐ TOTAL IMAGES IN DATASET: {len(df)}\")\nprint(\"-\" * 50)\n\n# --- Metadata Preview ---\nprint(\"\\n๐ Metadata Quick Inspection (First 5 records):\")\n# Professional styling for the table: Black headers, white background\nstyled_df = df.head().style.set_properties(**{\n 'background-color': '#ffffff',\n 'color': '#000000',\n 'border-color': '#dddddd'\n})\ndisplay(styled_df)\n\n# --- Species Distribution Visualization ---\nif 'species' in df.columns:\n plt.figure(figsize=(10, 5))\n # Using 'Greys_r' palette for a sleek black-and-white professional look\n sns.countplot(data=df, y='species', palette='Greys_r')\n \n plt.title('Species Distribution Analysis', fontsize=14, fontweight='bold', color='#000000')\n plt.xlabel('Number of Images', fontsize=12, color='#000000')\n plt.ylabel('Species Name', fontsize=12, color='#000000')\n \n sns.despine() # Removes top and right borders for a cleaner look\n plt.grid(axis='x', linestyle='--', alpha=0.3)\n plt.tight_layout()\n plt.show()","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2026-03-11T12:33:55.778352Z","iopub.execute_input":"2026-03-11T12:33:55.778859Z","iopub.status.idle":"2026-03-11T12:34:57.229845Z","shell.execute_reply.started":"2026-03-11T12:33:55.778834Z","shell.execute_reply":"2026-03-11T12:34:57.229273Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"# Visual Inspection & Image Quality Assessment\n\nThis section focuses on qualitative and quantitative analysis of the dataset samples. We utilize the **Laplacian Variance** method to estimate image sharpness and identify potential outliers such as blurred or low-quality captures.\n\n### ๐ Key Procedures:\n* **Sharpness Metric:** Calculating the variance of the Laplacian to quantify edge density (Blur detection).\n* **Random Sampling:** Selecting representative samples from each species for a diverse visual audit.\n* **Dynamic Thresholding:** Highlighting images with low quality scores to understand potential challenges for the deep learning model.\n* **Visual Validation:** Cross-referencing numerical quality scores with actual image clarity.\n\n---","metadata":{}},{"cell_type":"code","source":"def get_image_quality(img_path):\n \"\"\"Calculates image sharpness using the Laplacian variance method.\"\"\"\n img = cv2.imread(img_path)\n if img is None: \n return 0\n # Convert to grayscale for consistent variance calculation\n gray = cv2.cvtColor(img, cv2.COLOR_BGR2GRAY)\n return cv2.Laplacian(gray, cv2.CV_64F).var()\n\n# Selection: 2 samples per species for a balanced audit\nsample_df = df.groupby('species').sample(2, random_state=42)\n\nplt.figure(figsize=(16, 8))\nplt.suptitle('๐ Image Quality Audit & Sample Preview', fontsize=18, fontweight='bold', color='#000000')\n\nfor i, (idx, row) in enumerate(sample_df.iterrows()):\n img_path = os.path.join(root, row['path'])\n quality = get_image_quality(img_path)\n\n plt.subplot(2, 4, i+1)\n img = Image.open(img_path)\n plt.imshow(img)\n\n # Visual indicators: Black for good quality, Gray for low quality/blur\n # Using 100 as a standard baseline for sharpness\n color = '#000000' if quality > 100 else '#7f8c8d' \n plt.title(f\"Species: {row['species']}\\nSharpness: {quality:.2f}\", \n color=color, fontsize=11, fontweight='bold' if quality < 100 else 'normal')\n plt.axis('off')\n\nplt.tight_layout()\nplt.show()\n\n# --- Insight Section (Professional Minimalist Style) ---\nprint(\"-\" * 100)\nprint(\"๐ก TECHNICAL INSIGHT:\")\nprint(\"Images marked with LOWER sharpness scores (Gray titles) indicate potential motion blur or low focus.\")\nprint(\"The Deep Learning model might require robust data augmentation to handle these specific variations.\")\nprint(\"-\" * 100)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2026-03-11T12:34:57.230713Z","iopub.execute_input":"2026-03-11T12:34:57.230995Z","iopub.status.idle":"2026-03-11T12:34:58.965163Z","shell.execute_reply.started":"2026-03-11T12:34:57.230971Z","shell.execute_reply":"2026-03-11T12:34:58.964459Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"# Species-Specific Data Stratification\n\nTo ensure high-precision Re-Identification, we organize the dataset into specific subsets based on species. This stratification allows for localized feature extraction and more accurate similarity matching within each biological group.\n\n### ๐ Strategy:\n* **Dictionary Mapping:** Creating a structured `data_by_species` dictionary for efficient retrieval.\n* **Subset Initialization:** Instantiating dedicated `AnimalCLEF2026` objects for each unique species.\n* **Visual Verification:** Generating a grid preview for each category to confirm data integrity and label consistency.\n\n---","metadata":{}},{"cell_type":"code","source":"# --- Data Stratification ---\n# Dictionary to store datasets categorized by species\ndata_by_species = {}\n\nfor species in df['species'].unique():\n # Filter metadata for the specific species\n species_df = df[df['species'] == species]\n # Initialize a dedicated dataset object\n data_by_species[species] = AnimalCLEF2026(root=root, df=species_df)\n\n# Status Update\nprint(\"-\" * 50)\nprint(f\"โ Success: {len(data_by_species)} species-specific datasets initialized.\")\nprint(\"-\" * 50)\n\n# --- Visual Audit per Species ---\nfor name, dset in data_by_species.items():\n if len(dset.metadata) > 0:\n # Title formatting: Bold and Uppercase for professional look\n print(f\"\\n๐พ SPECIES: {name.upper()} | Sample Size: 3\")\n \n # Plotting the grid (Assuming plot_grid is a built-in method of the dataset)\n dset.plot_grid(n_rows=1, n_cols=3, rotate=False)\n \n # Styling the plot to match our theme\n plt.gcf().set_facecolor('white') # Keeping it clean\n plt.show()\n \n # Divider for readability\n print('ยท' * 60)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2026-03-11T12:34:58.966130Z","iopub.execute_input":"2026-03-11T12:34:58.966433Z","iopub.status.idle":"2026-03-11T12:34:59.846130Z","shell.execute_reply.started":"2026-03-11T12:34:58.966408Z","shell.execute_reply":"2026-03-11T12:34:59.845453Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"# Final Dataset Partitioning & Test Set Stratification\n\nIn this phase, we finalize the dataset structure by isolating the **Test Split** and further subdividing it based on specific dataset origins. This ensures that our evaluation is granular and accounts for variations across different data sources.\n\n### ๐ Key Steps:\n* **Final Initialization:** Instantiating the `AnimalCLEF2026` object with factorized labels for optimized processing.\n* **Subset Isolation:** Extracting the `test` split from the main dataset metadata.\n* **Source-Based Stratification:** Creating a specialized dictionary to store subsets organized by their original dataset labels.\n\n---","metadata":{}},{"cell_type":"code","source":"# --- Final Dataset Configuration ---\n# Initializing the final dataset object with factorized labels for consistency\ndataset_final = AnimalCLEF2026(\n root=root,\n transform=None,\n load_label=True,\n factorize_label=True,\n check_files=False\n)\n\n# --- Test Split Extraction ---\n# Isolating only the 'test' partition from the metadata\ndataset_test = dataset_final.get_subset(dataset_final.df['split'] == 'test')\n\n# --- Source-Based Sub-Partitioning ---\n# Stratifying the test set into individual datasets for granular evaluation\ndatasets = {}\nunique_sources = dataset_test.metadata['dataset'].unique()\n\nfor name in unique_sources:\n datasets[name] = dataset_test.get_subset(dataset_test.df['dataset'] == name)\n\n# --- Final Status Report ---\nprint(\"-\" * 60)\nprint(\"โ PREPARATION COMPLETE\")\nprint(f\"๐ฆ Unique Test Subsets Generated: {list(datasets.keys())}\")\nprint(f\"๐ Total Test Images Found: {len(dataset_test)}\")\nprint(\"-\" * 60)\n","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2026-03-11T12:34:59.847029Z","iopub.execute_input":"2026-03-11T12:34:59.847323Z","iopub.status.idle":"2026-03-11T12:34:59.904216Z","shell.execute_reply.started":"2026-03-11T12:34:59.847301Z","shell.execute_reply":"2026-03-11T12:34:59.903650Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"# Deep Feature Extraction & Similarity Computation\n\nThis core stage involves deploying specialized **MegaDescriptor** models tailored to each species. We transform raw imagery into high-dimensional embeddings and compute the **Cosine Similarity** matrices that will serve as the foundation for our Re-ID clustering.\n\n### ๐ Technical Pipeline:\n* **Model Selection:** Dynamic switching between `MegaDescriptor-L` and `MegaDescriptor-T` based on species-specific requirements.\n* **Pre-processing:** Automated image resizing and ImageNet-standard normalization.\n* **Feature Extraction:** Leveraging GPU acceleration to generate deep neural representations.\n* **Similarity Mapping:** Applying `CosineSimilarity` to calculate the pairwise distance between all images in the subset.\n\n---","metadata":{}},{"cell_type":"code","source":"# --- Deep Feature Extraction & Similarity Mapping Pipeline ---\n\n# Configuration for hardware and processing\ndevice = 'cuda' # Using GPU for high-speed tensor computations\nbatch_size = 32 # Processing images in batches to optimize VRAM usage\nsimilarities = {} # Dictionary to store the resulting similarity matrices\n\nfor name, dataset in datasets.items():\n print(f\"๐ Processing Dataset: {name}\")\n\n # 1. Dynamic Model Selection based on species requirements\n # High-resolution models for complex patterns (Salamanders/Turtles)\n if name in ['SalamanderID2025', 'SeaTurtleID2022']:\n model_name = 'hf-hub:BVRA/MegaDescriptor-L-384'\n size = 384\n # Standard resolution for fur and scale patterns (Lynx/Lizards)\n elif name in ['LynxID2025', 'TexasHornedLizards']:\n model_name = \"hf-hub:BVRA/MegaDescriptor-T-224\"\n size = 224\n else:\n model_name = \"hf-hub:BVRA/MegaDescriptor-L-384\"\n size = 384\n\n # 2. Initialize the Neural Network (Backbone)\n # Using timm to create the pre-trained MegaDescriptor model\n model = timm.create_model(model_name, pretrained=True).to(device).eval()\n\n # 3. Setup the Extraction and Matching Engines\n matcher = CosineSimilarity()\n extractor = DeepFeatures(model=model, device=device, batch_size=batch_size)\n\n # 4. Define Image Transformations (Standardization)\n transform = T.Compose([\n T.Resize(size=(size, size)),\n T.ToTensor(),\n T.Normalize(mean=(0.485, 0.456, 0.406), std=(0.229, 0.224, 0.225)),\n ])\n dataset.set_transform(transform)\n\n # 5. Execute Feature Extraction\n # Converts images into numerical embeddings (Feature Vectors)\n features = extractor(dataset)\n\n # 6. Compute Similarity Matrix (The Fix)\n # Comparing every image feature vector against others to find matches\n similarity = matcher(features, features)\n similarities[name] = similarity\n\n # Success Output for the current species\n print(f\"โ Similarity Matrix for {name} is ready. Shape: {similarity.shape}\")\n\nprint('\\n' + '='*50)\nprint('โจ MISSION ACCOMPLISHED: All similarity matrices are generated!')\nprint('='*50)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2026-03-11T12:36:03.035297Z","iopub.execute_input":"2026-03-11T12:36:03.035908Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"# Visual Similarity Audit (Sanity Check)\n\nTo validate our **Feature Extraction** and **Similarity Matrices**, we perform a random audit. This step picks a \"Query Image\" and retrieves its top matches based on the calculated similarity scores. It allows us to manually verify if the model is correctly identifying the same individuals across different frames.\n\n### ๐ Audit Methodology:\n* **Diagonal Masking:** Setting self-similarity to zero to focus only on relationships with other images.\n* **Top-K Retrieval:** Using `np.argsort` to find the most similar candidates.\n* **Visual Benchmarking:** Plotting the query alongside its top 3 matches to observe model confidence and potential false positives.\n\n---","metadata":{}},{"cell_type":"code","source":"def random_audit(species_name, num_samples=3):\n # 1. Retrieve the similarity matrix and dataset for the specific species\n sim_data = similarities[species_name]\n \n # Handle tensor to numpy conversion efficiently\n sim_matrix = sim_data.detach().cpu().numpy().copy() if hasattr(sim_data, 'cpu') else sim_data.copy()\n dataset = datasets[species_name]\n\n # 2. Mask the diagonal (self-similarity) to avoid matching an image with itself\n np.fill_diagonal(sim_matrix, 0)\n\n num_images = sim_matrix.shape[0]\n # Ensure we don't sample more than the available images\n sample_indices = random.sample(range(num_images), min(num_samples, num_images))\n\n # 3. Iterate through sampled query images\n for query_idx in sample_indices:\n # Get the indices of the top 3 highest similarity scores\n top_matches = np.argsort(sim_matrix[query_idx])[-3:][::-1]\n \n fig, axes = plt.subplots(1, 4, figsize=(20, 5))\n \n # Display the Query Image\n img_path_query = os.path.join(root, dataset.metadata.iloc[query_idx]['path'])\n axes[0].imshow(Image.open(img_path_query))\n axes[0].set_title(f'Query (Idx {query_idx})', color='#000000', fontweight='bold')\n axes[0].axis('off')\n\n # Display Top 3 Matches\n for i, match_idx in enumerate(top_matches):\n score = sim_matrix[query_idx][match_idx]\n img_path_match = os.path.join(root, dataset.metadata.iloc[match_idx]['path']) \n \n axes[i+1].imshow(Image.open(img_path_match))\n \n # Color logic: Black for strong matches, Gray for weak/uncertain matches\n match_color = '#000000' if score > 0.5 else '#7f8c8d'\n axes[i+1].set_title(f'Match {i+1}\\nScore: {score:.3f}', color=match_color)\n axes[i+1].axis('off')\n\n plt.suptitle(f'Visual Audit Report: {species_name}', fontsize=16, color='#000000', fontweight='bold')\n plt.tight_layout()\n plt.show()\n\n# Execute the audit for Lynx dataset\nrandom_audit('LynxID2025', num_samples=3)","metadata":{"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"# Confidence Analysis & Similarity Distribution\n\nBefore proceeding to identity clustering, it is essential to analyze the statistical distribution of our similarity scores. This histogram visualizes the \"Confidence\" of the model by tracking the highest similarity match for every image in the dataset.\n\n### ๐ Key Insights:\n* **Max Similarity per Image:** We calculate the strongest match for each individual frame (excluding itself).\n* **Confidence Threshold:** A vertical reference line at **0.5** helps distinguish between high-confidence matches (potential Re-ID) and low-confidence outliers.\n* **Density Patterns:** Clusters on the right side of the graph indicate a high presence of recurring individuals, while peaks on the left suggest unique individuals or challenging captures.\n\n---","metadata":{}},{"cell_type":"code","source":"def plot_confidence_audit(species_name):\n # 1. Retrieve and prepare similarity matrix\n sim_data = similarities[species_name]\n # Ensure data is converted to a NumPy array for visualization\n sim_matrix = sim_data.detach().cpu().numpy() if hasattr(sim_data, 'cpu') else sim_data\n\n # 2. Mask self-similarity (Diagonal) to find the best alternative match\n temp_matrix = sim_matrix.copy()\n np.fill_diagonal(temp_matrix, 0)\n \n # Extract the maximum similarity score for each image\n max_similarities = np.max(temp_matrix, axis=1)\n\n # 3. Visualization Setup (Minimalist Professional Style)\n plt.figure(figsize=(12, 5))\n \n # Plotting the histogram with a sleek gray/black palette\n plt.hist(max_similarities, bins=30, color='#34495e', alpha=0.8, edgecolor='#ffffff')\n \n # Adding a threshold line to guide the clustering decision\n plt.axvline(x=0.5, color='#c0392b', linestyle='--', linewidth=2, label='Decision Threshold (0.5)')\n \n # Aesthetic enhancements\n plt.title(f'Confidence Distribution: {species_name}', fontsize=14, fontweight='bold', color='#000000')\n plt.xlabel('Maximum Similarity Score (Confidence)', fontsize=12)\n plt.ylabel('Frequency (Count of Images)', fontsize=12)\n \n plt.grid(axis='y', linestyle=':', alpha=0.5)\n plt.legend(frameon=False)\n \n sns.despine() # Remove axis spines for a cleaner look\n plt.show()\n\n# Execute confidence audit for LynxID2025\nplot_confidence_audit('LynxID2025')","metadata":{"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"# Quantitative Performance Summary\n\nThis function provides a statistical breakdown of the model's performance for a specific species. By applying a **Decision Threshold**, we categorize the images into \"Identified\" (matched with others) and \"Unique/Uncertain\" (lonely images).\n\n### ๐ Metrics Explained:\n* **High Confidence Matches:** Images that found at least one partner with a similarity score above the threshold.\n* **Lonely Images:** Images that remain isolated, either representing a new individual or a capture with insufficient features for matching.\n* **Match Rate (%):** The overall percentage of the dataset that the model successfully linked, indicating the density of re-identifiable individuals.\n\n---","metadata":{}},{"cell_type":"code","source":"def species_summary(species_name, threshold=0.5):\n \"\"\"Generates a technical summary of matching performance for a given species.\"\"\"\n \n # 1. Fetch similarity data and convert to NumPy\n sim_data = similarities[species_name]\n sim_matrix = sim_data.detach().cpu().numpy() if hasattr(sim_data, 'cpu') else sim_data\n\n # 2. Process matrix to find the best match per image (excluding self)\n temp_matrix = sim_matrix.copy()\n np.fill_diagonal(temp_matrix, 0)\n max_sims = np.max(temp_matrix, axis=1)\n\n # 3. Calculate Key Performance Indicators (KPIs)\n total_images = len(max_sims)\n found_match = np.sum(max_sims >= threshold)\n lonely_images = np.sum(max_sims < threshold) \n match_percentage = (found_match / total_images) * 100\n\n # 4. Professional Terminal Output (Clean Black & White Style)\n print(\"-\" * 45)\n print(f\"๐ SUMMARY REPORT: {species_name.upper()}\")\n print(\"-\" * 45)\n print(f\"โ Linked Individuals (>= {threshold}): {found_match}\")\n print(f\"โ Lonely/Unique Images (< {threshold}): {lonely_images}\")\n print(f\"๐ Total Match Rate: {match_percentage:.1f}%\")\n print(\"-\" * 45)\n\n# Execute the summary for LynxID2025\nspecies_summary('LynxID2025', threshold=0.5)","metadata":{"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"# Identity Clustering via DBSCAN\n\nThis function transforms similarity scores into concrete individual identities. By converting **Cosine Similarity** into **Distance Metrics**, we leverage the **DBSCAN** (Density-Based Spatial Clustering of Applications with Noise) algorithm to group images belonging to the same individual.\n\n### ๐ Clustering Strategy:\n* **Distance Transformation:** Converting similarity ($S$) to distance ($D$) using the formula $D = 1 - S$.\n* **Precomputed Metric:** We feed the entire distance matrix directly into the model for maximum efficiency.\n* **Epsilon ($\\epsilon$):** This parameter acts as the \"Radius of Identity.\" A smaller $\\epsilon$ creates more unique individuals, while a larger $\\epsilon$ merges similar captures.\n* **Min Samples = 1:** Ensures that even a single sighting of a rare individual is assigned a unique ID (no data is discarded as noise).\n\n---","metadata":{}},{"cell_type":"code","source":"def get_final_labels(species_name, eps_value):\n \"\"\"\n Groups images into unique biological identities based on a distance threshold.\n \n Args:\n species_name (str): Key for the similarity matrix in the 'similarities' dict.\n eps_value (float): The maximum distance between two samples for one to be \n considered as in the neighborhood of the other.\n \"\"\"\n # 1. Access the specific similarity matrix\n sim_data = similarities[species_name]\n sim_matrix = sim_data.detach().cpu().numpy() if hasattr(sim_data, 'cpu') else sim_data\n\n # 2. Convert Similarity Matrix to Distance Matrix\n # We clip values between 0 and 1 to prevent numerical instability\n dist_matrix = 1 - np.clip(sim_matrix, 0, 1)\n\n # 3. Initialize and run DBSCAN\n # 'precomputed' metric is used because we've already calculated the distances\n model = DBSCAN(eps=eps_value, metric='precomputed', min_samples=1)\n labels = model.fit_predict(dist_matrix)\n\n return labels\n\n# Example Usage:\n#labels = get_final_labels('LynxID2025', eps_value=0.4)","metadata":{"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"# Final Submission Synthesis & Identity Labeling\n\nIn this culminating phase, we apply the tuned **Epsilon ($\\epsilon$)** values for each species to generate precise individual identities. Each image is assigned a globally unique string identifier, combining the species name with the discovered cluster index.\n\n### ๐ Finalization Logic:\n* **Configurable Epsilon:** Utilizing a specialized `final_configs` dictionary to apply the optimal distance threshold for each biological group.\n* **Identity Formatting:** Creating descriptive labels (e.g., `Lynx_ind_0`) to ensure clear separation between individuals across different datasets.\n* **Metadata Mapping:** Re-aligning the predicted clusters with the original `image_id` for submission compliance.\n\n---","metadata":{}},{"cell_type":"code","source":"# --- Final Clustering Configuration ---\n# Species-specific epsilon values tuned via distribution analysis\nfinal_configs = {\n 'LynxID2025': 0.35,\n 'SalamanderID2025': 0.25,\n 'SeaTurtleID2022': 0.40,\n 'TexasHornedLizards': 0.30\n}\n\nall_submissions = []\n\n# --- Synthesis Loop ---\nfor name in similarities.keys():\n print(\"-\" * 50)\n print(f\"๐ฆ Packaging Results: {name}\")\n\n # 1. Retrieve the optimal epsilon (defaulting to 0.3 if not specified)\n eps = final_configs.get(name, 0.3)\n \n # 2. Execute DBSCAN clustering to obtain identity labels\n cluster_labels = get_final_labels(name, eps)\n\n # 3. Construct a temporary DataFrame for the current species\n # We prefix labels with the species name to avoid ID collisions\n species_df = pd.DataFrame({\n 'image_id': datasets[name].metadata['image_id'],\n 'cluster': [f\"{name}_ind_{c}\" for c in cluster_labels]\n })\n \n all_submissions.append(species_df)\n print(f\"โ Successfully processed {len(species_df)} records for {name}.\")\n\nprint(\"\\n\" + \"=\"*50)\nprint(\"โจ All species sub-results have been synthesized.\")\nprint(\"=\"*50)","metadata":{"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"# Final Export: Generating Submission File\n\nThis is the final stage of our pipeline. We aggregate all individual species predictions into a single, unified structure and export the data to a CSV file compliant with the competition's submission format.\n\n### ๐ Final Operations:\n* **DataFrame Concatenation:** Merging all species-specific results into one comprehensive global DataFrame.\n* **CSV Serialization:** Writing the final results to `submission.csv` without row indices.\n* **Integrity Check:** Validating the final record count to ensure no data was lost during the process.\n\n---","metadata":{}},{"cell_type":"code","source":"# --- Global Aggregation ---\n# Combining all individual species DataFrames into one final master file\nfull_submission = pd.concat(all_submissions)\n\n# --- CSV Export ---\n# Saving the results to 'submission.csv' \n# index=False is crucial to avoid extra columns in the final file\nfull_submission.to_csv('submission.csv', index=False)\n\n# --- Final Deployment Report ---\nprint(\"-\" * 60)\nprint(f\"๐ SUCCESS: Mission Accomplished!\")\nprint(f\"๐ File 'submission.csv' is now generated and ready for upload.\")\nprint(f\"๐ Total processed records: {len(full_submission)}\")\nprint(\"-\" * 60)\n\n# Previewing the first few rows of the final submission\ndisplay(full_submission.head())","metadata":{"trusted":true},"outputs":[],"execution_count":null}]}