'.\n\n Parameters\n ----------\n species : dataset species key (e.g. 'SeaTurtleID2022')\n raw_label : str (known identity) or int (discovered cluster)\n identity_encoder : dict that maps known identity strings ā int\n\n Returns\n -------\n formatted label string\n \"\"\"\n prefix = SPECIES_CONFIG[species] # e.g. 'cluster_SeaTurtleID2022'\n\n if isinstance(raw_label, str):\n # Known train identity ā encode to a consistent integer\n cluster_id = identity_encoder[raw_label]\n else:\n # Discovered cluster ā raw_label is already a unique integer\n cluster_id = int(raw_label)\n\n return f\"{prefix}_{cluster_id}\"\n\n\n# āā Build per-species identity encoder āāāāāāāāāāāāāāāāāāāā\n# Maps each known train identity string ā small unique int\n# Starting at 1 to keep things readable.\nidentity_encoders: dict[str, dict] = {}\nfor species, dfs in SPECIES_DFS.items():\n if dfs[\"train\"] is not None:\n unique_ids = sorted(dfs[\"train\"][\"identity\"].unique())\n identity_encoders[species] = {uid: (i + 1) for i, uid in enumerate(unique_ids)}\n else:\n identity_encoders[species] = {}\n\n\n# āā Convert all predictions ā formatted strings āāāāāāāāāāāā\nrows = [] # list of {\"image_id\": ..., \"cluster\": ...}\nfor species, preds in all_predictions.items():\n enc = identity_encoders[species]\n for img_id, raw_label in preds.items():\n cluster_str = format_cluster_label(species, raw_label, enc)\n rows.append({\"image_id\": img_id, \"cluster\": cluster_str})\n\nsubmission_raw = pd.DataFrame(rows)\nprint(f\"Raw submission shape : {submission_raw.shape}\")\nprint(submission_raw.head(10).to_string(index=False))\n\n# āā Align to sample_submission.csv order (REQUIRED) āāāāāāā\nsample_sub = pd.read_csv(SAMPLE_SUB)\nprint(f\"\\nSample submission rows: {len(sample_sub)}\")\n\n# Left-join to preserve sample_submission row order exactly\nfinal_submission = sample_sub[[\"image_id\"]].merge(\n submission_raw, on=\"image_id\", how=\"left\"\n)\n\n# Safety: if any image_id is still missing a prediction, fill with a\n# unique singleton cluster so ARI is not hurt by NaN\nnull_mask = final_submission[\"cluster\"].isna()\nif null_mask.sum() > 0:\n print(f\"\\nā ļø {null_mask.sum()} image(s) missing predictions ā assigning fallback singletons.\")\n base_id = global_cluster_counter\n fallback_ids = range(base_id, base_id + null_mask.sum())\n final_submission.loc[null_mask, \"cluster\"] = [\n f\"cluster_fallback_{fid}\" for fid in fallback_ids\n ]\n\nassert final_submission[\"cluster\"].notna().all(), \"ā Missing cluster assignments!\"\nassert len(final_submission) == len(sample_sub), \"ā Row count mismatch!\"\n\n# āā Save āāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāā\nout_path = OUTPUT_DIR / \"submission.csv\"\nfinal_submission.to_csv(out_path, index=False)\nprint(f\"\\nā
Submission saved ā {out_path}\")\nprint(f\" Rows : {len(final_submission):,}\")\nprint(f\" Unique clusters : {final_submission['cluster'].nunique():,}\")\nprint(\"\\nCluster prefix distribution:\")\nprefix_counts = final_submission[\"cluster\"].str.extract(r\"(cluster_[^_]+_[^_]+)\")[0].str.rsplit(\"_\", n=1).str[0].value_counts()\nprint(prefix_counts.to_string())","metadata":{"trusted":true},"outputs":[],"execution_count":null},{"id":"cell-validate","cell_type":"code","source":"# āā Quick validation: spot-check label format āāāāāāāāāāāāāā\nimport re\n\nVALID_PATTERNS = [\n re.compile(r\"^cluster_LynxID2025_\\d+$\"),\n re.compile(r\"^cluster_SalamanderID2025_\\d+$\"),\n re.compile(r\"^cluster_SeaTurtleID2022_\\d+$\"),\n re.compile(r\"^cluster_TexasHornedLizards_\\d+$\"),\n re.compile(r\"^cluster_fallback_\\d+$\"), # safety fallback pattern\n]\n\ndef is_valid_label(label: str) -> bool:\n return any(p.match(label) for p in VALID_PATTERNS)\n\ninvalid_rows = final_submission[~final_submission[\"cluster\"].apply(is_valid_label)]\nif len(invalid_rows) == 0:\n print(\"ā
All cluster labels pass format validation.\")\nelse:\n print(f\"ā {len(invalid_rows)} rows have invalid cluster label formats:\")\n print(invalid_rows.head(20))\n\n# āā Preview āāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāā\nprint(\"\\nš Final submission preview:\")\nprint(final_submission.head(20).to_string(index=False))","metadata":{"trusted":true},"outputs":[],"execution_count":null},{"id":"cell-viz-distribution","cell_type":"code","source":"# āā Visualise cluster size distribution āāāāāāāāāāāāāāāāāāā\nfinal_submission[\"species_prefix\"] = final_submission[\"cluster\"].str.extract(\n r\"(cluster_\\w+)_\\d+$\"\n)[0]\n\ncluster_sizes = (\n final_submission.groupby([\"species_prefix\", \"cluster\"])\n .size()\n .reset_index(name=\"size\")\n)\n\nfig3 = px.histogram(\n cluster_sizes,\n x=\"size\",\n color=\"species_prefix\",\n nbins=40,\n barmode=\"overlay\",\n opacity=0.75,\n title=\"Cluster Size Distribution (images per predicted individual)\",\n labels={\"size\": \"Images per Cluster\", \"species_prefix\": \"Species\"},\n color_discrete_sequence=SPECIES_COLORS,\n template=\"plotly_dark\",\n height=420,\n)\nfig3.update_layout(\n paper_bgcolor=PALETTE[\"bg\"],\n plot_bgcolor=PALETTE[\"bg\"],\n xaxis=dict(gridcolor=PALETTE[\"grid\"]),\n yaxis=dict(gridcolor=PALETTE[\"grid\"]),\n)\nfig3.show()\n\nprint(\"\\nCluster size statistics:\")\nprint(cluster_sizes[\"size\"].describe().round(2).to_string())","metadata":{"trusted":true},"outputs":[],"execution_count":null},{"id":"cell-md-conclusion","cell_type":"markdown","source":"---\n## 6. Conclusion & Next Steps \n\n\n\n### What We Built\n\n| Stage | Method | Purpose |\n|---|---|---|\n| Feature Extraction | **MegaDescriptor-L-384** (ViT-L, 384px) | State-of-the-art wildlife embeddings |\n| Step A ā Lookup | **Cosine sim ā„ 0.85** ā known identity | Match to train individuals |\n| Step B ā Discovery | **DBSCAN** (cosine dist, eps=0.30) | Cluster unseen individuals |\n| Noise Handling | **Singleton assignment** for DBSCAN noise | Critical ARI fix |\n| TexasHornedLizards | **100% DBSCAN** (no train data) | Pure discovery |\n\n### š Next Steps \n\n1. **Fine-tune MegaDescriptor on WildlifeReID-10k** using **ArcFace loss** (margin = 0.5, scale = 64). The 140k images across 10k identities will dramatically close the domain gap and yield tighter embedding clusters, improving both lookup precision and discovery quality.\n\n2. **Threshold calibration**: Use a small held-out validation split to sweep `LOOKUP_THRESHOLD` ā [0.75, 0.95] and find the value that maximises ARI ā the optimal threshold varies between species.\n\n3. **HDBSCAN** (hierarchical DBSCAN) instead of DBSCAN for the discovery step: HDBSCAN is more robust to varying density and typically produces fewer noise points ā reducing the number of singletons and thus reducing ARI penalty from over-clustering.\n\n4. **Test-Time Augmentation (TTA)**: Average embeddings across 4ā8 augmented views of each image (horizontal flip, colour jitter, random crop) to obtain more robust feature representations.\n\n5. **Re-ranking with DBA (Database-side Augmentation)**: Before the lookup step, augment train embeddings by averaging each identity's embeddings with its k-nearest neighbours to produce a smoother, more generalizable reference prototype per individual.\n\n6. **Ensemble**: Combine **MegaDescriptor-L-384** with **miewid-msv3** embeddings (concatenate + re-normalise) for a stronger retrieval signal.\n\n
\n\n","metadata":{}},{"id":"4c83210e-8a46-4399-bdf2-f127b13296cd","cell_type":"code","source":"","metadata":{"trusted":true},"outputs":[],"execution_count":null}]}