{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"name":"python","version":"3.11.13","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"},"kaggle":{"accelerator":"none","dataSources":[{"sourceId":116062,"databundleVersionId":14084779,"sourceType":"competition"}],"dockerImageVersionId":31192,"isInternetEnabled":true,"language":"python","sourceType":"notebook","isGpuEnabled":false}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"code","source":"!pip install mermaidian\n\n\n# This Python 3 environment comes with many helpful analytics libraries installed\n# It is defined by the kaggle/python Docker image: https://github.com/kaggle/docker-python\n# For example, here's several helpful packages to load\n\nimport numpy as np # linear algebra\nimport pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\n\n\n# Input data files are available in the read-only \"../input/\" directory\n# For example, running this (by clicking run or pressing Shift+Enter) will list all files under the input directory\n\nimport os\nfor dirname, _, filenames in os.walk('/kaggle/input'):\n    for filename in filenames:\n        print(os.path.join(dirname, filename))\n\n# You can write up to 20GB to the current directory (/kaggle/working/) that gets preserved as output when you create a version using \"Save & Run All\" \n# You can also write temporary files to /kaggle/temp/, but they won't be saved outside of the current session\n\n#hide_input\n#hide_output\n","metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true,"execution":{"iopub.status.busy":"2025-11-09T21:55:18.111370Z","iopub.execute_input":"2025-11-09T21:55:18.111750Z","iopub.status.idle":"2025-11-09T21:55:22.268290Z","shell.execute_reply.started":"2025-11-09T21:55:18.111720Z","shell.execute_reply":"2025-11-09T21:55:22.266934Z"},"_kg_hide-input":true,"_kg_hide-output":true},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## 🧬 **FANTASIA – CAFA-6 Functional Annotation Framework**\n\n---\n\n### **1. Introduction**\n\nThis notebook documents the system developed for **CAFA-6**, integrating two complementary components designed to perform large-scale, temporally consistent protein function prediction:\n\n* **[Protein Information System (PIS)](https://github.com/CBBIO/protein-information-system):**\n  A structured and versioned protein information framework capable of maintaining multiple **temporal instances** containing proteins, Gene Ontology (GO) annotations, structural data, and embeddings.\n  The system can ingest data from the **UniProt API** or directly from **Gene Annotation Files (GAFs)**, preserving all associated metadata and evidence codes.\n\n* **[FANTASIA](https://github.com/CBBIO/FANTASIA):**\n  *Functional ANnoTAtion based on embedding space SImilArity*, a configurable inference engine that predicts protein function by measuring similarity in embedding space.\n  FANTASIA supports **taxonomy-based** and **redundancy-based** filtering strategies and integrates seamlessly with PIS for reproducible analysis.\n\nAdditional exploratory work was carried out using the Kaggle notebook\n**[CAFA5 Private Test Set Discovery](https://www.kaggle.com/code/xaxipirulazo/cafa5-private-test-set-discovery)**,\nwhere Gene Annotation Files from **releases 214 to 226** — spanning nearly **six years of biological knowledge evolution** — were processed to define temporal validation environments.\n\n---\n\n### **2. Current System State**\n\nAfter parsing and filtering, only **experimentally supported annotations** (as defined by the CAFA committee’s holdout protocol) were retained.\nThe system now contains:\n\n| Metric             |                                                            Value |\n| ------------------ | ---------------------------------------------------------------: |\n| Total proteins     |                                                      **224 309** |\n| Annotated proteins |                                                      **216 275** |\n| Total annotations  | **1 711 334** (including IBA + IEA; both can be disabled in PIS) |\n\n---\n\n### **3. Embedding Infrastructure**\n\nEmbeddings have been generated and stored using **ESM-3c** and **Ankh-3**, applying **multi-layer mean pooling** across model layers.\nThese representations are linked to their corresponding proteins in the PIS graph, enabling **embedding-space inference** by transferring GO annotations from nearest neighbors.\n\nEach inference result includes not only predicted GO terms but also **rich metadata** about the source proteins and alignment metrics:\n\n| Category                | Key Attributes                                                                      |\n| ----------------------- | ----------------------------------------------------------------------------------- |\n| **Embedding metadata**  | `accession`, `model_name`, `embedding_type_id`, `layer_index`                       |\n| **Functional metrics**  | `distance`, `go_id`, `category`, `evidence_code`                                    |\n| **Protein context**     | `protein_id`, `organism`, `gene_name`                                               |\n| **Alignment scores**    | `reliability_index`, `identity`, `similarity`, `alignment_score`, `gaps_percentage` |\n| **Distance parameters** | `distance_metric`, `distance_threshold`                                             |\n\n---\n\n### **4. Embedding Generation Configuration**\n\nThe embedding generation pipeline defines how protein sequences are encoded using large protein language models (PLMs).\nThis configuration controls **truncation**, **batch size**, **device allocation**, and **layer selection**, which together determine computational performance and representation quality.\n\n#### **Configuration Overview**\n\n```yaml\nembedding:\n  device: cuda                 # enum{cuda,cpu} | Primary device for PLMs; set to cpu to force CPU execution\n  queue_batch_size: 100        # int | Number of sequences per published batch to RabbitMQ\n  max_sequence_length: 1516    # int | 0 disables truncation; otherwise sequences are truncated to this length\n\n  # Per-model configuration.\n  # name: logical model identifier used across the pipeline (case-sensitive).\n  # enabled: if false, model is ignored at runtime.\n  # batch_size: PLM forward batch size (beware of VRAM limits).\n  # layer_index: list[int] of layers to export; multiple indices will produce several representations (multi-layer mean pooling).\n  # LAYER INDEXING NOTE: 0 = last (output) layer, 1 = penultimate, 2 = second-to-last, and so on.\n\n  models:\n    ESM: # 34 layers: 0..33\n      enabled: False\n      batch_size: 1\n      layer_index: [ 0 ]\n\n    ESM3c: # 36 layers: 0..35\n      enabled: True\n      batch_size: 1\n      layer_index: [ 0 ]\n      distance_threshold: 0\n\n    Ankh3-Large: # 49 layers: 0..48\n      enabled: False\n      batch_size: 1\n      layer_index: [ 0 ]\n      distance_threshold: 0\n\n    Prot-T5: # 25 layers: 0..24\n      enabled: False\n      batch_size: 1\n      layer_index: [ 0 ]\n      distance_threshold: 0\n\n    Prost-T5: # 25 layers: 0..24 (same backbone as Prot-T5)\n      enabled: False\n      batch_size: 1\n      layer_index: [ 0, 1, 2, 11, 12, 13 ]\n      distance_threshold: 0\n```\n\n#### **Technical Notes**\n\n* **Device selection (`device`)** – Configures the main execution device. CUDA is preferred when available for PLM inference; CPU fallback is supported for debugging or low-resource environments.\n* **Truncation (`max_sequence_length`)** – Sequences exceeding this length are truncated to avoid GPU memory overflow; `0` disables truncation entirely.\n* **Batch parameters (`batch_size`, `queue_batch_size`)** – Control throughput and memory footprint. Small batches (1–4) ensure stable performance within limited VRAM; the queue batch size governs the number of sequences sent per processing batch.\n* **Layer indexing (`layer_index`)** – Allows selective extraction of representations from deeper or intermediate PLM layers. Combining multiple indices yields averaged (“mean-pooled”) embeddings that capture both structural and functional information.\n* **Model selection** – Only models marked as `enabled: True` are loaded. In this configuration, **ESM3c** is the active model with its final layer exported.\n\n---\n\n### **5. LookUp Configuration**\n\nThe lookup stage consumes precomputed embeddings (from Stage A) and reference tables loaded in memory from the Protein Information System (PIS).\nIt performs pairwise similarity searches, redundancy and taxonomy filtering, and exports per-query results in CSV/TSV format.\n\n```yaml\n# ==============================================================================\n# Stage B — Lookup\n# Consumes embeddings.h5 + in-memory references (IDs, vectors, GO) to produce CSV/TSV.\n# ==============================================================================\nlookup:\n  use_gpu: True               # bool | If true, run vector distances on GPU when available\n  batch_size: 216             # int  | Vector distance batch size (tune to GPU memory)\n  distance_metric: cosine     # enum{cosine,euclidean} | Distance for nearest-neighbor search\n  limit_per_entry: 3          # int  | k neighbors returned per query (a.k.a. “k”)\n  lookup_cache_max: 4         # int  | Max entries per (model,layer) in-memory cache (tune to RAM)\n  topgo: true                 # bool | If true, emit TopGO-compatible TSV alongside CSV outputs\n\n  precision: 4                # int | Número de decimales a usar en la exportación de resultados.\n\n  # Redundancy filtering (optional pre-filter on reference side, e.g., MMseqs2).\n  redundancy:\n    identity: 0               # float in [0,1] | 0 disables; 1.0 = 100% identity (strict deduplication)\n    coverage: 0.7             # float in (0,1]  | Alignment coverage threshold used in deduplication\n    threads: 10               # int  | CPU threads for redundancy filtering tools\n\n  # Taxonomy filters (applied after NN retrieval to prune/keep specific taxa).\n  taxonomy:\n    exclude: [ ]              # list[str] | Taxonomy IDs to exclude (e.g., [\"559292\",\"6239\"])\n    include_only: [ ]         # list[str] | If non-empty, restrict results to these IDs (takes precedence)\n    get_descendants: false    # bool | If true, expand filters to include descendants\n```\n\n#### **Key Parameters**\n\n| Parameter          | Description                                                                            |\n| ------------------ | -------------------------------------------------------------------------------------- |\n| `use_gpu`          | Enables GPU-based nearest neighbor computation for accelerated lookup.                 |\n| `batch_size`       | Controls memory usage and parallelism during distance calculations.                    |\n| `distance_metric`  | Defines similarity function (`cosine` or `euclidean`).                                 |\n| `limit_per_entry`  | Number of top hits per protein (k).                                                    |\n| `lookup_cache_max` | Controls RAM caching per model/layer.                                                  |\n| `topgo`            | Generates additional `.topgo` files compatible with downstream GO enrichment analysis. |\n| `redundancy`       | Optional prefiltering step to remove redundant reference proteins.                     |\n| `taxonomy`         | Applies organism-level filters after similarity retrieval.                             |\n| `precision`        | Number of decimals in exported numeric results.                                        |\n\n---\n\n### **5. Post-Processing Configuration**\n\nThe aggregation of prediction attributes is governed by a weighted scoring schema defined as follows:\n\n```yaml\npostprocess:\n  keep_sequences: true\n  summary:\n    normalize_count_by_limit_per_entry: true\n    export_raw_count: true\n    metrics:\n      reliability_index: [max]\n      identity: [max]\n      identity_sw: [max]\n    aliases:\n      reliability_index: ri\n      identity: id_g\n      identity_sw: id_l\n    weights:\n      reliability_index: { max: 0.3 }\n      mean_id_g: 0.35\n      mean_id_l: 0.35\n      count: 0\n    weighted_prefix: \"w_\"\n```\n\nThe **post-processing stage** integrates results across multiple **layers and models** by applying a *weighted aggregation schema*.\nFor each query, FANTASIA combines the **maximum reliability index** and **mean global/local identities** from all active models and exported layers.\nScores are normalized per entry, aliased for clarity (`ri`, `id_g`, `id_l`), and merged using the specified weights\n*(0.3 × RI + 0.35 × ID₍g₎ + 0.35 × ID₍l₎)* to produce a unified prediction output (`w_*`).\n\n---\n\n### **6. Research framework schema**\n\n","metadata":{"_kg_hide-input":false}},{"cell_type":"code","source":"import mermaidian as mm\n\n# Configuración base (idéntica al ejemplo oficial)\nout_path = '/kaggle/working/'\n\nconfig0 = {'fontSize': '24px'}\noptions0 = {'bgColor': '#ffffff', 'width': '900'}\npad_data0 = {'pad_top': 80, 'pad_bottom': 40, 'border_thickness': 6, 'border_color': \"#aaaaaa\", 'pad_color': '#ffffff'}\ntitle_data0 = {'position': 'tc', 'font_scale': 1.0, 'font_thickness': 1, 'font_color': \"#000000\", 'font_name': 'duplex'}\n\ndef show_mermaid(diagram_code, file_name, title, theme='forest', config=config0, options=options0, pad_data=pad_data0, title_data=title_data0):\n    options['bgColor'] = options['bgColor'].replace('#', '')\n    diagram = mm.get_mermaid_diagram('svg', diagram_code, theme, config, options)\n    title_data_svg = {\n        'title': title,\n        'position': 'tc',\n        'font_name': 'Arial, sans-serif',\n        'font_size': 24,\n        'font_color': '#000000',\n        'font_bg_color': '',\n        'font_weight': 'bold'\n    }\n    diagramPBT = mm.add_paddings_border_and_title_to_svg(diagram, pad_data, title_data_svg)\n    mm.show_svg_ipython_centered(diagramPBT)\n    mm.save_diagram_as_svg(f'{out_path}/{file_name}.svg', diagramPBT)\n\n\nresearch_framework_code = \"\"\"\nflowchart LR\n    subgraph PIS[\"Protein Information System (PIS)\"]\n        B1[\"T₀ – CAFA5 baseline:<br>proteins, annotations, structures, embeddings\"]\n        B2[\"T₁ – CAFA6 baseline:<br>updated annotations and embeddings\"]\n    end\n\n    subgraph FANTASIA_T0[\"FANTASIA (T₀ instance)\"]\n        C1[\"Compute embedding distances<br>(cosine, L2)\"]\n        C2[\"Transfer GO terms from nearest neighbors\"]\n        C3[\"Weighted post-processing<br>(RI·0.3 + ID₍g₎·0.35 + ID₍l₎·0.35)\"]\n    end\n\n    subgraph FANTASIA_T1[\"FANTASIA (T₁ instance)\"]\n        C4[\"Compute embedding distances<br>(cosine, L2)\"]\n        C5[\"Transfer GO terms from nearest neighbors\"]\n        C6[\"Weighted post-processing<br>(RI·0.3 + ID₍g₎·0.35 + ID₍l₎·0.35)\"]\n    end\n\n    subgraph PIPELINE[\"Validation & Submission Pipeline\"]\n        D1[\"Holdout generation between T₀ → T₁\"]\n        D2[\"Internal validation on multiple temporal holdouts\"]\n        D3[\"Model consolidation\"]\n        D4[\"CAFA6 final submission\"]\n    end\n\n    A1[\"UniProt & GOA GAF archives (2014–2026)\"] --> A2[\"Parsing & filtering by evidence codes<br>(EXP, IDA, IMP, IPI, IBA, IEA)\"]\n    A2 --> B1 & B2\n    B1 --> C1 --> C2 --> C3 --> D1 --> D2 --> D3\n    B2 --> C4 --> C5 --> C6 --> D4\n\n    classDef data fill:#202830,stroke:#89b4fa,stroke-width:2px,color:#f8f9fa\n    classDef pis fill:#1e1e2e,stroke:#f38ba8,stroke-width:2px,color:#f8f9fa\n    classDef fantasia fill:#282a36,stroke:#94e2d5,stroke-width:2px,color:#f8f9fa\n    classDef pipe fill:#2b303b,stroke:#a6e3a1,stroke-width:2px,color:#f8f9fa\n    class A1,A2 data\n    class B1,B2 pis\n    class C1,C2,C3,C4,C5,C6 fantasia\n    class D1,D2,D3,D4 pipe\n\"\"\"\n\ntheme = {'primaryColor': '#1e1e2e',\n         'primaryTextColor': '#ffffff',\n         'secondaryColor': '#11111b',\n         'tertiaryColor': '#4c4f69',\n         'lineColor': '#aaaaaa',\n         'fontSize': '20px'}\n\nshow_mermaid(\n    research_framework_code,\n    file_name='fantasia_cafa6_schema',\n    title='FANTASIA–CAFA6 Research Framework',\n    theme=theme\n)\n\n","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-11-09T21:51:39.314782Z","iopub.execute_input":"2025-11-09T21:51:39.315394Z","iopub.status.idle":"2025-11-09T21:51:40.701137Z","shell.execute_reply.started":"2025-11-09T21:51:39.315357Z","shell.execute_reply":"2025-11-09T21:51:40.700348Z"},"_kg_hide-input":true},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"---\n\n### **7. Future Work**\n\n* Implement multiple **temporal holdouts** leveraging the historical sequence of GAF releases to rigorously evaluate model generalization.\n* Establish a benchmark milestone: **surpass the CAFA-5 #1 score** under both *leakage-free* and *standard* conditions.\n* Extend validation beyond FANTASIA to other prediction pipelines and embeddings.\n* Conduct **local hyperparameter optimization**, prioritizing robust validation over leaderboard fitting.\n* Explore integration of **structural data** to enhance annotation precision.\n* Continue investigating **multifunctionality and embedding-based inference reliability** within the PIS–FANTASIA framework.\n\n---\n\n### **8. Immediate Tasks**\n\nThis section summarizes the short-term objectives to consolidate the CAFA-6 system and prepare the official validation and submission pipeline.\n\n#### **1. Repository Creation – GAF Filtering Mechanism**\n\n* **Objective:** Create a dedicated GitHub repository for the **massive GAF filtering pipeline**.\n* **Purpose:** Centralize the code responsible for parsing, cleaning, and filtering historical GAFs (releases 214–226).\n* **Actions:**\n\n  * Publish the preprocessing scripts with version control and clear documentation.\n  * Include configuration templates for evidence-based filtering (`EXP`, `IDA`, `IMP`, `IPI`, `IBA`, `IEA`).\n  * Ensure reproducibility via environment files (`pyproject.toml` or `environment.yml`).\n\n#### **2. FANTASIA Results Dataset – CAFA-6**\n\n* **Objective:** Upload the **FANTASIA inference results** as a Kaggle dataset for CAFA-6.\n* **Actions:**\n\n  * Share FANTASIA results sample.\n  * Compress and upload into a Dataset.\n  * Include metadata and README with some exploratory analysis.\n  * Provide an **example submission notebook** demonstrating leaderboard formatting and metric calculation from raw results.\n\n#### **3. Information System Instance – Pre-Evaluation Snapshot** \n\n* **Objective:** Generate a **PIS instance** representing the biological knowledge *prior to the CAFA-6 evaluation period*.\n* **Actions:**\n\n  * Freeze the temporal database snapshot (proteins, annotations, structures, embeddings).\n  * Tag it as `PIS_T₀_local` for internal validation use.\n  * Store version metadata (GAF source, release IDs, evidence filters).\n\n#### **3b. Information System Instance –  Latest Snapshot** \n\n* **Objective:** Generate a **PIS instance** representing the most updated biological knowledge to predict the current holdout.\n* **Actions:**\n\n  * Freeze the temporal database snapshot (proteins, annotations, structures, embeddings).\n  * Tag it as `PIS_T₀_local` for internal validation use.\n  * Store version metadata (GAF source, release IDs, evidence filters).\n\n#### **4. Automated Evaluation and Hyperparameter Search**\n\n* **Objective:** Implement an **automated evaluation engine** to explore hyperparameters across different holdouts.\n* **Scope:**\n\n  * Automate internal validation loops using subsets of proteins with recent GO updates.\n  * Log metrics such as F-score, precision, recall, and semantic similarity.\n  * Integrate search strategies (grid/random/Bayesian) for model parameter tuning.\n  * Output results to structured reports or dashboards.\n\n#### **5. Controlled Holdout Evaluation**\n\n* **Rationale:**\n  Changes between GAF releases are **modest compared to the test supersets**, so validation should rely on **smaller, well-defined sets**.\n* **Plan:**\n\n  * Focus on reduced and traceable subsets where annotation updates are verified.\n  * Use these controlled holdouts to test new post-processing, weighting schemes, and model comparison consistency.\n\n**Outcome:**\nCompletion of these tasks will finalize the **CAFA-6 validation framework**, enabling fully reproducible experiments, model optimization, and submission deployment within the PIS–FANTASIA ecosystem.\n","metadata":{}},{"cell_type":"code","source":"","metadata":{"trusted":true},"outputs":[],"execution_count":null}]}