Good morning,
\nWe are conducting a research project titled “Hallucination Risks of Large Language Models in Clinical Proteomics”, recently formalized as a prospective evaluation study. The work systematically assesses how frontier LLMs (GPT-4, Claude Sonnet, Gemini Pro) behave when faced with quantitative, domain-specific proteomics data, particularly focusing on hallucination frequency, severity, and risk factors.
\nOur results demonstrate that even state-of-the-art models exhibit 27–35% baseline hallucination rates, escalating beyond 50% for complex or rare-protein queries. These findings underline a critical gap between general medical-AI benchmarks and the reality of high-precision clinical data interpretation.
\nWe are looking for a research collaborator experienced in:
\nThe collaboration aims to extend this framework toward hallucination-detection mechanisms and domain-specific evaluation benchmarks for biomedical AI.
\nIf this area aligns with your background, please feel free to reply here or reach out for discussion.
\nFull codebase, query datasets, and validation framework are openly available:\nhttps://github.com/olaflaitinen/llm-proteomics-hallucination
\nLooking forward to connecting with colleagues working at the intersection of AI reliability, biomedical NLP, and proteomics.
", "messages": [ { "id": 3314475, "postDate": "2025-11-10T07:45:45.487Z", "content": "Good morning,
\nWe are conducting a research project titled “Hallucination Risks of Large Language Models in Clinical Proteomics”, recently formalized as a prospective evaluation study. The work systematically assesses how frontier LLMs (GPT-4, Claude Sonnet, Gemini Pro) behave when faced with quantitative, domain-specific proteomics data, particularly focusing on hallucination frequency, severity, and risk factors.
\nOur results demonstrate that even state-of-the-art models exhibit 27–35% baseline hallucination rates, escalating beyond 50% for complex or rare-protein queries. These findings underline a critical gap between general medical-AI benchmarks and the reality of high-precision clinical data interpretation.
\nWe are looking for a research collaborator experienced in:
\nThe collaboration aims to extend this framework toward hallucination-detection mechanisms and domain-specific evaluation benchmarks for biomedical AI.
\nIf this area aligns with your background, please feel free to reply here or reach out for discussion.
\nFull codebase, query datasets, and validation framework are openly available:\nhttps://github.com/olaflaitinen/llm-proteomics-hallucination
\nLooking forward to connecting with colleagues working at the intersection of AI reliability, biomedical NLP, and proteomics.
", "rawMarkdown": "Good morning,\n\nWe are conducting a research project titled **“Hallucination Risks of Large Language Models in Clinical Proteomics”**, recently formalized as a **prospective evaluation study**. The work systematically assesses how frontier LLMs (GPT-4, Claude Sonnet, Gemini Pro) behave when faced with **quantitative, domain-specific proteomics data**, particularly focusing on **hallucination frequency, severity, and risk factors**.\n\nOur results demonstrate that even state-of-the-art models exhibit **27–35% baseline hallucination rates**, escalating beyond **50% for complex or rare-protein queries**. These findings underline a critical gap between general medical-AI benchmarks and the reality of **high-precision clinical data interpretation**.\n\nWe are looking for a **research collaborator** experienced in:\n\n* **Proteomics or mass-spectrometry data analysis**, and\n* **Large language model evaluation, uncertainty quantification, or AI safety.**\n\nThe collaboration aims to extend this framework toward **hallucination-detection mechanisms and domain-specific evaluation benchmarks** for biomedical AI.\n\nIf this area aligns with your background, please feel free to reply here or reach out for discussion.\n\nFull codebase, query datasets, and validation framework are openly available:\n[https://github.com/olaflaitinen/llm-proteomics-hallucination](https://github.com/olaflaitinen/llm-proteomics-hallucination)\n\nLooking forward to connecting with colleagues working at the intersection of **AI reliability, biomedical NLP, and proteomics.**", "votes": 9 }, { "id": 3319544, "postDate": "2025-11-11T20:47:24.930Z", "content": "I constantly iterate over generate code with chatgpt5, test and debug. Each script of the program I evaluate with my own eyes by picking some examples and evaluate their correctness. I want to finish this task of the challenge in 100 hours progamming and probably a lot more compute time experimenting with different neural network architectures but at the end the results will tell me if chatgpt5 and my method to use chatgpt5 is a fine method. I makes mistakes for sure but I got my AI engineering and biomedical degree before the LLM explosion 😅
", "rawMarkdown": "I constantly iterate over generate code with chatgpt5, test and debug. Each script of the program I evaluate with my own eyes by picking some examples and evaluate their correctness. I want to finish this task of the challenge in 100 hours progamming and probably a lot more compute time experimenting with different neural network architectures but at the end the results will tell me if chatgpt5 and my method to use chatgpt5 is a fine method. I makes mistakes for sure but I got my AI engineering and biomedical degree before the LLM explosion 😅", "votes": -1 }, { "id": 3394761, "postDate": "2026-01-21T16:29:17.397Z", "content": "i'm interested, how can i contact you?
", "rawMarkdown": "i'm interested, how can i contact you? " }, { "id": 3323218, "postDate": "2025-11-14T04:17:33.493Z", "content": "Hello Dr. Imanov,\nThank you for sharing this open-source project. I find that your systematic evaluations of LLM hallucination risks in mass-spectrometry and protein function analysis resonate strongly with several aspects of my background.
\nI am an undergraduate double major in biochemistry and philosophy, pre-law at the Catholic University of America in Washington, DC, United States. I am a freelance AI Evaluation and Alignment Engineer at Alignerr.ai, where I evaluate and validate generative AI outputs to ensure quality, coherence, and ethical alignment with product goals. Also, I implement and maintain API-driven test harnesses, author evaluation protocols, and provide actionable scoring and feedback that guide model updates and product decisions.
\nIn addition, my research as a co-investigator on an NIH-approved protein folding study (PLCO-1699) and prior experience optimizing quantum protein folding models gives me a unique perspective that bridges clinical proteomics and model reliability. My technical toolkit spans Python (PyTorch, TensorFlow, HuggingFace), bioinformatics (ColabFold, AlphaFold3, RDKit), and experiment reproducibility (Docker, AWS), which matches the modular and rigorous approach of your framework.
\nI am particularly interested in extending your hallucination-detection benchmarks with advanced uncertainty quantification and real-time detection methods. My prior work includes ML pipelines for molecular complexity scoring, the development of domain-specific benchmarks, and recursive transformer search architectures for biological data. I do my best to ensure robust statistical reporting, and have presented these solutions at venues such as Johns Hopkins Rising Stars in Cellular, the Catholic University of America Graduate Biology Symposium, and every Catholic University of America Research Day since I began my undergraduate degrees.
\nI would be delighted to contribute collaboratively, both methodologically and in expanding dataset/evaluation coverage. Let me know if you’d like to explore a discussion on integrating novel detection metrics, leveraging quantum-assisted models, or deploying real-world clinical validation (or something else!). I’m excited by the prospect of working at the intersection of proteomics, AI safety, and biomedical model evaluation.
\nGratefully,\nTommaso R. Marena
\nP.S. Here’s my LinkedIn: www.linkedin.com/in/tommaso-marena
", "rawMarkdown": "Hello Dr. Imanov,\nThank you for sharing this open-source project. I find that your systematic evaluations of LLM hallucination risks in mass-spectrometry and protein function analysis resonate strongly with several aspects of my background.\n\nI am an undergraduate double major in biochemistry and philosophy, pre-law at the Catholic University of America in Washington, DC, United States. I am a freelance AI Evaluation and Alignment Engineer at Alignerr.ai, where I evaluate and validate generative AI outputs to ensure quality, coherence, and ethical alignment with product goals. Also, I implement and maintain API-driven test harnesses, author evaluation protocols, and provide actionable scoring and feedback that guide model updates and product decisions.\n\nIn addition, my research as a co-investigator on an NIH-approved protein folding study (PLCO-1699) and prior experience optimizing quantum protein folding models gives me a unique perspective that bridges clinical proteomics and model reliability. My technical toolkit spans Python (PyTorch, TensorFlow, HuggingFace), bioinformatics (ColabFold, AlphaFold3, RDKit), and experiment reproducibility (Docker, AWS), which matches the modular and rigorous approach of your framework.\n\nI am particularly interested in extending your hallucination-detection benchmarks with advanced uncertainty quantification and real-time detection methods. My prior work includes ML pipelines for molecular complexity scoring, the development of domain-specific benchmarks, and recursive transformer search architectures for biological data. I do my best to ensure robust statistical reporting, and have presented these solutions at venues such as Johns Hopkins Rising Stars in Cellular, the Catholic University of America Graduate Biology Symposium, and every Catholic University of America Research Day since I began my undergraduate degrees.\n\nI would be delighted to contribute collaboratively, both methodologically and in expanding dataset/evaluation coverage. Let me know if you’d like to explore a discussion on integrating novel detection metrics, leveraging quantum-assisted models, or deploying real-world clinical validation (or something else!). I’m excited by the prospect of working at the intersection of proteomics, AI safety, and biomedical model evaluation.\n\nGratefully,\nTommaso R. Marena\n\nP.S. Here’s my LinkedIn: www.linkedin.com/in/tommaso-marena\n\n\n" }, { "id": 3318499, "postDate": "2025-11-11T07:40:50.903Z", "content": "That's so cool man.
", "rawMarkdown": "That's so cool man." }, { "id": 3315044, "postDate": "2025-11-10T13:18:47.157Z", "content": "What's the goal for your project, what rate are you targeting or do you consider acceptable?
", "rawMarkdown": "What's the goal for your project, what rate are you targeting or do you consider acceptable?" }, { "id": 3347339, "postDate": "2025-11-25T05:23:06Z", "content": "Hello Olaf. I am interested to join. The main reason: learn a bit about Biomedical Engineering and also how you researchers elevate the use AI and ML in this field. I am curious to learn and will help you to contribute. Honestly speaking, I don't have any formal education or degree in Biomedical or Bioscience field but I am curious to work along your side and contribute to the work.
", "rawMarkdown": "Hello Olaf. I am interested to join. The main reason: learn a bit about Biomedical Engineering and also how you researchers elevate the use AI and ML in this field. I am curious to learn and will help you to contribute. Honestly speaking, I don't have any formal education or degree in Biomedical or Bioscience field but I am curious to work along your side and contribute to the work.", "isDeleted": true } ], "comments": [ { "id": 3319544, "author_name": "Joost Ertryckx", "author_url": "", "post_date": "2025-11-11T20:47:24.930000", "content": "I constantly iterate over generate code with chatgpt5, test and debug. Each script of the program I evaluate with my own eyes by picking some examples and evaluate their correctness. I want to finish this task of the challenge in 100 hours progamming and probably a lot more compute time experimenting with different neural network architectures but at the end the results will tell me if chatgpt5 and my method to use chatgpt5 is a fine method. I makes mistakes for sure but I got my AI engineering and biomedical degree before the LLM explosion 😅
", "votes": -1, "replies": [] }, { "id": 3394761, "author_name": "wakaG", "author_url": "", "post_date": "2026-01-21T16:29:17.397000", "content": "i'm interested, how can i contact you?
", "votes": 0, "replies": [] }, { "id": 3323218, "author_name": "Tommaso Marena", "author_url": "", "post_date": "2025-11-14T04:17:33.493000", "content": "Hello Dr. Imanov,\nThank you for sharing this open-source project. I find that your systematic evaluations of LLM hallucination risks in mass-spectrometry and protein function analysis resonate strongly with several aspects of my background.
\nI am an undergraduate double major in biochemistry and philosophy, pre-law at the Catholic University of America in Washington, DC, United States. I am a freelance AI Evaluation and Alignment Engineer at Alignerr.ai, where I evaluate and validate generative AI outputs to ensure quality, coherence, and ethical alignment with product goals. Also, I implement and maintain API-driven test harnesses, author evaluation protocols, and provide actionable scoring and feedback that guide model updates and product decisions.
\nIn addition, my research as a co-investigator on an NIH-approved protein folding study (PLCO-1699) and prior experience optimizing quantum protein folding models gives me a unique perspective that bridges clinical proteomics and model reliability. My technical toolkit spans Python (PyTorch, TensorFlow, HuggingFace), bioinformatics (ColabFold, AlphaFold3, RDKit), and experiment reproducibility (Docker, AWS), which matches the modular and rigorous approach of your framework.
\nI am particularly interested in extending your hallucination-detection benchmarks with advanced uncertainty quantification and real-time detection methods. My prior work includes ML pipelines for molecular complexity scoring, the development of domain-specific benchmarks, and recursive transformer search architectures for biological data. I do my best to ensure robust statistical reporting, and have presented these solutions at venues such as Johns Hopkins Rising Stars in Cellular, the Catholic University of America Graduate Biology Symposium, and every Catholic University of America Research Day since I began my undergraduate degrees.
\nI would be delighted to contribute collaboratively, both methodologically and in expanding dataset/evaluation coverage. Let me know if you’d like to explore a discussion on integrating novel detection metrics, leveraging quantum-assisted models, or deploying real-world clinical validation (or something else!). I’m excited by the prospect of working at the intersection of proteomics, AI safety, and biomedical model evaluation.
\nGratefully,\nTommaso R. Marena
\nP.S. Here’s my LinkedIn: www.linkedin.com/in/tommaso-marena
", "votes": 0, "replies": [] }, { "id": 3318499, "author_name": "PhenomSG", "author_url": "", "post_date": "2025-11-11T07:40:50.903000", "content": "That's so cool man.
", "votes": 0, "replies": [] }, { "id": 3315044, "author_name": "Marek Liyanage", "author_url": "", "post_date": "2025-11-10T13:18:47.157000", "content": "What's the goal for your project, what rate are you targeting or do you consider acceptable?
", "votes": 0, "replies": [] }, { "id": 3347339, "author_name": "", "author_url": "", "post_date": "2025-11-25T05:23:06", "content": "Hello Olaf. I am interested to join. The main reason: learn a bit about Biomedical Engineering and also how you researchers elevate the use AI and ML in this field. I am curious to learn and will help you to contribute. Honestly speaking, I don't have any formal education or degree in Biomedical or Bioscience field but I am curious to work along your side and contribute to the work.
", "votes": 0, "replies": [] } ], "raw_markdown_by_id": { "3314475": "Good morning,\n\nWe are conducting a research project titled **“Hallucination Risks of Large Language Models in Clinical Proteomics”**, recently formalized as a **prospective evaluation study**. The work systematically assesses how frontier LLMs (GPT-4, Claude Sonnet, Gemini Pro) behave when faced with **quantitative, domain-specific proteomics data**, particularly focusing on **hallucination frequency, severity, and risk factors**.\n\nOur results demonstrate that even state-of-the-art models exhibit **27–35% baseline hallucination rates**, escalating beyond **50% for complex or rare-protein queries**. These findings underline a critical gap between general medical-AI benchmarks and the reality of **high-precision clinical data interpretation**.\n\nWe are looking for a **research collaborator** experienced in:\n\n* **Proteomics or mass-spectrometry data analysis**, and\n* **Large language model evaluation, uncertainty quantification, or AI safety.**\n\nThe collaboration aims to extend this framework toward **hallucination-detection mechanisms and domain-specific evaluation benchmarks** for biomedical AI.\n\nIf this area aligns with your background, please feel free to reply here or reach out for discussion.\n\nFull codebase, query datasets, and validation framework are openly available:\n[https://github.com/olaflaitinen/llm-proteomics-hallucination](https://github.com/olaflaitinen/llm-proteomics-hallucination)\n\nLooking forward to connecting with colleagues working at the intersection of **AI reliability, biomedical NLP, and proteomics.**", "3319544": "I constantly iterate over generate code with chatgpt5, test and debug. Each script of the program I evaluate with my own eyes by picking some examples and evaluate their correctness. I want to finish this task of the challenge in 100 hours progamming and probably a lot more compute time experimenting with different neural network architectures but at the end the results will tell me if chatgpt5 and my method to use chatgpt5 is a fine method. I makes mistakes for sure but I got my AI engineering and biomedical degree before the LLM explosion 😅", "3394761": "i'm interested, how can i contact you? ", "3323218": "Hello Dr. Imanov,\nThank you for sharing this open-source project. I find that your systematic evaluations of LLM hallucination risks in mass-spectrometry and protein function analysis resonate strongly with several aspects of my background.\n\nI am an undergraduate double major in biochemistry and philosophy, pre-law at the Catholic University of America in Washington, DC, United States. I am a freelance AI Evaluation and Alignment Engineer at Alignerr.ai, where I evaluate and validate generative AI outputs to ensure quality, coherence, and ethical alignment with product goals. Also, I implement and maintain API-driven test harnesses, author evaluation protocols, and provide actionable scoring and feedback that guide model updates and product decisions.\n\nIn addition, my research as a co-investigator on an NIH-approved protein folding study (PLCO-1699) and prior experience optimizing quantum protein folding models gives me a unique perspective that bridges clinical proteomics and model reliability. My technical toolkit spans Python (PyTorch, TensorFlow, HuggingFace), bioinformatics (ColabFold, AlphaFold3, RDKit), and experiment reproducibility (Docker, AWS), which matches the modular and rigorous approach of your framework.\n\nI am particularly interested in extending your hallucination-detection benchmarks with advanced uncertainty quantification and real-time detection methods. My prior work includes ML pipelines for molecular complexity scoring, the development of domain-specific benchmarks, and recursive transformer search architectures for biological data. I do my best to ensure robust statistical reporting, and have presented these solutions at venues such as Johns Hopkins Rising Stars in Cellular, the Catholic University of America Graduate Biology Symposium, and every Catholic University of America Research Day since I began my undergraduate degrees.\n\nI would be delighted to contribute collaboratively, both methodologically and in expanding dataset/evaluation coverage. Let me know if you’d like to explore a discussion on integrating novel detection metrics, leveraging quantum-assisted models, or deploying real-world clinical validation (or something else!). I’m excited by the prospect of working at the intersection of proteomics, AI safety, and biomedical model evaluation.\n\nGratefully,\nTommaso R. Marena\n\nP.S. Here’s my LinkedIn: www.linkedin.com/in/tommaso-marena\n\n\n", "3318499": "That's so cool man.", "3315044": "What's the goal for your project, what rate are you targeting or do you consider acceptable?", "3347339": "Hello Olaf. I am interested to join. The main reason: learn a bit about Biomedical Engineering and also how you researchers elevate the use AI and ML in this field. I am curious to learn and will help you to contribute. Honestly speaking, I don't have any formal education or degree in Biomedical or Bioscience field but I am curious to work along your side and contribute to the work." } }