The main idea behind this solution is to extract embeddings from different transformer layers and compute the average for each text. We then label the resulting vectors as fake or real and use an SVM to learn the decision boundaries. To reduce dimensionality, I apply PCA.
\nAn improvement I explored was ensembling across different models and layers. Interestingly, using embeddings from a single layer already yields strong performance, I didn’t need to rely heavily on ensembling. The key was extracting richer information from individual layers.
\nI also save a confidence score for each prediction and ran an additional analysis based on LLM reasoning (rather than embeddings). For this, I used the LLaMA-8B-Instruct model, which performs well with instructions. I provided it with two texts simultaneously and tasked it with identifying which one was fake, including few-shot examples for guidance.
\nI experimented with smaller models, but they tended to lose context and hallucinate. My next goal is to try larger models or alternative LLM processing strategies to reduce hallucination. But I was surprised that taking only embeddings worked better than refining this analysis with an LLM.
", "comments": [], "comment_count": 0, "messages": [ { "id": 3293331, "postDate": "2025-09-23T15:20:39.297Z", "content": "The main idea behind this solution is to extract embeddings from different transformer layers and compute the average for each text. We then label the resulting vectors as fake or real and use an SVM to learn the decision boundaries. To reduce dimensionality, I apply PCA.
\nAn improvement I explored was ensembling across different models and layers. Interestingly, using embeddings from a single layer already yields strong performance, I didn’t need to rely heavily on ensembling. The key was extracting richer information from individual layers.
\nI also save a confidence score for each prediction and ran an additional analysis based on LLM reasoning (rather than embeddings). For this, I used the LLaMA-8B-Instruct model, which performs well with instructions. I provided it with two texts simultaneously and tasked it with identifying which one was fake, including few-shot examples for guidance.
\nI experimented with smaller models, but they tended to lose context and hallucinate. My next goal is to try larger models or alternative LLM processing strategies to reduce hallucination. But I was surprised that taking only embeddings worked better than refining this analysis with an LLM.
", "rawMarkdown": "The main idea behind this solution is to extract embeddings from different transformer layers and compute the average for each text. We then label the resulting vectors as fake or real and use an SVM to learn the decision boundaries. To reduce dimensionality, I apply PCA.\n\nAn improvement I explored was ensembling across different models and layers. Interestingly, using embeddings from a single layer already yields strong performance, I didn’t need to rely heavily on ensembling. The key was extracting richer information from individual layers.\n\nI also save a confidence score for each prediction and ran an additional analysis based on LLM reasoning (rather than embeddings). For this, I used the LLaMA-8B-Instruct model, which performs well with instructions. I provided it with two texts simultaneously and tasked it with identifying which one was fake, including few-shot examples for guidance.\n\nI experimented with smaller models, but they tended to lose context and hallucinate. My next goal is to try larger models or alternative LLM processing strategies to reduce hallucination. But I was surprised that taking only embeddings worked better than refining this analysis with an LLM." } ], "complete": true, "source": "live", "fetched_at": "2026-10-19T21:43:56Z", "competition_id": 105874, "competition_slug": "fake-or-real-the-impostor-hunt", "team_id": 14393511, "solution_url": "/competitions/fake-or-real-the-impostor-hunt/writeups/llm-embedding-based-solution" }