Edit : a more detailed notebook has now been published and can be found here: https://www.kaggle.com/code/epiktistes/for-impostor-hunt-in-texts-3rd-place-solution
\nThank you to the organizers for this fun challenge. This solution that got third place is a bit messy since it's a mix of two draft solutions and some things got left at the exploration stage.
\nIn short, initially, I started to work with DSPy. I gave some training examples to Gemini for it to draft a rough prompt as a starting point, which gave this:
\n typing import Literal\nimport dspy\n dspy import Example\n\n Categorize(dspy.Signature):\n The Messenger,AcceptRejectDave chickenepic,whoa!Alright, picture this...I was sleepy when I wrote this...Chinese languageDinosaur dinosaur dinosaur...Jargon Salad... ancient during periods like ancient ...... inception inception...examplesaccepted_textThe project aims achieve an accuracy level dex analyzing elemental abundances like alpha elements iron peak elements when signal--noise ratios exceed per Angstrom (A). A 'figure merit' (FoM), calculated dividing successfully observed targets all listed targets within their catalog, serves an indicator measuring survey success.rejected_textDinosaur eggshells offer clues what dinosaurs ate long ago; similarly paleontologists can use fossils extinct species like dinosaurs marine reptiles piece together what ancient ecosystems looked like millions billions years ago!accepted_textThe azimuth drive system experienced problems oscillation under specific conditions (very slow rotation), leading unwanted 'sticking' images telescope structure. This issue was solved adding an electronic filter blocked this oscillation. After extensive analysis experimentation, new servo parameters were implemented August rejected_textThe wrybeasts observatory, particularly their intricate mechanics, presented a persistent challenge! Under specific circumstances, particularly when azimuth axis moved very slowly, azimuth servo would enter a dance oscillation, setting up a -Hz rhythm vibrated like a startled squirrel telescope structure.accepted_textALMA has produced a significant amount research Early Science Proposal Cycle (Cycle ) ended two years ago. Data Cycle currently being used researchers, Band ( GHz) making up roughly % publications. Most programs funded under Cycle resulted least one published paper, an average citation rate per publication citations per .rejected_textAlright, picture this: we're blasting off – literally – analyze cosmos ALMA, a revolutionary observatory basically slingshots us past normal limits what we can see! The results are pretty epic, most scientists hitting their publication targets – a staggering %!accepted_textThe design phase CTAs ( Cherenkov Telescope Array) has ended; now 's transitioning preconstruction telescope designs have been finalized manufacturing readiness review experts... Construction activities are expected start , leading up data availability partially assembled sections starting rejected_texttext_1text_2\n\n \n text_to_classify_1: str = dspy.InputField(\n desc=\n )\n text_to_classify_2: str = dspy.InputField(\n desc=\n )\n\n \n analysis_of_text_1: str = dspy.OutputField(\n desc=\n )\n analysis_of_text_2: str = dspy.OutputField(\n desc=\n )\n answer: Literal['text_1', 'text_2'] = dspy.OutputField(\n desc=\n )\n\nclassify = dspy.Predict(Categorize)\n\nexamples = [Example(text_to_classify_1=row.text_1, text_to_classify_2=row.text_2).with_inputs('text_to_classify_1', 'text_to_classify_2') _, row df.iterrows()]\n\nresults = classify.batch(examples)\nI ran it using Qwen3-4B, Qwen3-8B and DeepSeek Chat to get pre-optimization baselines, and that got 0.9 on the leaderboard.
\nThen, reading the rules, I saw that external data was allowed as long as it was publicly available at no cost. Since the website for The Messenger seemed to fit those criteria and had sequential URLs (the first issue being at https://messenger.eso.org/1/, the second at being at https://messenger.eso.org/2/, etc.), rather than starting to optimize with DSPy straight away, I looped through the available issues to download the articles as PDFs.
\nI then used SentenceTransformers to match the competition summaries (the file_1.txt and file_2.txt) to full articles.
\nWhen the two summaries (file_1.txt and file_2.txt) matched the same article, I considered that it was probably the right one, and those were sent to DeepSeek Chat and DeepSeek Reasoner with the following prompt:
\nf\"\"\"You are helpful assistant that can analyze two summaries of the same and determine which one is more accurate.\n\nLong : {}\n\nHere are the two summaries:\nSummary : {}\n : {}\n\nWhich of these two summaries best summarizes the long ?\n\nAnswer in JSON in the following format:\n\n{{\n \"summary_1_score\": <score>,\n : <score>,\n : <explanation>\n}}\n\"\"\"\nFor the rows where both summaries matched with the same full article, I used those predictions, and for the others, I fell back to the earlier predictions that had been obtained with the first approach. That got 0.937 on the public leaderboard, and 0.938 on the private one.
", "comments": [], "comment_count": 0, "messages": [ { "id": 3304649, "postDate": "2025-10-21T02:38:12.643Z", "content": "Edit : a more detailed notebook has now been published and can be found here: https://www.kaggle.com/code/epiktistes/for-impostor-hunt-in-texts-3rd-place-solution
\nThank you to the organizers for this fun challenge. This solution that got third place is a bit messy since it's a mix of two draft solutions and some things got left at the exploration stage.
\nIn short, initially, I started to work with DSPy. I gave some training examples to Gemini for it to draft a rough prompt as a starting point, which gave this:
\n typing import Literal\nimport dspy\n dspy import Example\n\n Categorize(dspy.Signature):\n The Messenger,AcceptRejectDave chickenepic,whoa!Alright, picture this...I was sleepy when I wrote this...Chinese languageDinosaur dinosaur dinosaur...Jargon Salad... ancient during periods like ancient ...... inception inception...examplesaccepted_textThe project aims achieve an accuracy level dex analyzing elemental abundances like alpha elements iron peak elements when signal--noise ratios exceed per Angstrom (A). A 'figure merit' (FoM), calculated dividing successfully observed targets all listed targets within their catalog, serves an indicator measuring survey success.rejected_textDinosaur eggshells offer clues what dinosaurs ate long ago; similarly paleontologists can use fossils extinct species like dinosaurs marine reptiles piece together what ancient ecosystems looked like millions billions years ago!accepted_textThe azimuth drive system experienced problems oscillation under specific conditions (very slow rotation), leading unwanted 'sticking' images telescope structure. This issue was solved adding an electronic filter blocked this oscillation. After extensive analysis experimentation, new servo parameters were implemented August rejected_textThe wrybeasts observatory, particularly their intricate mechanics, presented a persistent challenge! Under specific circumstances, particularly when azimuth axis moved very slowly, azimuth servo would enter a dance oscillation, setting up a -Hz rhythm vibrated like a startled squirrel telescope structure.accepted_textALMA has produced a significant amount research Early Science Proposal Cycle (Cycle ) ended two years ago. Data Cycle currently being used researchers, Band ( GHz) making up roughly % publications. Most programs funded under Cycle resulted least one published paper, an average citation rate per publication citations per .rejected_textAlright, picture this: we're blasting off – literally – analyze cosmos ALMA, a revolutionary observatory basically slingshots us past normal limits what we can see! The results are pretty epic, most scientists hitting their publication targets – a staggering %!accepted_textThe design phase CTAs ( Cherenkov Telescope Array) has ended; now 's transitioning preconstruction telescope designs have been finalized manufacturing readiness review experts... Construction activities are expected start , leading up data availability partially assembled sections starting rejected_texttext_1text_2\n\n \n text_to_classify_1: str = dspy.InputField(\n desc=\n )\n text_to_classify_2: str = dspy.InputField(\n desc=\n )\n\n \n analysis_of_text_1: str = dspy.OutputField(\n desc=\n )\n analysis_of_text_2: str = dspy.OutputField(\n desc=\n )\n answer: Literal['text_1', 'text_2'] = dspy.OutputField(\n desc=\n )\n\nclassify = dspy.Predict(Categorize)\n\nexamples = [Example(text_to_classify_1=row.text_1, text_to_classify_2=row.text_2).with_inputs('text_to_classify_1', 'text_to_classify_2') _, row df.iterrows()]\n\nresults = classify.batch(examples)\nI ran it using Qwen3-4B, Qwen3-8B and DeepSeek Chat to get pre-optimization baselines, and that got 0.9 on the leaderboard.
\nThen, reading the rules, I saw that external data was allowed as long as it was publicly available at no cost. Since the website for The Messenger seemed to fit those criteria and had sequential URLs (the first issue being at https://messenger.eso.org/1/, the second at being at https://messenger.eso.org/2/, etc.), rather than starting to optimize with DSPy straight away, I looped through the available issues to download the articles as PDFs.
\nI then used SentenceTransformers to match the competition summaries (the file_1.txt and file_2.txt) to full articles.
\nWhen the two summaries (file_1.txt and file_2.txt) matched the same article, I considered that it was probably the right one, and those were sent to DeepSeek Chat and DeepSeek Reasoner with the following prompt:
\nf\"\"\"You are helpful assistant that can analyze two summaries of the same and determine which one is more accurate.\n\nLong : {}\n\nHere are the two summaries:\nSummary : {}\n : {}\n\nWhich of these two summaries best summarizes the long ?\n\nAnswer in JSON in the following format:\n\n{{\n \"summary_1_score\": <score>,\n : <score>,\n : <explanation>\n}}\n\"\"\"\nFor the rows where both summaries matched with the same full article, I used those predictions, and for the others, I fell back to the earlier predictions that had been obtained with the first approach. That got 0.937 on the public leaderboard, and 0.938 on the private one.
", "rawMarkdown": "*Edit : a more detailed notebook has now been published and can be found here: [https://www.kaggle.com/code/epiktistes/for-impostor-hunt-in-texts-3rd-place-solution](https://www.kaggle.com/code/epiktistes/for-impostor-hunt-in-texts-3rd-place-solution)*\n\nThank you to the organizers for this fun challenge. This solution that got third place is a bit messy since it's a mix of two draft solutions and some things got left at the exploration stage.\n\nIn short, initially, I started to work with DSPy. I gave some training examples to Gemini for it to draft a rough prompt as a starting point, which gave this:\n\n```\nfrom typing import Literal\nimport dspy\nfrom dspy import Example\n\nclass Categorize(dspy.Signature):\n \"\"\"**Your Role:** You are an expert editor for \"The Messenger,\" the scientific and technical journal of the European Southern Observatory (ESO).\n\n **The Task:** You will be given a JSON object containing two texts to classify. In every pair, one article is suitable for publication and must be **Accepted**, while the other is unsuitable and must be **Rejected**. Your decisions should be based on the following rules and the provided JSON examples.\n\n ---\n\n ### **Classification Rules**\n\n **Criteria for an \"Accept\" Decision:**\n The article must be a well-written, professional piece suitable for an ESO journal.\n 1. **Subject Matter:** The topic must be professional astronomy, astrophysics, or related astronomical technology and instrumentation (e.g., telescopes, surveys, data analysis).\n 2. **Language and Tone:** The language must be formal, objective, and scientifically precise.\n 3. **Content and Coherence:** The article must be factual, coherent, and scientifically sound.\n\n **Criteria for a \"Reject\" Decision (Immediate Rejection):**\n The article should be rejected if it displays **any** of the following characteristics. These texts are often poorly written or automatically generated.\n 1. **Silly or Fictional Subjects:** The article discusses non-scientific topics like dinosaurs, Santa Claus, rainbow unicorns, fictional characters, or irrelevant mascots (e.g., \"Dave the chicken\").\n 2. **Unprofessional Language:** The tone is overly casual, uses slang (\"epic,\" \"whoa!\"), tells a story (\"Alright, picture this...\"), or includes self-deprecating author notes (\"I was sleepy when I wrote this...\").\n 3. **Keyword Swapping:** Key scientific terms are systematically replaced with absurd words (e.g., cheese names like `Gruyère`, or the phrase \"Chinese language\").\n 4. **Nonsensical or Repetitive Text:** The text is gibberish, just a list of repeating words (\"Dinosaur dinosaur dinosaur...\"), or contains random, incoherent characters.\n 5. **Empty or Incomplete Text:** The text is blank or clearly unfinished.\n 6. **Incoherent Jargon or \"Jargon Salad\":** The text uses real scientific terms but combines them in sentences that are grammatically broken, logically circular, or nonsensical. The sentences may be long and convoluted but lack a clear, coherent meaning.\n * **Example Sign:** Redundant, looping phrases like `\"...since ancient times during periods like ancient times...\"`.\n * **Example Sign:** Repetitive and unnatural phrasing like `\"...since around the year of its inception in the year of its inception...\"`.\n\n ---\n\n ### **JSON Examples for Classification**\n Here are examples of correctly classified texts in JSON format.\n\n ```json\n {\n \"examples\": [\n {\n \"accepted_text\": \"The project aims to achieve an accuracy level of 0.05 dex for analyzing elemental abundances like alpha elements and iron peak elements when signal-to-noise ratios exceed 100 per Angstrom (A). A 'figure of merit' (FoM), calculated by dividing successfully observed targets by all listed targets within their catalog, serves as an indicator measuring survey success.\",\n \"rejected_text\": \"Dinosaur eggshells offer clues about what dinosaurs ate long ago; similarly paleontologists can use fossils from extinct species like dinosaurs or marine reptiles to piece together what ancient ecosystems looked like millions or billions years ago!\"\n },\n {\n \"accepted_text\": \"The azimuth drive system experienced problems with oscillation under specific conditions (very slow rotation), leading to unwanted 'sticking' images in the telescope structure. This issue was solved by adding an electronic filter that blocked this oscillation. After extensive analysis and experimentation, new servo parameters were implemented in August 2009.\",\n \"rejected_text\": \"The wrybeasts in the observatory, particularly their intricate mechanics, presented a persistent challenge! Under specific circumstances, particularly when the azimuth axis moved very slowly, the azimuth servo would enter a dance of oscillation, setting up a 11.6-Hz rhythm that vibrated like a startled squirrel through the telescope structure.\"\n },\n {\n \"accepted_text\": \"ALMA has produced a significant amount of research since its first Early Science Proposal Cycle (Cycle 0) ended two years ago. Data from Cycle 0 is currently being used by researchers, with Band 7 and 9 (275-950 GHz) making up roughly 64% of the publications. Most programs funded under Cycle 0 resulted in at least one published paper, with an average citation rate per publication of 10 citations per year.\",\n \"rejected_text\": \"Alright, picture this: we're blasting off into space – literally – to analyze the cosmos with ALMA, a revolutionary observatory that basically slingshots us past the normal limits of what we can see! The results are pretty epic, with most scientists hitting their publication targets – a staggering 85%!\"\n },\n {\n \"accepted_text\": \"The design phase for CTAs (the Cherenkov Telescope Array) has ended; now it's transitioning into preconstruction where telescope designs have been finalized for manufacturing readiness review by experts... Construction activities are expected to start around 2018, leading up to data availability from partially assembled sections starting around 2021.\",\n \"rejected_text\": \"\"\n }\n ]\n }\n ```\n\n ---\n\n ### **Instructions for Your Output**\n\n For the classification task, analyze each text based on the rules above and provide:\n 1. A brief analysis of text_1 explaining why it should be accepted or rejected\n 2. A brief analysis of text_2 explaining why it should be accepted or rejected \n 3. Your decision on which text to accept (either \"text_1\" or \"text_2\")\n\n Remember: Exactly one text must be accepted and one must be rejected.\n \"\"\"\n \n # Input fields\n text_to_classify_1: str = dspy.InputField(\n desc=\"First text to classify for ESO journal publication\"\n )\n text_to_classify_2: str = dspy.InputField(\n desc=\"Second text to classify for ESO journal publication\"\n )\n \n # Output fields\n analysis_of_text_1: str = dspy.OutputField(\n desc=\"Brief analysis explaining why text_1 should be accepted or rejected based on the classification rules\"\n )\n analysis_of_text_2: str = dspy.OutputField(\n desc=\"Brief analysis explaining why text_2 should be accepted or rejected based on the classification rules\"\n )\n answer: Literal['text_1', 'text_2'] = dspy.OutputField(\n desc=\"Which text to accept for publication (must be either 'text_1' or 'text_2')\"\n )\n\nclassify = dspy.Predict(Categorize)\n\nexamples = [Example(text_to_classify_1=row.text_1, text_to_classify_2=row.text_2).with_inputs('text_to_classify_1', 'text_to_classify_2') for _, row in df.iterrows()]\n\nresults = classify.batch(examples)\n```\n\nI ran it using Qwen3-4B, Qwen3-8B and DeepSeek Chat to get pre-optimization baselines, and that got 0.9 on the leaderboard.\n\nThen, reading the rules, I saw that external data was allowed as long as it was publicly available at no cost. Since the website for The Messenger seemed to fit those criteria and had sequential URLs (the first issue being at https://messenger.eso.org/1/, the second at being at https://messenger.eso.org/2/, etc.), rather than starting to optimize with DSPy straight away, I looped through the available issues to download the articles as PDFs.\n\nI then used SentenceTransformers to match the competition summaries (the file_1.txt and file_2.txt) to full articles.\n\nWhen the two summaries (file_1.txt and file_2.txt) matched the same article, I considered that it was probably the right one, and those were sent to DeepSeek Chat and DeepSeek Reasoner with the following prompt:\n\n```\nf\"\"\"You are a helpful assistant that can analyze two summaries of the same text and determine which one is more accurate.\n\nLong text: {x.matched_longtext}\n\nHere are the two summaries:\nSummary 1: {x.text_1}\nSummary 2: {x.text_2}\n\nWhich of these two summaries best summarizes the long text?\n\nAnswer in JSON in the following format:\n\n{{\n \"summary_1_score\":