Author: Lonnie Qin
\nCompetition: AI Mathematical Olympiad — Progress Prize 3
\nPublic LB Score: ≈ 40 / 50 (no improvement over simpler baselines)
\nStatus: Awaiting Private LB results (hopeful for a competitive score)
This solution implements a retrieval-augmented, multi-attempt reasoning pipeline built around a 120B-class model, a high-quality olympiad corpus, and a persistent Python sandbox for tool use.
\nCore Idea (which I still believe is strong):
\nGive the model a rich set of similar, fully-solved competition problems (via hybrid semantic + keyword retrieval) and let it reason multiple times in parallel with tool access. Combine the attempts using entropy-weighted voting. This approach mimics how strong human contestants solve problems — by recalling analogous examples and verifying rigorously.
While the public leaderboard score stabilized around 40, I am optimistic that the method’s depth will translate better on the private test set.
\nNotebook: https://www.kaggle.com/code/lonnieqin/aimo-math-corpus-creation-embedding-generation
\nDatasets Used
\nAI-MO/NuminaMath-1.5): ~900k problems → filtered to hard competition-style only (aime, amc_10/12, imo, usamo, putnam, hmmt, etc.)hendrycks/competition_math): Level 4 & 5 onlyFiltering & Cleaning
\nEmbedding Generation
\nmodel = SentenceTransformer('sentence-transformers/all-mpnet-base-v2') # 768-dim\nembeddings = model.encode(corpus_texts, normalize_embeddings=True, batch_size=32)\nIndexFlatL2 (exact search)faiss_index.bin, corpus.pkl, embeddings.npy, metadataFinal corpus size: ~120k high-quality solved problems.
\nNotebook: https://www.kaggle.com/code/lonnieqin/aimo-3-solution
\ngpt-oss-120b served via vLLMfp8_e4m3 KV cache, 65k context, gpu-memory-utilization=0.96, prefix cachingclass SimilarProblemRetriever:\n def get_similar(self, problem: str, k: int = 5) -> str:\n # 1. Embedding retrieval (3× candidates)\n # 2. Keyword retrieval using MATH_KEYWORDS (divisor, prime, gcd, permutation, triangle…)\n # 3. Hybrid score = 0.6×embedding_sim + 0.4×keyword_overlap\nReturns 5 reference problems with full solutions and answers — dramatically enriching the context.
\n\\boxed{} output format (0–99999)sympy, numpy, mpmath, etc.python tool with automatic print() injectiontemperature=1.0, min_p=0.021 / max(entropy, 1e-9)All prompts, retrieved examples, and final answers are logged to prompt_logs.jsonl for analysis.
Public LB: ≈ 40
\nThe approach did not improve the public leaderboard score relative to simpler baselines. However, the core components (high-quality corpus, hybrid retrieval, persistent tool use, and entropy-weighted ensembling) remain technically sound and closely mirror successful human problem-solving strategies.
I am hopeful that the private test set will reward the method’s depth and robustness more than the public set did.
\n| Component | \nValue Even at Current Score | \n
|---|---|
| Hybrid retrieval | \nGuarantees relevant examples | \n
| 120B model + vLLM | \nRaw reasoning power | \n
| Persistent Python sandbox | \nExact verification | \n
| Entropy-weighted voting | \nFilters low-confidence outputs | \n
| Multi-attempt parallelism | \nRobustness | \n
Conclusion
\nWhile the public leaderboard result was not the breakthrough I hoped for, I remain convinced that retrieval-augmented multi-attempt reasoning with tool use is a powerful direction for mathematical Olympiad AI. The private LB will be the real test.
Notebooks
\nCredits to notebooks:\nhttps://www.kaggle.com/code/nihilisticneuralnet/44-50-let-me-over-cook
", "comments": [], "comment_count": 0, "messages": [ { "id": 3445343, "postDate": "2026-04-19T15:15:43.083Z", "content": "Author: Lonnie Qin
\nCompetition: AI Mathematical Olympiad — Progress Prize 3
\nPublic LB Score: ≈ 40 / 50 (no improvement over simpler baselines)
\nStatus: Awaiting Private LB results (hopeful for a competitive score)
This solution implements a retrieval-augmented, multi-attempt reasoning pipeline built around a 120B-class model, a high-quality olympiad corpus, and a persistent Python sandbox for tool use.
\nCore Idea (which I still believe is strong):
\nGive the model a rich set of similar, fully-solved competition problems (via hybrid semantic + keyword retrieval) and let it reason multiple times in parallel with tool access. Combine the attempts using entropy-weighted voting. This approach mimics how strong human contestants solve problems — by recalling analogous examples and verifying rigorously.
While the public leaderboard score stabilized around 40, I am optimistic that the method’s depth will translate better on the private test set.
\nNotebook: https://www.kaggle.com/code/lonnieqin/aimo-math-corpus-creation-embedding-generation
\nDatasets Used
\nAI-MO/NuminaMath-1.5): ~900k problems → filtered to hard competition-style only (aime, amc_10/12, imo, usamo, putnam, hmmt, etc.)hendrycks/competition_math): Level 4 & 5 onlyFiltering & Cleaning
\nEmbedding Generation
\nmodel = SentenceTransformer('sentence-transformers/all-mpnet-base-v2') # 768-dim\nembeddings = model.encode(corpus_texts, normalize_embeddings=True, batch_size=32)\nIndexFlatL2 (exact search)faiss_index.bin, corpus.pkl, embeddings.npy, metadataFinal corpus size: ~120k high-quality solved problems.
\nNotebook: https://www.kaggle.com/code/lonnieqin/aimo-3-solution
\ngpt-oss-120b served via vLLMfp8_e4m3 KV cache, 65k context, gpu-memory-utilization=0.96, prefix cachingclass SimilarProblemRetriever:\n def get_similar(self, problem: str, k: int = 5) -> str:\n # 1. Embedding retrieval (3× candidates)\n # 2. Keyword retrieval using MATH_KEYWORDS (divisor, prime, gcd, permutation, triangle…)\n # 3. Hybrid score = 0.6×embedding_sim + 0.4×keyword_overlap\nReturns 5 reference problems with full solutions and answers — dramatically enriching the context.
\n\\boxed{} output format (0–99999)sympy, numpy, mpmath, etc.python tool with automatic print() injectiontemperature=1.0, min_p=0.021 / max(entropy, 1e-9)All prompts, retrieved examples, and final answers are logged to prompt_logs.jsonl for analysis.
Public LB: ≈ 40
\nThe approach did not improve the public leaderboard score relative to simpler baselines. However, the core components (high-quality corpus, hybrid retrieval, persistent tool use, and entropy-weighted ensembling) remain technically sound and closely mirror successful human problem-solving strategies.
I am hopeful that the private test set will reward the method’s depth and robustness more than the public set did.
\n| Component | \nValue Even at Current Score | \n
|---|---|
| Hybrid retrieval | \nGuarantees relevant examples | \n
| 120B model + vLLM | \nRaw reasoning power | \n
| Persistent Python sandbox | \nExact verification | \n
| Entropy-weighted voting | \nFilters low-confidence outputs | \n
| Multi-attempt parallelism | \nRobustness | \n
Conclusion
\nWhile the public leaderboard result was not the breakthrough I hoped for, I remain convinced that retrieval-augmented multi-attempt reasoning with tool use is a powerful direction for mathematical Olympiad AI. The private LB will be the real test.
Notebooks
\nCredits to notebooks:\nhttps://www.kaggle.com/code/nihilisticneuralnet/44-50-let-me-over-cook
", "rawMarkdown": "**Author:** Lonnie Qin \n**Competition:** AI Mathematical Olympiad — Progress Prize 3 \n**Public LB Score:** ≈ 40 / 50 (no improvement over simpler baselines) \n**Status:** Awaiting Private LB results (hopeful for a competitive score)\n\n---\n\n### 1. Overview\n\nThis solution implements a **retrieval-augmented, multi-attempt reasoning pipeline** built around a 120B-class model, a high-quality olympiad corpus, and a persistent Python sandbox for tool use.\n\n**Core Idea (which I still believe is strong):** \nGive the model a rich set of similar, fully-solved competition problems (via hybrid semantic + keyword retrieval) and let it reason multiple times in parallel with tool access. Combine the attempts using entropy-weighted voting. This approach mimics how strong human contestants solve problems — by recalling analogous examples and verifying rigorously.\n\nWhile the public leaderboard score stabilized around **40**, I am optimistic that the method’s depth will translate better on the private test set.\n\n---\n\n### 2. Corpus Creation & Embedding Pipeline\n\n**Notebook:** https://www.kaggle.com/code/lonnieqin/aimo-math-corpus-creation-embedding-generation\n\n**Datasets Used**\n- **NuminaMath-1.5** (`AI-MO/NuminaMath-1.5`): ~900k problems → filtered to hard competition-style only (`aime`, `amc_10/12`, `imo`, `usamo`, `putnam`, `hmmt`, etc.)\n- **Hendrycks MATH** (`hendrycks/competition_math`): Level 4 & 5 only\n\n**Filtering & Cleaning**\n- Minimum solution length = 100 characters\n- Problem-type filter (Number Theory, Counting & Probability, Intermediate Algebra, Precalculus, Geometry, Algebra)\n- Strict deduplication on normalized problem text\n- Aggressive cleaning of AoPS navigation boilerplate, markdown links, ellipsis, etc.\n\n**Embedding Generation**\n```python\nmodel = SentenceTransformer('sentence-transformers/all-mpnet-base-v2') # 768-dim\nembeddings = model.encode(corpus_texts, normalize_embeddings=True, batch_size=32)\n```\n- FAISS `IndexFlatL2` (exact search)\n- Saved: `faiss_index.bin`, `corpus.pkl`, `embeddings.npy`, metadata\n\n**Final corpus size:** ~120k high-quality solved problems.\n\n---\n\n### 3. Main Inference Pipeline\n\n**Notebook:** https://www.kaggle.com/code/lonnieqin/aimo-3-solution\n\n#### 3.1 Model & Serving\n- Model: `gpt-oss-120b` served via vLLM\n- Configuration: `fp8_e4m3` KV cache, 65k context, `gpu-memory-utilization=0.96`, prefix caching\n\n#### 3.2 Hybrid Similar-Problem Retriever\n```python\nclass SimilarProblemRetriever:\n def get_similar(self, problem: str, k: int = 5) -> str:\n # 1. Embedding retrieval (3× candidates)\n # 2. Keyword retrieval using MATH_KEYWORDS (divisor, prime, gcd, permutation, triangle…)\n # 3. Hybrid score = 0.6×embedding_sim + 0.4×keyword_overlap\n```\nReturns 5 reference problems with full solutions and answers — dramatically enriching the context.\n\n#### 3.3 System Prompt Highlights\n- Elite IMO-level identity\n- Explicit instructions to study and adapt techniques from the retrieved examples\n- Strict `\\boxed{}` output format (0–99999)\n\n#### 3.4 Tool Use — Persistent Jupyter Sandbox\n- 16 pre-warmed kernels with `sympy`, `numpy`, `mpmath`, etc.\n- Custom `python` tool with automatic `print()` injection\n- State persists across reasoning turns within an attempt\n\n#### 3.5 Multi-Attempt Parallel Solving\n- 8 parallel attempts per problem (different seeds)\n- Up to 128 reasoning turns each\n- `temperature=1.0`, `min_p=0.02`\n- Early stopping when one answer reaches 4 votes\n\n#### 3.6 Answer Selection\n- Weighted voting: weight = `1 / max(entropy, 1e-9)`\n- Mean token entropy used as confidence signal\n- Transparent vote table displayed for each problem\n\n---\n\n### 4. Time & Resource Management\n- Global notebook timeout: 17,400 s\n- Adaptive per-problem budget (300–900 s) with 50-problem lookahead safety\n- Fully reliable within competition constraints\n\n---\n\n### 5. Prompt Logging\nAll prompts, retrieved examples, and final answers are logged to `prompt_logs.jsonl` for analysis.\n\n---\n\n### 6. Results & Reflection\n\n**Public LB:** ≈ 40 \nThe approach did not improve the public leaderboard score relative to simpler baselines. However, the core components (high-quality corpus, hybrid retrieval, persistent tool use, and entropy-weighted ensembling) remain technically sound and closely mirror successful human problem-solving strategies.\n\nI am hopeful that the private test set will reward the method’s depth and robustness more than the public set did.\n\n---\n\n### 7. Why This Approach Still Has Strong Potential\n\n| Component | Value Even at Current Score |\n|-------------------------------|-----------------------------|\n| Hybrid retrieval | Guarantees relevant examples |\n| 120B model + vLLM | Raw reasoning power |\n| Persistent Python sandbox | Exact verification |\n| Entropy-weighted voting | Filters low-confidence outputs |\n| Multi-attempt parallelism | Robustness |\n\n---\n\n### 8. Future Improvements (if continuing)\n- Stronger reranking (cross-encoder)\n- Difficulty-aware example selection\n- Longer context or MoE models\n- Self-verification / critique loops\n\n---\n\n**Conclusion** \nWhile the public leaderboard result was not the breakthrough I hoped for, I remain convinced that retrieval-augmented multi-attempt reasoning with tool use is a powerful direction for mathematical Olympiad AI. The private LB will be the real test.\n\n**Notebooks** \n- Corpus creation: https://www.kaggle.com/code/lonnieqin/aimo-math-corpus-creation-embedding-generation\n- Full solution: https://www.kaggle.com/code/lonnieqin/aimo-3-solution\n\nCredits to notebooks:\nhttps://www.kaggle.com/code/nihilisticneuralnet/44-50-let-me-over-cook\n" } ], "complete": true, "source": "live", "fetched_at": "2026-10-20T04:21:21Z", "competition_id": 118448, "competition_slug": "ai-mathematical-olympiad-progress-prize-3", "team_id": 14800917, "solution_url": "/competitions/ai-mathematical-olympiad-progress-prize-3/writeups/hybrid-rag-multi-attempt-reasoning-with-persistent" }