{
  "fetched_at": "2026-10-24T10:59:57Z",
  "source_url": "https://www.kaggle.com/competitions/code-retrieval-for-hugging-face-transformers/data",
  "pages": [
    {
      "name": "data-description",
      "content": "This dataset consists of natural language questions about the [Hugging Face Transformers](https://github.com/huggingface/transformers) library, together with the file names of 1-3 relevant documents that can help answer the question. All documents associated with a query are equally relevant, and their order in the list is irrelevant.\n\nThe dataset was created by [Storia AI](https://storia.ai/) in collaboration with [Morph Labs](https://morph.so/).\n\n## Files\n\n*   **train.csv** - The training set. Consists of 700 pairs of natural language questions and relevant file names.\n*   **test.csv** - The test set. Consists of 300 natural language questions.\n*   **sample_submission.csv** - A sample submission file in the correct format (with fake documents).\n\n## Columns\n\n*   `id` - A unique identifier for the data instance.\n*   `question` - A question in natural language.\n*   `documents` - A list of relevant documents, encoded as a JSON string. Each document is represented by the file path in the [Hugging Face Transformers](https://github.com/huggingface/transformers) repository, including the `huggingface/transformers` prefix.",
      "isPublished": false,
      "hasIsPublished": false,
      "postTitle": "",
      "hasPostTitle": false,
      "mimeType": "",
      "hasMimeType": false
    },
    {
      "name": "Description",
      "content": "## A typical code retrieval pipeline\nSetting up a code retrieval pipeline requires multiple steps:\n\n1. Download the document library.\n2. Embed each document and place it in a vector store.\n3. Define a retriever which, given a query, embeds it and then looks up relevant documents in the vector store.\n4. (Optional) Define a reranker which takes top N retrieved documents and reorders them.\n\n## Use [Sage](https://github.com/Storia-AI/sage) to bootstrap your code retrieval pipeline\nWhile you are free to implement the entire pipeline from scratch, you can use [Sage](https://github.com/Storia-AI/sage) for an off-the-shelf solution.\n\n### Installation\nTo install Sage, create a virtual environment and run `pip install`:\n```bash\npython -m venv my-venv\nsource activate my-venv/bin/activate\npip install git+https://github.com/Storia-AI/sage.git@main\n```\n\n### Prerequisites\nFollow the [Prerequisites](https://github.com/Storia-AI/sage?tab=readme-ov-file#prerequisites) section to set up your environment (including, for instance, the desired API keys).\n\n### Indexing huggingface/transformers\nTo clone + embed + store the repository in a vector store, run this command:\n```bash\nsage-index huggingface/transformers --commit-hash=7bb1c99800d235791dace10305731f377db8077b\n```\nNote that the dataset was created using the commit above. If you choose to clone the `huggingface/transformers` library manually, make sure to use the same commit.\n\nWe defaulted `sage-index` to run locally on your machine. For higher quality, you might want to use state-of-the-art third-party providers like OpenAI for embeddings, etc. You can overwrite these defaults from the command line. For instance:\n\n```bash\nsage-index huggingface/transformers \\\n--commit-hash=7bb1c99800d235791dace10305731f377db8077b\n--mode=remote \\\n--embeddings-provider=openai \\\n--embeddings-model=text-embedding-3-small\n```\n\nPlease refer directly to Sage's documentation for more details. You're also welcome to fork the repository to add your own extensions (and we accept pull requests if you're particularly proud of your contributions!).\n\n### Optional: chat with the codebase as a sanity-check\n\nIf you want to gain an intuition of how well your retrieval algorithm is working, you can bring up a chat interface by running the following command. This will bring up a Gradio app:\n\n```bash\nsage-chat huggingface/transformers --commit-hash=7bb1c99800d235791dace10305731f377db8077b\n```\n\n### Making a submission file\nYou can easily produce a submission file in the right format with the [retrieve_kaggle.py](https://github.com/Storia-AI/sage/blob/main/benchmarks/retrieval/retrieve_kaggle.py) script.",
      "isPublished": false,
      "hasIsPublished": false,
      "postTitle": "",
      "hasPostTitle": false,
      "mimeType": "",
      "hasMimeType": false
    },
    {
      "name": "Evaluation",
      "content": "Submissions are evaluated on [Recall@3](https://ir-measur.es/en/latest/measures.html#r), since the dataset contains at most 3 relevant documents per query.\n\n## Submission File\nThe submission file must contain two columns: `id` (taken from the column with the same name in the `test.csv` file) and `documents` (a JSON-encoded list of file paths). Documents will be treated as a set (i.e. their order in the list is irrelevant). See the `sample_submission.csv` file for an example.\n\nIf using Sage, you can simply run the [retrieve_kaggle.py](https://github.com/Storia-AI/sage/blob/main/benchmarks/retrieval/retrieve_kaggle.py) script to get a submission file in the right format.",
      "isPublished": false,
      "hasIsPublished": false,
      "postTitle": "",
      "hasPostTitle": false,
      "mimeType": "",
      "hasMimeType": false
    },
    {
      "name": "rules",
      "content": "Just have fun!",
      "isPublished": false,
      "hasIsPublished": false,
      "postTitle": "",
      "hasPostTitle": false,
      "mimeType": "",
      "hasMimeType": false
    },
    {
      "name": "abstract",
      "content": "The goal of this competition is to design a code retrieval system that can effectively find relevant files to help answer questions about [Hugging Face's Transformers](https://github.com/huggingface/transformers) library. Our dataset consists of natural language questions paired with relevant file names from the repository.\n\nThis will give you the chance to experiment with many of the moving pieces involved in a code retrieval pipeline: file chunking, dense vs sparse embeddings, embedding types, rerankers, etc. While we do provide a training set, we expect to see improvements over the provided baseline without any fine-tuning.",
      "isPublished": false,
      "hasIsPublished": false,
      "postTitle": "",
      "hasPostTitle": false,
      "mimeType": "",
      "hasMimeType": false
    }
  ]
}