# Agentic Retrieval Grand Challenge (ACM-ICAIF '25) — Data

来源：https://www.kaggle.com/competitions/acm-icaif-25-ai-agentic-retrieval-grand-challenge/data

本地 Meta Kaggle 快照；不是本次实时页面。

# Dataset Description

## Overview  

This competition is designed to evaluate *agentic (multi-step) retrieval* capabilities in financial AI. The central goal is to identify grounded evidence from large-scale SEC filings in order to answer complex institutional finance questions. To this end, participants are tasked to ***optimize a system prompt that maximize the ranking performance***. Unlike traditional retrieval tasks, this challenge is framed as a two-stage process.

In the first stage, **Document-Level Ranking**, participants must determine which type of SEC filing—such as 10-K, 10-Q, 8-K, DEF 14A, or an Earnings Transcript—is most likely to contain the relevant information for a given query. In the second stage, **Chunk-Level Ranking**, the task shifts to locating the most relevant passages within the selected document, with participants required to rank the Top-5 chunks that best support the answer.  

The dataset is drawn from expert-curated examples based on real SEC filings from the period 2023–2024. This ensures that questions and contexts are realistic and grounded in actual financial disclosure practices.  

All results must be submitted as a single consolidated **CSV file** that combines outputs from both retrieval tasks. Detailed specifications for the submission format are provided in a separate section.  

To ensure reproducibility, every submission must include a working notebook that can be executed in the **Databricks Free Edition** environment. The notebook link should be posted in the competition’s discussion tab as the standardized evaluation interface. For participants interested in enhanced functionality, optional integration with the **Upstage Solar LLM API** is also supported.  

***Notice: The dataset for competition will be accessable at the start of the competition.***

## Files

*   **document_ranking_kaggle_dev.jsonl** - the development set of document ranking task.
*   **chunk_ranking_kaggle_dev.jsonl** - the development set of chunk ranking task.

*   **document_ranking_kaggle_eval.jsonl** - the development set of document ranking task. (containing 200 samples)
*   **chunk_ranking_kaggle_eval.jsonl** - the development set of chunk ranking task. (containing 200 samples)

## Columns - Document Ranking Task

The following columns are provided for the **document-level ranking** dataset. This task requires ranking entire SEC filing types by their relative likelihood of containing the answer to a given financial question.

- `_id`: An unique identifier for each sample.

- `messages`: A JSON-style field containing the complete **user prompt**. This includes the instruction, the financial question, and the list of candidate document types to be ranked. The prompt specifies that the output must be a list of indices (e.g., `[4, 2, 1, 0, 3]`), covering all document types in ranked order.  

- `qrel`: A dictionary mapping document indices (`0` … `N-1`) to integer scores between 0 and 4, matches roughly with the **relative importance of each document type**.  
  - `4` = most relevant document type  
  - `3` = next most relevant  
  - `2` = moderately relevant  
  - `1` = weakly relevant  
  - `0` = least relevant document type  
  - In other words, every candidate document type receives a unique score from 0–4, reflecting its position in the ranking.  
  - **Important**: *qrel values are provided only in the development/training set. For the official test set, `qrel` is hidden and participants must predict the ranking themselves.*  

These fields define the input (prompt + candidate document types), the relative ground-truth ranking used in development, and the contextual metadata needed to interpret each example.

## Columns - Chunk Ranking Task

The following columns are provided for the **chunk-level ranking** dataset. This task requires ranking paragraph-level chunks within a single SEC filing to identify which passages contain the supporting evidence needed to answer the question.

- `messages`: A JSON-style field containing the full user prompt. This includes a simple instruction, the financial question, and the context of the document or chunks to be ranked.  
- `qrel`: A dictionary mapping chunk indices (`0` … `N-1`) to graded relevance labels, following a TREC-style convention.  
  - `0` = non-relevant (negative chunk)  
  - `1` = relevant (positive chunk)  
  - `2` = highly relevant (strong positive chunk)  
  These labels provide a finer-grained supervision signal compared to binary annotation.  
  - **Note**: `qrel` is only included in the development/training set for evaluation and model tuning. In the official test set, `qrel` values are hidden and participants must predict the relevance scores themselves.  

These fields collectively define the input (prompt + candidate chunks), the ground-truth labels used for evaluation in development, and the contextual metadata describing the example.