{
  "fetched_at": "2026-10-19T08:03:51Z",
  "source_url": "https://www.kaggle.com/competitions/audio-driven-co-speech-gesture-video-generation/data",
  "pages": [
    {
      "name": "data-description",
      "content": "~~Our co-speech generation dataset contains training and validation videos for **4** different IDs (PersonA, PersonB, PersonC, and PersonD), each video contains a single speaker, PersonA and PersonB speak Chinese, PersonC and PersonD speak English. The training videos last for at least 5 minutes with a resolution of 2160×3840 and the validation videos last for at least 1 minute with a resolution of 2160×3840. ~~\n\n**Update: 2023-12-01**\n\n\nOur co-speech generation dataset contains training and validation videos for **2** different IDs (PersonA, PersonB), each video contains a single speaker, PersonA and PersonB speak Chinese. The training videos last for at least 5 minutes with a resolution of 2160×3840 and the validation videos last for at least 1 minute with a resolution of 2160×3840. \n\nThe audio of the test set is also released for participants to generate their results for the competition.\n\n###Note\n1. We also notice that directly generating high-resolution 2D videos is very challenging and time-consuming, so in this competition, the participants may downsample to video resolution to 288×512 to train their model, we will evaluate the results with a resolution of 288×512.\n1. This dataset can only be used for this competition.\n1. Due to some legal issues, we fail to collect a suitable English dataset for this competition. As the competition will end within about 1 month, given little time left, we decide to use the 2 uploaded IDs only in this competition.",
      "isPublished": false,
      "hasIsPublished": false,
      "postTitle": "",
      "hasPostTitle": false,
      "mimeType": "",
      "hasMimeType": false
    },
    {
      "name": "Evaluation",
      "content": "In this task, participants are supposed to generate co-speech gesture videos for given audio.\nTo evaluate the quality of the generated  co-speech gesture videos, we use both objective metrics and subjective metrics.\n\n###Objective metrics\n- **Fréchet Video Distance** (**FVD**) [3] to measure the similarity between two video sequences. It takes into account both spatial and temporal relationships between the videos, allowing for a more accurate assessment of their similarity.\n- **Fréchet Gesture Distance** (**FGD**) [4] to evaluate the distance between the real and synthetic gesture distribution. We train an auto-encoder on poses extracted from both the competition dataset and PATS dataset to get motion features. Fréchet distance between the real and synthetic gesture is calculated in motion feature space.\n- **Beat Consistency Score** (**BC**) [5] to measure the speech-motion alignment. \n- **Diversity** (**Div.**) [6] to measure the diversity among generated gestures.\n\n**Note 1**:  **FGD**, **BC** and **Div.**  *evaluate generated videos from a human motion perspective. We first use [MMPose](https://github.com/open-mmlab/mmpose/tree/main) to estimate poses of videos.*\n\n###Subjective metrics\n- **Realness** to evaluate the generation quality on the video level, whether the video is generated or captured. \n- **Human-likeness** to evaluate whether the generated gestures look natural and are difficult to distinguish from captured human motions.\n- **Gesture-speech appropriateness** to evaluate whether the generated gestures are consistent with the input audio, in terms of both rhythm and semantics.\n- **Diversity (Div.)** to evaluate whether the generated gestures are diverse enough.\n\n**Note 2**: *The rating scale for the subjective metrics is 1 to 5, with 1 being the poorest and 5 being the best.*\n**Note 3**: *The Leaderboard only shows the ranking of **FVD**.  Other metrics will only be evaluated on the final submission.*\n\nThe final ranking would be determined by the number of \"first place\" across all metrics. If multiple teams win the same number of \"first place\" metrics, the final ranking would be determined by the number of \"second place\" across all metrics, and so on. \n\n## Submission File\nYour team ultimately needs to submit a compressed zip file(*.zip), for instance, \"**submission.zip**\".\nThe compressed zip file contains two parts: submission.csv and your generated videos.\n```\n|--submission.zip\n|----submission.csv\n|----PersonA_test.mp4\n|----PersonB_test.mp4\n```\nFor submission CSV files, you should use the \"**extraction_FVD_to_submission.py**\" (submission_form.zip from Dataset) tool to process your generated videos. \n**!!!!! Tips: Please ensure that the file name of the video you generate matches our requirements: PersonA_test.mp4, PersonB_test.mp4**\nFor each ID in the test set, you must generate a video for a given audio and the video should be named according to the following format \"ID_test.mp4\", for example, \"PersonA_test.mp4\".\n\n##Reference\n[1] Audio-Driven Co-Speech Gesture Video Generation, NeurIPS 2022: ANGIE https://github.com/alvinliu0/ANGIE.\n[2] Audio-Driven Neural Gesture Reenactment with Video Motion Graphs, CVPR 2022:  vid-reenact https://github.com/yzhou359/vid-reenact.\n[3] Towards accurate generative models of video: A new metric & challenges. arXiv:1812.01717, 2018.\n[4] Learn to Dance with AIST++: Music Conditioned 3D Dance Generation, ICCV2021: https://github.com/google-research/mint\n[5] Speech Gesture Generation from the Trimodal Context of Text, Audio, and Speaker Identity, TOG2020: https://github.com/ai4r/Gesture-Generation-from-Trimodal-Context#gesture-generation-from-trimodal-context\n[6] Learning Hierarchical Cross-Modal Association for Co-Speech Gesture Generation, CVPR2022: HA2G https://github.com/alvinliu0/HA2G\n",
      "isPublished": false,
      "hasIsPublished": false,
      "postTitle": "",
      "hasPostTitle": false,
      "mimeType": "",
      "hasMimeType": false
    },
    {
      "name": "rules",
      "content": "- Participants must create a submission that matches the description provided by the organizers.\n- The submissions will be judged based on the evaluation metrics listed in the task. Winners will be announced and awarded a prize.\n- The dataset can only be used for this competition, please contact the organizer for commercial usage.",
      "isPublished": false,
      "hasIsPublished": false,
      "postTitle": "",
      "hasPostTitle": false,
      "mimeType": "",
      "hasMimeType": false
    },
    {
      "name": "abstract",
      "content": "**！！！[New]！！！ Update: The final ranking is announced and the leader of the top 3 teams please send a confirmation email to [zenghaoqi@huawei.com]. Please also indicate your team name in the email.**\n\n\n| #Rank|Team Name | Total 1-Place Number | Total 2-Place Number | The highest ranking except for first and second place |\n|:---:|:---:|:---:|:---:|:---:|\n| #1|DragonBooOOM | 4 | - | - |\n| #2|USTC_IAT_United|  1| 1 | - |\n| #3|damNull|  1| 0 | 5 |\n\n\nCongratulations to all the winners! For ranking details, please refer to the following Public Per-Metric Leaderboard Results.\n\n------------------------------------------\n\n**Public Per-Metric Leaderboard Results**\n\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F7542929%2Fbaf4731e9a496885fae29a5d1b40f0c5%2Fresult.png?generation=1704960241973483&alt=media)\n\n**Note**: Only valid submissions that meet the correct format were evaluated, we recruited over 30 participants to do subjective evaluations. Moreover,  we also observe that objective metrics are not consistent with human subjective evaluations, which is the same as the findings of Kucherenko et al. [1].\n\n**Reference**\n[1] Kucherenko, Taras, et al. \"Evaluating gesture-generation in a large-scale open challenge: The GENEA Challenge 2022.\" arXiv preprint arXiv:2303.08737 (2023).\n\n------------------------------------------\n\n\n**Happy New Year! The competition is now closed for submission and we will evaluate the results and announce the final results once we finish all the evaluations.**\n\n**Please verify that your submission meets the correct format (a zip file containing videos with a resolution of 288x512 and a CSV file). Submissions with videos of other resolutions or in illegal formats will be rejected.**\n\nNonverbal behaviors, especially co-speech gestures, are vital for face-to-face communication. Studies have shown that gestures can help express intentions, convey emotions, and complement the information conveyed by speech. Recent research has demonstrated that machine learning-based approaches play an increasingly critical role in motion synthesis, which has become the primary method for co-speech gesture synthesis.\n\nIn this task, participants are required to synthesize co-speech gesture videos of a target person based on any given audio. The final output should be a rendered video, instead of motion sequences. Generally, the synthesized gesture motions in this task are expected to be natural, difficult to distinguish from captured videos, and consistent with audio in terms of rhythm, semantics, and style. We encourage participants to propose novel ideas for synthesizing high-fidelity co-speech gestures.\n\nSpecifically, we will provide two video clips for each target person, one (about 5 minutes) for training, and one (about 1 minute) for validation. We will also provide a testing audio for the participants to generate the final result video for the target person.\n\n### Prizes\n$1,800 will be awarded to the Top 3 teams with the highest scores at the conclusion of the competition:\n- 1st Place - $ 1,000\n- 2nd Place - $ 500\n- 3rd Place - $ 300\n\n**Note that teams that win the prize should submit their code and technical reports at the end of the competition.**",
      "isPublished": false,
      "hasIsPublished": false,
      "postTitle": "",
      "hasPostTitle": false,
      "mimeType": "",
      "hasMimeType": false
    }
  ]
}