{ "fetched_at": "2026-10-19T07:51:18Z", "source_url": "https://www.kaggle.com/competitions/audio-driven-talking-head-generation/data", "pages": [ { "name": "data-description", "content": "Our talking head dataset contains **10 videos**, where each video contains a single speaker. These videos are in different languages and all the videos last for at least two minutes with a resolution of 512×512. \n\n- For the **Single image setting**, the first frame of each video's testing set is selected as the driving image.\n- For the **video setting**, about 9/10 of the videos are given as the training set.\n\nWe choose the last 1/10 of the video to be the testing set. The audio of the testing set will be released for participants to generate their results for both the image and video settings.\n\nTraining videos and testing audios for video setting are in **Video.zip** while driving images and testing audios for single image setting are in **Single_Image.zip**.\n\nWe've also uploaded the **ground-truth camera poses** for testing set so that the PNSR and SSIM metrics would be more effective. The camera poses are generated following the convention of AD-NeRF. Please raise an issue in the discussion if you have any questions related to the dataset.", "isPublished": false, "hasIsPublished": false, "postTitle": "", "hasPostTitle": false, "mimeType": "", "hasMimeType": false }, { "name": "rules", "content": "- Participants can choose to participate in either the single image setting or the video setting to tackle the task. \n- Participants must create a submission that matches the description provided by the organizers. \n- The submissions will be judged based on the evaluation metrics listed in each task. Winners will be announced and awarded a prize.\n- The dataset including the videos, images and audios should **only be used in this competition**.", "isPublished": false, "hasIsPublished": false, "postTitle": "", "hasPostTitle": false, "mimeType": "", "hasMimeType": false }, { "name": "abstract", "content": "###Final Announcement\n**Happy new year!** The competition is now closed for submission and we would announce the most realistic results for both settings once we finish all the evaluations.\n**Update: The final ranking is announced and the leader of the top3 team please send a confirmation email to [gjn0310@sjtu.edu.cn]. Please also indicate your team name in the email.**\n\n\nIn this task, we specifically focus on the ability to generate talking heads with realistic facial expressions and natural head poses that match the accompanying audio. By learning from both the audio and visual data, this task focuses on developing a **multi-modal learning model for talking head generation**. We further design the following two settings to motivate the participants to tackle this task with 2D and 3D methods, respectively.\n\n- **Single image setting**: Participants are allowed to train their model with external data, and we will provide a **single image** for their models to animate.\n\n- **Video setting**: Participants are not allowed to train their model with external data, while a **video** is provided for training a personalized talking head model. Participants are allowed to use pretrained audio models to process audios but are not allowed to use pretrained talking head (visual) model and finetune under this setting.\n\nThe final output of both settings should be a talking head model that can be driven by any input audio. For the image setting, the synthesized talking head videos are expected to be lip-synchronized and with high fidelity. As for the video setting, we encourage the participants to design 3D-based models so that the speakers can have various natural poses during talking while preserving their identity. \n\nThe official website of the workshop is provided as https://digitalhumanworkshop.github.io/\n\n###Prizes\n1st Place - $ 1000\n2nd Place - $ 500\n3rd Place - $ 300\nThe first place of the competition would win 1000$ as the prize, the second place would win 500$ and the third place would win 300$. An additional \"Most Realistic\" Award would be chosen at the end of the competition by the host for both tracks. **Note that teams that win the prize should submit their code and a brief summary at the end of the competition.**\n\nIf you have any questions, please raise an issue in the discussion.\n\n**Please check all the requirements in the submission section before you submit your results.**", "isPublished": false, "hasIsPublished": false, "postTitle": "", "hasPostTitle": false, "mimeType": "", "hasMimeType": false }, { "name": "Evaluation", "content": "In this task, participants are supposed to generate talking head videos for both the image and video settings. \nTo evaluate the quality of the generated talking heads, we use the following metrics:\n- **SSIM** and **PSNR** to measure the generation quality on the image level for **Video Setting**. \n- **FID** to measure the generation quality on thefeature level for **Single Image Setting**.\n- **AV offset and AV confidence** to evaluate the lip synchronization. Note that we use the absolute value of AV offset for evalution.\n- **L1 Distance** of 3DMM landmarks to evaluate the generated head motion quality (Only be used in Video Setting).\n\n**The ranking would be determined by the number of “first place” across all metrics.** \n**IMPORTANT NOTICE**: Note that you could participate in either settings but the ranking would be based on the number of “first place” **across all metrics of both settings**. For example, Team A only participate in the Single Image setting and get 4 \"first place\" while B participates in both settings and got 4 \"first place\" in Video setting and 1 in Single Image setting, the score of B is then 5, which is greater than A.", "isPublished": false, "hasIsPublished": false, "postTitle": "", "hasPostTitle": false, "mimeType": "", "hasMimeType": false }, { "name": "Submission", "content": "We recommend participants to zip all the generated results and save it to your own **Google Drive**. Since Kaggle accepts a **csv** file normally, participants should submit a **csv** file including the team name, the link of the google drive, and the setting that you choose to participate in. A general submission format is provided as follows:\n| ID | Usage | Vid | Setting | Team_Name |\n|:-------:|:---------:|:------------:|:---------:|:-----------------:|\n| Task3 | Public | [Your Link] | 0/1 | [Your Team Name]|\n\nYou need to fill in the **link** and **team name** in the table above, and fill in 0 for single image setting and 1 for video setting, **the rest of the table should be kept**. \n- The video in the zip file should **be named as xxx_pred.mp4**. The resolution should be **512×512** and the frame rate should bd **25FPS**, where xxx is the original name of the training video or driving image. For example, for the result for the data named \"French\" would be French_pred.mp4.\n- Note that after your submission,**the score would be the choice of your setting**, so please don't be worry if you get 0.0 or 1.0 for your score, we would evaluate the results as soon as possible and update the results in the **Public Per-Metric Leaderboard** in this page. \n- Participants can only submit **up to 5 \"valid\" results** during the competition. If the submission could not be evaluated (e.g. Participant does not submit in the request format), the submission would not be counted as \"valid\" submission. The final ranking would be based on the best submissions.\n- If you met the some errors like \"ID column ID not found in submission\", you can check whether you type an additional space after the column name. For example, please use \"ID\" instead of \"ID \".\n- If you encounter \"Submission has been administratively invalidated\", you can check whether you have submitted **all 10 videos** in the zip file or whether you have successfully **shared the zip file**. In addition, please check whether you have **concated the audio** and video together and **name the video in the correct format**.", "isPublished": false, "hasIsPublished": false, "postTitle": "", "hasPostTitle": false, "mimeType": "", "hasMimeType": false }, { "name": "Baseline", "content": "In this task, we provide reference implementations of the baseline methods:\n-Single image setting: **SadTalker** https://github.com/Winfredy/SadTalker.\n-Video setting: **AD-NeRF** https://github.com/YudongGuo/AD-NeRF.\nWe acknowledge the authors for these two wonderful projects.", "isPublished": false, "hasIsPublished": false, "postTitle": "", "hasPostTitle": false, "mimeType": "", "hasMimeType": false }, { "name": "Public Per-Metric Leaderboard", "content": "Since we employ multiple metrics for evaluation, we would maintain a per-metric leaderboard here as reference. **The best evaluation results would be chosen**. \nIf the final ranking appears to be the same for two team, FID would be used for video setting ranking. Subjective score would be given based on the realism of the generated results for both settings if ranking is still the same for tied teams.\n###Top3 Overall Ranking\n| #Rank | TeamName |Total First-Place Number|\n|:----:|:---------:|:------------:|\n|1|BDIV Lab|4|\n|2|USTC-IAT-United|3|\n|3|ExcitAI|2|\n\n###Single Image Setting\n| #Rank | TeamName | FID ↓ | AVconf ↑ | AVoffset↓ |\n|:----:|:---------:|:------------:|:-----------------:|:-----------------:|\n| 1|USTC-IAT-United|**3.397**|4.788 |**0.1** |\n| 2 | ExcitAI |4.156|**9.583** | 2.0|\n| |BDIV Lab|4.017|1.602|0.5|\n| |gsdata|3.872|5.069|2.9|\n| |fdwjtzsbjrpz|4.361|9.564|2.0|\n| |team-zz| 4.397| 7.206| 0.8| \n| |TA| 4.919 | 9.092 | 1.0| \n| |testint| 4.434 | 8.881 | 1.0| \n| |moss| 4.421 | 5.152 | 1.8| \n| -|Baseline(Sadtalker)| 8.634 | 6.914 | 0.4| \n\n\n###Video Setting\n| #Rank | TeamName | PSNR ↑ | SSIM ↑ | AVconf ↑ | AVoffset ↓ | L1trans↓ | L1angle↓ |\n|:-------:|:---------:|:------------:|:---------:|:-----------------:|:-----------------:|:-----------------:|:-----------------:|\n| 1 |BDIV Lab|**24.913**|**0.831**|6.605|2.1|**0.039**|**0.007**|\n| 2 |ExcitAI|17.940|0.613|**8.848**|2.0|0.052|0.055|\n| 2 |USTC-IAT-United|17.984 |0.620 |3.381 | **0.1**| 0.058|0.053 | \n| 2|Scut DT Lab| 22.163| 0.791 | 5.122| **0.1**|0.049 | 0.011| \n| 2 |VastGen| 18.167 | 0.616 |6.989 | **0.1** | 0.047|0.052 | \n| |Li |23.450 |0.794 | 6.961| 2.6|0.044 | 0.014| \n| |fdwjtzsbjrpz|23.184|0.803|6.660|2.1|0.043|0.008|\n| |gsdata|23.813|0.807|6.959|2.6|0.043|0.008|\n| |TA|18.801|0.642 |4.632 | 2.2 | 0.047|0.050 | \n| | wzw|17.818|0.614|2.976|0.1|0.081|0.053|\n| -|Baseline(AD-NeRF)| 18.071 | 0.631 | 4.908 | 1.0 | 0.053|0.053 | \n\n####Latest Update: 2024.1.1\n", "isPublished": false, "hasIsPublished": false, "postTitle": "", "hasPostTitle": false, "mimeType": "", "hasMimeType": false } ] }