Thanks for your efforts in hosting the good competition!
\nI'm interested in joining the competition. But I'm concerned about the evaluation metric since there is no head pose in the validation set provided. What camera pose should we assign to the trained model to produce the final results to compare with the GT val data? Without GT camera pose provided, the produced and GT data are not aligned, which means that the PSNR, SSIM (error-based metric) do not make sense. Just imagine: when the produced and GT video are controlled by different head pose, the PSNR may be computed from a predicted teeth and GT hair. In this way, I think the computed PSNR and other pixel-wise error-based metric can be simply regarded as \"noise\" and cannot reflect the video quality.
\nIn my opinion, there are two potential solutions to address this issue:
\nThanks again for your precious time and effort in organizing the competition and workshop!
", "messages": [ { "id": "2508314", "postDate": "11/01/2023 15:50:41", "content": "Thanks for your efforts in hosting the good competition!
\nI'm interested in joining the competition. But I'm concerned about the evaluation metric since there is no head pose in the validation set provided. What camera pose should we assign to the trained model to produce the final results to compare with the GT val data? Without GT camera pose provided, the produced and GT data are not aligned, which means that the PSNR, SSIM (error-based metric) do not make sense. Just imagine: when the produced and GT video are controlled by different head pose, the PSNR may be computed from a predicted teeth and GT hair. In this way, I think the computed PSNR and other pixel-wise error-based metric can be simply regarded as \"noise\" and cannot reflect the video quality.
\nIn my opinion, there are two potential solutions to address this issue:
\nThanks again for your precious time and effort in organizing the competition and workshop!
", "rawMarkdown": "Thanks for your efforts in hosting the good competition! \n\nI'm interested in joining the competition. But I'm concerned about the evaluation metric since there is no head pose in the validation set provided. What camera pose should we assign to the trained model to produce the final results to compare with the GT val data? Without GT camera pose provided, the produced and GT data are not aligned, which means that the PSNR, SSIM (error-based metric) do not make sense. Just imagine: when the produced and GT video are controlled by different head pose, the PSNR may be computed from a predicted teeth and GT hair. In this way, I think the computed PSNR and other pixel-wise error-based metric can be simply regarded as \"noise\" and cannot reflect the video quality.\n\nIn my opinion, there are two potential solutions to address this issue:\n1. Provide the GT camera pose of the validate dataset so the PSNR/SSIM comparison makes sense.\n2. Or give up the error-based metric, instead use mean opinion score (MOS) or perceptual metric (LPIPS, FID).\n\nThanks again for your precious time and effort in organizing the competition and workshop!", "votes": null }, { "id": "2508750", "postDate": "11/02/2023 01:22:29", "content": "Thank you for the reponse! Indeed, PSNR and SSIM would be less effective without GT camera poses. We would provide GT camera poses as soon as possible.
", "rawMarkdown": "Thank you for the reponse! Indeed, PSNR and SSIM would be less effective without GT camera poses. We would provide GT camera poses as soon as possible.", "votes": null }, { "id": "2508889", "postDate": "11/02/2023 04:32:41", "content": "Thanks for you quick response. As for the GT camera poses, I'm little worried about the camera convention. Since different code base can predict significantly different camera pose (For instance, Deep3DFaceRecon and AD-NeRF have different conventions and you cannot use the Deep3d predicted ones to drive the model trained on AD-NeRF.).
", "rawMarkdown": "Thanks for you quick response. As for the GT camera poses, I'm little worried about the camera convention. Since different code base can predict significantly different camera pose (For instance, Deep3DFaceRecon and AD-NeRF have different conventions and you cannot use the Deep3d predicted ones to drive the model trained on AD-NeRF.).", "votes": null }, { "id": "2508900", "postDate": "11/02/2023 04:46:41", "content": "Thanks for the additional comment. We have already generated the camera poses following the convention of AD-NeRF. We would organize the data and release it as soon as possible. As for Deep3DFaceRecon, we would try to organize it later.
", "rawMarkdown": "Thanks for the additional comment. We have already generated the camera poses following the convention of AD-NeRF. We would organize the data and release it as soon as possible. As for Deep3DFaceRecon, we would try to organize it later.", "votes": null }, { "id": "2508944", "postDate": "11/02/2023 05:20:27", "content": "I've just uploaded the testing GT camera poses generated following the AD-NeRF convention, thanks for the great comment! Besides, we decided to use FID instead of PSNR and SSIM to evaluate the results for Single Image Setting.
", "rawMarkdown": "I've just uploaded the testing GT camera poses generated following the AD-NeRF convention, thanks for the great comment! Besides, we decided to use FID instead of PSNR and SSIM to evaluate the results for Single Image Setting.", "votes": null }, { "id": "2508964", "postDate": "11/02/2023 05:40:14", "content": "That's great! Thanks again for your quick response and solution!
", "rawMarkdown": "That's great! Thanks again for your quick response and solution!", "votes": null }, { "id": "2508993", "postDate": "11/02/2023 05:50:57", "content": "You're welcome, thanks again for your valuable comments.
", "rawMarkdown": "You're welcome, thanks again for your valuable comments.", "votes": null } ], "comments": [ { "id": 2508750, "author_name": "jingnangao", "author_url": "", "post_date": "11/02/2023 01:22:29", "content": "Thank you for the reponse! Indeed, PSNR and SSIM would be less effective without GT camera poses. We would provide GT camera poses as soon as possible.
", "votes": null, "replies": [] }, { "id": 2508889, "author_name": "zhenhuiye", "author_url": "", "post_date": "11/02/2023 04:32:41", "content": "Thanks for you quick response. As for the GT camera poses, I'm little worried about the camera convention. Since different code base can predict significantly different camera pose (For instance, Deep3DFaceRecon and AD-NeRF have different conventions and you cannot use the Deep3d predicted ones to drive the model trained on AD-NeRF.).
", "votes": null, "replies": [ { "id": 2508944, "author_name": "jingnangao", "author_url": "", "post_date": "11/02/2023 05:20:27", "content": "I've just uploaded the testing GT camera poses generated following the AD-NeRF convention, thanks for the great comment! Besides, we decided to use FID instead of PSNR and SSIM to evaluate the results for Single Image Setting.
", "votes": null, "replies": [] } ] }, { "id": 2508900, "author_name": "jingnangao", "author_url": "", "post_date": "11/02/2023 04:46:41", "content": "Thanks for the additional comment. We have already generated the camera poses following the convention of AD-NeRF. We would organize the data and release it as soon as possible. As for Deep3DFaceRecon, we would try to organize it later.
", "votes": null, "replies": [ { "id": 2508964, "author_name": "zhenhuiye", "author_url": "", "post_date": "11/02/2023 05:40:14", "content": "That's great! Thanks again for your quick response and solution!
", "votes": null, "replies": [ { "id": 2508993, "author_name": "jingnangao", "author_url": "", "post_date": "11/02/2023 05:50:57", "content": "You're welcome, thanks again for your valuable comments.
", "votes": null, "replies": [] } ] } ] } ], "raw_markdown_by_id": { "2508314": "Thanks for your efforts in hosting the good competition! \n\nI'm interested in joining the competition. But I'm concerned about the evaluation metric since there is no head pose in the validation set provided. What camera pose should we assign to the trained model to produce the final results to compare with the GT val data? Without GT camera pose provided, the produced and GT data are not aligned, which means that the PSNR, SSIM (error-based metric) do not make sense. Just imagine: when the produced and GT video are controlled by different head pose, the PSNR may be computed from a predicted teeth and GT hair. In this way, I think the computed PSNR and other pixel-wise error-based metric can be simply regarded as \"noise\" and cannot reflect the video quality.\n\nIn my opinion, there are two potential solutions to address this issue:\n1. Provide the GT camera pose of the validate dataset so the PSNR/SSIM comparison makes sense.\n2. Or give up the error-based metric, instead use mean opinion score (MOS) or perceptual metric (LPIPS, FID).\n\nThanks again for your precious time and effort in organizing the competition and workshop!", "2508750": "Thank you for the reponse! Indeed, PSNR and SSIM would be less effective without GT camera poses. We would provide GT camera poses as soon as possible.", "2508889": "Thanks for you quick response. As for the GT camera poses, I'm little worried about the camera convention. Since different code base can predict significantly different camera pose (For instance, Deep3DFaceRecon and AD-NeRF have different conventions and you cannot use the Deep3d predicted ones to drive the model trained on AD-NeRF.).", "2508900": "Thanks for the additional comment. We have already generated the camera poses following the convention of AD-NeRF. We would organize the data and release it as soon as possible. As for Deep3DFaceRecon, we would try to organize it later.", "2508944": "I've just uploaded the testing GT camera poses generated following the AD-NeRF convention, thanks for the great comment! Besides, we decided to use FID instead of PSNR and SSIM to evaluate the results for Single Image Setting.", "2508964": "That's great! Thanks again for your quick response and solution!", "2508993": "You're welcome, thanks again for your valuable comments." }, "source": "meta" }