{
  "id": 579446,
  "title": "Phase II Evaluation Procedure and Required Actions",
  "url": "/competitions/3d-reconstruction-from-monocular-multi-food-images/discussion/579446",
  "author_name": "",
  "post_date": "2025-05-17T14:13:34.983728Z",
  "votes": null,
  "comment_count": 8,
  "views": 0,
  "content": "<p>Dear Participants,<br>\nThank you for your food model submissions. We are now advancing to Phase II of the competition, which will focus on chamfer distance evaluation.</p>\n<p><strong>Phase II Materials Available</strong><br>\nWe have uploaded all necessary Phase II files to the \"<strong>phase_2_data_and_scripts</strong>\" folder, which you should now have access to.</p>\n<p><strong>Required Action: Submit transform.json Files</strong><br>\nBefore we can proceed with distance measurements between ground truth and predicted models, <strong>you must provide a transform.json file for each of your models</strong> to ensure proper alignment with the ground truth.</p>\n<ul>\n<li>An example transformation file is available in the \"<strong>example_transform</strong>\" folder for your reference</li>\n<li><strong>The transform.json file must be submitted to us no later than May 21st, 2025</strong></li>\n</ul>\n<p><strong>Evaluation Process Clarification</strong><br>\nThe \"phase_2_eval_scripts\" folder contains three evaluation scripts that will be used to determine Phase II rankings:</p>\n<ol>\n<li><strong>eval_obj_cd.py</strong>: Measures chamfer distance without any transformation</li>\n<li><strong>eval_obj_cd_with_transform.py</strong>: Measures distance after applying your transform.json files to predicted models</li>\n<li><strong>eval_obj_cd_with_transform_open3d.py</strong>: Uses an alternative Python library to measure distance and apply transformation matrices</li>\n</ol>\n<p><strong>Important</strong>: Your final evaluation score will be the lowest value produced by either the second or third script (eval_obj_cd_with_transform.py or eval_obj_cd_with_transform_open3d.py).</p>\n<p>Please contact us if you have any questions about this procedure.</p>\n<p>Thanks,<br>\nMetaFood Workshop Organizing Committee</p>",
  "messages": [
    {
      "id": "3203935",
      "postDate": "05/17/2025 14:13:34",
      "content": "<p>Dear Participants,<br>\nThank you for your food model submissions. We are now advancing to Phase II of the competition, which will focus on chamfer distance evaluation.</p>\n<p><strong>Phase II Materials Available</strong><br>\nWe have uploaded all necessary Phase II files to the \"<strong>phase_2_data_and_scripts</strong>\" folder, which you should now have access to.</p>\n<p><strong>Required Action: Submit transform.json Files</strong><br>\nBefore we can proceed with distance measurements between ground truth and predicted models, <strong>you must provide a transform.json file for each of your models</strong> to ensure proper alignment with the ground truth.</p>\n<ul>\n<li>An example transformation file is available in the \"<strong>example_transform</strong>\" folder for your reference</li>\n<li><strong>The transform.json file must be submitted to us no later than May 21st, 2025</strong></li>\n</ul>\n<p><strong>Evaluation Process Clarification</strong><br>\nThe \"phase_2_eval_scripts\" folder contains three evaluation scripts that will be used to determine Phase II rankings:</p>\n<ol>\n<li><strong>eval_obj_cd.py</strong>: Measures chamfer distance without any transformation</li>\n<li><strong>eval_obj_cd_with_transform.py</strong>: Measures distance after applying your transform.json files to predicted models</li>\n<li><strong>eval_obj_cd_with_transform_open3d.py</strong>: Uses an alternative Python library to measure distance and apply transformation matrices</li>\n</ol>\n<p><strong>Important</strong>: Your final evaluation score will be the lowest value produced by either the second or third script (eval_obj_cd_with_transform.py or eval_obj_cd_with_transform_open3d.py).</p>\n<p>Please contact us if you have any questions about this procedure.</p>\n<p>Thanks,<br>\nMetaFood Workshop Organizing Committee</p>",
      "rawMarkdown": "Dear Participants,\nThank you for your food model submissions. We are now advancing to Phase II of the competition, which will focus on chamfer distance evaluation.\n\n**Phase II Materials Available**\nWe have uploaded all necessary Phase II files to the \"**phase_2_data_and_scripts**\" folder, which you should now have access to.\n\n**Required Action: Submit transform.json Files**\nBefore we can proceed with distance measurements between ground truth and predicted models, **you must provide a transform.json file for each of your models** to ensure proper alignment with the ground truth.\n\n- An example transformation file is available in the \"**example_transform**\" folder for your reference\n- **The transform.json file must be submitted to us no later than May 21st, 2025**\n\n**Evaluation Process Clarification**\nThe \"phase_2_eval_scripts\" folder contains three evaluation scripts that will be used to determine Phase II rankings:\n\n1. **eval_obj_cd.py**: Measures chamfer distance without any transformation\n2. **eval_obj_cd_with_transform.py**: Measures distance after applying your transform.json files to predicted models\n3. **eval_obj_cd_with_transform_open3d.py**: Uses an alternative Python library to measure distance and apply transformation matrices\n\n**Important**: Your final evaluation score will be the lowest value produced by either the second or third script (eval_obj_cd_with_transform.py or eval_obj_cd_with_transform_open3d.py).\n\nPlease contact us if you have any questions about this procedure.\n\nThanks,\nMetaFood Workshop Organizing Committee",
      "votes": null
    },
    {
      "id": "3205221",
      "postDate": "05/19/2025 13:44:13",
      "content": "<p>Hello Organizers,</p>\n<p>We want to highlight an observation about the ground truths you provided. According to your Phase 1 Evaluation script, the mesh unit was in meters, but the ground truth meshes you provided are not in the same unit. Can you please advise us on what we need to do in this case? Should we scale our meshes by 10 to match both mesh units?</p>\n<p>Looking forward to hearing from you soon!</p>",
      "rawMarkdown": "Hello Organizers,\n\nWe want to highlight an observation about the ground truths you provided. According to your Phase 1 Evaluation script, the mesh unit was in meters, but the ground truth meshes you provided are not in the same unit. Can you please advise us on what we need to do in this case? Should we scale our meshes by 10 to match both mesh units?\n\nLooking forward to hearing from you soon!",
      "votes": null
    },
    {
      "id": "3205239",
      "postDate": "05/19/2025 14:03:48",
      "content": "<p>Hi Umair,</p>\n<p>In Phase I, we use the posted calculate_volumes.py script and the meshes to generate GT. The units for the meshes are in fact mm^3, but the volume is evaluated in ml. You can see this by looking at the script. There is a line saying</p>\n<pre><code>         = mesh.volume * e-  # Assuming mesh.volume is in mm^\n</code></pre>\n<p>We used the same meshes and scripts to calculate volumes as well as calculate Chamfer distances. As long as you use our scripts, it should align how we evaluate in the end.</p>",
      "rawMarkdown": "Hi Umair,\n\nIn Phase I, we use the posted calculate_volumes.py script and the meshes to generate GT. The units for the meshes are in fact mm^3, but the volume is evaluated in ml. You can see this by looking at the script. There is a line saying\n```\n        volume_ml = mesh.volume * 1e-3  # Assuming mesh.volume is in mm^3\n\n```\n\nWe used the same meshes and scripts to calculate volumes as well as calculate Chamfer distances. As long as you use our scripts, it should align how we evaluate in the end.",
      "votes": null
    },
    {
      "id": "3205307",
      "postDate": "05/19/2025 16:15:28",
      "content": "<p>Thank you so much for the clarification!</p>\n<p>We also had the same situation last year, and you provided us with the \"ml\" meshes. <br>\nReference:<br>\n<a href=\"https://www.kaggle.com/competitions/cvpr-metafood-3d-food-reconstruction-challenge/discussion/508282#2843346\" target=\"_blank\">https://www.kaggle.com/competitions/cvpr-metafood-3d-food-reconstruction-challenge/discussion/508282#2843346</a></p>\n<p>If you want, we can scale up our meshes to match the GT meshes and then estimate the Transformation Matrices. Please let me know if this is okay so I can proceed accordingly.</p>\n<p>Thank you :)</p>",
      "rawMarkdown": "Thank you so much for the clarification!\n\nWe also had the same situation last year, and you provided us with the \"ml\" meshes. \nReference:\nhttps://www.kaggle.com/competitions/cvpr-metafood-3d-food-reconstruction-challenge/discussion/508282#2843346\n\nIf you want, we can scale up our meshes to match the GT meshes and then estimate the Transformation Matrices. Please let me know if this is okay so I can proceed accordingly.\n\nThank you :)",
      "votes": null
    },
    {
      "id": "3205314",
      "postDate": "05/19/2025 16:34:17",
      "content": "<p>you can provide me with the modified evaluation script with scaling separately added. I will run the script and select the lowest Chamfer distance. I will do the same for all the teams if they want it this way. Scaling can be achieved by modifying diagonal terms of your 4x4 transformation matrix.</p>",
      "rawMarkdown": "you can provide me with the modified evaluation script with scaling separately added. I will run the script and select the lowest Chamfer distance. I will do the same for all the teams if they want it this way. Scaling can be achieved by modifying diagonal terms of your 4x4 transformation matrix.",
      "votes": null
    },
    {
      "id": "3205328",
      "postDate": "05/19/2025 16:57:42",
      "content": "<p>OK Perfect! I understand, I'll estimate the Transformation Matrices accordingly.</p>",
      "rawMarkdown": "OK Perfect! I understand, I'll estimate the Transformation Matrices accordingly.",
      "votes": null
    },
    {
      "id": "3205651",
      "postDate": "05/20/2025 07:56:49",
      "content": "<p>Hello Organizers,<br>\nWe would like to ask when is the final deadline for submitting the Transformation Matrices, by 0:00 on May 21st?</p>",
      "rawMarkdown": "Hello Organizers,\nWe would like to ask when is the final deadline for submitting the Transformation Matrices, by 0:00 on May 21st?",
      "votes": null
    },
    {
      "id": "3205937",
      "postDate": "05/20/2025 15:13:37",
      "content": "<p>We use end of day 11:59PM AOE time. </p>",
      "rawMarkdown": "We use end of day 11:59PM AOE time.",
      "votes": null
    },
    {
      "id": "3207611",
      "postDate": "05/23/2025 03:50:18",
      "content": "<p>Hello Organizers,<br>\nWe would like to ask when is the final deadline for submitting the Transformation Matrices, by 0:00 on May 21st?</p>",
      "rawMarkdown": "Hello Organizers,\nWe would like to ask when is the final deadline for submitting the Transformation Matrices, by 0:00 on May 21st?",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 3205221,
      "author_name": "umairharoon",
      "author_url": "",
      "post_date": "05/19/2025 13:44:13",
      "content": "<p>Hello Organizers,</p>\n<p>We want to highlight an observation about the ground truths you provided. According to your Phase 1 Evaluation script, the mesh unit was in meters, but the ground truth meshes you provided are not in the same unit. Can you please advise us on what we need to do in this case? Should we scale our meshes by 10 to match both mesh units?</p>\n<p>Looking forward to hearing from you soon!</p>",
      "votes": null,
      "replies": [
        {
          "id": 3205239,
          "author_name": "metafoodcvpr",
          "author_url": "",
          "post_date": "05/19/2025 14:03:48",
          "content": "<p>Hi Umair,</p>\n<p>In Phase I, we use the posted calculate_volumes.py script and the meshes to generate GT. The units for the meshes are in fact mm^3, but the volume is evaluated in ml. You can see this by looking at the script. There is a line saying</p>\n<pre><code>         = mesh.volume * e-  # Assuming mesh.volume is in mm^\n</code></pre>\n<p>We used the same meshes and scripts to calculate volumes as well as calculate Chamfer distances. As long as you use our scripts, it should align how we evaluate in the end.</p>",
          "votes": null,
          "replies": [
            {
              "id": 3205307,
              "author_name": "umairharoon",
              "author_url": "",
              "post_date": "05/19/2025 16:15:28",
              "content": "<p>Thank you so much for the clarification!</p>\n<p>We also had the same situation last year, and you provided us with the \"ml\" meshes. <br>\nReference:<br>\n<a href=\"https://www.kaggle.com/competitions/cvpr-metafood-3d-food-reconstruction-challenge/discussion/508282#2843346\" target=\"_blank\">https://www.kaggle.com/competitions/cvpr-metafood-3d-food-reconstruction-challenge/discussion/508282#2843346</a></p>\n<p>If you want, we can scale up our meshes to match the GT meshes and then estimate the Transformation Matrices. Please let me know if this is okay so I can proceed accordingly.</p>\n<p>Thank you :)</p>",
              "votes": null,
              "replies": [
                {
                  "id": 3205314,
                  "author_name": "metafoodcvpr",
                  "author_url": "",
                  "post_date": "05/19/2025 16:34:17",
                  "content": "<p>you can provide me with the modified evaluation script with scaling separately added. I will run the script and select the lowest Chamfer distance. I will do the same for all the teams if they want it this way. Scaling can be achieved by modifying diagonal terms of your 4x4 transformation matrix.</p>",
                  "votes": null,
                  "replies": [
                    {
                      "id": 3205328,
                      "author_name": "umairharoon",
                      "author_url": "",
                      "post_date": "05/19/2025 16:57:42",
                      "content": "<p>OK Perfect! I understand, I'll estimate the Transformation Matrices accordingly.</p>",
                      "votes": null,
                      "replies": []
                    }
                  ]
                }
              ]
            }
          ]
        }
      ]
    },
    {
      "id": 3205651,
      "author_name": "wuhu648648",
      "author_url": "",
      "post_date": "05/20/2025 07:56:49",
      "content": "<p>Hello Organizers,<br>\nWe would like to ask when is the final deadline for submitting the Transformation Matrices, by 0:00 on May 21st?</p>",
      "votes": null,
      "replies": [
        {
          "id": 3205937,
          "author_name": "metafoodcvpr",
          "author_url": "",
          "post_date": "05/20/2025 15:13:37",
          "content": "<p>We use end of day 11:59PM AOE time. </p>",
          "votes": null,
          "replies": []
        }
      ]
    },
    {
      "id": 3207611,
      "author_name": "taurenus",
      "author_url": "",
      "post_date": "05/23/2025 03:50:18",
      "content": "<p>Hello Organizers,<br>\nWe would like to ask when is the final deadline for submitting the Transformation Matrices, by 0:00 on May 21st?</p>",
      "votes": null,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "3203935": "Dear Participants,\nThank you for your food model submissions. We are now advancing to Phase II of the competition, which will focus on chamfer distance evaluation.\n\n**Phase II Materials Available**\nWe have uploaded all necessary Phase II files to the \"**phase_2_data_and_scripts**\" folder, which you should now have access to.\n\n**Required Action: Submit transform.json Files**\nBefore we can proceed with distance measurements between ground truth and predicted models, **you must provide a transform.json file for each of your models** to ensure proper alignment with the ground truth.\n\n- An example transformation file is available in the \"**example_transform**\" folder for your reference\n- **The transform.json file must be submitted to us no later than May 21st, 2025**\n\n**Evaluation Process Clarification**\nThe \"phase_2_eval_scripts\" folder contains three evaluation scripts that will be used to determine Phase II rankings:\n\n1. **eval_obj_cd.py**: Measures chamfer distance without any transformation\n2. **eval_obj_cd_with_transform.py**: Measures distance after applying your transform.json files to predicted models\n3. **eval_obj_cd_with_transform_open3d.py**: Uses an alternative Python library to measure distance and apply transformation matrices\n\n**Important**: Your final evaluation score will be the lowest value produced by either the second or third script (eval_obj_cd_with_transform.py or eval_obj_cd_with_transform_open3d.py).\n\nPlease contact us if you have any questions about this procedure.\n\nThanks,\nMetaFood Workshop Organizing Committee",
    "3205221": "Hello Organizers,\n\nWe want to highlight an observation about the ground truths you provided. According to your Phase 1 Evaluation script, the mesh unit was in meters, but the ground truth meshes you provided are not in the same unit. Can you please advise us on what we need to do in this case? Should we scale our meshes by 10 to match both mesh units?\n\nLooking forward to hearing from you soon!",
    "3205239": "Hi Umair,\n\nIn Phase I, we use the posted calculate_volumes.py script and the meshes to generate GT. The units for the meshes are in fact mm^3, but the volume is evaluated in ml. You can see this by looking at the script. There is a line saying\n```\n        volume_ml = mesh.volume * 1e-3  # Assuming mesh.volume is in mm^3\n\n```\n\nWe used the same meshes and scripts to calculate volumes as well as calculate Chamfer distances. As long as you use our scripts, it should align how we evaluate in the end.",
    "3205307": "Thank you so much for the clarification!\n\nWe also had the same situation last year, and you provided us with the \"ml\" meshes. \nReference:\nhttps://www.kaggle.com/competitions/cvpr-metafood-3d-food-reconstruction-challenge/discussion/508282#2843346\n\nIf you want, we can scale up our meshes to match the GT meshes and then estimate the Transformation Matrices. Please let me know if this is okay so I can proceed accordingly.\n\nThank you :)",
    "3205314": "you can provide me with the modified evaluation script with scaling separately added. I will run the script and select the lowest Chamfer distance. I will do the same for all the teams if they want it this way. Scaling can be achieved by modifying diagonal terms of your 4x4 transformation matrix.",
    "3205328": "OK Perfect! I understand, I'll estimate the Transformation Matrices accordingly.",
    "3205651": "Hello Organizers,\nWe would like to ask when is the final deadline for submitting the Transformation Matrices, by 0:00 on May 21st?",
    "3205937": "We use end of day 11:59PM AOE time.",
    "3207611": "Hello Organizers,\nWe would like to ask when is the final deadline for submitting the Transformation Matrices, by 0:00 on May 21st?"
  },
  "source": "meta"
}