{ "id": 417388, "title": "Instructions for valid submission", "url": "/competitions/budgeted-model-inference-iccv-2023-rcv-workshop/discussion/417388", "author_name": "Gowreesh Mago", "post_date": "2023-06-15T13:53:42.135000", "votes": 3, "comment_count": 21, "views": 0, "content": "

We received some invlaid submissions so here are the set of instructions to be followed for a valid one.
\nThe second Phase of the competition involves testing the code submissions on UltraMNIST test data created. Scores obtained on this dataset will decide the final position on the Phase 2 leaderboard. Participants are requested to follow the instructions provided here. Submissions violating the instructions will not find their positions on the leaderboard.

\n
    \n
  1. Ensure that the code submitted runs on the environment provided in the yml file inside the dataset. For participants using their machines to train the model, they are requested to set up this environment and test their code before submitting.
    2. \n
  2. The submissions must be done in a zip file with the team name attached as in TEAM_NAME.zip. This TEAM_NAME will be displayed on the Phase 2 leaderboard.
    3. \n
  3. The zip file should only contain the model and training script with the name main.py. Participants are requested to hardcode the parameters inside the script as no config files will be accepted.
    4. \n
  4. The script will be tested against the data directory containing data for Phase 2. The data directory contains 1 folder: test, which will be used for inference. Participants are requested to create dataloaders accordingly or use the ones provided in the main.py script inside the inference_submission dataset
    5. \n
  5. Participants are requested to include the mandatory arguments root_dir and output_dir inside their script. root_dir will be used to provide the path to the dataset and output_dir will be used to dump submission.csv.
    6. \n
  6. The script is expected to fetch the model from the same directory where the script is located and test data from the data directory, complete inference on the test data provided, and dump the predictions as submission.csv inside the output_dir provided as the argument. Reference for data loaders is provided in the main.py script.
    7. \n
  7. The competition puts forward the inference constraint:
    8. \n
  8. To ensure the memory constraint is not violated participants are requested to include the following snippet inside their code.
    \ntup = torch.cuda.mem_get_info()\ntorch.cuda.set_per_process_memory_fraction((16*(1024)**3)/tup[1], 0)
    9. \n
  9. It is expected that the participants include the logic to read the model weights from the same directory where the main.py script is located. To find the directory where the script is located one can use the following snippet as used in the main.py script:
    \nMODEL_LOAD_DIR = os.path.dirname(os.path.abspath(__file__))
    10. \n
  10. So in summary:
    11. \n
", "messages": [ { "id": 2303829, "postDate": "2023-06-15T13:53:42.137Z", "content": "

We received some invlaid submissions so here are the set of instructions to be followed for a valid one.
\nThe second Phase of the competition involves testing the code submissions on UltraMNIST test data created. Scores obtained on this dataset will decide the final position on the Phase 2 leaderboard. Participants are requested to follow the instructions provided here. Submissions violating the instructions will not find their positions on the leaderboard.

\n
    \n
  1. Ensure that the code submitted runs on the environment provided in the yml file inside the dataset. For participants using their machines to train the model, they are requested to set up this environment and test their code before submitting.
    2. \n
  2. The submissions must be done in a zip file with the team name attached as in TEAM_NAME.zip. This TEAM_NAME will be displayed on the Phase 2 leaderboard.
    3. \n
  3. The zip file should only contain the model and training script with the name main.py. Participants are requested to hardcode the parameters inside the script as no config files will be accepted.
    4. \n
  4. The script will be tested against the data directory containing data for Phase 2. The data directory contains 1 folder: test, which will be used for inference. Participants are requested to create dataloaders accordingly or use the ones provided in the main.py script inside the inference_submission dataset
    5. \n
  5. Participants are requested to include the mandatory arguments root_dir and output_dir inside their script. root_dir will be used to provide the path to the dataset and output_dir will be used to dump submission.csv.
    6. \n
  6. The script is expected to fetch the model from the same directory where the script is located and test data from the data directory, complete inference on the test data provided, and dump the predictions as submission.csv inside the output_dir provided as the argument. Reference for data loaders is provided in the main.py script.
    7. \n
  7. The competition puts forward the inference constraint:
    8. \n
  8. To ensure the memory constraint is not violated participants are requested to include the following snippet inside their code.
    \ntup = torch.cuda.mem_get_info()\ntorch.cuda.set_per_process_memory_fraction((16*(1024)**3)/tup[1], 0)
    9. \n
  9. It is expected that the participants include the logic to read the model weights from the same directory where the main.py script is located. To find the directory where the script is located one can use the following snippet as used in the main.py script:
    \nMODEL_LOAD_DIR = os.path.dirname(os.path.abspath(__file__))
    10. \n
  10. So in summary:
    11. \n
", "rawMarkdown": "We received some invlaid submissions so here are the set of instructions to be followed for a valid one.\nThe second Phase of the competition involves testing the code submissions on UltraMNIST test data created. Scores obtained on this dataset will decide the final position on the Phase 2 leaderboard. Participants are requested to follow the instructions provided here. Submissions violating the instructions will not find their positions on the leaderboard.\n1. Ensure that the code submitted runs on the environment provided in the [yml](https://www.kaggle.com/competitions/budgeted-model-inference-iccv-2023-rcv-workshop/data?select=rcv.yml) file inside the dataset. For participants using their machines to train the model, they are requested to set up this environment and test their code before submitting.\n2. The submissions must be done in a zip file with the team name attached as in **TEAM_NAME.zip**. This TEAM_NAME will be displayed on the Phase 2 leaderboard.\n3. The zip file should only contain the **model** and **training script** with the name main.py. Participants are requested to hardcode the parameters inside the script as **no config files will be accepted.** \n4. The script will be tested against the data directory containing data for Phase 2. The data directory contains 1 folder: test, which will be used for inference. Participants are requested to create dataloaders accordingly or use the ones provided in the main.py script inside the [inference_submission](https://www.kaggle.com/datasets/mlnoob327/inference-submission) dataset \n5. Participants are requested to include the mandatory arguments **root_dir** and **output_dir** inside their script. root_dir will be used to provide the path to the dataset and output_dir will be used to dump submission.csv.\n6. The script is expected to fetch the model from the same directory where the script is located and test data from the data directory, complete inference on the test data provided, and dump the predictions as **submission.csv** inside the output_dir provided as the argument. Reference for data loaders is provided in the main.py script.\n7. The competition puts forward the inference constraint: \n - Inferring with a budgeted CUDA memory of 16GB.\n8. To ensure the memory constraint is not violated participants are requested to include the following snippet inside their code. \n`tup = torch.cuda.mem_get_info()\ntorch.cuda.set_per_process_memory_fraction((16*(1024)**3)/tup[1], 0)`\n9. It is expected that the participants include the logic to read the model weights from the same directory where the main.py script is located. To find the directory where the script is located one can use the following snippet as used in the main.py script: \n`MODEL_LOAD_DIR = os.path.dirname(os.path.abspath(__file__))`\n10. So in summary:\n - Set up the environment.\n - Write the inference script main.py which can accept root_dir and output_dir.\n - Name the output csv as submission.csv.\n - Include code snippets to ensure memory constraint.\n - Ensure that script is able to load model weights in the same directory as the main.py.\n - Create a zip file with TEAM_NAME.zip, containing the model and main.py.\n - Submit the zip file on the [submission link](https://forms.gle/CwfbtZd4k6FwmEFj7):\n", "votes": 3 }, { "id": 2365529, "postDate": "2023-07-30T10:01:17.787Z", "content": "

Hi, the competition deadline is approaching, I have some questions about the Final submission and hope to get your answers;

\n
    \n
  1. Which time zone (2023-8-1) will be used for the submission time of the final competition. And what is the submission way, whether it is consistent with phase2;
    2. \n
  2. Whether the final submission is a must, or we can directly inherit the phase2 leaderboard score;
    3. \n
  3. How many solutions can each contestant submit? Because it is mentioned in the 【Overview-Description】 section of the competition kaggle website -- \"Every team will be able to submit their two best solutions in one instance.\"
    4. \n
\n

Thank you for your support.

", "rawMarkdown": "Hi, the competition deadline is approaching, I have some questions about the Final submission and hope to get your answers;\n\n1. Which time zone (2023-8-1) will be used for the submission time of the final competition. And what is the submission way, whether it is consistent with phase2;\n2. Whether the final submission is a must, or we can directly inherit the phase2 leaderboard score;\n3. How many solutions can each contestant submit? Because it is mentioned in the **【Overview-Description】** section of the competition kaggle website -- \"Every team will be able to submit their two best solutions in one instance.\"\n\nThank you for your support.\n", "replies": [ { "id": 2366158, "postDate": "2023-07-30T19:19:35.767Z", "content": "

Hi @bh230522,

\n
    \n
  1. We will be considering submissions made through the form till 2023-7-30 11:59 PM UTC. The last submission will be considered as the final score.
    2. \n
  2. Winners will be required to submit their full training + inference code that led to the leaderboard to ensure reproducibility. They will get separate emails for that.
    3. \n
  3. You can ignore that comment it was only relevant when the leaderboard frequency was very low.
    4. \n
\n

Thanks!

", "rawMarkdown": "Hi @bh230522,\n1. We will be considering submissions made through the form till 2023-7-30 11:59 PM UTC. The last submission will be considered as the final score. \n2. Winners will be required to submit their full training + inference code that led to the leaderboard to ensure reproducibility. They will get separate emails for that. \n3. You can ignore that comment it was only relevant when the leaderboard frequency was very low.\n\nThanks!", "replies": [ { "id": 2367249, "postDate": "2023-07-31T12:57:52.747Z", "content": "

Thanks for your reply, I misunderstood that we need to resubmit a code in 8.1 as an evaluation of the final result

", "rawMarkdown": "Thanks for your reply, I misunderstood that we need to resubmit a code in 8.1 as an evaluation of the final result" } ] } ] }, { "id": 2365255, "postDate": "2023-07-30T05:38:42.997Z", "content": "

Hi, I submitted my code two days ago, but the leaderboard is not yet updated. It would be helpful if you updated the leaderboard, as we can make further changes to our code. Thank you.

", "rawMarkdown": "Hi, I submitted my code two days ago, but the leaderboard is not yet updated. It would be helpful if you updated the leaderboard, as we can make further changes to our code. Thank you.", "replies": [ { "id": 2365282, "postDate": "2023-07-30T06:11:04.200Z", "content": "

Hi, the lb has beed updated twice since 2 days can you check again. Thanks.

", "rawMarkdown": "Hi, the lb has beed updated twice since 2 days can you check again. Thanks.", "replies": [ { "id": 2365290, "postDate": "2023-07-30T06:16:34.013Z", "content": "

Hi we actually missed to update your score, its updated now, thanks for pointing out.

", "rawMarkdown": "Hi we actually missed to update your score, its updated now, thanks for pointing out." }, { "id": 2365291, "postDate": "2023-07-30T06:16:45.393Z", "content": "

Thank you for your response. But I cannot find my name on the leaderboard. I submitted my code using the Google Form link provided. Can you please let me know if you have received my submission? Thank you. 

", "rawMarkdown": "Thank you for your response. But I cannot find my name on the leaderboard. I submitted my code using the Google Form link provided. Can you please let me know if you have received my submission? Thank you. ", "votes": 1 }, { "id": 2365293, "postDate": "2023-07-30T06:18:16.837Z", "content": "

Thank you for your response.

", "rawMarkdown": "Thank you for your response." } ] } ] }, { "id": 2350131, "postDate": "2023-07-19T03:13:24.463Z", "content": "

Hello, I have submit a better version of my code with checkpoint but the leaderboard is not updated.

", "rawMarkdown": "Hello, I have submit a better version of my code with checkpoint but the leaderboard is not updated. ", "replies": [ { "id": 2351722, "postDate": "2023-07-20T10:49:54.627Z", "content": "

hi can you please recheck, they should have been updated.

", "rawMarkdown": "hi can you please recheck, they should have been updated.", "replies": [ { "id": 2351723, "postDate": "2023-07-20T10:51:16.447Z", "content": "

If you still believe the score is not latest you can resubmit now, the leaderboard will be updated tonight again.

", "rawMarkdown": "If you still believe the score is not latest you can resubmit now, the leaderboard will be updated tonight again." }, { "id": 2352438, "postDate": "2023-07-21T01:27:12.940Z", "content": "

Hi, i did resubmit but it is still. I used the same model with better performance, therefore I believe there is no way the leaderboard score is lower than the previous submission.

", "rawMarkdown": "Hi, i did resubmit but it is still. I used the same model with better performance, therefore I believe there is no way the leaderboard score is lower than the previous submission." }, { "id": 2352448, "postDate": "2023-07-21T01:51:03.997Z", "content": "

Hi Binh, just want to point out that the accuracy being calculated is on the private test set. So there is possibility of getting lower leaderboard score even if the score increases on the public test set. Hope this clarifies your doubt.

", "rawMarkdown": "Hi Binh, just want to point out that the accuracy being calculated is on the private test set. So there is possibility of getting lower leaderboard score even if the score increases on the public test set. Hope this clarifies your doubt." }, { "id": 2352472, "postDate": "2023-07-21T02:51:34.187Z", "content": "

Hi, what is the update frequency of the phase2 leaderboard now? Is there a fixed update time point?

", "rawMarkdown": "Hi, what is the update frequency of the phase2 leaderboard now? Is there a fixed update time point?" }, { "id": 2352605, "postDate": "2023-07-21T06:13:34.150Z", "content": "

Update frequency is once per day, but time point is not fixed (i am trying to do this though). For few days its been around evening-night.

", "rawMarkdown": "Update frequency is once per day, but time point is not fixed (i am trying to do this though). For few days its been around evening-night. " }, { "id": 2356213, "postDate": "2023-07-24T03:15:39.357Z", "rawMarkdown": "", "isDeleted": true }, { "id": 2356754, "postDate": "2023-07-24T11:06:10.103Z", "content": "

Hello, it has been about two days since the last leaderboard update. I wonder if something went wrong? If the leaderboard can be updated faster, it will be very helpful for us to adjust the experiment strategy. Thanks a lot for the efforts you put into the competition!

", "rawMarkdown": "Hello, it has been about two days since the last leaderboard update. I wonder if something went wrong? If the leaderboard can be updated faster, it will be very helpful for us to adjust the experiment strategy. Thanks a lot for the efforts you put into the competition!" }, { "id": 2357465, "postDate": "2023-07-24T20:04:03.040Z", "content": "

Hi everything is running fine, the lb is updated just now. Will try to update it every night. Thanks for your patience.

", "rawMarkdown": "Hi everything is running fine, the lb is updated just now. Will try to update it every night. Thanks for your patience." }, { "id": 2359077, "postDate": "2023-07-26T01:13:53.223Z", "content": "

hello. What is the final submission deadline for submitting through google form instead of kaggle submission? The time difference is different, so I need the exact time.

", "rawMarkdown": "hello. What is the final submission deadline for submitting through google form instead of kaggle submission? The time difference is different, so I need the exact time." }, { "id": 2360400, "postDate": "2023-07-26T18:00:10.310Z", "content": "

we will only consider submissions made till 30/07/2023 11:59 PM IST. Thanks!

", "rawMarkdown": "we will only consider submissions made till 30/07/2023 11:59 PM IST. Thanks!" } ] } ] }, { "id": 2360933, "postDate": "2023-07-27T05:37:17.980Z", "rawMarkdown": "", "isDeleted": true } ], "comments": [ { "id": 2365529, "author_name": "yyk", "author_url": "", "post_date": "2023-07-30T10:01:17.787000", "content": "

Hi, the competition deadline is approaching, I have some questions about the Final submission and hope to get your answers;

\n
    \n
  1. Which time zone (2023-8-1) will be used for the submission time of the final competition. And what is the submission way, whether it is consistent with phase2;
    2. \n
  2. Whether the final submission is a must, or we can directly inherit the phase2 leaderboard score;
    3. \n
  3. How many solutions can each contestant submit? Because it is mentioned in the 【Overview-Description】 section of the competition kaggle website -- \"Every team will be able to submit their two best solutions in one instance.\"
    4. \n
\n

Thank you for your support.

", "votes": 0, "replies": [ { "id": 2366158, "author_name": "Rishabh Tiwari", "author_url": "", "post_date": "2023-07-30T19:19:35.767000", "content": "

Hi @bh230522,

\n
    \n
  1. We will be considering submissions made through the form till 2023-7-30 11:59 PM UTC. The last submission will be considered as the final score.
    2. \n
  2. Winners will be required to submit their full training + inference code that led to the leaderboard to ensure reproducibility. They will get separate emails for that.
    3. \n
  3. You can ignore that comment it was only relevant when the leaderboard frequency was very low.
    4. \n
\n

Thanks!

", "votes": 0, "replies": [ { "id": 2367249, "author_name": "yyk", "author_url": "", "post_date": "2023-07-31T12:57:52.747000", "content": "

Thanks for your reply, I misunderstood that we need to resubmit a code in 8.1 as an evaluation of the final result

", "votes": 0, "replies": [] } ] } ] }, { "id": 2365255, "author_name": "Abinila Siva", "author_url": "", "post_date": "2023-07-30T05:38:42.997000", "content": "

Hi, I submitted my code two days ago, but the leaderboard is not yet updated. It would be helpful if you updated the leaderboard, as we can make further changes to our code. Thank you.

", "votes": 0, "replies": [ { "id": 2365282, "author_name": "Rishabh Tiwari", "author_url": "", "post_date": "2023-07-30T06:11:04.200000", "content": "

Hi, the lb has beed updated twice since 2 days can you check again. Thanks.

", "votes": 0, "replies": [ { "id": 2365290, "author_name": "Rishabh Tiwari", "author_url": "", "post_date": "2023-07-30T06:16:34.013000", "content": "

Hi we actually missed to update your score, its updated now, thanks for pointing out.

", "votes": 0, "replies": [] }, { "id": 2365291, "author_name": "Abinila Siva", "author_url": "", "post_date": "2023-07-30T06:16:45.393000", "content": "

Thank you for your response. But I cannot find my name on the leaderboard. I submitted my code using the Google Form link provided. Can you please let me know if you have received my submission? Thank you. 

", "votes": 1, "replies": [] }, { "id": 2365293, "author_name": "Abinila Siva", "author_url": "", "post_date": "2023-07-30T06:18:16.837000", "content": "

Thank you for your response.

", "votes": 0, "replies": [] } ] } ] }, { "id": 2350131, "author_name": "Binh Nguyen", "author_url": "", "post_date": "2023-07-19T03:13:24.463000", "content": "

Hello, I have submit a better version of my code with checkpoint but the leaderboard is not updated.

", "votes": 0, "replies": [ { "id": 2351722, "author_name": "Rishabh Tiwari", "author_url": "", "post_date": "2023-07-20T10:49:54.627000", "content": "

hi can you please recheck, they should have been updated.

", "votes": 0, "replies": [ { "id": 2351723, "author_name": "Rishabh Tiwari", "author_url": "", "post_date": "2023-07-20T10:51:16.447000", "content": "

If you still believe the score is not latest you can resubmit now, the leaderboard will be updated tonight again.

", "votes": 0, "replies": [] }, { "id": 2352438, "author_name": "Binh Nguyen", "author_url": "", "post_date": "2023-07-21T01:27:12.940000", "content": "

Hi, i did resubmit but it is still. I used the same model with better performance, therefore I believe there is no way the leaderboard score is lower than the previous submission.

", "votes": 0, "replies": [] }, { "id": 2352448, "author_name": "Rishabh Tiwari", "author_url": "", "post_date": "2023-07-21T01:51:03.997000", "content": "

Hi Binh, just want to point out that the accuracy being calculated is on the private test set. So there is possibility of getting lower leaderboard score even if the score increases on the public test set. Hope this clarifies your doubt.

", "votes": 0, "replies": [] }, { "id": 2352472, "author_name": "yyk", "author_url": "", "post_date": "2023-07-21T02:51:34.187000", "content": "

Hi, what is the update frequency of the phase2 leaderboard now? Is there a fixed update time point?

", "votes": 0, "replies": [] }, { "id": 2352605, "author_name": "Rishabh Tiwari", "author_url": "", "post_date": "2023-07-21T06:13:34.150000", "content": "

Update frequency is once per day, but time point is not fixed (i am trying to do this though). For few days its been around evening-night.

", "votes": 0, "replies": [] }, { "id": 2356213, "author_name": "", "author_url": "", "post_date": "2023-07-24T03:15:39.357000", "content": "", "votes": 0, "replies": [] }, { "id": 2356754, "author_name": "yyk", "author_url": "", "post_date": "2023-07-24T11:06:10.103000", "content": "

Hello, it has been about two days since the last leaderboard update. I wonder if something went wrong? If the leaderboard can be updated faster, it will be very helpful for us to adjust the experiment strategy. Thanks a lot for the efforts you put into the competition!

", "votes": 0, "replies": [] }, { "id": 2357465, "author_name": "Rishabh Tiwari", "author_url": "", "post_date": "2023-07-24T20:04:03.040000", "content": "

Hi everything is running fine, the lb is updated just now. Will try to update it every night. Thanks for your patience.

", "votes": 0, "replies": [] }, { "id": 2359077, "author_name": "hominjung", "author_url": "", "post_date": "2023-07-26T01:13:53.223000", "content": "

hello. What is the final submission deadline for submitting through google form instead of kaggle submission? The time difference is different, so I need the exact time.

", "votes": 0, "replies": [] }, { "id": 2360400, "author_name": "Rishabh Tiwari", "author_url": "", "post_date": "2023-07-26T18:00:10.310000", "content": "

we will only consider submissions made till 30/07/2023 11:59 PM IST. Thanks!

", "votes": 0, "replies": [] } ] } ] }, { "id": 2360933, "author_name": "", "author_url": "", "post_date": "2023-07-27T05:37:17.980000", "content": "", "votes": 0, "replies": [] } ], "raw_markdown_by_id": { "2303829": "We received some invlaid submissions so here are the set of instructions to be followed for a valid one.\nThe second Phase of the competition involves testing the code submissions on UltraMNIST test data created. Scores obtained on this dataset will decide the final position on the Phase 2 leaderboard. Participants are requested to follow the instructions provided here. Submissions violating the instructions will not find their positions on the leaderboard.\n1. Ensure that the code submitted runs on the environment provided in the [yml](https://www.kaggle.com/competitions/budgeted-model-inference-iccv-2023-rcv-workshop/data?select=rcv.yml) file inside the dataset. For participants using their machines to train the model, they are requested to set up this environment and test their code before submitting.\n2. The submissions must be done in a zip file with the team name attached as in **TEAM_NAME.zip**. This TEAM_NAME will be displayed on the Phase 2 leaderboard.\n3. The zip file should only contain the **model** and **training script** with the name main.py. Participants are requested to hardcode the parameters inside the script as **no config files will be accepted.** \n4. The script will be tested against the data directory containing data for Phase 2. The data directory contains 1 folder: test, which will be used for inference. Participants are requested to create dataloaders accordingly or use the ones provided in the main.py script inside the [inference_submission](https://www.kaggle.com/datasets/mlnoob327/inference-submission) dataset \n5. Participants are requested to include the mandatory arguments **root_dir** and **output_dir** inside their script. root_dir will be used to provide the path to the dataset and output_dir will be used to dump submission.csv.\n6. The script is expected to fetch the model from the same directory where the script is located and test data from the data directory, complete inference on the test data provided, and dump the predictions as **submission.csv** inside the output_dir provided as the argument. Reference for data loaders is provided in the main.py script.\n7. The competition puts forward the inference constraint: \n - Inferring with a budgeted CUDA memory of 16GB.\n8. To ensure the memory constraint is not violated participants are requested to include the following snippet inside their code. \n`tup = torch.cuda.mem_get_info()\ntorch.cuda.set_per_process_memory_fraction((16*(1024)**3)/tup[1], 0)`\n9. It is expected that the participants include the logic to read the model weights from the same directory where the main.py script is located. To find the directory where the script is located one can use the following snippet as used in the main.py script: \n`MODEL_LOAD_DIR = os.path.dirname(os.path.abspath(__file__))`\n10. So in summary:\n - Set up the environment.\n - Write the inference script main.py which can accept root_dir and output_dir.\n - Name the output csv as submission.csv.\n - Include code snippets to ensure memory constraint.\n - Ensure that script is able to load model weights in the same directory as the main.py.\n - Create a zip file with TEAM_NAME.zip, containing the model and main.py.\n - Submit the zip file on the [submission link](https://forms.gle/CwfbtZd4k6FwmEFj7):\n", "2365529": "Hi, the competition deadline is approaching, I have some questions about the Final submission and hope to get your answers;\n\n1. Which time zone (2023-8-1) will be used for the submission time of the final competition. And what is the submission way, whether it is consistent with phase2;\n2. Whether the final submission is a must, or we can directly inherit the phase2 leaderboard score;\n3. How many solutions can each contestant submit? Because it is mentioned in the **【Overview-Description】** section of the competition kaggle website -- \"Every team will be able to submit their two best solutions in one instance.\"\n\nThank you for your support.\n", "2365255": "Hi, I submitted my code two days ago, but the leaderboard is not yet updated. It would be helpful if you updated the leaderboard, as we can make further changes to our code. Thank you.", "2350131": "Hello, I have submit a better version of my code with checkpoint but the leaderboard is not updated. ", "2360933": "" } }