{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# Instructions:\n* This notebook attempts to simulate the caller script which will be used to evaluate the submissions for Phase 2 of the competition. A part of the code used to evaluate is presented here. \n* The submission must be done in a zip file which should indicate the team name in the following format, **TEAM_NAME.zip** and should contain **only the models and main.py**, the script to be used for inference and not training. It is expected that the **models are in the same directory as the script** and the code is equipped with the capability to load the model weights. For more reference as to how this can be achieved, one may refer to the main.py attached with the dataset.\n* The expected input for the script is only the root_dir and output_dir argument, which will  be the only arguments that are passed for running the script. The other arguments provided are for faster experimentation for training. If the participants wish to use values different from what are provided in the training script, they are requested to modify the same in the default args or use them as hardcoded constants. **No config files will be accepted**, as submisison of a zip file containing only the 'main.py' and models is expected.\n* The output directory is expected to have the final 'submission.csv', which is the output of the model after inference on the test folder inside the 'root_dir'. This submission will be used to grade the team on the leaderboard. The time calculated here will be incorporated in the metric to finalize the position on the leaderboard.","metadata":{}},{"cell_type":"code","source":"import time\nimport os\nimport pandas as pd\nfrom glob import glob","metadata":{"execution":{"iopub.status.busy":"2023-06-07T13:45:06.191899Z","iopub.execute_input":"2023-06-07T13:45:06.192532Z","iopub.status.idle":"2023-06-07T13:45:06.204281Z","shell.execute_reply.started":"2023-06-07T13:45:06.192500Z","shell.execute_reply":"2023-06-07T13:45:06.203104Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"directories = ['/kaggle/input/inference-submission/']\nfor directory in directories:\n    begin = time.time()\n    os.system(f'python {directory}/main.py --root_dir /kaggle/input/budgeted-model-inference-iccv-2023-rcv-workshop/umnist_iccv_1024/umnist_iccv_1024 --output_dir ./')\n    end = time.time()\n    time_taken = end-begin\n    print(f'Time taken was {time_taken}')","metadata":{"execution":{"iopub.status.busy":"2023-06-07T13:45:06.206204Z","iopub.execute_input":"2023-06-07T13:45:06.206850Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"","metadata":{},"execution_count":null,"outputs":[]}]}