## Motivation

Despite the success of CNNs and, more recently, transformers, the scale of deep learning models has grown exponentially due to the rapid development of computational resources. Going forward, it is critical to focus on the practical training and inference efficiency of these models, as well as ensuring that even the largest models have viable real-world applications. Moreover, compute efficiency of computer vision models is becoming increasingly crucial for downstream tasks such as segmentation, object tracking, and action recognition, among others.

There exist several different directions towards making deep learning efficient for computer vision, thereby leading to a reduction in the required computational memory or the associated training and inference time. This competition will aim at bringing research efforts into practice for one such direction: *budget-aware model inference*. Most research that exists focuses on making deep learning methods efficient, however, the resource associated with real-world AI devices can vary drastically, and a method termed efficient for a certain choice of resource budgets might be completely inefficient for a different one. This competition will also focus on methods that aim at performing budget-aware model inference in best possible ways under limited GPU memory memory as well as limited inference time.

**This challenge is part of the ICCV 2023 Resource-Efficient Deep Learning (RCV) Workshop to be held between Oct 2-5, 2023 in Paris, France.**

## Problem description
As part of this budgeted inference challenge, we pose the problem of the classification of UltraMNIST digits. The problem at test time needs to be solved using limited GPU compute memory as well as time. The UltraMNIST dataset used here is an adapted variant of [UltraMNIST dataset](https://www.kaggle.com/competitions/ultra-mnist/data) comprises images of size  with 3-5 digits per image. Each of these digits has been extracted from the original MNIST dataset. Your task is to predict the sum of the digits per image, and this number can be anything from 0 to 27. However there are certain rules to be respected to qualify as a winning solution.

1. For final evaluation, each submitted inference script along with trained models weights will be used to evaluate on a separate private test set than those revealed here. The inference speed and accuracy score on this corresponding test set will be used to decide the final score of the model.
2. All the models will be inferenced on a V100 GPU with a memory constraint of 16 GB.
3. Together with your code (only python scripts + one model weights file), you will be required to submit the run environment as a .yml that will be used to run the code. (we provide a basic .yml file environment in the code section; use this a starting point and only add packages if you need to.)
4. The main script should be named main.py and should take only one command line argument --data_dir to produce submission.csv in the current directory.

This competition will be running in two phases: **Phase I** and **Phase II**.

**Phase I:** This phase is a standard classification problem where the goal is to maximize the accuracy on the test set and improve the ranking on the leaderboard. This phase of the competition will be hosted here on Kaggle and the leaderboard of Kaggle will be used to rank the solutions. Note that this phase will only use accuracy as the evaluation measure.  Although this phase does not take the GPU memory and inference speed into consideration, participants are recommended to adhere to an approximate estimate and use the leaderboard to assess the performance of their efficient models that could pass on to the next Phase. Using very large networks that has slow inference speed, is highly discouraged since this will improve your ranking in Phase I, but will not be able to qualify you for Phase II.

**Phase II:** We will post a baseline solution and all solutions that obtain an accuracy higher than baseline accuracy will be eligible to submit their solutions for Phase II. For details on the evaluation of the solutions for this phase, see the final evaluation details describe for the final evaluation above. This phase will be in an offline mode and results can be viewed at [Phase II Leaderboard](https://sites.google.com/view/rcv2023/leaderboard-workshop-challenges). We will keep updating the leaderboard on a rolling basis, however, every solution can be expected to get reflected on the leaderboard in 5 days from the day of the submission. Every team will be able to submit their two best solutions in one instance. For the start date of this phase, check the timeline page.

## Challenge Organizer team
For details related to the organizing team, see https://sites.google.com/view/rcv2023/organizing-committee