In critical applications such as **medical diagnostics**, **autonomous driving**, and **financial systems**, the risks associated with incorrect predictions can be significant. For instance, a self-driving car that fails to detect an obstacle may cause a serious accident, while an incorrect medical diagnosis could lead to harmful treatments. Thus, models must make predictions only when they are highly confident. If they lack sufficient confidence, they should abstain from predicting.

### Assignment Objective

This assignment focuses on developing machine learning models that achieve **high overall accuracy** while ensuring **high-confidence predictions**. You will train models using the **CIFAR-100 dataset**, consisting of **60,000 32x32 color images** across **100 classes**. Out of these, **40,000 images will be used for training**. Each image is associated with fine-grained class labels like "apple," "car," or "tree."

You will be given an **"accuracy threshold"** as input. Your goal is to make predictions on the test set such that:
1. The overall test accuracy for all classes exceeds the specified **accuracy threshold**.
2. The number of examples classified into any class is maximized. 

If your model cannot confidently predict a class label, it should indicate a **failure to predict** by assigning a label of **-1**.

- The custom metric rewards confident predictions and penalizes uncertain ones.
- Class-wise accuracy is calculated to distinguish between high and low accuracy classes.
- The final score is based on the counts of high and low accuracy predictions.

---

### Evaluation Metric

Model performance will be evaluated using a custom metric that rewards confident predictions and penalizes cases of uncertainty. Predictions with a class label of **-1** will not be considered for evaluation, focusing on reliability and accuracy.

### Metric Components

**Notations:**
- **ŷᵢ**: Predicted label for the i-th example.
- **yᵢ**: True label for the i-th example.
- **𝒫ₜ**: Set of examples predicted as class c.
- **α**: Accuracy threshold distinguishing high and low accuracy classes.
- **γ**: Penalty applied to low-accuracy counts.

**Class-wise Accuracy:**

The accuracy for each class **c** is calculated as:

```
Accuracy(c) = (Σ [ŷᵢ = yᵢ] for i ∈ 𝒫ₜ) / |𝒫ₜ|
```

**Classification Metrics:**

- **High Accuracy Class Count:**

```
High Accuracy Count = Σ [1 if Accuracy(c) ≥ α] × |𝒫ₜ|
```

- **Low Accuracy Class Count:**

```
Low Accuracy Count = Σ [1 if Accuracy(c) < α] × |𝒫ₜ|
```

**Final Evaluation Score:**

The overall score is calculated as:

```
Final Score = High Accuracy Count - γ × Low Accuracy Count
```

For the **Kaggle leaderboard**, use **α = 0.99** and **γ = 5**. For private testing, **α** may range from **[0.98, 0.999]**. Aim to minimize the low accuracy count to make the performance independent of **γ**.

Your submission file should include two columns:
- `ID`: Unique identifier for each test example, matching IDs from the `test.pkl` file.
- `Predicted_label`: Class label in the range of [-1, 99].

The file format should be:

```
ID,Predicted_label
1,42
2,15
3,-1
...
```

**Important**: Do not shuffle the IDs in your submission file to ensure correct alignment for evaluation.

### Tentative Submission Deadlines
1. **Submission 1**: 6th October
2. **Submission 2**: 20th October
3. **Submission 3**: 27th October
4. **Submission 4**: 10th November

All deadlines are inclusive, and submissions must be made by 7 PM on the respective date.

### Rules

1. Use any open-source tool or library but mention it in your report. Pre-trained weights are not allowed.
2. Training must not exceed 5 hours on the provided HPC.
3. Prediction must not exceed 30 minutes on the provided HPC.
4. Model file (model.pth) must be under 1 GB.
5. Ensure reproducibility by using fixed seed values in your code.
6. Any misconduct, including hard-coding results or using disallowed models, will result in severe penalties.

Good luck, and happy coding!