Grading will be explained below, but we want to make clear that even though Kaggle presents things as a leaderboard, you are NOT being graded competitively. The leaderboard is supposed to be a fun way to see how your models compare to others and is not meant to imply that you are being graded based on your performance relative to others; as mentioned below, your grade is based on the accuracy of your model.

#Evaluation
The evaluation metric used for this assignment is classification accuracy

![](https://latex2image-output.s3.amazonaws.com/img-4EArwe6TMvhs.png)

#Submit to Kaggle
For your submission to Kaggle, you only need to submit a CSV file that has the Id and Category for each row in the test dataset. It will use the userid_DI and certified columns to compare to the true answers in order to evaluate your model.

| userid_DI | certified |
|---|---|
| 123 | 1 |
| 45 | 0 |
| 678 | 0 |
|...|...|

To simplify getting the Kaggle format right, we have provided a code snippet that you should use to save the Kaggle submission. You will need to write the code in the middle to make the predictions on the data, but there should be no need to modify the code to save the DataFrame to a file. This code is already included in the [Google Colab starter code](https://colab.research.google.com/drive/1VxAARGUpYPNU4640Q5RB0CxHjN2Wns9H?usp=sharing).

```
df_test = pd.read_csv('edx_test.csv')

# Any code to pre-process the data 
...

# The code to make the predictions on the test data 
# (likely something like model.predict(...))
predictions = ...

to_save = df_test[['userid_DI']].copy()
to_save.loc[:, 'certified'] = predictions
to_save.to_csv('submission.csv', index=False)
```
You can then upload `submission.csv` to Kaggle to evaluate the model after running this code.

**Important Note**: Kaggle only lets you submit 5 times a day, resetting at 00:00 UTC. This will keep the test accuracies as correct as possible. We recommend you start early so you don't run up against the submission limit close to the deadline.

**Important Note**: Kaggle uses the Id field for each example for evaluation. It's very important that your code that pre-processes and predicts the test data does not change the order that the examples appear in the test dataset. The code snippet above assumes the predictions variable stores a series of length `len(dftest)` that is in the same order as the original `dftest`. If you were to change the order of the dataset or predictions, you will likely get very wild accuracies on Kaggle.

If you want to test out if you can submit everything correctly, you can try a baseline classifier that selects labels at random. Such a classifier is already written in the [starter code](https://colab.research.google.com/drive/1VxAARGUpYPNU4640Q5RB0CxHjN2Wns9H?usp=sharing).

#Grading
The majority of your grade for this assignment will be determined by your accuracy on the private test data (see below), but also includes a portion for your written report. Your grade breakdown for this assignment will be:

40 points: Accuracy on the private test data
20 points: Programming portion and written report
(Recall that there is still a Conceptual portion of HW4 on Gradescope, as usual. The Kaggle portion is only the Programming portion.)

#Private Test Data
Kaggle has the notion of a public leaderboard and a private leaderboard both based on portions of the test dataset provided.

The public leaderboard is viewable while the assignment is open. The score in the public leaderboard is based on the accuracy on 1/3 of the test data. The test accuracy of your model for this 1/3 of the test data is shown to you each time you submit.

Once the assignment has closed, the rankings are computed based on the accuracy on the remaining 2/3 of the test data; this is called the private leaderboard. This is similar to "shipping" your model and seeing how it performs in the wild on unseen data. This is the accuracy score that will be used for grading.

#Evaluation
You do not need to get 100% accuracy on the test data for full credit. We will give full credit for the "accuracy" portion of the assignment for any test accuracy on the private dataset that is 95% or above. Don't stress over the numbers, focus on explaining your strategy on Gradescope and polishing your notebooks.

Here are some examples for how grades can work for this assignment, ignoring the Concept portion:

* 75% accuracy on private test data is an 75/95 which will be converted to 31.58 / 40, 15/20 on the programming report => 46.58/60 total
* 95% accuracy on private test data is a 95/95 which will be converted to 40/40, 20/20 on the programming report => 60/60 total
* 98% accuracy on private test data is a 95/95 which will be scaled to 40/40, 15/20 on the programming report => 55/60 total

#Additional Credit
As a small incentive to encourage you to make the best models you can, we will provide 5 additional points to the top 3 teams on the private leader board. Therefore, the max grade on this assignment is 65/60. Recall that extra credit will not have a huge impact on your final grade and is really meant to be something for fun and is not to stress you out.