AUC is a metric used to judge predictions in binary response (0/1) problem. It is only sensitive to the order determined by the predictions and not their magnitudes.
AUC was first used by the American army after the attack on Pearl Harbour, to detect Japanese aircraft from radar signals.
Today, it is a commonly used evaluation method for binary choice problems, which involve classifying an instance as either positive or negative. Its main advantages over other evaluation methods, such as the simpler misclassification error, are:
It's insensitive to unbalanced datasets (datasets that have more installeds than not-installeds or vice versa).
For other evaluation methods, a user has to choose a cut-off point above which the target variable is part of the positive class (e.g. a logistic regression model returns any real number between 0 and 1 - the modeler might decide that predictions greater than 0.5 mean a positive class prediction while a prediction of less than 0.5 mean a negative class prediction). AUC evaluates entries at all cut-off points, giving better insight into how well the classifier is able to separate the two classes.
Submission Format
For each student, each problem, each submission
(
except for the first submission to a problem, and cases where there is no forum or video event between two submissions
), your submission file should include two columns:
uniqRowID
and
improved
.
uniqRowID:
a unique identifier string with the following format:
UserID_ProblemID_SubmissionNumber
improved:
1
if there is a grade improvement, 0 otherwise
uniqRowID, improved
5e774bfc69f_2_5, 0
db5e774bfc69f_2_6, 0
db5e774bfc69f_4_13, 1
db5e774bfc69f_7_39, 1
6f0db5e774bfx_8_2, 0
000e3646f969f_8_4, 1