{ "id": 919, "title": "AUC implementation", "url": "/competitions/SemiSupervisedFeatureLearning/discussion/919", "author_name": "", "post_date": "2011-10-04T08:07:43.400Z", "votes": null, "comment_count": 12, "views": 15312, "content": "

Hi! What kind of tool is used to compute the leaderboard scores? Is it `perf` [1], libsvmtools [2], or some custom code?

\r\n

I wonder how score ties are handled...

\r\n

[1] http://osmot.cs.cornell.edu/kddcup/software.html 

\r\n

[2] http://www.csie.ntu.edu.tw/~cjlin/libsvmtools/ 

\r\n

thanks!

\r\n

 

\r\n

 

\r\n

 

", "messages": [ { "id": "5886", "postDate": "10/04/2011 08:07:43", "content": "

Hi! What kind of tool is used to compute the leaderboard scores? Is it `perf` [1], libsvmtools [2], or some custom code?

\r\n

I wonder how score ties are handled...

\r\n

[1] http://osmot.cs.cornell.edu/kddcup/software.html 

\r\n

[2] http://www.csie.ntu.edu.tw/~cjlin/libsvmtools/ 

\r\n

thanks!

\r\n

 

\r\n

 

\r\n

 

", "rawMarkdown": "", "votes": null }, { "id": "5892", "postDate": "10/04/2011 17:51:25", "content": "

Jeremy, can you answer this?

", "rawMarkdown": "", "votes": null }, { "id": "5896", "postDate": "10/04/2011 21:24:25", "content": "

[quote=Peter Prettenhofer;5886]

\r\n

Hi! What kind of tool is used to compute the leaderboard scores? Is it `perf` [1], libsvmtools [2], or some custom code?

\r\n

[/quote]

\r\n

It's custom C# code that I wrote very similar to \r\nthis post. Submissions are sorted by their predicted value.

", "rawMarkdown": "", "votes": null }, { "id": "5898", "postDate": "10/04/2011 21:32:15", "content": "

Here's Octave/Matlab code that gets the same results:

\r\n

function auc = scoreAUC(category,posterior)
\r\n% auc = scoreAUC(category,posterior)
\r\n%
\r\n% Calculates the area under the ROC for a given set
\r\n% of posterior predictions and labels. Currently limited to two classes.
\r\n%
\r\n% posterior: n1 matrix of posterior probabilities for class 1
\r\n% category: n1 matrix of categories {0,1}
\r\n% auc: Area under the curve
\r\n%
\r\n% Author: Benjamin Hamner
\r\n% Date Modified: October 14, 2010
\r\n%
\r\n% Algorithm found in
\r\n% A Simple Generalisation of the Area Under the ROC
\r\n% Curve for Multiple Class Classification Problems
\r\n% David Hand and Robert Till
\r\n% http://www.springerlink.com/content/nn141j42838n7u21/fulltext.pdf

\r\n

r = tiedrank(posterior);
\r\nauc = (sum(r(category==1)) - sum(category==1) * (sum(category==1)+1)/2) / ...
\r\n( sum(category<1) * sum(category==1));

", "rawMarkdown": "", "votes": null }, { "id": "5901", "postDate": "10/05/2011 06:59:28", "content": "

Thanks! What strucks me is that my cross validation results (3-Fold CV on train) are very different from the leader board results. Do you experience a similar behaviour? 

\r\n

 

\r\n

E.g. for example_transform.public_train_data.csv (using liblinear) I get an AUC of 0.8644 (averaged over the 3 folds). 

", "rawMarkdown": "", "votes": null }, { "id": "5905", "postDate": "10/05/2011 16:57:32", "content": "

Since you've run this experiment and can likely figure this out from your results, I'll go ahead and reveal one hidden factor in the training data.

\r\n

The training labels have some label noise synthetically injected into them, to better mirror real-world cases where labels are unreliable in some way. (One of the things we're interested in seeing is if the large unlabeled data set can be used to help improve\r\n results in the presence of class label noise.)

\r\n

The test labels used for evaluation are clean ground-truth, in the sense that no noise has been injected. This explains why your cross-validation results on training data shows lower AUC than for the test data.

", "rawMarkdown": "", "votes": null }, { "id": "6123", "postDate": "10/18/2011 08:40:04", "content": "

Hi,

\r\n

Now that the labels for test data have been released, is it possible to provide your script/program that calculates AUC so that we can get the same results that we were getting when submitting? As you know there are several ways to calculate AUC (see e.g. http://www.cs.ru.nl/~tomh/onderwijs/dm/dm_files/roc_auc.pdf)\r\n and having consistency in the results is essential for the describing our methods in the joint contestants paper (see thread "About\r\n acknowledgement of contestants").

\r\n

Thanks,

\r\n

Nicholas

", "rawMarkdown": "", "votes": null }, { "id": "6130", "postDate": "10/18/2011 13:45:10", "content": "

Nicholas - while there are multiple ways to implement AUC calculations, AUC is precisely defined, and all exact methods get the same result (up to roundoff error). There are several ways to approximate it as well, but approximations are not necessary for\r\n this dataset. Look at the M-code I posted earlier on this thread for a 2-line example of an exact way to calculate AUC.

", "rawMarkdown": "", "votes": null }, { "id": "6132", "postDate": "10/18/2011 13:56:34", "content": "

Hi Ben,

\r\n

 

\r\n

First of all congratulations for your winning approach. Good job indeed!

\r\n

 

\r\n

The reason that I asked Jeff for the code is because I get different results on my private score when I use the M-code that you posted. This is weird since the difference is large and cannot possibly be attributed to roundoff errors.

\r\n

Thanks,

\r\n

Nicholas 

", "rawMarkdown": "", "votes": null }, { "id": "6134", "postDate": "10/18/2011 14:27:34", "content": "

Not every package deals consistently with the case of tied scores, which is kind of a degenerate case for AUC computation. If you have a lot of tied scores this could account for differences.

\r\n

I don't have access to kaggle's internal code for AUC computation. (Jeremy, can you release this?)

\r\n

But the other code I've run has given consistent results with those posted (admittedly, for methods that produce basically no tied scores). You can of course test this by comparing your results to those posted.

\r\n

Some standard open source code for metrics computation is here:

\r\n

http://osmot.cs.cornell.edu/kddcup/software.html

\r\n

Hope this is helpful.

", "rawMarkdown": "", "votes": null }, { "id": "6135", "postDate": "10/18/2011 14:37:51", "content": "

Thank you for the link.

\r\n

Perf and Ben's M-code give me exactly the same results! Unfortunately these are still quite far from what has been reported as my private score by kaggle's internal code for AUC computation.

", "rawMarkdown": "", "votes": null }, { "id": "6136", "postDate": "10/18/2011 14:50:19", "content": "

Note that we order by predicted value, but as of yet don't take into account ties

\r\n
// From 'AUC Calculation Check' post in IJCNN Social Network Challenge forum\r\n// Credit: B Yang - original C++ code\r\npublic static double Auc(this IList a, IList p) {\r\n    // AUC requires int array as dependent\r\n\r\n    var all = a.Zip(p, \r\n                    (actual, pred) => new {actualValue = actual < 0.5 ? 0 : 1, predictedValue = pred})\r\n               .OrderBy(ap => ap.predictedValue)\r\n               .ToArray();\r\n\r\n    long n = all.Length;\r\n    long ones = all.Sum(v => v.actualValue);\r\n    if (0 == ones || n == ones) return 1;\r\n\r\n    long tp0, tn;\r\n    long truePos = tp0 = ones; long accum = tn = 0; double threshold = all[0].predictedValue;\r\n    for (int i = 0; i < n; i++) {\r\n        if (all[i].predictedValue != threshold) { // threshold changes\r\n            threshold = all[i].predictedValue;\r\n            accum += tn * (truePos + tp0); //2* the area of  trapezoid\r\n            tp0 = truePos;\r\n            tn = 0;\r\n        }\r\n        tn += 1 - all[i].actualValue; // x-distance between adjacent points\r\n        truePos -= all[i].actualValue;\r\n    }\r\n    accum += tn * (truePos + tp0); // 2 * the area of trapezoid\r\n    return (double)accum / (2 * ones * (n - ones));\r\n}\r\n
", "rawMarkdown": "", "votes": null }, { "id": "6139", "postDate": "10/18/2011 15:45:01", "content": "

Thank you Jeff. I'll check it out

", "rawMarkdown": "", "votes": null } ], "comments": [ { "id": 5892, "author_name": "argv", "author_url": "", "post_date": "10/04/2011 17:51:25", "content": "

Jeremy, can you answer this?

", "votes": null, "replies": [] }, { "id": 5896, "author_name": "jeffmoser", "author_url": "", "post_date": "10/04/2011 21:24:25", "content": "

[quote=Peter Prettenhofer;5886]

\r\n

Hi! What kind of tool is used to compute the leaderboard scores? Is it `perf` [1], libsvmtools [2], or some custom code?

\r\n

[/quote]

\r\n

It's custom C# code that I wrote very similar to \r\nthis post. Submissions are sorted by their predicted value.

", "votes": null, "replies": [] }, { "id": 5898, "author_name": "benhamner", "author_url": "", "post_date": "10/04/2011 21:32:15", "content": "

Here's Octave/Matlab code that gets the same results:

\r\n

function auc = scoreAUC(category,posterior)
\r\n% auc = scoreAUC(category,posterior)
\r\n%
\r\n% Calculates the area under the ROC for a given set
\r\n% of posterior predictions and labels. Currently limited to two classes.
\r\n%
\r\n% posterior: n1 matrix of posterior probabilities for class 1
\r\n% category: n1 matrix of categories {0,1}
\r\n% auc: Area under the curve
\r\n%
\r\n% Author: Benjamin Hamner
\r\n% Date Modified: October 14, 2010
\r\n%
\r\n% Algorithm found in
\r\n% A Simple Generalisation of the Area Under the ROC
\r\n% Curve for Multiple Class Classification Problems
\r\n% David Hand and Robert Till
\r\n% http://www.springerlink.com/content/nn141j42838n7u21/fulltext.pdf

\r\n

r = tiedrank(posterior);
\r\nauc = (sum(r(category==1)) - sum(category==1) * (sum(category==1)+1)/2) / ...
\r\n( sum(category<1) * sum(category==1));

", "votes": null, "replies": [] }, { "id": 5901, "author_name": "pprett", "author_url": "", "post_date": "10/05/2011 06:59:28", "content": "

Thanks! What strucks me is that my cross validation results (3-Fold CV on train) are very different from the leader board results. Do you experience a similar behaviour? 

\r\n

 

\r\n

E.g. for example_transform.public_train_data.csv (using liblinear) I get an AUC of 0.8644 (averaged over the 3 folds). 

", "votes": null, "replies": [] }, { "id": 5905, "author_name": "argv", "author_url": "", "post_date": "10/05/2011 16:57:32", "content": "

Since you've run this experiment and can likely figure this out from your results, I'll go ahead and reveal one hidden factor in the training data.

\r\n

The training labels have some label noise synthetically injected into them, to better mirror real-world cases where labels are unreliable in some way. (One of the things we're interested in seeing is if the large unlabeled data set can be used to help improve\r\n results in the presence of class label noise.)

\r\n

The test labels used for evaluation are clean ground-truth, in the sense that no noise has been injected. This explains why your cross-validation results on training data shows lower AUC than for the test data.

", "votes": null, "replies": [] }, { "id": 6123, "author_name": "ampazis", "author_url": "", "post_date": "10/18/2011 08:40:04", "content": "

Hi,

\r\n

Now that the labels for test data have been released, is it possible to provide your script/program that calculates AUC so that we can get the same results that we were getting when submitting? As you know there are several ways to calculate AUC (see e.g. http://www.cs.ru.nl/~tomh/onderwijs/dm/dm_files/roc_auc.pdf)\r\n and having consistency in the results is essential for the describing our methods in the joint contestants paper (see thread "About\r\n acknowledgement of contestants").

\r\n

Thanks,

\r\n

Nicholas

", "votes": null, "replies": [] }, { "id": 6130, "author_name": "benhamner", "author_url": "", "post_date": "10/18/2011 13:45:10", "content": "

Nicholas - while there are multiple ways to implement AUC calculations, AUC is precisely defined, and all exact methods get the same result (up to roundoff error). There are several ways to approximate it as well, but approximations are not necessary for\r\n this dataset. Look at the M-code I posted earlier on this thread for a 2-line example of an exact way to calculate AUC.

", "votes": null, "replies": [] }, { "id": 6132, "author_name": "ampazis", "author_url": "", "post_date": "10/18/2011 13:56:34", "content": "

Hi Ben,

\r\n

 

\r\n

First of all congratulations for your winning approach. Good job indeed!

\r\n

 

\r\n

The reason that I asked Jeff for the code is because I get different results on my private score when I use the M-code that you posted. This is weird since the difference is large and cannot possibly be attributed to roundoff errors.

\r\n

Thanks,

\r\n

Nicholas 

", "votes": null, "replies": [] }, { "id": 6134, "author_name": "argv", "author_url": "", "post_date": "10/18/2011 14:27:34", "content": "

Not every package deals consistently with the case of tied scores, which is kind of a degenerate case for AUC computation. If you have a lot of tied scores this could account for differences.

\r\n

I don't have access to kaggle's internal code for AUC computation. (Jeremy, can you release this?)

\r\n

But the other code I've run has given consistent results with those posted (admittedly, for methods that produce basically no tied scores). You can of course test this by comparing your results to those posted.

\r\n

Some standard open source code for metrics computation is here:

\r\n

http://osmot.cs.cornell.edu/kddcup/software.html

\r\n

Hope this is helpful.

", "votes": null, "replies": [] }, { "id": 6135, "author_name": "ampazis", "author_url": "", "post_date": "10/18/2011 14:37:51", "content": "

Thank you for the link.

\r\n

Perf and Ben's M-code give me exactly the same results! Unfortunately these are still quite far from what has been reported as my private score by kaggle's internal code for AUC computation.

", "votes": null, "replies": [] }, { "id": 6136, "author_name": "jeffmoser", "author_url": "", "post_date": "10/18/2011 14:50:19", "content": "

Note that we order by predicted value, but as of yet don't take into account ties

\r\n
// From 'AUC Calculation Check' post in IJCNN Social Network Challenge forum\r\n// Credit: B Yang - original C++ code\r\npublic static double Auc(this IList a, IList p) {\r\n    // AUC requires int array as dependent\r\n\r\n    var all = a.Zip(p, \r\n                    (actual, pred) => new {actualValue = actual < 0.5 ? 0 : 1, predictedValue = pred})\r\n               .OrderBy(ap => ap.predictedValue)\r\n               .ToArray();\r\n\r\n    long n = all.Length;\r\n    long ones = all.Sum(v => v.actualValue);\r\n    if (0 == ones || n == ones) return 1;\r\n\r\n    long tp0, tn;\r\n    long truePos = tp0 = ones; long accum = tn = 0; double threshold = all[0].predictedValue;\r\n    for (int i = 0; i < n; i++) {\r\n        if (all[i].predictedValue != threshold) { // threshold changes\r\n            threshold = all[i].predictedValue;\r\n            accum += tn * (truePos + tp0); //2* the area of  trapezoid\r\n            tp0 = truePos;\r\n            tn = 0;\r\n        }\r\n        tn += 1 - all[i].actualValue; // x-distance between adjacent points\r\n        truePos -= all[i].actualValue;\r\n    }\r\n    accum += tn * (truePos + tp0); // 2 * the area of trapezoid\r\n    return (double)accum / (2 * ones * (n - ones));\r\n}\r\n
", "votes": null, "replies": [] }, { "id": 6139, "author_name": "ampazis", "author_url": "", "post_date": "10/18/2011 15:45:01", "content": "

Thank you Jeff. I'll check it out

", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "5886": "", "5892": "", "5896": "", "5898": "", "5901": "", "5905": "", "6123": "", "6130": "", "6132": "", "6134": "", "6135": "", "6136": "", "6139": "" }, "source": "meta" }