{ "id": 23322, "title": "AUC vs Matthews correlation coefficient", "url": "/competitions/bosch-production-line-performance/discussion/23322", "author_name": "Dalaska", "post_date": "2016-09-03T18:13:17.317000", "votes": 0, "comment_count": 12, "views": 2436, "content": "

Is it a good idea to first build a model with the best AUC score. \nThen do grid_search to find the best cut-off probability for the best Matthews correlation coefficient?

\n\n

Does a better AUC score always guarantee a better Matthews correlation coefficient?

", "messages": [ { "id": 133804, "postDate": "2016-09-03T21:30:06.830Z", "content": "

I don't exactly get what you mean with that, but the special thing about MCC is, that it describes the confusion matrix regardless of the class sizes. Try it out:

\n\n
TN <-50\nTP <-5000\nFN <-8\nFP <-123\n\n(TP * TN - FP * FN) / sqrt((TP+FP) * (TP+FN) * (FP+TN) * (TN+FN))
\n\n

Versus:

\n\n
TP <-50\nTN <-5000\nFP <-8\nFN <-123\n\n(TP * TN - FP * FN) / sqrt((TP+FP) * (TP+FN) * (FP+TN) * (TN+FN))
", "rawMarkdown": "I don't exactly get what you mean with that, but the special thing about MCC is, that it describes the confusion matrix regardless of the class sizes. Try it out:\r\n\r\n
TN <-50\r\nTP <-5000\r\nFN <-8\r\nFP <-123\r\n\r\n(TP * TN - FP * FN) / sqrt((TP+FP) * (TP+FN) * (FP+TN) * (TN+FN))
\r\n\r\nVersus:\r\n\r\n
TP <-50\r\nTN <-5000\r\nFP <-8\r\nFN <-123\r\n\r\n(TP * TN - FP * FN) / sqrt((TP+FP) * (TP+FN) * (FP+TN) * (TN+FN))
\r\n\r\n", "votes": 1 }, { "id": 133792, "postDate": "2016-09-03T20:02:09.137Z", "content": "

1) Try and find out, you will learn in the process

\n\n

2) No

", "rawMarkdown": "1) Try and find out, you will learn in the process\r\n\r\n2) No", "votes": -1 }, { "id": 135047, "postDate": "2016-09-10T14:43:47.650Z", "content": "

mcc <- function(preds, real) {\n preds[preds>0.5] <- 1\n preds[preds<=0.5] <- 0\n TN <- sum(real==0 & preds==0)\n TP <- sum(real==1 & preds==1)\n FN <- sum(real==0 & preds==1)\n FP <- sum(real==1 & preds==0)\n N <- TN+TP+FN+FP\n S <- (TP+FN)/N\n P <- (TP+FP)/N\n mcc <- as.numeric((TP/N-S*P) / (sqrt(P*S*(1-S)*(1-P))))\n return(list(metric = "error", value = mcc))\n}

", "rawMarkdown": "mcc <- function(preds, real) {\r\n preds[preds>0.5] <- 1\r\n preds[preds<=0.5] <- 0\r\n TN <- sum(real==0 & preds==0)\r\n TP <- sum(real==1 & preds==1)\r\n FN <- sum(real==0 & preds==1)\r\n FP <- sum(real==1 & preds==0)\r\n N <- TN+TP+FN+FP\r\n S <- (TP+FN)/N\r\n P <- (TP+FP)/N\r\n mcc <- as.numeric((TP/N-S*P) / (sqrt(P*S*(1-S)*(1-P))))\r\n return(list(metric = \"error\", value = mcc))\r\n}" }, { "id": 134880, "postDate": "2016-09-09T01:10:57.240Z", "content": "

@Jesse - Thanks for sharing your insights in to the metric. Much appreciated.

", "rawMarkdown": "@Jesse - Thanks for sharing your insights in to the metric. Much appreciated." }, { "id": 133839, "postDate": "2016-09-04T00:00:35.427Z", "content": "

Sure, you have to optimize the threshhold to maximize MCC!

", "rawMarkdown": "Sure, you have to optimize the threshhold to maximize MCC!" }, { "id": 133828, "postDate": "2016-09-03T23:00:05.307Z", "content": "

Hi Charly:\nThat's interesting. I didn't know that before.

\n\n

What I mean is if you predict in terms of probability instead of 0,1. You need to choose a threshold to get TP,TN, FP,FN. You can tune the probability to get better Matthews correlation coefficient.

", "rawMarkdown": "Hi Charly:\r\nThat's interesting. I didn't know that before.\r\n\r\nWhat I mean is if you predict in terms of probability instead of 0,1. You need to choose a threshold to get TP,TN, FP,FN. You can tune the probability to get better Matthews correlation coefficient." }, { "id": 133798, "postDate": "2016-09-03T20:45:55.040Z", "content": "

[quote=NxGTR;133792]

\n\n

1) Try and find out, you will learn in the process

\n\n

2) No

\n\n

[/quote]

\n\n

Thanks NxGTR\nThe Matthews correlation coefficient is kind of weird. \nUsually when the training data size increases, the performance of the classifier increases.
\nBut Matthews correlation coefficient doesn't aways increase with the training data set.

", "rawMarkdown": "[quote=NxGTR;133792]\r\n\r\n1) Try and find out, you will learn in the process\r\n\r\n2) No\r\n\r\n[/quote]\r\n\r\nThanks NxGTR\r\nThe Matthews correlation coefficient is kind of weird. \r\nUsually when the training data size increases, the performance of the classifier increases. \r\nBut Matthews correlation coefficient doesn't aways increase with the training data set. " }, { "id": 133772, "postDate": "2016-09-03T18:13:17.317Z", "content": "

Is it a good idea to first build a model with the best AUC score. \nThen do grid_search to find the best cut-off probability for the best Matthews correlation coefficient?

\n\n

Does a better AUC score always guarantee a better Matthews correlation coefficient?

", "rawMarkdown": "Is it a good idea to first build a model with the best AUC score. \r\nThen do grid_search to find the best cut-off probability for the best Matthews correlation coefficient?\r\n\r\nDoes a better AUC score always guarantee a better Matthews correlation coefficient?" }, { "id": 135068, "postDate": "2016-09-10T18:30:33.677Z", "rawMarkdown": "", "votes": 1, "isDeleted": true }, { "id": 134891, "postDate": "2016-09-09T03:14:31.387Z", "rawMarkdown": "", "isDeleted": true }, { "id": 134874, "postDate": "2016-09-09T00:49:46.150Z", "rawMarkdown": "", "isDeleted": true }, { "id": 134546, "postDate": "2016-09-07T07:12:30.237Z", "rawMarkdown": "", "votes": 1, "isDeleted": true }, { "id": 133983, "postDate": "2016-09-04T16:41:15.587Z", "rawMarkdown": "", "votes": 3, "isDeleted": true } ], "comments": [ { "id": 133804, "author_name": "Charly", "author_url": "", "post_date": "2016-09-03T21:30:06.830000", "content": "

I don't exactly get what you mean with that, but the special thing about MCC is, that it describes the confusion matrix regardless of the class sizes. Try it out:

\n\n
TN <-50\nTP <-5000\nFN <-8\nFP <-123\n\n(TP * TN - FP * FN) / sqrt((TP+FP) * (TP+FN) * (FP+TN) * (TN+FN))
\n\n

Versus:

\n\n
TP <-50\nTN <-5000\nFP <-8\nFN <-123\n\n(TP * TN - FP * FN) / sqrt((TP+FP) * (TP+FN) * (FP+TN) * (TN+FN))
", "votes": 1, "replies": [] }, { "id": 133792, "author_name": "NxGTR", "author_url": "", "post_date": "2016-09-03T20:02:09.137000", "content": "

1) Try and find out, you will learn in the process

\n\n

2) No

", "votes": -1, "replies": [] }, { "id": 135047, "author_name": "goodman", "author_url": "", "post_date": "2016-09-10T14:43:47.650000", "content": "

mcc <- function(preds, real) {\n preds[preds>0.5] <- 1\n preds[preds<=0.5] <- 0\n TN <- sum(real==0 & preds==0)\n TP <- sum(real==1 & preds==1)\n FN <- sum(real==0 & preds==1)\n FP <- sum(real==1 & preds==0)\n N <- TN+TP+FN+FP\n S <- (TP+FN)/N\n P <- (TP+FP)/N\n mcc <- as.numeric((TP/N-S*P) / (sqrt(P*S*(1-S)*(1-P))))\n return(list(metric = "error", value = mcc))\n}

", "votes": 0, "replies": [] }, { "id": 134880, "author_name": "inversion", "author_url": "", "post_date": "2016-09-09T01:10:57.240000", "content": "

@Jesse - Thanks for sharing your insights in to the metric. Much appreciated.

", "votes": 0, "replies": [] }, { "id": 133839, "author_name": "Charly", "author_url": "", "post_date": "2016-09-04T00:00:35.427000", "content": "

Sure, you have to optimize the threshhold to maximize MCC!

", "votes": 0, "replies": [] }, { "id": 133828, "author_name": "Dalaska", "author_url": "", "post_date": "2016-09-03T23:00:05.307000", "content": "

Hi Charly:\nThat's interesting. I didn't know that before.

\n\n

What I mean is if you predict in terms of probability instead of 0,1. You need to choose a threshold to get TP,TN, FP,FN. You can tune the probability to get better Matthews correlation coefficient.

", "votes": 0, "replies": [] }, { "id": 133798, "author_name": "Dalaska", "author_url": "", "post_date": "2016-09-03T20:45:55.040000", "content": "

[quote=NxGTR;133792]

\n\n

1) Try and find out, you will learn in the process

\n\n

2) No

\n\n

[/quote]

\n\n

Thanks NxGTR\nThe Matthews correlation coefficient is kind of weird. \nUsually when the training data size increases, the performance of the classifier increases.
\nBut Matthews correlation coefficient doesn't aways increase with the training data set.

", "votes": 0, "replies": [] }, { "id": 135068, "author_name": "", "author_url": "", "post_date": "2016-09-10T18:30:33.677000", "content": "", "votes": 1, "replies": [] }, { "id": 134891, "author_name": "", "author_url": "", "post_date": "2016-09-09T03:14:31.387000", "content": "", "votes": 0, "replies": [] }, { "id": 134874, "author_name": "", "author_url": "", "post_date": "2016-09-09T00:49:46.150000", "content": "", "votes": 0, "replies": [] }, { "id": 134546, "author_name": "", "author_url": "", "post_date": "2016-09-07T07:12:30.237000", "content": "", "votes": 1, "replies": [] }, { "id": 133983, "author_name": "", "author_url": "", "post_date": "2016-09-04T16:41:15.587000", "content": "", "votes": 3, "replies": [] } ], "raw_markdown_by_id": { "133804": "I don't exactly get what you mean with that, but the special thing about MCC is, that it describes the confusion matrix regardless of the class sizes. Try it out:\r\n\r\n
TN <-50\r\nTP <-5000\r\nFN <-8\r\nFP <-123\r\n\r\n(TP * TN - FP * FN) / sqrt((TP+FP) * (TP+FN) * (FP+TN) * (TN+FN))
\r\n\r\nVersus:\r\n\r\n
TP <-50\r\nTN <-5000\r\nFP <-8\r\nFN <-123\r\n\r\n(TP * TN - FP * FN) / sqrt((TP+FP) * (TP+FN) * (FP+TN) * (TN+FN))
\r\n\r\n", "133792": "1) Try and find out, you will learn in the process\r\n\r\n2) No", "135047": "mcc <- function(preds, real) {\r\n preds[preds>0.5] <- 1\r\n preds[preds<=0.5] <- 0\r\n TN <- sum(real==0 & preds==0)\r\n TP <- sum(real==1 & preds==1)\r\n FN <- sum(real==0 & preds==1)\r\n FP <- sum(real==1 & preds==0)\r\n N <- TN+TP+FN+FP\r\n S <- (TP+FN)/N\r\n P <- (TP+FP)/N\r\n mcc <- as.numeric((TP/N-S*P) / (sqrt(P*S*(1-S)*(1-P))))\r\n return(list(metric = \"error\", value = mcc))\r\n}", "134880": "@Jesse - Thanks for sharing your insights in to the metric. Much appreciated.", "133839": "Sure, you have to optimize the threshhold to maximize MCC!", "133828": "Hi Charly:\r\nThat's interesting. I didn't know that before.\r\n\r\nWhat I mean is if you predict in terms of probability instead of 0,1. You need to choose a threshold to get TP,TN, FP,FN. You can tune the probability to get better Matthews correlation coefficient.", "133798": "[quote=NxGTR;133792]\r\n\r\n1) Try and find out, you will learn in the process\r\n\r\n2) No\r\n\r\n[/quote]\r\n\r\nThanks NxGTR\r\nThe Matthews correlation coefficient is kind of weird. \r\nUsually when the training data size increases, the performance of the classifier increases. \r\nBut Matthews correlation coefficient doesn't aways increase with the training data set. ", "133772": "Is it a good idea to first build a model with the best AUC score. \r\nThen do grid_search to find the best cut-off probability for the best Matthews correlation coefficient?\r\n\r\nDoes a better AUC score always guarantee a better Matthews correlation coefficient?", "135068": "", "134891": "", "134874": "", "134546": "", "133983": "" } }