Just found out how to calculate mcc in r and how to find the best split to optimize mcc value.
\n\nlibrary(ROCR)\ndata(ROCR.simple)\npred <- prediction( ROCR.simple$predictions, ROCR.simple$labels)\nperf2 <- performance(pred, "mat")\nplot(perf2)\nbest_split = perf2@x.values[[1]][which.max(perf2@y.values[[1]])]\nHope it helps. Please correct me if I am wrong.
", "messages": [ { "id": "134074", "postDate": "09/05/2016 00:06:53", "content": "Just found out how to calculate mcc in r and how to find the best split to optimize mcc value.
\n\nlibrary(ROCR)\ndata(ROCR.simple)\npred <- prediction( ROCR.simple$predictions, ROCR.simple$labels)\nperf2 <- performance(pred, "mat")\nplot(perf2)\nbest_split = perf2@x.values[[1]][which.max(perf2@y.values[[1]])]\nHope it helps. Please correct me if I am wrong.
", "rawMarkdown": "Just found out how to calculate mcc in r and how to find the best split to optimize mcc value.\r\n\r\n library(ROCR)\r\n data(ROCR.simple)\r\n pred <- prediction( ROCR.simple$predictions, ROCR.simple$labels)\r\n perf2 <- performance(pred, \"mat\")\r\n plot(perf2)\r\n best_split = perf2@x.values[[1]][which.max(perf2@y.values[[1]])]\r\n\r\nHope it helps. Please correct me if I am wrong.", "votes": null }, { "id": "134085", "postDate": "09/05/2016 01:36:26", "content": "Hi xyz,
\n\nVery nice. I've been using my own custom metric and letting the model determine cutoff. I get the same results as the ROCR calculation. The graph is a nice touch to help determine if the model is stable. I'll have to experiment with this.
\n\nmcc <- function (actual, predicted)\n{\n # Compute the Matthews correlation coefficient (MCC) score\n # Jeff Hebert 9/1/2016\n #\n # actual = vector of true outcomes, 1 = Positive, 0 = Negative\n # predicted = vector of predicted outcomes, 1 = Positive, 0 = Negative\n # function returns MCC\n\n TP <- sum(actual == 1 & predicted == 1)\n TN <- sum(actual == 0 & predicted == 0)\n FP <- sum(actual == 0 & predicted == 1)\n FN <- sum(actual == 1 & predicted == 0)\n\n mcc <- ((TP*TN)-(FP*FN)) / sqrt((TP+FP)*(TP+FN)*(TN+FP)*(TN+FN))\n return(mcc)\n}\n[quote=JeffH;134085]
\n\nHi xyz,
\n\nVery nice. I've been using my own custom metric and letting the model determine cutoff. I get the same results as the ROCR calculation. The graph is a nice touch to help determine if the model is stable. I'll have to experiment with this.
\n\nmcc <- function (actual, predicted)\n{\n # Compute the Matthews correlation coefficient (MCC) score\n # Jeff Hebert 9/1/2016\n #\n # actual = vector of true outcomes, 1 = Positive, 0 = Negative\n # predicted = vector of predicted outcomes, 1 = Positive, 0 = Negative\n # function returns MCC\n\n TP <- sum(actual == 1 & predicted == 1)\n TN <- sum(actual == 0 & predicted == 0)\n FP <- sum(actual == 0 & predicted == 1)\n FN <- sum(actual == 1 & predicted == 0)\n\n mcc <- ((TP*TN)-(FP*FN)) / sqrt((TP+FP)*(TP+FN)*(TN+FP)*(TN+FN))\n return(mcc)\n}\n[/quote]
\n\nHey Jeff, thanks. Did you try to customize the loss function when using xgboost?
", "rawMarkdown": "[quote=JeffH;134085]\r\n\r\nHi xyz,\r\n\r\nVery nice. I've been using my own custom metric and letting the model determine cutoff. I get the same results as the ROCR calculation. The graph is a nice touch to help determine if the model is stable. I'll have to experiment with this.\r\n\r\n\r\n mcc <- function (actual, predicted)\r\n {\r\n # Compute the Matthews correlation coefficient (MCC) score\r\n # Jeff Hebert 9/1/2016\r\n #\r\n # actual = vector of true outcomes, 1 = Positive, 0 = Negative\r\n # predicted = vector of predicted outcomes, 1 = Positive, 0 = Negative\r\n # function returns MCC\r\n \r\n TP <- sum(actual == 1 & predicted == 1)\r\n TN <- sum(actual == 0 & predicted == 0)\r\n FP <- sum(actual == 0 & predicted == 1)\r\n FN <- sum(actual == 1 & predicted == 0)\r\n \r\n mcc <- ((TP*TN)-(FP*FN)) / sqrt((TP+FP)*(TP+FN)*(TN+FP)*(TN+FN))\r\n return(mcc)\r\n }\r\n\r\n\r\n\r\n[/quote]\r\n\r\n\r\nHey Jeff, thanks. Did you try to customize the loss function when using xgboost?", "votes": null }, { "id": "139524", "postDate": "10/14/2016 17:08:35", "content": "Added fixes for defects that surfaced.
\n\nRegards,
\n\nga
\n\nmcc <- function (actual, predicted)\n{\n # Compute the Matthews correlation coefficient (MCC) score\n # Jeff Hebert 9/1/2016\n # Geoffrey Anderson 10/14/2016 \n # Added zero denominator handling.\n # Avoided overflow error on large-ish products in denominator.\n #\n # actual = vector of true outcomes, 1 = Positive, 0 = Negative\n # predicted = vector of predicted outcomes, 1 = Positive, 0 = Negative\n # function returns MCC\n\n TP <- sum(actual == 1 & predicted == 1)\n TN <- sum(actual == 0 & predicted == 0)\n FP <- sum(actual == 0 & predicted == 1)\n FN <- sum(actual == 1 & predicted == 0)\n #TP;TN;FP;FN # for debugging\n sum1 <- TP+FP; sum2 <-TP+FN ; sum3 <-TN+FP ; sum4 <- TN+FN;\n denom <- as.double(sum1)*sum2*sum3*sum4 # as.double to avoid overflow error on large products\n if (any(sum1==0, sum2==0, sum3==0, sum4==0)) {\n denom <- 1\n }\n mcc <- ((TP*TN)-(FP*FN)) / sqrt(denom)\n return(mcc)\n}\nHi xyz,
\n\nVery nice. I've been using my own custom metric and letting the model determine cutoff. I get the same results as the ROCR calculation. The graph is a nice touch to help determine if the model is stable. I'll have to experiment with this.
\n\nmcc <- function (actual, predicted)\n{\n # Compute the Matthews correlation coefficient (MCC) score\n # Jeff Hebert 9/1/2016\n #\n # actual = vector of true outcomes, 1 = Positive, 0 = Negative\n # predicted = vector of predicted outcomes, 1 = Positive, 0 = Negative\n # function returns MCC\n\n TP <- sum(actual == 1 & predicted == 1)\n TN <- sum(actual == 0 & predicted == 0)\n FP <- sum(actual == 0 & predicted == 1)\n FN <- sum(actual == 1 & predicted == 0)\n\n mcc <- ((TP*TN)-(FP*FN)) / sqrt((TP+FP)*(TP+FN)*(TN+FP)*(TN+FN))\n return(mcc)\n}\n[quote=JeffH;134085]
\n\nHi xyz,
\n\nVery nice. I've been using my own custom metric and letting the model determine cutoff. I get the same results as the ROCR calculation. The graph is a nice touch to help determine if the model is stable. I'll have to experiment with this.
\n\nmcc <- function (actual, predicted)\n{\n # Compute the Matthews correlation coefficient (MCC) score\n # Jeff Hebert 9/1/2016\n #\n # actual = vector of true outcomes, 1 = Positive, 0 = Negative\n # predicted = vector of predicted outcomes, 1 = Positive, 0 = Negative\n # function returns MCC\n\n TP <- sum(actual == 1 & predicted == 1)\n TN <- sum(actual == 0 & predicted == 0)\n FP <- sum(actual == 0 & predicted == 1)\n FN <- sum(actual == 1 & predicted == 0)\n\n mcc <- ((TP*TN)-(FP*FN)) / sqrt((TP+FP)*(TP+FN)*(TN+FP)*(TN+FN))\n return(mcc)\n}\n[/quote]
\n\nHey Jeff, thanks. Did you try to customize the loss function when using xgboost?
", "votes": null, "replies": [] }, { "id": 139524, "author_name": "ganderson", "author_url": "", "post_date": "10/14/2016 17:08:35", "content": "Added fixes for defects that surfaced.
\n\nRegards,
\n\nga
\n\nmcc <- function (actual, predicted)\n{\n # Compute the Matthews correlation coefficient (MCC) score\n # Jeff Hebert 9/1/2016\n # Geoffrey Anderson 10/14/2016 \n # Added zero denominator handling.\n # Avoided overflow error on large-ish products in denominator.\n #\n # actual = vector of true outcomes, 1 = Positive, 0 = Negative\n # predicted = vector of predicted outcomes, 1 = Positive, 0 = Negative\n # function returns MCC\n\n TP <- sum(actual == 1 & predicted == 1)\n TN <- sum(actual == 0 & predicted == 0)\n FP <- sum(actual == 0 & predicted == 1)\n FN <- sum(actual == 1 & predicted == 0)\n #TP;TN;FP;FN # for debugging\n sum1 <- TP+FP; sum2 <-TP+FN ; sum3 <-TN+FP ; sum4 <- TN+FN;\n denom <- as.double(sum1)*sum2*sum3*sum4 # as.double to avoid overflow error on large products\n if (any(sum1==0, sum2==0, sum3==0, sum4==0)) {\n denom <- 1\n }\n mcc <- ((TP*TN)-(FP*FN)) / sqrt(denom)\n return(mcc)\n}\n