I had to write the scoring evaluation metric for my Xgboost models. I know there are plenty of examples on how to do this in R or Python, but this is here for anyone to use and not reinvent the wheel.
\n\nconf = .5\nmcc_eval <- function(preds, dtrain) {\n labels <- getinfo(dtrain, "label")\n tp <- as.numeric(sum((labels == 1) & (preds >= conf)))\n tn <- as.numeric(sum((labels == 0) & (preds < conf)))\n fp <- as.numeric(sum((labels == 0) & (preds >= conf)))\n fn <- as.numeric(sum((labels == 1) & (preds < conf)))\n\n if(tp+fp == 0) {\n # There are no valid TP or FP, nothing has been classified\n # either write or wrong. This will make a NaN, so set to 0.\n mcc <- 0\n } else {\n mcc <- (tp*tn-fp*fn)/(sqrt((tp+fp)*(tp+fn)*(tn+fp)*(tn+fn)))\n }\n return(list(metric="mcc", value=mcc))\n}\nto use it, just stick eval_metric=mcc_eval instead of one of the existing metrics in xgboost.train or cv parameters.
EDIT: I have added conf parameter to threshold logistic curves at 0.5. Change this to 1 if your preds are already binned into 0 or 1
\n\nEDIT2: I found that == did not work and changed to &. Binary AND is the same in this context. Also added a catch for NaN.
I had to write the scoring evaluation metric for my Xgboost models. I know there are plenty of examples on how to do this in R or Python, but this is here for anyone to use and not reinvent the wheel.
\n\nconf = .5\nmcc_eval <- function(preds, dtrain) {\n labels <- getinfo(dtrain, "label")\n tp <- as.numeric(sum((labels == 1) & (preds >= conf)))\n tn <- as.numeric(sum((labels == 0) & (preds < conf)))\n fp <- as.numeric(sum((labels == 0) & (preds >= conf)))\n fn <- as.numeric(sum((labels == 1) & (preds < conf)))\n\n if(tp+fp == 0) {\n # There are no valid TP or FP, nothing has been classified\n # either write or wrong. This will make a NaN, so set to 0.\n mcc <- 0\n } else {\n mcc <- (tp*tn-fp*fn)/(sqrt((tp+fp)*(tp+fn)*(tn+fp)*(tn+fn)))\n }\n return(list(metric="mcc", value=mcc))\n}\nto use it, just stick eval_metric=mcc_eval instead of one of the existing metrics in xgboost.train or cv parameters.
EDIT: I have added conf parameter to threshold logistic curves at 0.5. Change this to 1 if your preds are already binned into 0 or 1
\n\nEDIT2: I found that == did not work and changed to &. Binary AND is the same in this context. Also added a catch for NaN.
Thanks
", "rawMarkdown": "Thanks", "votes": null }, { "id": "134585", "postDate": "09/07/2016 11:42:15", "content": "Apologies if I'm being stupid here, but wouldn't that only count the predictions that XGBoost output as exactly 1 or 0? Wouldn't you need to deal with the probablities?
", "rawMarkdown": "Apologies if I'm being stupid here, but wouldn't that only count the predictions that XGBoost output as exactly 1 or 0? Wouldn't you need to deal with the probablities?", "votes": null }, { "id": "134586", "postDate": "09/07/2016 11:43:54", "content": "@anokas: at least within the sklearn API, you can switch from 'predict_proba' to 'predict' - although i'm not sure where does xgboost apply the threshold internally => the results could be horrible. But the code as such will work.
", "rawMarkdown": "anokas: at least within the sklearn API, you can switch from 'predict_proba' to 'predict' - although i'm not sure where does xgboost apply the threshold internally => the results could be horrible. But the code as such *will* work.", "votes": null }, { "id": "134591", "postDate": "09/07/2016 12:10:17", "content": "instead of preds == 1 and preds == 0 you can define conf = .5 outside the mcc_eval and then use preds >= conf for 1s and preds < conf for 0s. Better still you can define a conversion function classify() and use classify(preds) == 1 or classify(preds) == 0 for 1s and 0s respectively.
conf =.5\nmcc_eval <- function(preds, dtrain) {\n labels <- getinfo(dtrain, "label")\n tp <- as.numeric(sum((labels == 1) == (preds >= conf)))\n tn <- as.numeric(sum((labels == 0) == (preds < conf)))\n fp <- as.numeric(sum((labels == 0) == (preds >= conf)))\n fn <- as.numeric(sum((labels == 1) == (preds < conf)))\n mcc <- (tp*tn-fp*fn)/(sqrt((tp+fp)*(tp+fn)*(tn+fp)*(tn+fn)))\n return(list(metric="mcc", value=mcc))\n}\n[quote=Konrad Banachewicz;134586]
\n\n@anokas: at least within the sklearn API, you can switch from 'predict_proba' to 'predict' - although i'm not sure where does xgboost apply the threshold internally => the results could be horrible. But the code as such will work.
\n\n[/quote]
\n\nWhenever ive used xgboost custom metrics, it does not apply a threshold and the predictions that are provided are probabilities - OP should probably check that it is actually giving binary predictions otherwise his code might be quite misleading :)
\n\nEDIT: Too late, you fixed it! Looks good now!
", "rawMarkdown": "[quote=Konrad Banachewicz;134586]\r\n\r\n@anokas: at least within the sklearn API, you can switch from 'predict_proba' to 'predict' - although i'm not sure where does xgboost apply the threshold internally => the results could be horrible. But the code as such *will* work.\r\n\r\n[/quote]\r\n\r\nWhenever ive used xgboost custom metrics, it does not apply a threshold and the predictions that are provided are probabilities - OP should probably check that it is actually giving binary predictions otherwise his code might be quite misleading :)\r\n\r\nEDIT: Too late, you fixed it! Looks good now!", "votes": null }, { "id": "134596", "postDate": "09/07/2016 12:33:27", "content": "For this metric it needs to be maximized, not minimised. Minimizing I think the default behaviour with the existing metrics (but I could be wrong).
", "rawMarkdown": "For this metric it needs to be maximized, not minimised. Minimizing I think the default behaviour with the existing metrics (but I could be wrong).", "votes": null }, { "id": "134598", "postDate": "09/07/2016 12:49:40", "content": "Hi AlexJironkin,
\n\nThank you for sharing your code.
\nThe next thing we should think about is the way to adjust threshold.
\nIs 'conf =.5' reasonable threshold ?
Yes if you have logistic model, and No otherwise. This really needs to be tailored for the type of model used.
", "rawMarkdown": "Yes if you have logistic model, and No otherwise. This really needs to be tailored for the type of model used.", "votes": null }, { "id": "134628", "postDate": "09/07/2016 16:25:30", "content": "With labels cached/transformed before usage for R:
\n\nthreshold = 0.5\nmcc_eval <- function(preds, dtrain) {\n labels <- getinfo(dtrain, "label")\n # positives <- (labels == 1) # might be faster depending on your CPU (and how you ordered labels)\n positives <- as.logical(labels) # label to boolean\n counter <- sum(positives) # get the amount of positive labels\n tp <- as.numeric(sum(preds[positives] >= threshold))\n fp <- as.numeric(sum(preds[!positives] >= threshold))\n tn <- as.numeric(length(labels) - counter - fp) # avoid computing he opposite\n fn <- as.numeric(counter - tp) # avoid computing the opposite\n mcc <- (tp*tn-fp*fn)/(sqrt((tp+fp)*(tp+fn)*(tn+fp)*(tn+fn)))\n return(list(metric="mcc", value=ifelse(is.na(mcc), -1, mcc)))\n}\nas.numeric because integers in R have a range limit (there is a coercion to NA if you reach it).
\n\nAlso, we can compare twice only each pairs for faster computation. In the list return, ifelse is allowed to return only an element of size 1 in any case (ifelse cannot return a different thing), hence there is no slowdown. It returns -1 if the denominator is 0.
", "rawMarkdown": "With labels cached/transformed before usage for R:\r\n\r\n threshold = 0.5\r\n mcc_eval <- function(preds, dtrain) {\r\n labels <- getinfo(dtrain, \"label\")\r\n # positives <- (labels == 1) # might be faster depending on your CPU (and how you ordered labels)\r\n positives <- as.logical(labels) # label to boolean\r\n counter <- sum(positives) # get the amount of positive labels\r\n tp <- as.numeric(sum(preds[positives] >= threshold))\r\n fp <- as.numeric(sum(preds[!positives] >= threshold))\r\n tn <- as.numeric(length(labels) - counter - fp) # avoid computing he opposite\r\n fn <- as.numeric(counter - tp) # avoid computing the opposite\r\n mcc <- (tp*tn-fp*fn)/(sqrt((tp+fp)*(tp+fn)*(tn+fp)*(tn+fn)))\r\n return(list(metric=\"mcc\", value=ifelse(is.na(mcc), -1, mcc)))\r\n }\r\n\r\nas.numeric because integers in R have a range limit (there is a coercion to NA if you reach it).\r\n\r\nAlso, we can compare twice only each pairs for faster computation. In the list return, ifelse is allowed to return only an element of size 1 in any case (ifelse cannot return a different thing), hence there is no slowdown. It returns -1 if the denominator is 0.", "votes": null }, { "id": "134656", "postDate": "09/07/2016 18:31:17", "content": "Why I have this error?
\n\nError in as.character(x) : \n cannot coerce type 'closure' to vector of type 'character'
", "rawMarkdown": "Why I have this error?\r\n\r\nError in as.character(x) : \r\n cannot coerce type 'closure' to vector of type 'character'", "votes": null }, { "id": "134711", "postDate": "09/08/2016 03:03:23", "content": "[quote=Allenbj;134656]
\n\nWhy I have this error?
\n\nError in as.character(x) : \n cannot coerce type 'closure' to vector of type 'character'
\n\n[/quote]
\n\nThis happens to me when I have wrong syntax around a function. Check for missing () in your code. Or post more of the code to help.
\n\nHere is an example
\n\nmy_function <- function(x) sum(x)\nas.character(my_function) # this will fail\nas.character(my_function(1:3)) # this will work\nGood luck,\nJeff
", "rawMarkdown": "[quote=Allenbj;134656]\r\n\r\nWhy I have this error?\r\n\r\nError in as.character(x) : \r\n cannot coerce type 'closure' to vector of type 'character'\r\n\r\n[/quote]\r\n\r\nThis happens to me when I have wrong syntax around a function. Check for missing () in your code. Or post more of the code to help.\r\n\r\nHere is an example\r\n\r\n\r\n my_function <- function(x) sum(x)\r\n as.character(my_function) # this will fail\r\n as.character(my_function(1:3)) # this will work\r\n\r\nGood luck,\r\nJeff", "votes": null }, { "id": "134721", "postDate": "09/08/2016 05:08:56", "content": "Thanks, Jeff.
\n\n[quote=JeffH;134711]
\n\n[quote=Allenbj;134656]
\n\nWhy I have this error?
\n\nError in as.character(x) : \n cannot coerce type 'closure' to vector of type 'character'
\n\n[/quote]
\n\nThis happens to me when I have wrong syntax around a function. Check for missing () in your code. Or post more of the code to help.
\n\nHere is an example
\n\nmy_function <- function(x) sum(x)\nas.character(my_function) # this will fail\nas.character(my_function(1:3)) # this will work\nGood luck,\nJeff
\n\n[/quote]
", "rawMarkdown": "Thanks, Jeff.\r\n\r\n[quote=JeffH;134711]\r\n\r\n[quote=Allenbj;134656]\r\n\r\nWhy I have this error?\r\n\r\nError in as.character(x) : \r\n cannot coerce type 'closure' to vector of type 'character'\r\n\r\n[/quote]\r\n\r\nThis happens to me when I have wrong syntax around a function. Check for missing () in your code. Or post more of the code to help.\r\n\r\nHere is an example\r\n\r\n\r\n my_function <- function(x) sum(x)\r\n as.character(my_function) # this will fail\r\n as.character(my_function(1:3)) # this will work\r\n\r\nGood luck,\r\nJeff\r\n\r\n\r\n\r\n[/quote]", "votes": null } ], "comments": [ { "id": 134569, "author_name": "chinta", "author_url": "", "post_date": "09/07/2016 09:46:59", "content": "Thanks
", "votes": null, "replies": [] }, { "id": 134585, "author_name": "anokas", "author_url": "", "post_date": "09/07/2016 11:42:15", "content": "Apologies if I'm being stupid here, but wouldn't that only count the predictions that XGBoost output as exactly 1 or 0? Wouldn't you need to deal with the probablities?
", "votes": null, "replies": [] }, { "id": 134586, "author_name": "konradb", "author_url": "", "post_date": "09/07/2016 11:43:54", "content": "@anokas: at least within the sklearn API, you can switch from 'predict_proba' to 'predict' - although i'm not sure where does xgboost apply the threshold internally => the results could be horrible. But the code as such will work.
", "votes": null, "replies": [] }, { "id": 134591, "author_name": "alexjironkin", "author_url": "", "post_date": "09/07/2016 12:10:17", "content": "instead of preds == 1 and preds == 0 you can define conf = .5 outside the mcc_eval and then use preds >= conf for 1s and preds < conf for 0s. Better still you can define a conversion function classify() and use classify(preds) == 1 or classify(preds) == 0 for 1s and 0s respectively.
conf =.5\nmcc_eval <- function(preds, dtrain) {\n labels <- getinfo(dtrain, "label")\n tp <- as.numeric(sum((labels == 1) == (preds >= conf)))\n tn <- as.numeric(sum((labels == 0) == (preds < conf)))\n fp <- as.numeric(sum((labels == 0) == (preds >= conf)))\n fn <- as.numeric(sum((labels == 1) == (preds < conf)))\n mcc <- (tp*tn-fp*fn)/(sqrt((tp+fp)*(tp+fn)*(tn+fp)*(tn+fn)))\n return(list(metric="mcc", value=mcc))\n}\n[quote=Konrad Banachewicz;134586]
\n\n@anokas: at least within the sklearn API, you can switch from 'predict_proba' to 'predict' - although i'm not sure where does xgboost apply the threshold internally => the results could be horrible. But the code as such will work.
\n\n[/quote]
\n\nWhenever ive used xgboost custom metrics, it does not apply a threshold and the predictions that are provided are probabilities - OP should probably check that it is actually giving binary predictions otherwise his code might be quite misleading :)
\n\nEDIT: Too late, you fixed it! Looks good now!
", "votes": null, "replies": [] }, { "id": 134596, "author_name": "alexjironkin", "author_url": "", "post_date": "09/07/2016 12:33:27", "content": "For this metric it needs to be maximized, not minimised. Minimizing I think the default behaviour with the existing metrics (but I could be wrong).
", "votes": null, "replies": [] }, { "id": 134598, "author_name": "toshik", "author_url": "", "post_date": "09/07/2016 12:49:40", "content": "Hi AlexJironkin,
\n\nThank you for sharing your code.
\nThe next thing we should think about is the way to adjust threshold.
\nIs 'conf =.5' reasonable threshold ?
Yes if you have logistic model, and No otherwise. This really needs to be tailored for the type of model used.
", "votes": null, "replies": [] }, { "id": 134628, "author_name": "laurae2", "author_url": "", "post_date": "09/07/2016 16:25:30", "content": "With labels cached/transformed before usage for R:
\n\nthreshold = 0.5\nmcc_eval <- function(preds, dtrain) {\n labels <- getinfo(dtrain, "label")\n # positives <- (labels == 1) # might be faster depending on your CPU (and how you ordered labels)\n positives <- as.logical(labels) # label to boolean\n counter <- sum(positives) # get the amount of positive labels\n tp <- as.numeric(sum(preds[positives] >= threshold))\n fp <- as.numeric(sum(preds[!positives] >= threshold))\n tn <- as.numeric(length(labels) - counter - fp) # avoid computing he opposite\n fn <- as.numeric(counter - tp) # avoid computing the opposite\n mcc <- (tp*tn-fp*fn)/(sqrt((tp+fp)*(tp+fn)*(tn+fp)*(tn+fn)))\n return(list(metric="mcc", value=ifelse(is.na(mcc), -1, mcc)))\n}\nas.numeric because integers in R have a range limit (there is a coercion to NA if you reach it).
\n\nAlso, we can compare twice only each pairs for faster computation. In the list return, ifelse is allowed to return only an element of size 1 in any case (ifelse cannot return a different thing), hence there is no slowdown. It returns -1 if the denominator is 0.
", "votes": null, "replies": [] }, { "id": 134656, "author_name": "", "author_url": "", "post_date": "09/07/2016 18:31:17", "content": "Why I have this error?
\n\nError in as.character(x) : \n cannot coerce type 'closure' to vector of type 'character'
", "votes": null, "replies": [] }, { "id": 134711, "author_name": "jeffhebert", "author_url": "", "post_date": "09/08/2016 03:03:23", "content": "[quote=Allenbj;134656]
\n\nWhy I have this error?
\n\nError in as.character(x) : \n cannot coerce type 'closure' to vector of type 'character'
\n\n[/quote]
\n\nThis happens to me when I have wrong syntax around a function. Check for missing () in your code. Or post more of the code to help.
\n\nHere is an example
\n\nmy_function <- function(x) sum(x)\nas.character(my_function) # this will fail\nas.character(my_function(1:3)) # this will work\nGood luck,\nJeff
", "votes": null, "replies": [] }, { "id": 134721, "author_name": "", "author_url": "", "post_date": "09/08/2016 05:08:56", "content": "Thanks, Jeff.
\n\n[quote=JeffH;134711]
\n\n[quote=Allenbj;134656]
\n\nWhy I have this error?
\n\nError in as.character(x) : \n cannot coerce type 'closure' to vector of type 'character'
\n\n[/quote]
\n\nThis happens to me when I have wrong syntax around a function. Check for missing () in your code. Or post more of the code to help.
\n\nHere is an example
\n\nmy_function <- function(x) sum(x)\nas.character(my_function) # this will fail\nas.character(my_function(1:3)) # this will work\nGood luck,\nJeff
\n\n[/quote]
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "134563": "I had to write the scoring evaluation metric for my Xgboost models. I know there are plenty of examples on how to do this in [R][1] or [Python][2], but this is here for anyone to use and not reinvent the wheel.\r\n\r\n conf = .5\r\n mcc_eval <- function(preds, dtrain) {\r\n labels <- getinfo(dtrain, \"label\")\r\n tp <- as.numeric(sum((labels == 1) & (preds >= conf)))\r\n tn <- as.numeric(sum((labels == 0) & (preds < conf)))\r\n fp <- as.numeric(sum((labels == 0) & (preds >= conf)))\r\n fn <- as.numeric(sum((labels == 1) & (preds < conf)))\r\n \r\n if(tp+fp == 0) {\r\n # There are no valid TP or FP, nothing has been classified\r\n # either write or wrong. This will make a NaN, so set to 0.\r\n mcc <- 0\r\n } else {\r\n mcc <- (tp*tn-fp*fn)/(sqrt((tp+fp)*(tp+fn)*(tn+fp)*(tn+fn)))\r\n }\r\n return(list(metric=\"mcc\", value=mcc))\r\n }\r\n\r\nto use it, just stick `eval_metric=mcc_eval` instead of one of the existing metrics in xgboost.train or cv parameters.\r\n\r\nEDIT: I have added conf parameter to threshold logistic curves at 0.5. Change this to 1 if your preds are already binned into 0 or 1\r\n\r\nEDIT2: I found that `==` did not work and changed to `&`. Binary AND is the same in this context. Also added a catch for NaN.\r\n\r\n [1]: https://github.com/dmlc/xgboost/blob/master/R-package/demo/custom_objective.R\r\n [2]: https://github.com/dmlc/xgboost/blob/master/demo/guide-python/custom_objective.py", "134569": "Thanks", "134585": "Apologies if I'm being stupid here, but wouldn't that only count the predictions that XGBoost output as exactly 1 or 0? Wouldn't you need to deal with the probablities?", "134586": "anokas: at least within the sklearn API, you can switch from 'predict_proba' to 'predict' - although i'm not sure where does xgboost apply the threshold internally => the results could be horrible. But the code as such *will* work.", "134591": "instead of `preds == 1` and `preds == 0` you can define `conf = .5` outside the *mcc_eval* and then use `preds >= conf` for 1s and `preds < conf` for 0s. Better still you can define a conversion function `classify()` and use `classify(preds) == 1` or `classify(preds) == 0` for 1s and 0s respectively. \r\n\r\n conf =.5\r\n mcc_eval <- function(preds, dtrain) {\r\n labels <- getinfo(dtrain, \"label\")\r\n tp <- as.numeric(sum((labels == 1) == (preds >= conf)))\r\n tn <- as.numeric(sum((labels == 0) == (preds < conf)))\r\n fp <- as.numeric(sum((labels == 0) == (preds >= conf)))\r\n fn <- as.numeric(sum((labels == 1) == (preds < conf)))\r\n mcc <- (tp*tn-fp*fn)/(sqrt((tp+fp)*(tp+fn)*(tn+fp)*(tn+fn)))\r\n return(list(metric=\"mcc\", value=mcc))\r\n }", "134592": "[quote=Konrad Banachewicz;134586]\r\n\r\n@anokas: at least within the sklearn API, you can switch from 'predict_proba' to 'predict' - although i'm not sure where does xgboost apply the threshold internally => the results could be horrible. But the code as such *will* work.\r\n\r\n[/quote]\r\n\r\nWhenever ive used xgboost custom metrics, it does not apply a threshold and the predictions that are provided are probabilities - OP should probably check that it is actually giving binary predictions otherwise his code might be quite misleading :)\r\n\r\nEDIT: Too late, you fixed it! Looks good now!", "134596": "For this metric it needs to be maximized, not minimised. Minimizing I think the default behaviour with the existing metrics (but I could be wrong).", "134598": "Hi AlexJironkin,\r\n\r\nThank you for sharing your code.