{ "id": 23909, "title": "Question on how to implement Matthews Correlation Coefficient as xgboost eval metrics", "url": "/competitions/bosch-production-line-performance/discussion/23909", "author_name": "", "post_date": "2016-09-26T13:20:40.143Z", "votes": 1, "comment_count": 5, "views": 1392, "content": "

Hi there,

\n\n

I am currently trying to use xgboost on Python. I have been reading the documentation trying to use the Matthews Correlation Coefficient (MCC) as evaluation metric but I'm having trouble.

\n\n

First question is on the xgboost.cv function. The call for that function is as follows:

\n\n
xgboost.cv(params, dtrain, num_boost_round=10, nfold=3, stratified=False, folds=None, metrics=(), obj=None, feval=None, maximize=False, early_stopping_rounds=None, fpreproc=None, as_pandas=True, verbose_eval=None, show_stdv=True, seed=0, callbacks=None)\n
\n\n

I have used this in the past with standard metrics. I would pass metrics=['error', 'auc'] and it would work. But the MCC is not a standard metric. So, I thought that maybe I could pass it to feval by first importing the metric:

\n\n
from sklearn.metrics import matthews_corrcoef\n
\n\n

and then passing in the xgboost.cv function above feval=matthews_corrcoef and add maximize=True. However, that does not work. Is it possible to use the MCC as a metric within this function? Can anyone explain how it would be done?

\n\n

Second question is on the fit function. A basic example of my code would be:

\n\n
clf = xgboost.XGBClassifier(objective='binary:logistic', subsample=0.8, colsample_bytree=0.8, base_score=0.005, seed=1234)\n\nclf.fit(trainX, trainy, eval_set=[(trainX, trainy), (evalX, evaly)], eval_metric='auc', early_stopping_rounds=50, verbose=True)\n
\n\n

However, again, I don't want to use auc as evaluation metric, but MCC. The documentation says that eval_metric can be either a str and therefore a built-in evaluation metric, or callable (a custom evaluation metric). In this second case, the call signature is func(y_pred, y_true) where y_true will be a DMatrix so the get_label method can be used. It also says that it must return a (str, value) pair where the str is a name for the evaluation and value is the value of the evaluation function. It also says that this objective will be minimized.

\n\n

I am a bit confuse as how to implement this. Could anyone suggest a solution?

\n\n

Huge thanks!

", "messages": [ { "id": "136914", "postDate": "09/26/2016 13:20:40", "content": "

Hi there,

\n\n

I am currently trying to use xgboost on Python. I have been reading the documentation trying to use the Matthews Correlation Coefficient (MCC) as evaluation metric but I'm having trouble.

\n\n

First question is on the xgboost.cv function. The call for that function is as follows:

\n\n
xgboost.cv(params, dtrain, num_boost_round=10, nfold=3, stratified=False, folds=None, metrics=(), obj=None, feval=None, maximize=False, early_stopping_rounds=None, fpreproc=None, as_pandas=True, verbose_eval=None, show_stdv=True, seed=0, callbacks=None)\n
\n\n

I have used this in the past with standard metrics. I would pass metrics=['error', 'auc'] and it would work. But the MCC is not a standard metric. So, I thought that maybe I could pass it to feval by first importing the metric:

\n\n
from sklearn.metrics import matthews_corrcoef\n
\n\n

and then passing in the xgboost.cv function above feval=matthews_corrcoef and add maximize=True. However, that does not work. Is it possible to use the MCC as a metric within this function? Can anyone explain how it would be done?

\n\n

Second question is on the fit function. A basic example of my code would be:

\n\n
clf = xgboost.XGBClassifier(objective='binary:logistic', subsample=0.8, colsample_bytree=0.8, base_score=0.005, seed=1234)\n\nclf.fit(trainX, trainy, eval_set=[(trainX, trainy), (evalX, evaly)], eval_metric='auc', early_stopping_rounds=50, verbose=True)\n
\n\n

However, again, I don't want to use auc as evaluation metric, but MCC. The documentation says that eval_metric can be either a str and therefore a built-in evaluation metric, or callable (a custom evaluation metric). In this second case, the call signature is func(y_pred, y_true) where y_true will be a DMatrix so the get_label method can be used. It also says that it must return a (str, value) pair where the str is a name for the evaluation and value is the value of the evaluation function. It also says that this objective will be minimized.

\n\n

I am a bit confuse as how to implement this. Could anyone suggest a solution?

\n\n

Huge thanks!

", "rawMarkdown": "Hi there,\r\n\r\nI am currently trying to use xgboost on Python. I have been reading the documentation trying to use the Matthews Correlation Coefficient (MCC) as evaluation metric but I'm having trouble.\r\n\r\n**First question** is on the `xgboost.cv` function. The call for that function is as follows:\r\n\r\n xgboost.cv(params, dtrain, num_boost_round=10, nfold=3, stratified=False, folds=None, metrics=(), obj=None, feval=None, maximize=False, early_stopping_rounds=None, fpreproc=None, as_pandas=True, verbose_eval=None, show_stdv=True, seed=0, callbacks=None)\r\n\r\nI have used this in the past with standard metrics. I would pass `metrics=['error', 'auc']` and it would work. But the MCC is not a standard metric. So, I thought that maybe I could pass it to `feval` by first importing the metric:\r\n\r\n from sklearn.metrics import matthews_corrcoef\r\n\r\nand then passing in the `xgboost.cv` function above `feval=matthews_corrcoef` and add `maximize=True`. However, that does not work. Is it possible to use the MCC as a metric within this function? Can anyone explain how it would be done?\r\n\r\n\r\n**Second question** is on the `fit` function. A basic example of my code would be:\r\n\r\n clf = xgboost.XGBClassifier(objective='binary:logistic', subsample=0.8, colsample_bytree=0.8, base_score=0.005, seed=1234)\r\n\r\n clf.fit(trainX, trainy, eval_set=[(trainX, trainy), (evalX, evaly)], eval_metric='auc', early_stopping_rounds=50, verbose=True)\r\n\r\nHowever, again, I don't want to use auc as evaluation metric, but MCC. The documentation says that `eval_metric` can be either a str and therefore a built-in evaluation metric, or callable (a custom evaluation metric). In this second case, the call signature is `func(y_pred, y_true)` where `y_true` will be a DMatrix so the `get_label` method can be used. It also says that it must return a (str, value) pair where the str is a name for the evaluation and value is the value of the evaluation function. It also says that this objective will be minimized. \r\n\r\nI am a bit confuse as how to implement this. Could anyone suggest a solution?\r\n\r\nHuge thanks!", "votes": null }, { "id": "136918", "postDate": "09/26/2016 13:51:10", "content": "

I think the sklearn MCC function does not fit the XGB requirements for a custom loss fct. take a look at this example https://github.com/dmlc/xgboost/blob/master/demo/guide-python/cross_validation.py

\n\n

you can also find a script in the kernel section of this competition, which additionally tries to deal with the cutoff probability (unbalanced classes):\nhttps://www.kaggle.com/cpmpml/bosch-production-line-performance/optimizing-probabilities-for-best-mcc

", "rawMarkdown": "I think the sklearn MCC function does not fit the XGB requirements for a custom loss fct. take a look at this example https://github.com/dmlc/xgboost/blob/master/demo/guide-python/cross_validation.py\r\n\r\nyou can also find a script in the kernel section of this competition, which additionally tries to deal with the cutoff probability (unbalanced classes):\r\nhttps://www.kaggle.com/cpmpml/bosch-production-line-performance/optimizing-probabilities-for-best-mcc", "votes": null }, { "id": "136936", "postDate": "09/26/2016 17:25:49", "content": "

What about this method:

\n\n

import xgboost as xgb

\n\n

from sklearn.metrics import matthews_corrcoef

\n\n

from sklearn.grid_search import GridSearchCV

\n\n

def matthews_corrcoef3(estimator, X_test, y_test):

\n\n
   return matthews_corrcoef(y_test, estimator.predict(X_test))\n
\n\n

model = xgb.XGBClassifier()

\n\n

grid_search = GridSearchCV(model, {... model params...}, scoring=matthews_corrcoef3, ...)

\n\n

...

", "rawMarkdown": "What about this method:\r\n\r\nimport xgboost as xgb\r\n\r\nfrom sklearn.metrics import matthews_corrcoef\r\n\r\nfrom sklearn.grid_search import GridSearchCV\r\n\r\ndef matthews_corrcoef3(estimator, X_test, y_test):\r\n\r\n return matthews_corrcoef(y_test, estimator.predict(X_test))\r\n\r\nmodel = xgb.XGBClassifier()\r\n\r\ngrid_search = GridSearchCV(model, {... model params...}, scoring=matthews_corrcoef3, ...)\r\n\r\n...", "votes": null }, { "id": "136938", "postDate": "09/26/2016 18:15:08", "content": "
from sklearn.metrics import matthews_corrcoef\n\ndef evalerror(preds, dtrain):\n    labels = dtrain.get_label()\n    thresholds = np.linspace(0.01, 0.99, 50)\n    mcc = np.array([matthews_corrcoef(labels, preds>thr) for thr in thresholds])\n    best_score = mcc.max()\n    return 'error', -best_score\n
\n\n

num_round = 3

\n\n

bst = xgb.train(dtrain, num_round, watchlist, verbose_eval=True, early_stopping_rounds=20, feval = evalerror )

", "rawMarkdown": "from sklearn.metrics import matthews_corrcoef\r\n\r\n def evalerror(preds, dtrain):\r\n labels = dtrain.get_label()\r\n thresholds = np.linspace(0.01, 0.99, 50)\r\n mcc = np.array([matthews_corrcoef(labels, preds>thr) for thr in thresholds])\r\n best_score = mcc.max()\r\n return 'error', -best_score\r\n\r\nnum_round = 3\r\n\r\n\r\nbst = xgb.train(dtrain, num_round, watchlist, verbose_eval=True, early_stopping_rounds=20, feval = evalerror )", "votes": null }, { "id": "136940", "postDate": "09/26/2016 18:30:15", "content": "

Attached is a solution for using MCC as evaluation function with a fixed threshold. Hope it helps. :)

", "rawMarkdown": "Attached is a solution for using MCC as evaluation function with a fixed threshold. Hope it helps. :)", "votes": null }, { "id": "1102977", "postDate": "12/05/2020 14:31:51", "content": "

can you explain me why do you use a thresholds?

", "rawMarkdown": "can you explain me why do you use a thresholds?", "votes": null } ], "comments": [ { "id": 136918, "author_name": "stgrmks", "author_url": "", "post_date": "09/26/2016 13:51:10", "content": "

I think the sklearn MCC function does not fit the XGB requirements for a custom loss fct. take a look at this example https://github.com/dmlc/xgboost/blob/master/demo/guide-python/cross_validation.py

\n\n

you can also find a script in the kernel section of this competition, which additionally tries to deal with the cutoff probability (unbalanced classes):\nhttps://www.kaggle.com/cpmpml/bosch-production-line-performance/optimizing-probabilities-for-best-mcc

", "votes": null, "replies": [] }, { "id": 136936, "author_name": "dominique", "author_url": "", "post_date": "09/26/2016 17:25:49", "content": "

What about this method:

\n\n

import xgboost as xgb

\n\n

from sklearn.metrics import matthews_corrcoef

\n\n

from sklearn.grid_search import GridSearchCV

\n\n

def matthews_corrcoef3(estimator, X_test, y_test):

\n\n
   return matthews_corrcoef(y_test, estimator.predict(X_test))\n
\n\n

model = xgb.XGBClassifier()

\n\n

grid_search = GridSearchCV(model, {... model params...}, scoring=matthews_corrcoef3, ...)

\n\n

...

", "votes": null, "replies": [] }, { "id": 136938, "author_name": "notaapple", "author_url": "", "post_date": "09/26/2016 18:15:08", "content": "
from sklearn.metrics import matthews_corrcoef\n\ndef evalerror(preds, dtrain):\n    labels = dtrain.get_label()\n    thresholds = np.linspace(0.01, 0.99, 50)\n    mcc = np.array([matthews_corrcoef(labels, preds>thr) for thr in thresholds])\n    best_score = mcc.max()\n    return 'error', -best_score\n
\n\n

num_round = 3

\n\n

bst = xgb.train(dtrain, num_round, watchlist, verbose_eval=True, early_stopping_rounds=20, feval = evalerror )

", "votes": null, "replies": [ { "id": 1102977, "author_name": "zakariacheurfi", "author_url": "", "post_date": "12/05/2020 14:31:51", "content": "

can you explain me why do you use a thresholds?

", "votes": null, "replies": [] } ] }, { "id": 136940, "author_name": "mmueller", "author_url": "", "post_date": "09/26/2016 18:30:15", "content": "

Attached is a solution for using MCC as evaluation function with a fixed threshold. Hope it helps. :)

", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "136914": "Hi there,\r\n\r\nI am currently trying to use xgboost on Python. I have been reading the documentation trying to use the Matthews Correlation Coefficient (MCC) as evaluation metric but I'm having trouble.\r\n\r\n**First question** is on the `xgboost.cv` function. The call for that function is as follows:\r\n\r\n xgboost.cv(params, dtrain, num_boost_round=10, nfold=3, stratified=False, folds=None, metrics=(), obj=None, feval=None, maximize=False, early_stopping_rounds=None, fpreproc=None, as_pandas=True, verbose_eval=None, show_stdv=True, seed=0, callbacks=None)\r\n\r\nI have used this in the past with standard metrics. I would pass `metrics=['error', 'auc']` and it would work. But the MCC is not a standard metric. So, I thought that maybe I could pass it to `feval` by first importing the metric:\r\n\r\n from sklearn.metrics import matthews_corrcoef\r\n\r\nand then passing in the `xgboost.cv` function above `feval=matthews_corrcoef` and add `maximize=True`. However, that does not work. Is it possible to use the MCC as a metric within this function? Can anyone explain how it would be done?\r\n\r\n\r\n**Second question** is on the `fit` function. A basic example of my code would be:\r\n\r\n clf = xgboost.XGBClassifier(objective='binary:logistic', subsample=0.8, colsample_bytree=0.8, base_score=0.005, seed=1234)\r\n\r\n clf.fit(trainX, trainy, eval_set=[(trainX, trainy), (evalX, evaly)], eval_metric='auc', early_stopping_rounds=50, verbose=True)\r\n\r\nHowever, again, I don't want to use auc as evaluation metric, but MCC. The documentation says that `eval_metric` can be either a str and therefore a built-in evaluation metric, or callable (a custom evaluation metric). In this second case, the call signature is `func(y_pred, y_true)` where `y_true` will be a DMatrix so the `get_label` method can be used. It also says that it must return a (str, value) pair where the str is a name for the evaluation and value is the value of the evaluation function. It also says that this objective will be minimized. \r\n\r\nI am a bit confuse as how to implement this. Could anyone suggest a solution?\r\n\r\nHuge thanks!", "136918": "I think the sklearn MCC function does not fit the XGB requirements for a custom loss fct. take a look at this example https://github.com/dmlc/xgboost/blob/master/demo/guide-python/cross_validation.py\r\n\r\nyou can also find a script in the kernel section of this competition, which additionally tries to deal with the cutoff probability (unbalanced classes):\r\nhttps://www.kaggle.com/cpmpml/bosch-production-line-performance/optimizing-probabilities-for-best-mcc", "136936": "What about this method:\r\n\r\nimport xgboost as xgb\r\n\r\nfrom sklearn.metrics import matthews_corrcoef\r\n\r\nfrom sklearn.grid_search import GridSearchCV\r\n\r\ndef matthews_corrcoef3(estimator, X_test, y_test):\r\n\r\n return matthews_corrcoef(y_test, estimator.predict(X_test))\r\n\r\nmodel = xgb.XGBClassifier()\r\n\r\ngrid_search = GridSearchCV(model, {... model params...}, scoring=matthews_corrcoef3, ...)\r\n\r\n...", "136938": "from sklearn.metrics import matthews_corrcoef\r\n\r\n def evalerror(preds, dtrain):\r\n labels = dtrain.get_label()\r\n thresholds = np.linspace(0.01, 0.99, 50)\r\n mcc = np.array([matthews_corrcoef(labels, preds>thr) for thr in thresholds])\r\n best_score = mcc.max()\r\n return 'error', -best_score\r\n\r\nnum_round = 3\r\n\r\n\r\nbst = xgb.train(dtrain, num_round, watchlist, verbose_eval=True, early_stopping_rounds=20, feval = evalerror )", "136940": "Attached is a solution for using MCC as evaluation function with a fixed threshold. Hope it helps. :)", "1102977": "can you explain me why do you use a thresholds?" }, "source": "meta" }