{
  "id": 24884,
  "title": "XGBoost Custom Objective Function",
  "url": "/competitions/bosch-production-line-performance/discussion/24884",
  "author_name": "",
  "post_date": "2016-10-28T18:41:37.067Z",
  "votes": 2,
  "comment_count": 7,
  "views": 1370,
  "content": "<p>Did anyone build a working MCC objective function for xgboost and if so, did it pay off?</p>",
  "messages": [
    {
      "id": "141863",
      "postDate": "10/28/2016 18:41:37",
      "content": "<p>Did anyone build a working MCC objective function for xgboost and if so, did it pay off?</p>",
      "rawMarkdown": "Did anyone build a working MCC objective function for xgboost and if so, did it pay off?",
      "votes": null
    },
    {
      "id": "141901",
      "postDate": "10/29/2016 02:10:47",
      "content": "<p>I believe MCC is non-differentiable. therefore, it cannot be customized in xgboost.  </p>",
      "rawMarkdown": "I believe MCC is non-differentiable. therefore, it cannot be customized in xgboost.",
      "votes": null
    },
    {
      "id": "141965",
      "postDate": "10/29/2016 16:20:14",
      "content": "",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "141968",
      "postDate": "10/29/2016 16:33:02",
      "content": "<pre><code>def logregobj(preds, dtrain):\n\n\n       labels = dtrain.get_label()\n       preds = 1.0 / (1.0 + np.exp(-preds))\n       grad = preds - labels\n       hess = preds * (1.0-preds)\n       return grad, hess\n</code></pre>\n\n<p>I am wondering how to modify this for optimized mcc....</p>",
      "rawMarkdown": "def logregobj(preds, dtrain):\r\n          \r\n    \r\n           labels = dtrain.get_label()\r\n           preds = 1.0 / (1.0 + np.exp(-preds))\r\n           grad = preds - labels\r\n           hess = preds * (1.0-preds)\r\n           return grad, hess\r\n\r\nI am wondering how to modify this for optimized mcc....",
      "votes": null
    },
    {
      "id": "142071",
      "postDate": "10/30/2016 18:08:24",
      "content": "<pre><code>def mcc(y_preds, dtrain):\n    y_true = dtrain.get_label()\n    y_preds[y_preds&lt;0.5] = -1\n    y_preds[y_preds&gt;0.5] = 1\n    y_preds[y_preds==0.5] = -1\n    y_true[y_true==0] = -1\n    return 'Matthews Correlation Coefficient', matthews_corrcoef(y_true, y_preds)\n</code></pre>",
      "rawMarkdown": "def mcc(y_preds, dtrain):\r\n    \ty_true = dtrain.get_label()\r\n    \ty_preds[y_preds<0.5] = -1\r\n    \ty_preds[y_preds>0.5] = 1\r\n    \ty_preds[y_preds==0.5] = -1\r\n    \ty_true[y_true==0] = -1\r\n    \treturn 'Matthews Correlation Coefficient', matthews_corrcoef(y_true, y_preds)",
      "votes": null
    },
    {
      "id": "142140",
      "postDate": "10/31/2016 08:24:20",
      "content": "<p>Check the one used here, seems very efficient to me:</p>\n\n<p><a href=\"https://www.kaggle.com/mcosch/bosch-production-line-performance/leaky-36-in-r/code\">https://www.kaggle.com/mcosch/bosch-production-line-performance/leaky-36-in-r/code</a></p>",
      "rawMarkdown": "Check the one used here, seems very efficient to me:\r\n\r\nhttps://www.kaggle.com/mcosch/bosch-production-line-performance/leaky-36-in-r/code",
      "votes": null
    },
    {
      "id": "142148",
      "postDate": "10/31/2016 09:11:53",
      "content": "<p>You can use mcc as an evaluation metric (not the objective function) together with early termination.  I provided some code here: <a href=\"https://www.kaggle.com/cpmpml/bosch-production-line-performance/optimizing-probabilities-for-best-mcc\">https://www.kaggle.com/cpmpml/bosch-production-line-performance/optimizing-probabilities-for-best-mcc</a></p>",
      "rawMarkdown": "You can use mcc as an evaluation metric (not the objective function) together with early termination.  I provided some code here: https://www.kaggle.com/cpmpml/bosch-production-line-performance/optimizing-probabilities-for-best-mcc",
      "votes": null
    },
    {
      "id": "142423",
      "postDate": "11/02/2016 07:03:27",
      "content": "<p>Thanks guys, although some of you seem to be confusing the objective function with the evaluation metric.</p>\n\n<p>@CPMP I've been using essentially the same thing in R.  I also wrote a function to give me the theoretical optimal MCC and threshold given probabilities ONLY, but it doesn't seem to work quite as well as the determining the best MCC and threshold empirically.</p>\n\n<p>For anyone who wants it (and sorry, the forum text wrapping messed up my formatting)</p>\n\n<pre><code>optimal_mcc &lt;- function(probs, actuals=NULL, returnAll=FALSE){\n  # Find the theoretical optimal threshold and expected MCC value for a vector of probabilities\n  # Optionally, if a vector of actuals is provided, the empirical best threshold and MCC is returned\n\n  if(is.null(actuals)){\n    # Theoretical (based only on case probabilities)\n\n    thresholdsDT &lt;- data.table(Prob=probs)[order(Prob)]\n    thresholdsDT &lt;- thresholdsDT[, list(Samples=.N), keyby=Prob]\n    thresholdsDT[, Thresh := (c(tail(Prob, -1), 1) + Prob)/2]\n    thresholdsDT[, `:=`(ExpP = sum(Prob*Samples), ExpN = sum(Samples) - sum(Prob*Samples), ExpP.cum = cumsum(Prob*Samples), ExpN.cum = cumsum((1-Prob)*Samples))]\n    thresholdsDT[nrow(thresholdsDT), ExpN.cum := ExpN] # Force the last ExpTN value to = ExpN, otherwise decimal precision causes error\n    thresholdsDT[, `:=`(ExpTP = ExpP - ExpP.cum, ExpTN = ExpN.cum, ExpFP = ExpN - ExpN.cum, ExpFN = ExpP.cum)]\n    thresholdsDT[, `:=`(MCC.numerator = ExpTP * ExpTN - ExpFP * ExpFN, MCC.denominator = sqrt((ExpTP + ExpFP)*(ExpTP + ExpFN)*(ExpTN + ExpFP)*(ExpTN + ExpFN)))]\n    thresholdsDT[MCC.denominator &gt; 0, ExpMCC := MCC.numerator / MCC.denominator]\n    thresholdsDT[MCC.denominator == 0, ExpMCC := MCC.numerator]\n    if(returnAll) return(thresholdsDT) else return(thresholdsDT[which.max(ExpMCC)])\n\n  } else{\n    # Empirical (based on actual responses)\n\n    thresholdsDT &lt;- data.table(Prob=probs, Actual=actuals)[order(Prob)]\n    thresholdsDT &lt;- thresholdsDT[, list(Samples=.N, Samples.P=sum(Actual), Samples.N=sum(1-Actual)), keyby=Prob]\n    thresholdsDT[, Thresh := (c(tail(Prob, -1), 1) + Prob)/2]\n    thresholdsDT[, `:=`(P=sum(Samples.P), N=sum(Samples.N), P.cum = cumsum(Samples.P), N.cum = cumsum(Samples.N))]\n    thresholdsDT[, `:=`(TP = P - P.cum, TN = N.cum, FP = N - N.cum, FN = P.cum)]\n    thresholdsDT[, `:=`(MCC.numerator = TP * TN - FP * FN, MCC.denominator = sqrt((TP + FP)*(TP + FN)*(TN + FP)*(TN + FN)))]\n    thresholdsDT[MCC.denominator &gt; 0, MCC := MCC.numerator / MCC.denominator]\n    thresholdsDT[MCC.denominator == 0, MCC := MCC.numerator]\n    if(returnAll) return(thresholdsDT) else return(thresholdsDT[which.max(MCC)])\n  }\n}\n</code></pre>",
      "rawMarkdown": "Thanks guys, although some of you seem to be confusing the objective function with the evaluation metric.\r\n\r\n@CPMP I've been using essentially the same thing in R.  I also wrote a function to give me the theoretical optimal MCC and threshold given probabilities ONLY, but it doesn't seem to work quite as well as the determining the best MCC and threshold empirically.\r\n\r\nFor anyone who wants it (and sorry, the forum text wrapping messed up my formatting)\r\n\r\n    optimal_mcc <- function(probs, actuals=NULL, returnAll=FALSE){\r\n      # Find the theoretical optimal threshold and expected MCC value for a vector of probabilities\r\n      # Optionally, if a vector of actuals is provided, the empirical best threshold and MCC is returned\r\n      \r\n      if(is.null(actuals)){\r\n        # Theoretical (based only on case probabilities)\r\n        \r\n        thresholdsDT <- data.table(Prob=probs)[order(Prob)]\r\n        thresholdsDT <- thresholdsDT[, list(Samples=.N), keyby=Prob]\r\n        thresholdsDT[, Thresh := (c(tail(Prob, -1), 1) + Prob)/2]\r\n        thresholdsDT[, `:=`(ExpP = sum(Prob*Samples), ExpN = sum(Samples) - sum(Prob*Samples), ExpP.cum = cumsum(Prob*Samples), ExpN.cum = cumsum((1-Prob)*Samples))]\r\n        thresholdsDT[nrow(thresholdsDT), ExpN.cum := ExpN] # Force the last ExpTN value to = ExpN, otherwise decimal precision causes error\r\n        thresholdsDT[, `:=`(ExpTP = ExpP - ExpP.cum, ExpTN = ExpN.cum, ExpFP = ExpN - ExpN.cum, ExpFN = ExpP.cum)]\r\n        thresholdsDT[, `:=`(MCC.numerator = ExpTP * ExpTN - ExpFP * ExpFN, MCC.denominator = sqrt((ExpTP + ExpFP)*(ExpTP + ExpFN)*(ExpTN + ExpFP)*(ExpTN + ExpFN)))]\r\n        thresholdsDT[MCC.denominator > 0, ExpMCC := MCC.numerator / MCC.denominator]\r\n        thresholdsDT[MCC.denominator == 0, ExpMCC := MCC.numerator]\r\n        if(returnAll) return(thresholdsDT) else return(thresholdsDT[which.max(ExpMCC)])\r\n        \r\n      } else{\r\n        # Empirical (based on actual responses)\r\n        \r\n        thresholdsDT <- data.table(Prob=probs, Actual=actuals)[order(Prob)]\r\n        thresholdsDT <- thresholdsDT[, list(Samples=.N, Samples.P=sum(Actual), Samples.N=sum(1-Actual)), keyby=Prob]\r\n        thresholdsDT[, Thresh := (c(tail(Prob, -1), 1) + Prob)/2]\r\n        thresholdsDT[, `:=`(P=sum(Samples.P), N=sum(Samples.N), P.cum = cumsum(Samples.P), N.cum = cumsum(Samples.N))]\r\n        thresholdsDT[, `:=`(TP = P - P.cum, TN = N.cum, FP = N - N.cum, FN = P.cum)]\r\n        thresholdsDT[, `:=`(MCC.numerator = TP * TN - FP * FN, MCC.denominator = sqrt((TP + FP)*(TP + FN)*(TN + FP)*(TN + FN)))]\r\n        thresholdsDT[MCC.denominator > 0, MCC := MCC.numerator / MCC.denominator]\r\n        thresholdsDT[MCC.denominator == 0, MCC := MCC.numerator]\r\n        if(returnAll) return(thresholdsDT) else return(thresholdsDT[which.max(MCC)])\r\n      }\r\n    }",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 141901,
      "author_name": "lishubin",
      "author_url": "",
      "post_date": "10/29/2016 02:10:47",
      "content": "<p>I believe MCC is non-differentiable. therefore, it cannot be customized in xgboost.  </p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 141965,
      "author_name": "yuyuzhou",
      "author_url": "",
      "post_date": "10/29/2016 16:20:14",
      "content": "",
      "votes": null,
      "replies": []
    },
    {
      "id": 141968,
      "author_name": "yuyuzhou",
      "author_url": "",
      "post_date": "10/29/2016 16:33:02",
      "content": "<pre><code>def logregobj(preds, dtrain):\n\n\n       labels = dtrain.get_label()\n       preds = 1.0 / (1.0 + np.exp(-preds))\n       grad = preds - labels\n       hess = preds * (1.0-preds)\n       return grad, hess\n</code></pre>\n\n<p>I am wondering how to modify this for optimized mcc....</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 142071,
      "author_name": "shahnawazakhtar",
      "author_url": "",
      "post_date": "10/30/2016 18:08:24",
      "content": "<pre><code>def mcc(y_preds, dtrain):\n    y_true = dtrain.get_label()\n    y_preds[y_preds&lt;0.5] = -1\n    y_preds[y_preds&gt;0.5] = 1\n    y_preds[y_preds==0.5] = -1\n    y_true[y_true==0] = -1\n    return 'Matthews Correlation Coefficient', matthews_corrcoef(y_true, y_preds)\n</code></pre>",
      "votes": null,
      "replies": []
    },
    {
      "id": 142140,
      "author_name": "barasztamas",
      "author_url": "",
      "post_date": "10/31/2016 08:24:20",
      "content": "<p>Check the one used here, seems very efficient to me:</p>\n\n<p><a href=\"https://www.kaggle.com/mcosch/bosch-production-line-performance/leaky-36-in-r/code\">https://www.kaggle.com/mcosch/bosch-production-line-performance/leaky-36-in-r/code</a></p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 142148,
      "author_name": "cpmpml",
      "author_url": "",
      "post_date": "10/31/2016 09:11:53",
      "content": "<p>You can use mcc as an evaluation metric (not the objective function) together with early termination.  I provided some code here: <a href=\"https://www.kaggle.com/cpmpml/bosch-production-line-performance/optimizing-probabilities-for-best-mcc\">https://www.kaggle.com/cpmpml/bosch-production-line-performance/optimizing-probabilities-for-best-mcc</a></p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 142423,
      "author_name": "ben519",
      "author_url": "",
      "post_date": "11/02/2016 07:03:27",
      "content": "<p>Thanks guys, although some of you seem to be confusing the objective function with the evaluation metric.</p>\n\n<p>@CPMP I've been using essentially the same thing in R.  I also wrote a function to give me the theoretical optimal MCC and threshold given probabilities ONLY, but it doesn't seem to work quite as well as the determining the best MCC and threshold empirically.</p>\n\n<p>For anyone who wants it (and sorry, the forum text wrapping messed up my formatting)</p>\n\n<pre><code>optimal_mcc &lt;- function(probs, actuals=NULL, returnAll=FALSE){\n  # Find the theoretical optimal threshold and expected MCC value for a vector of probabilities\n  # Optionally, if a vector of actuals is provided, the empirical best threshold and MCC is returned\n\n  if(is.null(actuals)){\n    # Theoretical (based only on case probabilities)\n\n    thresholdsDT &lt;- data.table(Prob=probs)[order(Prob)]\n    thresholdsDT &lt;- thresholdsDT[, list(Samples=.N), keyby=Prob]\n    thresholdsDT[, Thresh := (c(tail(Prob, -1), 1) + Prob)/2]\n    thresholdsDT[, `:=`(ExpP = sum(Prob*Samples), ExpN = sum(Samples) - sum(Prob*Samples), ExpP.cum = cumsum(Prob*Samples), ExpN.cum = cumsum((1-Prob)*Samples))]\n    thresholdsDT[nrow(thresholdsDT), ExpN.cum := ExpN] # Force the last ExpTN value to = ExpN, otherwise decimal precision causes error\n    thresholdsDT[, `:=`(ExpTP = ExpP - ExpP.cum, ExpTN = ExpN.cum, ExpFP = ExpN - ExpN.cum, ExpFN = ExpP.cum)]\n    thresholdsDT[, `:=`(MCC.numerator = ExpTP * ExpTN - ExpFP * ExpFN, MCC.denominator = sqrt((ExpTP + ExpFP)*(ExpTP + ExpFN)*(ExpTN + ExpFP)*(ExpTN + ExpFN)))]\n    thresholdsDT[MCC.denominator &gt; 0, ExpMCC := MCC.numerator / MCC.denominator]\n    thresholdsDT[MCC.denominator == 0, ExpMCC := MCC.numerator]\n    if(returnAll) return(thresholdsDT) else return(thresholdsDT[which.max(ExpMCC)])\n\n  } else{\n    # Empirical (based on actual responses)\n\n    thresholdsDT &lt;- data.table(Prob=probs, Actual=actuals)[order(Prob)]\n    thresholdsDT &lt;- thresholdsDT[, list(Samples=.N, Samples.P=sum(Actual), Samples.N=sum(1-Actual)), keyby=Prob]\n    thresholdsDT[, Thresh := (c(tail(Prob, -1), 1) + Prob)/2]\n    thresholdsDT[, `:=`(P=sum(Samples.P), N=sum(Samples.N), P.cum = cumsum(Samples.P), N.cum = cumsum(Samples.N))]\n    thresholdsDT[, `:=`(TP = P - P.cum, TN = N.cum, FP = N - N.cum, FN = P.cum)]\n    thresholdsDT[, `:=`(MCC.numerator = TP * TN - FP * FN, MCC.denominator = sqrt((TP + FP)*(TP + FN)*(TN + FP)*(TN + FN)))]\n    thresholdsDT[MCC.denominator &gt; 0, MCC := MCC.numerator / MCC.denominator]\n    thresholdsDT[MCC.denominator == 0, MCC := MCC.numerator]\n    if(returnAll) return(thresholdsDT) else return(thresholdsDT[which.max(MCC)])\n  }\n}\n</code></pre>",
      "votes": null,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "141863": "Did anyone build a working MCC objective function for xgboost and if so, did it pay off?",
    "141901": "I believe MCC is non-differentiable. therefore, it cannot be customized in xgboost.",
    "141965": "",
    "141968": "def logregobj(preds, dtrain):\r\n          \r\n    \r\n           labels = dtrain.get_label()\r\n           preds = 1.0 / (1.0 + np.exp(-preds))\r\n           grad = preds - labels\r\n           hess = preds * (1.0-preds)\r\n           return grad, hess\r\n\r\nI am wondering how to modify this for optimized mcc....",
    "142071": "def mcc(y_preds, dtrain):\r\n    \ty_true = dtrain.get_label()\r\n    \ty_preds[y_preds<0.5] = -1\r\n    \ty_preds[y_preds>0.5] = 1\r\n    \ty_preds[y_preds==0.5] = -1\r\n    \ty_true[y_true==0] = -1\r\n    \treturn 'Matthews Correlation Coefficient', matthews_corrcoef(y_true, y_preds)",
    "142140": "Check the one used here, seems very efficient to me:\r\n\r\nhttps://www.kaggle.com/mcosch/bosch-production-line-performance/leaky-36-in-r/code",
    "142148": "You can use mcc as an evaluation metric (not the objective function) together with early termination.  I provided some code here: https://www.kaggle.com/cpmpml/bosch-production-line-performance/optimizing-probabilities-for-best-mcc",
    "142423": "Thanks guys, although some of you seem to be confusing the objective function with the evaluation metric.\r\n\r\n@CPMP I've been using essentially the same thing in R.  I also wrote a function to give me the theoretical optimal MCC and threshold given probabilities ONLY, but it doesn't seem to work quite as well as the determining the best MCC and threshold empirically.\r\n\r\nFor anyone who wants it (and sorry, the forum text wrapping messed up my formatting)\r\n\r\n    optimal_mcc <- function(probs, actuals=NULL, returnAll=FALSE){\r\n      # Find the theoretical optimal threshold and expected MCC value for a vector of probabilities\r\n      # Optionally, if a vector of actuals is provided, the empirical best threshold and MCC is returned\r\n      \r\n      if(is.null(actuals)){\r\n        # Theoretical (based only on case probabilities)\r\n        \r\n        thresholdsDT <- data.table(Prob=probs)[order(Prob)]\r\n        thresholdsDT <- thresholdsDT[, list(Samples=.N), keyby=Prob]\r\n        thresholdsDT[, Thresh := (c(tail(Prob, -1), 1) + Prob)/2]\r\n        thresholdsDT[, `:=`(ExpP = sum(Prob*Samples), ExpN = sum(Samples) - sum(Prob*Samples), ExpP.cum = cumsum(Prob*Samples), ExpN.cum = cumsum((1-Prob)*Samples))]\r\n        thresholdsDT[nrow(thresholdsDT), ExpN.cum := ExpN] # Force the last ExpTN value to = ExpN, otherwise decimal precision causes error\r\n        thresholdsDT[, `:=`(ExpTP = ExpP - ExpP.cum, ExpTN = ExpN.cum, ExpFP = ExpN - ExpN.cum, ExpFN = ExpP.cum)]\r\n        thresholdsDT[, `:=`(MCC.numerator = ExpTP * ExpTN - ExpFP * ExpFN, MCC.denominator = sqrt((ExpTP + ExpFP)*(ExpTP + ExpFN)*(ExpTN + ExpFP)*(ExpTN + ExpFN)))]\r\n        thresholdsDT[MCC.denominator > 0, ExpMCC := MCC.numerator / MCC.denominator]\r\n        thresholdsDT[MCC.denominator == 0, ExpMCC := MCC.numerator]\r\n        if(returnAll) return(thresholdsDT) else return(thresholdsDT[which.max(ExpMCC)])\r\n        \r\n      } else{\r\n        # Empirical (based on actual responses)\r\n        \r\n        thresholdsDT <- data.table(Prob=probs, Actual=actuals)[order(Prob)]\r\n        thresholdsDT <- thresholdsDT[, list(Samples=.N, Samples.P=sum(Actual), Samples.N=sum(1-Actual)), keyby=Prob]\r\n        thresholdsDT[, Thresh := (c(tail(Prob, -1), 1) + Prob)/2]\r\n        thresholdsDT[, `:=`(P=sum(Samples.P), N=sum(Samples.N), P.cum = cumsum(Samples.P), N.cum = cumsum(Samples.N))]\r\n        thresholdsDT[, `:=`(TP = P - P.cum, TN = N.cum, FP = N - N.cum, FN = P.cum)]\r\n        thresholdsDT[, `:=`(MCC.numerator = TP * TN - FP * FN, MCC.denominator = sqrt((TP + FP)*(TP + FN)*(TN + FP)*(TN + FN)))]\r\n        thresholdsDT[MCC.denominator > 0, MCC := MCC.numerator / MCC.denominator]\r\n        thresholdsDT[MCC.denominator == 0, MCC := MCC.numerator]\r\n        if(returnAll) return(thresholdsDT) else return(thresholdsDT[which.max(MCC)])\r\n      }\r\n    }"
  },
  "source": "meta"
}