{
  "id": 22917,
  "title": "Optimising probabilities -> binary prediction (script)",
  "url": "/competitions/bosch-production-line-performance/discussion/22917",
  "author_name": "",
  "post_date": "2016-08-17T19:23:25.103Z",
  "votes": 29,
  "comment_count": 8,
  "views": 3583,
  "content": "<p>Hi everyone,</p>\n\n<p>I wrote some quick python functions for automagically optimising the threshold for converting probabilities from an algorithm such as XGBoost into a binary prediction that can be submitted to this competition. This checks the validation score at many different thresholds and then selects the best one to apply to the testing data. If set to verbose, it also prints out a validation score.</p>\n\n<p>The <code>try_all</code> option will try every single value as a threshold, otherwise a step size of 0.01 is used (which should be sufficient).</p>\n\n<pre><code>import numpy as np\nfrom sklearn.metrics import matthews_corrcoef\n\ndef find_matthews_threshold(p_valid, y_valid, try_all=False, verbose=False):\n    p_valid, y_valid = np.array(p_valid), np.array(y_valid)\n\n    best = 0\n    best_score = -2\n    totry = np.arange(0,1,0.01) if try_all is False else np.unique(p_valid)\n    for t in totry:\n        score = matthews_corrcoef(y_valid, p_valid &gt; t)\n        if score &gt; best_score:\n            best_score = score\n            best = t\n    if verbose is True: \n        print('Best score: ', round(best_score, 5), ' @ threshold ', best)\n\n    return best\n\ndef best_threshold_submission(p_valid, y_valid, p_test, try_all=False, verbose=False):\n    p_test = np.array(p_test)\n    thresh = find_matthews_threshold(p_valid, y_valid, try_all, verbose)\n    return p_test &gt; thresh\n\nsubmission_values = best_threshold_submission(validation_probabilities, validation_groundtruth, test_probabilities)\n</code></pre>\n\n<p>Hopefully this helps, and obviously let me know if there are any improvements to be made/bugs to be fixed :)</p>\n\n<p>- Mikel</p>",
  "messages": [
    {
      "id": "131509",
      "postDate": "08/17/2016 19:23:25",
      "content": "<p>Hi everyone,</p>\n\n<p>I wrote some quick python functions for automagically optimising the threshold for converting probabilities from an algorithm such as XGBoost into a binary prediction that can be submitted to this competition. This checks the validation score at many different thresholds and then selects the best one to apply to the testing data. If set to verbose, it also prints out a validation score.</p>\n\n<p>The <code>try_all</code> option will try every single value as a threshold, otherwise a step size of 0.01 is used (which should be sufficient).</p>\n\n<pre><code>import numpy as np\nfrom sklearn.metrics import matthews_corrcoef\n\ndef find_matthews_threshold(p_valid, y_valid, try_all=False, verbose=False):\n    p_valid, y_valid = np.array(p_valid), np.array(y_valid)\n\n    best = 0\n    best_score = -2\n    totry = np.arange(0,1,0.01) if try_all is False else np.unique(p_valid)\n    for t in totry:\n        score = matthews_corrcoef(y_valid, p_valid &gt; t)\n        if score &gt; best_score:\n            best_score = score\n            best = t\n    if verbose is True: \n        print('Best score: ', round(best_score, 5), ' @ threshold ', best)\n\n    return best\n\ndef best_threshold_submission(p_valid, y_valid, p_test, try_all=False, verbose=False):\n    p_test = np.array(p_test)\n    thresh = find_matthews_threshold(p_valid, y_valid, try_all, verbose)\n    return p_test &gt; thresh\n\nsubmission_values = best_threshold_submission(validation_probabilities, validation_groundtruth, test_probabilities)\n</code></pre>\n\n<p>Hopefully this helps, and obviously let me know if there are any improvements to be made/bugs to be fixed :)</p>\n\n<p>- Mikel</p>",
      "rawMarkdown": "Hi everyone,\r\n\r\nI wrote some quick python functions for automagically optimising the threshold for converting probabilities from an algorithm such as XGBoost into a binary prediction that can be submitted to this competition. This checks the validation score at many different thresholds and then selects the best one to apply to the testing data. If set to verbose, it also prints out a validation score.\r\n\r\nThe `try_all` option will try every single value as a threshold, otherwise a step size of 0.01 is used (which should be sufficient).\r\n\r\n    import numpy as np\r\n    from sklearn.metrics import matthews_corrcoef\r\n    \r\n    def find_matthews_threshold(p_valid, y_valid, try_all=False, verbose=False):\r\n        p_valid, y_valid = np.array(p_valid), np.array(y_valid)\r\n    \r\n        best = 0\r\n        best_score = -2\r\n        totry = np.arange(0,1,0.01) if try_all is False else np.unique(p_valid)\r\n        for t in totry:\r\n            score = matthews_corrcoef(y_valid, p_valid > t)\r\n            if score > best_score:\r\n                best_score = score\r\n                best = t\r\n        if verbose is True: \r\n            print('Best score: ', round(best_score, 5), ' @ threshold ', best)\r\n    \r\n        return best\r\n    \r\n    def best_threshold_submission(p_valid, y_valid, p_test, try_all=False, verbose=False):\r\n        p_test = np.array(p_test)\r\n        thresh = find_matthews_threshold(p_valid, y_valid, try_all, verbose)\r\n        return p_test > thresh\r\n    \r\n    submission_values = best_threshold_submission(validation_probabilities, validation_groundtruth, test_probabilities)\r\n\r\nHopefully this helps, and obviously let me know if there are any improvements to be made/bugs to be fixed :)\r\n\r\n\\- Mikel",
      "votes": null
    },
    {
      "id": "131512",
      "postDate": "08/17/2016 19:38:07",
      "content": "<p>This looks nice and quick to optimize using a specific threshold.</p>\n\n<p>I got another idea which is different from yours, but uses probability outputs directly for defining the threshold. What about the following steps?</p>\n\n<ul>\n<li>Ordering distinct probabilities</li>\n<li>Computing TP/TN when assuming the threshold for each distinct probability</li>\n<li>Computing the MCC</li>\n<li>Returning the probability row with the maximum MCC</li>\n<li>Optional: plot the probability/MCC curve</li>\n</ul>",
      "rawMarkdown": "This looks nice and quick to optimize using a specific threshold.\r\n\r\nI got another idea which is different from yours, but uses probability outputs directly for defining the threshold. What about the following steps?\r\n\r\n* Ordering distinct probabilities\r\n* Computing TP/TN when assuming the threshold for each distinct probability\r\n* Computing the MCC\r\n* Returning the probability row with the maximum MCC\r\n* Optional: plot the probability/MCC curve",
      "votes": null
    },
    {
      "id": "131513",
      "postDate": "08/17/2016 19:44:48",
      "content": "<p>[quote=Laurae;131512]</p>\n\n<p>This looks nice and quick to optimize using a specific threshold.</p>\n\n<p>I got another idea which is different from yours, but uses probability outputs directly for defining the threshold. What about the following steps?</p>\n\n<ul>\n<li>Ordering distinct probabilities</li>\n<li>Computing TP/TN when assuming the threshold for each distinct probability</li>\n<li>Computing the MCC</li>\n<li>Returning the probability row with the maximum MCC</li>\n<li>Optional: plot the probability/MCC curve</li>\n</ul>\n\n<p>[/quote]</p>\n\n<p>@Laurae I'm probably misunderstanding you, but I think that's what my <code>find_matthews_threshold</code> function is doing when <code>try_all=True</code>. For every unique probability, it computes the MCC using this probability as a threshold and then returns the best threshold probability. That's disabled by default since I'm not sure how fast the MCC function in numpy is, and trying every probability may take an obscene amount of time. Please tell me if I'm wrong.</p>\n\n<p>As for plotting the probability/MCC curve, that's a very good idea - I'll add that soon :)</p>",
      "rawMarkdown": "[quote=Laurae;131512]\r\n\r\nThis looks nice and quick to optimize using a specific threshold.\r\n\r\nI got another idea which is different from yours, but uses probability outputs directly for defining the threshold. What about the following steps?\r\n\r\n* Ordering distinct probabilities\r\n* Computing TP/TN when assuming the threshold for each distinct probability\r\n* Computing the MCC\r\n* Returning the probability row with the maximum MCC\r\n* Optional: plot the probability/MCC curve\r\n\r\n[/quote]\r\n\r\n@Laurae I'm probably misunderstanding you, but I think that's what my `find_matthews_threshold` function is doing when `try_all=True`. For every unique probability, it computes the MCC using this probability as a threshold and then returns the best threshold probability. That's disabled by default since I'm not sure how fast the MCC function in numpy is, and trying every probability may take an obscene amount of time. Please tell me if I'm wrong.\r\n\r\nAs for plotting the probability/MCC curve, that's a very good idea - I'll add that soon :)",
      "votes": null
    },
    {
      "id": "131514",
      "postDate": "08/17/2016 19:55:17",
      "content": "<p>[quote=anokas;131513]</p>\n\n<p>@Laurae I'm probably misunderstanding you, but I think that's what my <code>find_matthews_threshold</code> function is doing when <code>try_all=True</code>. For every unique probability, it computes the MCC using this probability as a threshold and then returns the best threshold probability. That's disabled by default since I'm not sure how fast the MCC function in numpy is, and trying every probability may take an obscene amount of time. Please tell me if I'm wrong.</p>\n\n<p>[/quote]</p>\n\n<p>Woops! I was blind by the step size, but you are right in what you describe (code loses color while replying, makes reading hard) - thus my post was exactly the same as your code but with try_all = True.</p>\n\n<p>As for the slowness issue, matthews_corrcoef should not be slow. It does not use a loop, it encodes the inputs, then computes the coefficient as a human would do (averages to compute P/N total, then percentage to total, then numerator / denominator, then return either 0 if error or the MCC if no error).</p>\n\n<p>MCC code: <a href=\"https://github.com/scikit-learn/scikit-learn/blob/master/sklearn/metrics/classification.py\">https://github.com/scikit-learn/scikit-learn/blob/master/sklearn/metrics/classification.py</a></p>\n\n<p>You can make it even more faster if you assume correctness of inputs though.</p>\n\n<pre><code>def matthews_corrcoef(y_true, y_pred, sample_weight=None):\n\n    y_type, y_true, y_pred = _check_targets(y_true, y_pred)\n\n    if y_type != &quot;binary&quot;:\n        raise ValueError(&quot;%s is not supported&quot; % y_type)\n\n    lb = LabelEncoder()\n    lb.fit(np.hstack([y_true, y_pred]))\n    y_true = lb.transform(y_true)\n    y_pred = lb.transform(y_pred)\n    mean_yt = np.average(y_true, weights=sample_weight)\n    mean_yp = np.average(y_pred, weights=sample_weight)\n\n    y_true_u_cent = y_true - mean_yt\n    y_pred_u_cent = y_pred - mean_yp\n\n    cov_ytyp = np.average(y_true_u_cent * y_pred_u_cent, weights=sample_weight)\n    var_yt = np.average(y_true_u_cent ** 2, weights=sample_weight)\n    var_yp = np.average(y_pred_u_cent ** 2, weights=sample_weight)\n\n    mcc = cov_ytyp / np.sqrt(var_yt * var_yp)\n\n    if np.isnan(mcc):\n        return 0.\n    else:\n        return mcc\n</code></pre>",
      "rawMarkdown": "[quote=anokas;131513]\r\n\r\n@Laurae I'm probably misunderstanding you, but I think that's what my `find_matthews_threshold` function is doing when `try_all=True`. For every unique probability, it computes the MCC using this probability as a threshold and then returns the best threshold probability. That's disabled by default since I'm not sure how fast the MCC function in numpy is, and trying every probability may take an obscene amount of time. Please tell me if I'm wrong.\r\n\r\n[/quote]\r\n\r\nWoops! I was blind by the step size, but you are right in what you describe (code loses color while replying, makes reading hard) - thus my post was exactly the same as your code but with try_all = True.\r\n\r\nAs for the slowness issue, matthews_corrcoef should not be slow. It does not use a loop, it encodes the inputs, then computes the coefficient as a human would do (averages to compute P/N total, then percentage to total, then numerator / denominator, then return either 0 if error or the MCC if no error).\r\n\r\nMCC code: https://github.com/scikit-learn/scikit-learn/blob/master/sklearn/metrics/classification.py\r\n\r\nYou can make it even more faster if you assume correctness of inputs though.\r\n\r\n    def matthews_corrcoef(y_true, y_pred, sample_weight=None):\r\n\r\n        y_type, y_true, y_pred = _check_targets(y_true, y_pred)\r\n    \r\n        if y_type != \"binary\":\r\n            raise ValueError(\"%s is not supported\" % y_type)\r\n    \r\n        lb = LabelEncoder()\r\n        lb.fit(np.hstack([y_true, y_pred]))\r\n        y_true = lb.transform(y_true)\r\n        y_pred = lb.transform(y_pred)\r\n        mean_yt = np.average(y_true, weights=sample_weight)\r\n        mean_yp = np.average(y_pred, weights=sample_weight)\r\n    \r\n        y_true_u_cent = y_true - mean_yt\r\n        y_pred_u_cent = y_pred - mean_yp\r\n    \r\n        cov_ytyp = np.average(y_true_u_cent * y_pred_u_cent, weights=sample_weight)\r\n        var_yt = np.average(y_true_u_cent ** 2, weights=sample_weight)\r\n        var_yp = np.average(y_pred_u_cent ** 2, weights=sample_weight)\r\n    \r\n        mcc = cov_ytyp / np.sqrt(var_yt * var_yp)\r\n    \r\n        if np.isnan(mcc):\r\n            return 0.\r\n        else:\r\n            return mcc",
      "votes": null
    },
    {
      "id": "133343",
      "postDate": "09/01/2016 15:52:18",
      "content": "<p>I am currious  if some could  give  more expalantion   about the  thing  with the probabilities  and this uber duper   performance metric , cause  just arrived here  and want to    catch  up things  .\nAren't   we  allowed  to submit   usual probs  or    just classes?\nTnks ,\nAlexandru</p>",
      "rawMarkdown": "I am currious  if some could  give  more expalantion   about the  thing  with the probabilities  and this uber duper   performance metric , cause  just arrived here  and want to    catch  up things  .\r\nAren't   we  allowed  to submit   usual probs  or    just classes?\r\nTnks ,\r\nAlexandru",
      "votes": null
    },
    {
      "id": "133601",
      "postDate": "09/02/2016 21:28:12",
      "content": "<p>You can see from the evaluation metric that you'll have to submit classes as it is a function of true positives, false positives, true negatives, and false negatives.  So you need to choose a probability that splits the data that will optimize the evaluation function.</p>",
      "rawMarkdown": "You can see from the evaluation metric that you'll have to submit classes as it is a function of true positives, false positives, true negatives, and false negatives.  So you need to choose a probability that splits the data that will optimize the evaluation function.",
      "votes": null
    },
    {
      "id": "136855",
      "postDate": "09/25/2016 20:54:29",
      "content": "<p>I have noted a curious behavior of the optimal mcc in terms of the number of trees in xgboost. </p>\n\n<p>If the number of trees is small the optimal threshold is close to 0.5. As more trees are added, the optimal mcc for the train error has the tendency to decrease to a value that is drastically lower than 0.5.</p>\n\n<p>Why does the train-optimal mcc decrease as the classifier complexity increases? I would love to hear your explanations on this phenomenon!</p>",
      "rawMarkdown": "I have noted a curious behavior of the optimal mcc in terms of the number of trees in xgboost. \r\n\r\nIf the number of trees is small the optimal threshold is close to 0.5. As more trees are added, the optimal mcc for the train error has the tendency to decrease to a value that is drastically lower than 0.5.\r\n\r\nWhy does the train-optimal mcc decrease as the classifier complexity increases? I would love to hear your explanations on this phenomenon!",
      "votes": null
    },
    {
      "id": "136857",
      "postDate": "09/25/2016 21:10:10",
      "content": "<p>[quote=Christian Hirsch;136855]</p>\n\n<p>If the number of trees is small the optimal threshold is close to 0.5. As more trees are added, the optimal mcc for the train error has the tendency to decrease to a value that is drastically lower than 0.5.</p>\n\n<p>Why does the train-optimal mcc decrease as the classifier complexity increases?</p>\n\n<p>[/quote]</p>\n\n<p>In xgboost, you use gradient boosting to improve the predictions. For gradient boosting, you do use gradient descent optimization: it means you need a loss function.</p>\n\n<p>The loss function is what directs xgboost to optimize for: as the data is heavily unbalanced (about 6000 positives for 1+ million negatives), optimizing most loss functions (such as squared error, etc.) will lead to push predictions towards a specific value which is what a random number generator should generate to optimize that loss function.</p>\n\n<p>Hence, having your base_margin set to 0.5 initially to start, the mean of predictions will gradually drop to lower the loss function used in gradient descent.</p>\n\n<p>Imagine you need to lower the linear error in this:</p>\n\n<pre><code>Truth  Initial  Error (sum = 3)\n  0      0.5     0.5\n  1      0.5     0.5\n  0      0.5     0.5\n  1      0.5     0.5\n  0      0.5     0.5\n  0      0.5     0.5\n</code></pre>\n\n<p>You would go for this:</p>\n\n<pre><code>Truth    Pred   Error (sum = 2.66, which is less than our error of 3 initially!!)\n  0      0.33    0.33\n  1      0.33    0.66\n  0      0.33    0.33\n  1      0.33    0.66\n  0      0.33    0.33\n  0      0.33    0.33\n</code></pre>",
      "rawMarkdown": "[quote=Christian Hirsch;136855]\r\n\r\nIf the number of trees is small the optimal threshold is close to 0.5. As more trees are added, the optimal mcc for the train error has the tendency to decrease to a value that is drastically lower than 0.5.\r\n\r\nWhy does the train-optimal mcc decrease as the classifier complexity increases?\r\n\r\n[/quote]\r\n\r\nIn xgboost, you use gradient boosting to improve the predictions. For gradient boosting, you do use gradient descent optimization: it means you need a loss function.\r\n\r\nThe loss function is what directs xgboost to optimize for: as the data is heavily unbalanced (about 6000 positives for 1+ million negatives), optimizing most loss functions (such as squared error, etc.) will lead to push predictions towards a specific value which is what a random number generator should generate to optimize that loss function.\r\n\r\nHence, having your base_margin set to 0.5 initially to start, the mean of predictions will gradually drop to lower the loss function used in gradient descent.\r\n\r\nImagine you need to lower the linear error in this:\r\n\r\n    Truth  Initial  Error (sum = 3)\r\n      0      0.5     0.5\r\n      1      0.5     0.5\r\n      0      0.5     0.5\r\n      1      0.5     0.5\r\n      0      0.5     0.5\r\n      0      0.5     0.5\r\n\r\nYou would go for this:\r\n\r\n    Truth    Pred   Error (sum = 2.66, which is less than our error of 3 initially!!)\r\n      0      0.33    0.33\r\n      1      0.33    0.66\r\n      0      0.33    0.33\r\n      1      0.33    0.66\r\n      0      0.33    0.33\r\n      0      0.33    0.33",
      "votes": null
    },
    {
      "id": "136916",
      "postDate": "09/26/2016 13:42:34",
      "content": "<p>Many thanks for the excellent explanation! Now, I understand much better the connection between the choice of the objective function and the optimal value of the threshold.</p>",
      "rawMarkdown": "Many thanks for the excellent explanation! Now, I understand much better the connection between the choice of the objective function and the optimal value of the threshold.",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 131512,
      "author_name": "laurae2",
      "author_url": "",
      "post_date": "08/17/2016 19:38:07",
      "content": "<p>This looks nice and quick to optimize using a specific threshold.</p>\n\n<p>I got another idea which is different from yours, but uses probability outputs directly for defining the threshold. What about the following steps?</p>\n\n<ul>\n<li>Ordering distinct probabilities</li>\n<li>Computing TP/TN when assuming the threshold for each distinct probability</li>\n<li>Computing the MCC</li>\n<li>Returning the probability row with the maximum MCC</li>\n<li>Optional: plot the probability/MCC curve</li>\n</ul>",
      "votes": null,
      "replies": []
    },
    {
      "id": 131513,
      "author_name": "anokas",
      "author_url": "",
      "post_date": "08/17/2016 19:44:48",
      "content": "<p>[quote=Laurae;131512]</p>\n\n<p>This looks nice and quick to optimize using a specific threshold.</p>\n\n<p>I got another idea which is different from yours, but uses probability outputs directly for defining the threshold. What about the following steps?</p>\n\n<ul>\n<li>Ordering distinct probabilities</li>\n<li>Computing TP/TN when assuming the threshold for each distinct probability</li>\n<li>Computing the MCC</li>\n<li>Returning the probability row with the maximum MCC</li>\n<li>Optional: plot the probability/MCC curve</li>\n</ul>\n\n<p>[/quote]</p>\n\n<p>@Laurae I'm probably misunderstanding you, but I think that's what my <code>find_matthews_threshold</code> function is doing when <code>try_all=True</code>. For every unique probability, it computes the MCC using this probability as a threshold and then returns the best threshold probability. That's disabled by default since I'm not sure how fast the MCC function in numpy is, and trying every probability may take an obscene amount of time. Please tell me if I'm wrong.</p>\n\n<p>As for plotting the probability/MCC curve, that's a very good idea - I'll add that soon :)</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 131514,
      "author_name": "laurae2",
      "author_url": "",
      "post_date": "08/17/2016 19:55:17",
      "content": "<p>[quote=anokas;131513]</p>\n\n<p>@Laurae I'm probably misunderstanding you, but I think that's what my <code>find_matthews_threshold</code> function is doing when <code>try_all=True</code>. For every unique probability, it computes the MCC using this probability as a threshold and then returns the best threshold probability. That's disabled by default since I'm not sure how fast the MCC function in numpy is, and trying every probability may take an obscene amount of time. Please tell me if I'm wrong.</p>\n\n<p>[/quote]</p>\n\n<p>Woops! I was blind by the step size, but you are right in what you describe (code loses color while replying, makes reading hard) - thus my post was exactly the same as your code but with try_all = True.</p>\n\n<p>As for the slowness issue, matthews_corrcoef should not be slow. It does not use a loop, it encodes the inputs, then computes the coefficient as a human would do (averages to compute P/N total, then percentage to total, then numerator / denominator, then return either 0 if error or the MCC if no error).</p>\n\n<p>MCC code: <a href=\"https://github.com/scikit-learn/scikit-learn/blob/master/sklearn/metrics/classification.py\">https://github.com/scikit-learn/scikit-learn/blob/master/sklearn/metrics/classification.py</a></p>\n\n<p>You can make it even more faster if you assume correctness of inputs though.</p>\n\n<pre><code>def matthews_corrcoef(y_true, y_pred, sample_weight=None):\n\n    y_type, y_true, y_pred = _check_targets(y_true, y_pred)\n\n    if y_type != &quot;binary&quot;:\n        raise ValueError(&quot;%s is not supported&quot; % y_type)\n\n    lb = LabelEncoder()\n    lb.fit(np.hstack([y_true, y_pred]))\n    y_true = lb.transform(y_true)\n    y_pred = lb.transform(y_pred)\n    mean_yt = np.average(y_true, weights=sample_weight)\n    mean_yp = np.average(y_pred, weights=sample_weight)\n\n    y_true_u_cent = y_true - mean_yt\n    y_pred_u_cent = y_pred - mean_yp\n\n    cov_ytyp = np.average(y_true_u_cent * y_pred_u_cent, weights=sample_weight)\n    var_yt = np.average(y_true_u_cent ** 2, weights=sample_weight)\n    var_yp = np.average(y_pred_u_cent ** 2, weights=sample_weight)\n\n    mcc = cov_ytyp / np.sqrt(var_yt * var_yp)\n\n    if np.isnan(mcc):\n        return 0.\n    else:\n        return mcc\n</code></pre>",
      "votes": null,
      "replies": []
    },
    {
      "id": 133343,
      "author_name": "",
      "author_url": "",
      "post_date": "09/01/2016 15:52:18",
      "content": "<p>I am currious  if some could  give  more expalantion   about the  thing  with the probabilities  and this uber duper   performance metric , cause  just arrived here  and want to    catch  up things  .\nAren't   we  allowed  to submit   usual probs  or    just classes?\nTnks ,\nAlexandru</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 133601,
      "author_name": "mathandbikes",
      "author_url": "",
      "post_date": "09/02/2016 21:28:12",
      "content": "<p>You can see from the evaluation metric that you'll have to submit classes as it is a function of true positives, false positives, true negatives, and false negatives.  So you need to choose a probability that splits the data that will optimize the evaluation function.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 136855,
      "author_name": "dollardollar",
      "author_url": "",
      "post_date": "09/25/2016 20:54:29",
      "content": "<p>I have noted a curious behavior of the optimal mcc in terms of the number of trees in xgboost. </p>\n\n<p>If the number of trees is small the optimal threshold is close to 0.5. As more trees are added, the optimal mcc for the train error has the tendency to decrease to a value that is drastically lower than 0.5.</p>\n\n<p>Why does the train-optimal mcc decrease as the classifier complexity increases? I would love to hear your explanations on this phenomenon!</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 136857,
      "author_name": "laurae2",
      "author_url": "",
      "post_date": "09/25/2016 21:10:10",
      "content": "<p>[quote=Christian Hirsch;136855]</p>\n\n<p>If the number of trees is small the optimal threshold is close to 0.5. As more trees are added, the optimal mcc for the train error has the tendency to decrease to a value that is drastically lower than 0.5.</p>\n\n<p>Why does the train-optimal mcc decrease as the classifier complexity increases?</p>\n\n<p>[/quote]</p>\n\n<p>In xgboost, you use gradient boosting to improve the predictions. For gradient boosting, you do use gradient descent optimization: it means you need a loss function.</p>\n\n<p>The loss function is what directs xgboost to optimize for: as the data is heavily unbalanced (about 6000 positives for 1+ million negatives), optimizing most loss functions (such as squared error, etc.) will lead to push predictions towards a specific value which is what a random number generator should generate to optimize that loss function.</p>\n\n<p>Hence, having your base_margin set to 0.5 initially to start, the mean of predictions will gradually drop to lower the loss function used in gradient descent.</p>\n\n<p>Imagine you need to lower the linear error in this:</p>\n\n<pre><code>Truth  Initial  Error (sum = 3)\n  0      0.5     0.5\n  1      0.5     0.5\n  0      0.5     0.5\n  1      0.5     0.5\n  0      0.5     0.5\n  0      0.5     0.5\n</code></pre>\n\n<p>You would go for this:</p>\n\n<pre><code>Truth    Pred   Error (sum = 2.66, which is less than our error of 3 initially!!)\n  0      0.33    0.33\n  1      0.33    0.66\n  0      0.33    0.33\n  1      0.33    0.66\n  0      0.33    0.33\n  0      0.33    0.33\n</code></pre>",
      "votes": null,
      "replies": []
    },
    {
      "id": 136916,
      "author_name": "dollardollar",
      "author_url": "",
      "post_date": "09/26/2016 13:42:34",
      "content": "<p>Many thanks for the excellent explanation! Now, I understand much better the connection between the choice of the objective function and the optimal value of the threshold.</p>",
      "votes": null,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "131509": "Hi everyone,\r\n\r\nI wrote some quick python functions for automagically optimising the threshold for converting probabilities from an algorithm such as XGBoost into a binary prediction that can be submitted to this competition. This checks the validation score at many different thresholds and then selects the best one to apply to the testing data. If set to verbose, it also prints out a validation score.\r\n\r\nThe `try_all` option will try every single value as a threshold, otherwise a step size of 0.01 is used (which should be sufficient).\r\n\r\n    import numpy as np\r\n    from sklearn.metrics import matthews_corrcoef\r\n    \r\n    def find_matthews_threshold(p_valid, y_valid, try_all=False, verbose=False):\r\n        p_valid, y_valid = np.array(p_valid), np.array(y_valid)\r\n    \r\n        best = 0\r\n        best_score = -2\r\n        totry = np.arange(0,1,0.01) if try_all is False else np.unique(p_valid)\r\n        for t in totry:\r\n            score = matthews_corrcoef(y_valid, p_valid > t)\r\n            if score > best_score:\r\n                best_score = score\r\n                best = t\r\n        if verbose is True: \r\n            print('Best score: ', round(best_score, 5), ' @ threshold ', best)\r\n    \r\n        return best\r\n    \r\n    def best_threshold_submission(p_valid, y_valid, p_test, try_all=False, verbose=False):\r\n        p_test = np.array(p_test)\r\n        thresh = find_matthews_threshold(p_valid, y_valid, try_all, verbose)\r\n        return p_test > thresh\r\n    \r\n    submission_values = best_threshold_submission(validation_probabilities, validation_groundtruth, test_probabilities)\r\n\r\nHopefully this helps, and obviously let me know if there are any improvements to be made/bugs to be fixed :)\r\n\r\n\\- Mikel",
    "131512": "This looks nice and quick to optimize using a specific threshold.\r\n\r\nI got another idea which is different from yours, but uses probability outputs directly for defining the threshold. What about the following steps?\r\n\r\n* Ordering distinct probabilities\r\n* Computing TP/TN when assuming the threshold for each distinct probability\r\n* Computing the MCC\r\n* Returning the probability row with the maximum MCC\r\n* Optional: plot the probability/MCC curve",
    "131513": "[quote=Laurae;131512]\r\n\r\nThis looks nice and quick to optimize using a specific threshold.\r\n\r\nI got another idea which is different from yours, but uses probability outputs directly for defining the threshold. What about the following steps?\r\n\r\n* Ordering distinct probabilities\r\n* Computing TP/TN when assuming the threshold for each distinct probability\r\n* Computing the MCC\r\n* Returning the probability row with the maximum MCC\r\n* Optional: plot the probability/MCC curve\r\n\r\n[/quote]\r\n\r\n@Laurae I'm probably misunderstanding you, but I think that's what my `find_matthews_threshold` function is doing when `try_all=True`. For every unique probability, it computes the MCC using this probability as a threshold and then returns the best threshold probability. That's disabled by default since I'm not sure how fast the MCC function in numpy is, and trying every probability may take an obscene amount of time. Please tell me if I'm wrong.\r\n\r\nAs for plotting the probability/MCC curve, that's a very good idea - I'll add that soon :)",
    "131514": "[quote=anokas;131513]\r\n\r\n@Laurae I'm probably misunderstanding you, but I think that's what my `find_matthews_threshold` function is doing when `try_all=True`. For every unique probability, it computes the MCC using this probability as a threshold and then returns the best threshold probability. That's disabled by default since I'm not sure how fast the MCC function in numpy is, and trying every probability may take an obscene amount of time. Please tell me if I'm wrong.\r\n\r\n[/quote]\r\n\r\nWoops! I was blind by the step size, but you are right in what you describe (code loses color while replying, makes reading hard) - thus my post was exactly the same as your code but with try_all = True.\r\n\r\nAs for the slowness issue, matthews_corrcoef should not be slow. It does not use a loop, it encodes the inputs, then computes the coefficient as a human would do (averages to compute P/N total, then percentage to total, then numerator / denominator, then return either 0 if error or the MCC if no error).\r\n\r\nMCC code: https://github.com/scikit-learn/scikit-learn/blob/master/sklearn/metrics/classification.py\r\n\r\nYou can make it even more faster if you assume correctness of inputs though.\r\n\r\n    def matthews_corrcoef(y_true, y_pred, sample_weight=None):\r\n\r\n        y_type, y_true, y_pred = _check_targets(y_true, y_pred)\r\n    \r\n        if y_type != \"binary\":\r\n            raise ValueError(\"%s is not supported\" % y_type)\r\n    \r\n        lb = LabelEncoder()\r\n        lb.fit(np.hstack([y_true, y_pred]))\r\n        y_true = lb.transform(y_true)\r\n        y_pred = lb.transform(y_pred)\r\n        mean_yt = np.average(y_true, weights=sample_weight)\r\n        mean_yp = np.average(y_pred, weights=sample_weight)\r\n    \r\n        y_true_u_cent = y_true - mean_yt\r\n        y_pred_u_cent = y_pred - mean_yp\r\n    \r\n        cov_ytyp = np.average(y_true_u_cent * y_pred_u_cent, weights=sample_weight)\r\n        var_yt = np.average(y_true_u_cent ** 2, weights=sample_weight)\r\n        var_yp = np.average(y_pred_u_cent ** 2, weights=sample_weight)\r\n    \r\n        mcc = cov_ytyp / np.sqrt(var_yt * var_yp)\r\n    \r\n        if np.isnan(mcc):\r\n            return 0.\r\n        else:\r\n            return mcc",
    "133343": "I am currious  if some could  give  more expalantion   about the  thing  with the probabilities  and this uber duper   performance metric , cause  just arrived here  and want to    catch  up things  .\r\nAren't   we  allowed  to submit   usual probs  or    just classes?\r\nTnks ,\r\nAlexandru",
    "133601": "You can see from the evaluation metric that you'll have to submit classes as it is a function of true positives, false positives, true negatives, and false negatives.  So you need to choose a probability that splits the data that will optimize the evaluation function.",
    "136855": "I have noted a curious behavior of the optimal mcc in terms of the number of trees in xgboost. \r\n\r\nIf the number of trees is small the optimal threshold is close to 0.5. As more trees are added, the optimal mcc for the train error has the tendency to decrease to a value that is drastically lower than 0.5.\r\n\r\nWhy does the train-optimal mcc decrease as the classifier complexity increases? I would love to hear your explanations on this phenomenon!",
    "136857": "[quote=Christian Hirsch;136855]\r\n\r\nIf the number of trees is small the optimal threshold is close to 0.5. As more trees are added, the optimal mcc for the train error has the tendency to decrease to a value that is drastically lower than 0.5.\r\n\r\nWhy does the train-optimal mcc decrease as the classifier complexity increases?\r\n\r\n[/quote]\r\n\r\nIn xgboost, you use gradient boosting to improve the predictions. For gradient boosting, you do use gradient descent optimization: it means you need a loss function.\r\n\r\nThe loss function is what directs xgboost to optimize for: as the data is heavily unbalanced (about 6000 positives for 1+ million negatives), optimizing most loss functions (such as squared error, etc.) will lead to push predictions towards a specific value which is what a random number generator should generate to optimize that loss function.\r\n\r\nHence, having your base_margin set to 0.5 initially to start, the mean of predictions will gradually drop to lower the loss function used in gradient descent.\r\n\r\nImagine you need to lower the linear error in this:\r\n\r\n    Truth  Initial  Error (sum = 3)\r\n      0      0.5     0.5\r\n      1      0.5     0.5\r\n      0      0.5     0.5\r\n      1      0.5     0.5\r\n      0      0.5     0.5\r\n      0      0.5     0.5\r\n\r\nYou would go for this:\r\n\r\n    Truth    Pred   Error (sum = 2.66, which is less than our error of 3 initially!!)\r\n      0      0.33    0.33\r\n      1      0.33    0.66\r\n      0      0.33    0.33\r\n      1      0.33    0.66\r\n      0      0.33    0.33\r\n      0      0.33    0.33",
    "136916": "Many thanks for the excellent explanation! Now, I understand much better the connection between the choice of the objective function and the optimal value of the threshold."
  },
  "source": "meta"
}