Although it looks like feature construction is the most important aspect of this competition, good CV will also be important with only 6879 positive examples in the training set (and probably a similar number in the test set). With the public LB based on only about 2060 positives (30% of 6879), it would be easy to overfit to the public LB without good CV. Below I outline my method (most of it is just common sense). I encourage others to give suggestions.
\n\nI’m doing 5-fold cross-validation. The folds are created by putting every fifth line in a different fold. I didn’t stratify by Response, but that would probably be a good thing to do too.
\n\nWithin each fold, I find the number of TPs, FPs, TNs, and FNs. Then at the end I sum up the results of the folds to get total TPs, FPs, TNs, and FNs. I then calculate MCC from the totals. I was originally calculating MCC within each fold and averaging the 5 MCCs – that works too, and I found it gave similar results.
\n\nActually, within each fold I create predictions 210 times. I create predictions for 7 different values for ntreelimit (number of XGBoost rounds) and 30 different cutoff thresholds (threshold that converts continuous probability to a 0 or 1). 7 * 30 = 210. The reason I did this is A) It doesn’t add that much processing time – you only need to build the model once for each fold, B) I’ve found that the number of rounds that maximizes MCC is not necessarily the same as the number of rounds that maximizes AUC, C) The score is sensitive to the cutoff threshold so I calculate the score for every cutoff between 0.20 and 0.49 (30 cutoffs), and D) The best cutoff is different for different values of ntreelimit so you need to look at the relationship between these two. So for each fold I actually end up with 210 sets of TP, FP, TN, FN.
\n\nSo after the 5 folds are complete, I end up with 210 MCC scores, and this allows me to plot out how MCC varies as ntreelimit and cutoff are varied.
\n\nNow you might ask, “Why not just use MCC as the eval_metric?” I tried that. I found that it was a lot slower and only gave a small improvement over using AUC.
\n\nFinally, if you have constructed features that are based on % positives for various feature values, I would recommend calculating those features inside the CV loop and only based on the training folds. If you calculate the % positives on all the folds, there will be leakage from the test folds into the training folds, and your CV score will be higher than it should be.
", "messages": [ { "id": "138164", "postDate": "10/07/2016 13:28:58", "content": "Although it looks like feature construction is the most important aspect of this competition, good CV will also be important with only 6879 positive examples in the training set (and probably a similar number in the test set). With the public LB based on only about 2060 positives (30% of 6879), it would be easy to overfit to the public LB without good CV. Below I outline my method (most of it is just common sense). I encourage others to give suggestions.
\n\nI’m doing 5-fold cross-validation. The folds are created by putting every fifth line in a different fold. I didn’t stratify by Response, but that would probably be a good thing to do too.
\n\nWithin each fold, I find the number of TPs, FPs, TNs, and FNs. Then at the end I sum up the results of the folds to get total TPs, FPs, TNs, and FNs. I then calculate MCC from the totals. I was originally calculating MCC within each fold and averaging the 5 MCCs – that works too, and I found it gave similar results.
\n\nActually, within each fold I create predictions 210 times. I create predictions for 7 different values for ntreelimit (number of XGBoost rounds) and 30 different cutoff thresholds (threshold that converts continuous probability to a 0 or 1). 7 * 30 = 210. The reason I did this is A) It doesn’t add that much processing time – you only need to build the model once for each fold, B) I’ve found that the number of rounds that maximizes MCC is not necessarily the same as the number of rounds that maximizes AUC, C) The score is sensitive to the cutoff threshold so I calculate the score for every cutoff between 0.20 and 0.49 (30 cutoffs), and D) The best cutoff is different for different values of ntreelimit so you need to look at the relationship between these two. So for each fold I actually end up with 210 sets of TP, FP, TN, FN.
\n\nSo after the 5 folds are complete, I end up with 210 MCC scores, and this allows me to plot out how MCC varies as ntreelimit and cutoff are varied.
\n\nNow you might ask, “Why not just use MCC as the eval_metric?” I tried that. I found that it was a lot slower and only gave a small improvement over using AUC.
\n\nFinally, if you have constructed features that are based on % positives for various feature values, I would recommend calculating those features inside the CV loop and only based on the training folds. If you calculate the % positives on all the folds, there will be leakage from the test folds into the training folds, and your CV score will be higher than it should be.
", "rawMarkdown": "Although it looks like feature construction is the most important aspect of this competition, good CV will also be important with only 6879 positive examples in the training set (and probably a similar number in the test set). With the public LB based on only about 2060 positives (30% of 6879), it would be easy to overfit to the public LB without good CV. Below I outline my method (most of it is just common sense). I encourage others to give suggestions.\r\n\r\nI’m doing 5-fold cross-validation. The folds are created by putting every fifth line in a different fold. I didn’t stratify by Response, but that would probably be a good thing to do too.\r\n\r\nWithin each fold, I find the number of TPs, FPs, TNs, and FNs. Then at the end I sum up the results of the folds to get total TPs, FPs, TNs, and FNs. I then calculate MCC from the totals. I was originally calculating MCC within each fold and averaging the 5 MCCs – that works too, and I found it gave similar results.\r\n\r\nActually, within each fold I create predictions 210 times. I create predictions for 7 different values for ntreelimit (number of XGBoost rounds) and 30 different cutoff thresholds (threshold that converts continuous probability to a 0 or 1). 7 * 30 = 210. The reason I did this is A) It doesn’t add that much processing time – you only need to build the model once for each fold, B) I’ve found that the number of rounds that maximizes MCC is not necessarily the same as the number of rounds that maximizes AUC, C) The score is sensitive to the cutoff threshold so I calculate the score for every cutoff between 0.20 and 0.49 (30 cutoffs), and D) The best cutoff is different for different values of ntreelimit so you need to look at the relationship between these two. So for each fold I actually end up with 210 sets of TP, FP, TN, FN.\r\n\r\nSo after the 5 folds are complete, I end up with 210 MCC scores, and this allows me to plot out how MCC varies as ntreelimit and cutoff are varied.\r\n\r\nNow you might ask, “Why not just use MCC as the eval_metric?” I tried that. I found that it was a lot slower and only gave a small improvement over using AUC. \r\n\r\nFinally, if you have constructed features that are based on % positives for various feature values, I would recommend calculating those features inside the CV loop and only based on the training folds. If you calculate the % positives on all the folds, there will be leakage from the test folds into the training folds, and your CV score will be higher than it should be.", "votes": null }, { "id": "138165", "postDate": "10/07/2016 13:31:18", "content": "One more thing ... Picking the best cutoff threshold for your final submissions.
\n\nI’ve experimented with making multiple submissions from the same model, but just vary the cutoff threshold. The cutoff that was best for CV was not always the cutoff that was best for public LB. If you also do this, I recommend using a weighted average of the CV and LB scores to pick the cutoff threshold for your final submission. Weighted average = (1.0 * CV + 0.3 * public LB) / 1.3.
", "rawMarkdown": "One more thing ... Picking the best cutoff threshold for your final submissions.\r\n\r\nI’ve experimented with making multiple submissions from the same model, but just vary the cutoff threshold. The cutoff that was best for CV was not always the cutoff that was best for public LB. If you also do this, I recommend using a weighted average of the CV and LB scores to pick the cutoff threshold for your final submission. Weighted average = (1.0 * CV + 0.3 * public LB) / 1.3.", "votes": null }, { "id": "138212", "postDate": "10/07/2016 16:53:26", "content": "For me the best logloss eval for each CV has given the best MCC for overall prediction. I compute MCC score and threshold on whole train set (merging 5 separate parts of training set predicted by 5 CVs), and use that threshold on average of the probabilities from 5 CV's for test set submission. Working well so far !!! Got local CV score of 0.266 and the Public LB for same is 0.2713. If I try tinkering with this eval method, then I am getting public LB worse than local CV score.
", "rawMarkdown": "For me the best logloss eval for each CV has given the best MCC for overall prediction. I compute MCC score and threshold on whole train set (merging 5 separate parts of training set predicted by 5 CVs), and use that threshold on average of the probabilities from 5 CV's for test set submission. Working well so far !!! Got local CV score of 0.266 and the Public LB for same is 0.2713. If I try tinkering with this eval method, then I am getting public LB worse than local CV score.", "votes": null }, { "id": "138251", "postDate": "10/07/2016 18:59:45", "content": "oh my. this bring back a lot of memories from the prudental insurance competition. it is a not as bad in this one as you have only one cutoff to tune while in prudental it was 7 cutoffs (8 classes classification). i was one of the few that actually moved up in the private board (top 1%) but the shake up there was massive. i have to dig out my old code and notes to remember the tricks i used to not overfit the cutoff to the LB
", "rawMarkdown": "oh my. this bring back a lot of memories from the prudental insurance competition. it is a not as bad in this one as you have only one cutoff to tune while in prudental it was 7 cutoffs (8 classes classification). i was one of the few that actually moved up in the private board (top 1%) but the shake up there was massive. i have to dig out my old code and notes to remember the tricks i used to not overfit the cutoff to the LB", "votes": null }, { "id": "139607", "postDate": "10/15/2016 06:41:15", "content": "Anyone using data simulation to increase the sample-size of positive samples in training :)?
", "rawMarkdown": "Anyone using data simulation to increase the sample-size of positive samples in training :)?", "votes": null }, { "id": "139660", "postDate": "10/15/2016 17:19:10", "content": "I thought about it, BK. With the data set being as large as it is, I instead downsampled the 0s to get about a 95/5 mix. It gave some decent results but I didn't do a direct comparison of the scores. Have you tried it yet?
", "rawMarkdown": "I thought about it, BK. With the data set being as large as it is, I instead downsampled the 0s to get about a 95/5 mix. It gave some decent results but I didn't do a direct comparison of the scores. Have you tried it yet?", "votes": null }, { "id": "139979", "postDate": "10/17/2016 17:27:22", "content": "@Banana Kitty,
\n\nI tried various things but did not manage to get something better than what i have with original data.
", "rawMarkdown": "Banana Kitty,\r\n\r\nI tried various things but did not manage to get something better than what i have with original data.", "votes": null }, { "id": "140004", "postDate": "10/17/2016 20:51:32", "content": "For optimizing the threshold cutoff, I prefer to use scipy.optimize (code below). I found empirically that the default optimization method doesn't work with MCC (instabilities?) but the Powell method works fine. This code also brute-force searches for a good initial threshold to use.
\n\nimport numpy as np\nfrom scipy.optimize import minimize\n\ndef threshold(y, t):\n """Threshold a numpy array of probabilities.\n\n Args:\n y: the array of values\n t: the threshold. Map values >= t to 1, otherwise 0.\n\n Returns:\n The thresholded (copy of the) array.\n """\n assert 0 <= t <= 1\n y = y.copy()\n y[y >= t] = 1\n y[y < t] = 0\n return y\n\ndef optimize_threshold(ya, yp, score_func, initial_guess=None,\n method='Powell', options=None, verbose=False):\n """Explore possible thresholds to find one that works best.\n\n Args:\n ya: array of actual labels\n yp: array of predicted probabilities\n score_func: scoring function to use\n initial_guess: initial threshold to use. If None, search for one.\n method: optimization method (see scipy.minimize)\n options: optimization options (see scipy.minimize)\n verbose: if False, return the threshold only; otherwise,\n the full scipy.minimize result\n\n Returns:\n The optimal threshold or scipy.minimize results.\n """\n # Create a helper function to compute MCC at threshold t.\n # Use the negation because scipy.optimize handles only minimization.\n f = lambda t: -score_func(ya, threshold(yp, t))\n # Search for a suitable initial guess.\n if (initial_guess is None):\n initial_score = -1.\n ymin = min(1e-9, yp.min()) # Move away from 0\n for i in np.logspace(np.log10(ymin), 0, endpoint=False, num=10):\n if (i >= ymin):\n s = score_func(ya, threshold(yp, i))\n if (s > initial_score):\n initial_score = s\n initial_guess = i\n opt = minimize(f, initial_guess, method=method, options=options)\n return opt if verbose else opt.x\n@JohnM: I observed similar result. Downsampling seems to work and make sure to remove duplicates from training :) In term of simulation, I used this http://www.sciencedirect.com/science/article/pii/S0266892009000113
\n\n@CPMP: Gz to pushing to 0.46 :) Yes, I tried various things too and the original data seems to be the best to use.
\n\n@BreakfastPirate thanks for your tips (y)
", "rawMarkdown": "JohnM: I observed similar result. Downsampling seems to work and make sure to remove duplicates from training :) In term of simulation, I used this http://www.sciencedirect.com/science/article/pii/S0266892009000113\r\n\r\n@CPMP: Gz to pushing to 0.46 :) Yes, I tried various things too and the original data seems to be the best to use. \r\n\r\n@BreakfastPirate thanks for your tips (y)", "votes": null }, { "id": "140132", "postDate": "10/18/2016 14:13:50", "content": "@MichaelB You may want to look at this notebook ( https://www.kaggle.com/cpmpml/bosch-production-line-performance/optimizing-probabilities-for-best-mcc ) where I provide code to compute the optimal threshold exactly.
", "rawMarkdown": "MichaelB You may want to look at this notebook ( https://www.kaggle.com/cpmpml/bosch-production-line-performance/optimizing-probabilities-for-best-mcc ) where I provide code to compute the optimal threshold exactly.", "votes": null }, { "id": "460250", "postDate": "01/23/2019 09:35:13", "content": "Can anyone provide the sample Cross Validation code sample so that i can understand how it really works.Thanks
", "rawMarkdown": "Can anyone provide the sample Cross Validation code sample so that i can understand how it really works.Thanks", "votes": null }, { "id": "460313", "postDate": "01/23/2019 12:25:21", "content": "See https://scikit-learn.org/stable/modules/cross_validation.html
", "rawMarkdown": "See https://scikit-learn.org/stable/modules/cross_validation.html", "votes": null } ], "comments": [ { "id": 138165, "author_name": "breakfastpirate", "author_url": "", "post_date": "10/07/2016 13:31:18", "content": "One more thing ... Picking the best cutoff threshold for your final submissions.
\n\nI’ve experimented with making multiple submissions from the same model, but just vary the cutoff threshold. The cutoff that was best for CV was not always the cutoff that was best for public LB. If you also do this, I recommend using a weighted average of the CV and LB scores to pick the cutoff threshold for your final submission. Weighted average = (1.0 * CV + 0.3 * public LB) / 1.3.
", "votes": null, "replies": [] }, { "id": 138212, "author_name": "gullu39", "author_url": "", "post_date": "10/07/2016 16:53:26", "content": "For me the best logloss eval for each CV has given the best MCC for overall prediction. I compute MCC score and threshold on whole train set (merging 5 separate parts of training set predicted by 5 CVs), and use that threshold on average of the probabilities from 5 CV's for test set submission. Working well so far !!! Got local CV score of 0.266 and the Public LB for same is 0.2713. If I try tinkering with this eval method, then I am getting public LB worse than local CV score.
", "votes": null, "replies": [] }, { "id": 138251, "author_name": "samehif", "author_url": "", "post_date": "10/07/2016 18:59:45", "content": "oh my. this bring back a lot of memories from the prudental insurance competition. it is a not as bad in this one as you have only one cutoff to tune while in prudental it was 7 cutoffs (8 classes classification). i was one of the few that actually moved up in the private board (top 1%) but the shake up there was massive. i have to dig out my old code and notes to remember the tricks i used to not overfit the cutoff to the LB
", "votes": null, "replies": [] }, { "id": 139607, "author_name": "", "author_url": "", "post_date": "10/15/2016 06:41:15", "content": "Anyone using data simulation to increase the sample-size of positive samples in training :)?
", "votes": null, "replies": [] }, { "id": 139660, "author_name": "jpmiller", "author_url": "", "post_date": "10/15/2016 17:19:10", "content": "I thought about it, BK. With the data set being as large as it is, I instead downsampled the 0s to get about a 95/5 mix. It gave some decent results but I didn't do a direct comparison of the scores. Have you tried it yet?
", "votes": null, "replies": [] }, { "id": 139979, "author_name": "cpmpml", "author_url": "", "post_date": "10/17/2016 17:27:22", "content": "@Banana Kitty,
\n\nI tried various things but did not manage to get something better than what i have with original data.
", "votes": null, "replies": [] }, { "id": 140004, "author_name": "brundage", "author_url": "", "post_date": "10/17/2016 20:51:32", "content": "For optimizing the threshold cutoff, I prefer to use scipy.optimize (code below). I found empirically that the default optimization method doesn't work with MCC (instabilities?) but the Powell method works fine. This code also brute-force searches for a good initial threshold to use.
\n\nimport numpy as np\nfrom scipy.optimize import minimize\n\ndef threshold(y, t):\n """Threshold a numpy array of probabilities.\n\n Args:\n y: the array of values\n t: the threshold. Map values >= t to 1, otherwise 0.\n\n Returns:\n The thresholded (copy of the) array.\n """\n assert 0 <= t <= 1\n y = y.copy()\n y[y >= t] = 1\n y[y < t] = 0\n return y\n\ndef optimize_threshold(ya, yp, score_func, initial_guess=None,\n method='Powell', options=None, verbose=False):\n """Explore possible thresholds to find one that works best.\n\n Args:\n ya: array of actual labels\n yp: array of predicted probabilities\n score_func: scoring function to use\n initial_guess: initial threshold to use. If None, search for one.\n method: optimization method (see scipy.minimize)\n options: optimization options (see scipy.minimize)\n verbose: if False, return the threshold only; otherwise,\n the full scipy.minimize result\n\n Returns:\n The optimal threshold or scipy.minimize results.\n """\n # Create a helper function to compute MCC at threshold t.\n # Use the negation because scipy.optimize handles only minimization.\n f = lambda t: -score_func(ya, threshold(yp, t))\n # Search for a suitable initial guess.\n if (initial_guess is None):\n initial_score = -1.\n ymin = min(1e-9, yp.min()) # Move away from 0\n for i in np.logspace(np.log10(ymin), 0, endpoint=False, num=10):\n if (i >= ymin):\n s = score_func(ya, threshold(yp, i))\n if (s > initial_score):\n initial_score = s\n initial_guess = i\n opt = minimize(f, initial_guess, method=method, options=options)\n return opt if verbose else opt.x\n@JohnM: I observed similar result. Downsampling seems to work and make sure to remove duplicates from training :) In term of simulation, I used this http://www.sciencedirect.com/science/article/pii/S0266892009000113
\n\n@CPMP: Gz to pushing to 0.46 :) Yes, I tried various things too and the original data seems to be the best to use.
\n\n@BreakfastPirate thanks for your tips (y)
", "votes": null, "replies": [] }, { "id": 140132, "author_name": "cpmpml", "author_url": "", "post_date": "10/18/2016 14:13:50", "content": "@MichaelB You may want to look at this notebook ( https://www.kaggle.com/cpmpml/bosch-production-line-performance/optimizing-probabilities-for-best-mcc ) where I provide code to compute the optimal threshold exactly.
", "votes": null, "replies": [] }, { "id": 460250, "author_name": "sagardubey3", "author_url": "", "post_date": "01/23/2019 09:35:13", "content": "Can anyone provide the sample Cross Validation code sample so that i can understand how it really works.Thanks
", "votes": null, "replies": [ { "id": 460313, "author_name": "cpmpml", "author_url": "", "post_date": "01/23/2019 12:25:21", "content": "See https://scikit-learn.org/stable/modules/cross_validation.html
", "votes": null, "replies": [] } ] } ], "raw_markdown_by_id": { "138164": "Although it looks like feature construction is the most important aspect of this competition, good CV will also be important with only 6879 positive examples in the training set (and probably a similar number in the test set). With the public LB based on only about 2060 positives (30% of 6879), it would be easy to overfit to the public LB without good CV. Below I outline my method (most of it is just common sense). I encourage others to give suggestions.\r\n\r\nI’m doing 5-fold cross-validation. The folds are created by putting every fifth line in a different fold. I didn’t stratify by Response, but that would probably be a good thing to do too.\r\n\r\nWithin each fold, I find the number of TPs, FPs, TNs, and FNs. Then at the end I sum up the results of the folds to get total TPs, FPs, TNs, and FNs. I then calculate MCC from the totals. I was originally calculating MCC within each fold and averaging the 5 MCCs – that works too, and I found it gave similar results.\r\n\r\nActually, within each fold I create predictions 210 times. I create predictions for 7 different values for ntreelimit (number of XGBoost rounds) and 30 different cutoff thresholds (threshold that converts continuous probability to a 0 or 1). 7 * 30 = 210. The reason I did this is A) It doesn’t add that much processing time – you only need to build the model once for each fold, B) I’ve found that the number of rounds that maximizes MCC is not necessarily the same as the number of rounds that maximizes AUC, C) The score is sensitive to the cutoff threshold so I calculate the score for every cutoff between 0.20 and 0.49 (30 cutoffs), and D) The best cutoff is different for different values of ntreelimit so you need to look at the relationship between these two. So for each fold I actually end up with 210 sets of TP, FP, TN, FN.\r\n\r\nSo after the 5 folds are complete, I end up with 210 MCC scores, and this allows me to plot out how MCC varies as ntreelimit and cutoff are varied.\r\n\r\nNow you might ask, “Why not just use MCC as the eval_metric?” I tried that. I found that it was a lot slower and only gave a small improvement over using AUC. \r\n\r\nFinally, if you have constructed features that are based on % positives for various feature values, I would recommend calculating those features inside the CV loop and only based on the training folds. If you calculate the % positives on all the folds, there will be leakage from the test folds into the training folds, and your CV score will be higher than it should be.", "138165": "One more thing ... Picking the best cutoff threshold for your final submissions.\r\n\r\nI’ve experimented with making multiple submissions from the same model, but just vary the cutoff threshold. The cutoff that was best for CV was not always the cutoff that was best for public LB. If you also do this, I recommend using a weighted average of the CV and LB scores to pick the cutoff threshold for your final submission. Weighted average = (1.0 * CV + 0.3 * public LB) / 1.3.", "138212": "For me the best logloss eval for each CV has given the best MCC for overall prediction. I compute MCC score and threshold on whole train set (merging 5 separate parts of training set predicted by 5 CVs), and use that threshold on average of the probabilities from 5 CV's for test set submission. Working well so far !!! Got local CV score of 0.266 and the Public LB for same is 0.2713. If I try tinkering with this eval method, then I am getting public LB worse than local CV score.", "138251": "oh my. this bring back a lot of memories from the prudental insurance competition. it is a not as bad in this one as you have only one cutoff to tune while in prudental it was 7 cutoffs (8 classes classification). i was one of the few that actually moved up in the private board (top 1%) but the shake up there was massive. i have to dig out my old code and notes to remember the tricks i used to not overfit the cutoff to the LB", "139607": "Anyone using data simulation to increase the sample-size of positive samples in training :)?", "139660": "I thought about it, BK. With the data set being as large as it is, I instead downsampled the 0s to get about a 95/5 mix. It gave some decent results but I didn't do a direct comparison of the scores. Have you tried it yet?", "139979": "Banana Kitty,\r\n\r\nI tried various things but did not manage to get something better than what i have with original data.", "140004": "For optimizing the threshold cutoff, I prefer to use scipy.optimize (code below). I found empirically that the default optimization method doesn't work with MCC (instabilities?) but the Powell method works fine. This code also brute-force searches for a good initial threshold to use.\r\n\r\n import numpy as np\r\n from scipy.optimize import minimize\r\n \r\n def threshold(y, t):\r\n \"\"\"Threshold a numpy array of probabilities.\r\n \r\n Args:\r\n y: the array of values\r\n t: the threshold. Map values >= t to 1, otherwise 0.\r\n \r\n Returns:\r\n The thresholded (copy of the) array.\r\n \"\"\"\r\n assert 0 <= t <= 1\r\n y = y.copy()\r\n y[y >= t] = 1\r\n y[y < t] = 0\r\n return y\r\n \r\n def optimize_threshold(ya, yp, score_func, initial_guess=None,\r\n method='Powell', options=None, verbose=False):\r\n \"\"\"Explore possible thresholds to find one that works best.\r\n \r\n Args:\r\n ya: array of actual labels\r\n yp: array of predicted probabilities\r\n score_func: scoring function to use\r\n initial_guess: initial threshold to use. If None, search for one.\r\n method: optimization method (see scipy.minimize)\r\n options: optimization options (see scipy.minimize)\r\n verbose: if False, return the threshold only; otherwise,\r\n the full scipy.minimize result\r\n \r\n Returns:\r\n The optimal threshold or scipy.minimize results.\r\n \"\"\"\r\n # Create a helper function to compute MCC at threshold t.\r\n # Use the negation because scipy.optimize handles only minimization.\r\n f = lambda t: -score_func(ya, threshold(yp, t))\r\n # Search for a suitable initial guess.\r\n if (initial_guess is None):\r\n initial_score = -1.\r\n ymin = min(1e-9, yp.min()) # Move away from 0\r\n for i in np.logspace(np.log10(ymin), 0, endpoint=False, num=10):\r\n if (i >= ymin):\r\n s = score_func(ya, threshold(yp, i))\r\n if (s > initial_score):\r\n initial_score = s\r\n initial_guess = i\r\n opt = minimize(f, initial_guess, method=method, options=options)\r\n return opt if verbose else opt.x", "140125": "JohnM: I observed similar result. Downsampling seems to work and make sure to remove duplicates from training :) In term of simulation, I used this http://www.sciencedirect.com/science/article/pii/S0266892009000113\r\n\r\n@CPMP: Gz to pushing to 0.46 :) Yes, I tried various things too and the original data seems to be the best to use. \r\n\r\n@BreakfastPirate thanks for your tips (y)", "140132": "MichaelB You may want to look at this notebook ( https://www.kaggle.com/cpmpml/bosch-production-line-performance/optimizing-probabilities-for-best-mcc ) where I provide code to compute the optimal threshold exactly.", "460250": "Can anyone provide the sample Cross Validation code sample so that i can understand how it really works.Thanks", "460313": "See https://scikit-learn.org/stable/modules/cross_validation.html" }, "source": "meta" }