Anyone willing to share how many 1's are in their submission?
\n\nI've got 1826 with public LB score ~ 0.31. I don't think that's giving away too much information.
\n\nBy my calculation, this means I have roughly in the ballpark of 1120 TP's and about 706 FP's. (That is extrapolating to the whole test set, not just the 30% that the public leaderboard is based on. My CV score is about the same as my public LB score.)
\n\nBut then again, I'm only in the 0.3x club, not the 0.4x club. :-)
", "messages": [ { "id": 137943, "postDate": "2016-10-06T07:17:34.163Z", "content": "@ rohan... here you go
\n\n> table(sub[,2])\n\n 0 1 \n1180984 2764 \nY'all are way overthinking this. The range of positives for a given mcc score is a symptom of the mcc metric itself. It is dominated by the number of correct TPs first, and secondly by the # of FPs. You could easily game it out using Excel Solver, it's that easy.
\n\nGiven 1183748 total rows and (an assumed) 6900 true positives, an mcc score of 0.50 could be obtained with as few as 1,735 predicted "positives" (1734 TPs and 1 FP) all the way up to as many as 27,125 predicted "positives" (all 6900 TPs and all the rest FPs).
\n\nBasically, the more TPs you can get correct, the more sloppy your FPs can be. Setting up that simple solver can give you a pretty good hint as to where you are in terms of TPs for a given mcc score and number of positives in your submission.
", "rawMarkdown": "Y'all are way overthinking this. The range of positives for a given mcc score is a symptom of the mcc metric itself. It is dominated by the number of correct TPs first, and secondly by the # of FPs. You could easily game it out using Excel Solver, it's that easy.\r\n\r\nGiven 1183748 total rows and (an assumed) 6900 true positives, an mcc score of 0.50 could be obtained with as few as 1,735 predicted \"positives\" (1734 TPs and 1 FP) all the way up to as many as 27,125 predicted \"positives\" (all 6900 TPs and all the rest FPs).\r\n\r\nBasically, the more TPs you can get correct, the more sloppy your FPs can be. Setting up that simple solver can give you a pretty good hint as to where you are in terms of TPs for a given mcc score and number of positives in your submission.\r\n\r\n", "votes": 6 }, { "id": 138088, "postDate": "2016-10-07T05:01:09.117Z", "content": "[quote=aldente;138081]
\n\nGreat! I have questions. How is public/private split? How is distribution of true '1's across public/private, is it also 30/70%?
\n\nWith smaller amount of '1's shared above, 1900 or 2800 out of ~6879, is there chance most of submission's '1's are in the public test set, or vice versa (unbalanced)?
\n\nExample, X has submission with 2000 '1's, LB score is low, is it possible because of most of the correct '1's are in the private set? In other words, X will climb the tower at the end? \n(or the opposite.)
\n\nIf so, is it possible for big shake-up at the end? Thank you.
\n\n[/quote]
\n\nPublic/private split is random. So I would assume that the proportion of 1's is approximately the same in the private part as in the public part, but with only ~6879 1's, it is possible that the proportions might be a little different.
\n\nSo the public LB is based on ~2060 1's (30% of 6879). With that small a number of positive examples, it would certainly be possible to overfit to the public LB. Santander was a similar situation - small number of positive examples. In Santander, there was a big shake-up because people tried out a bunch of hand-coded rules that worked on the training data, and then they kept only the ones that increased their public LB score. But the rules were overfit to the training data and public LB data. I don't see people doing that on this contest (yet). I suspect that people will be tempted to overfit their cutoff-threshold (for converting continuous results to 0's and 1's) to the public LB. But the best way to pick that is to take the weighted average of your CV and LB scores for various cut-off thresholds.
", "rawMarkdown": "[quote=aldente;138081]\r\n\r\nGreat! I have questions. How is public/private split? How is distribution of true '1's across public/private, is it also 30/70%? \r\n\r\nWith smaller amount of '1's shared above, 1900 or 2800 out of ~6879, is there chance most of submission's '1's are in the public test set, or vice versa (unbalanced)? \r\n\r\nExample, X has submission with 2000 '1's, LB score is low, is it possible because of most of the correct '1's are in the private set? In other words, X will climb the tower at the end? \r\n(or the opposite.)\r\n\r\nIf so, is it possible for big shake-up at the end? Thank you. \r\n\r\n[/quote]\r\n\r\nPublic/private split is random. So I would assume that the proportion of 1's is approximately the same in the private part as in the public part, but with only ~6879 1's, it is possible that the proportions might be a little different.\r\n\r\nSo the public LB is based on ~2060 1's (30% of 6879). With that small a number of positive examples, it would certainly be possible to overfit to the public LB. Santander was a similar situation - small number of positive examples. In Santander, there was a big shake-up because people tried out a bunch of hand-coded rules that worked on the training data, and then they kept only the ones that increased their public LB score. But the rules were overfit to the training data and public LB data. I don't see people doing that on this contest (yet). I suspect that people will be tempted to overfit their cutoff-threshold (for converting continuous results to 0's and 1's) to the public LB. But the best way to pick that is to take the weighted average of your CV and LB scores for various cut-off thresholds.", "votes": 3 }, { "id": 137779, "postDate": "2016-10-05T02:10:51.833Z", "content": "I've got 2851 1's that scored ~ 0.44 on the LB. Thats 0.24%.
\n% of 1s in my CV is around the same proportion.
Anyone willing to share how many 1's are in their submission?
\n\nI've got 1826 with public LB score ~ 0.31. I don't think that's giving away too much information.
\n\nBy my calculation, this means I have roughly in the ballpark of 1120 TP's and about 706 FP's. (That is extrapolating to the whole test set, not just the 30% that the public leaderboard is based on. My CV score is about the same as my public LB score.)
\n\nBut then again, I'm only in the 0.3x club, not the 0.4x club. :-)
", "rawMarkdown": "Anyone willing to share how many 1's are in their submission?\r\n\r\nI've got 1826 with public LB score ~ 0.31. I don't think that's giving away too much information.\r\n\r\nBy my calculation, this means I have roughly in the ballpark of 1120 TP's and about 706 FP's. (That is extrapolating to the whole test set, not just the 30% that the public leaderboard is based on. My CV score is about the same as my public LB score.)\r\n\r\nBut then again, I'm only in the 0.3x club, not the 0.4x club. :-)", "votes": 3 }, { "id": 138277, "postDate": "2016-10-07T21:48:59.847Z", "content": "Thank you, @Taffey Lewis.
\n\n@toshi_k, would you like to elaborate more please?
\n\nThis is my Python 2 code to estimate number of TP and FP in the submission file based on its public LB score. (With assumptions explained by BreakfastPirate et.al. above.)
\n\nimport pandas as pd\nimport numpy as np\nfrom scipy.optimize import minimize\ndef mcc(tp, tn, fp, fn):\n sup = tp * tn - fp * fn\n inf = float(tp + fp) * float(tp + fn) * float(tn + fp) * float(tn + fn)\n if inf == 0:\n return 0\n else:\n return float(sup) / np.sqrt(inf)\n\n# fsub = submission file, lbscore = public LB score.\ndef estimate_tp_tn(fsub, lbscore):\n sub = pd.read_csv(fsub)\n numone = np.sum(sub['Response'].values)\n print 'num of one:', numone\n\n def fun(x):\n tp = x[0]\n fp = numone - tp\n fn = 6879 - tp\n tn = 1183748 - tp - fp - fn\n score = mcc(tp, tn, fp, fn)\n return abs(lbscore - score)\n\n res = minimize(fun, [1], method='L-BFGS-B', \n tol=0.00001, options={'maxiter': 100000})\n tp = int(round(res.x[0]))\n fp = numone - tp\n fn = 6879 - tp\n tn = 1183748 - tp - fp - fn\n print 'estimated tp:', tp, 'fp:', fp, 'tn:', tn, 'fn:', fn \n print 'estimated mcc:', lbscore-res.fun, 'lb-score:', lbscore\nExample usage,
\n\nestimate_tp_tn(fsub = './sub1/sub_xgb_v08.csv', lbscore = 0.28226)\nConsole's output is something like,
\n\nnum of one: 2108\nestimated tp: 1109 fp: 1099 tn: 1175770 fn: 5770\nestimated mcc: 0.282259999999 lb-score: 0.28226\n@aldente, nicely done.
\n\nThis was not converging for some of my files until I set the solver explicitly and increased the number of iterations in minimize line:
\n\nres = minimize(fun, [1], method='L-BFGS-B', tol=0.00001, options={'maxiter': 100000})\n@Rohan, you are Top-10 now :P
", "rawMarkdown": "@Rohan, you are Top-10 now :P", "votes": 1 }, { "id": 137855, "postDate": "2016-10-05T15:56:17.833Z", "content": "You should be mighty pleased with yourself now!
\n\nHhhmm... I double checked and sure, I have 2851 of those 'interesting 1s'. My previous submission which scored ~ 0.42 on the LB has 2012 1s, and it isn't a drastically different model.
\n\nNot really sure right now what to infer from it. Probably will make me feel better if someone else in Top-10 has 2500+ 1s. Anyone? :-)
", "rawMarkdown": "You should be mighty pleased with yourself now!\r\n\r\nHhhmm... I double checked and sure, I have 2851 of those 'interesting 1s'. My previous submission which scored ~ 0.42 on the LB has 2012 1s, and it isn't a drastically different model.\r\n\r\nNot really sure right now what to infer from it. Probably will make me feel better if someone else in Top-10 has 2500+ 1s. Anyone? :-)", "votes": 1 }, { "id": 137780, "postDate": "2016-10-05T02:18:58.513Z", "content": "I have more or less what you have, Pirate. One of my recent models gets 2000 or so 1's, with 1650 TP's, 350 FP's and 5200 FN's. A different configuration of that model gets more like 1600 1's, 1350 TP's, 250 FP's and 5500 FN's. Both of those are CV predictions.
\n\nI think everyone is predicting fewer positives than there actually are.
", "rawMarkdown": "I have more or less what you have, Pirate. One of my recent models gets 2000 or so 1's, with 1650 TP's, 350 FP's and 5200 FN's. A different configuration of that model gets more like 1600 1's, 1350 TP's, 250 FP's and 5500 FN's. Both of those are CV predictions. \r\n\r\nI think everyone is predicting fewer positives than there actually are.", "votes": 2 }, { "id": 138475, "postDate": "2016-10-09T02:22:11.140Z", "content": ".40886, 1921 positives in file.
", "rawMarkdown": ".40886, 1921 positives in file." }, { "id": 138403, "postDate": "2016-10-08T16:30:21.573Z", "content": "TP = 1452\nFP = 626\nTN = 1176242\nFN = 5427
\n\nLB Score -> 0.38867
", "rawMarkdown": "TP = 1452\r\nFP = 626\r\nTN = 1176242\r\nFN = 5427\r\n\r\nLB Score -> 0.38867" }, { "id": 138323, "postDate": "2016-10-08T02:37:10.333Z", "content": "[quote=aldente;138117]
\n\nDo you know if the random split is stratified?
\n\n[/quote]
\n\nI don't think so.
\n\n[quote=Taffey Lewis;138245]
\n\nan mcc score of 0.50 could be obtained with as few as 1,735 predicted "positives" (1734 TPs and 1 FP) all the way up to as many as 27,125 predicted "positives" (all 6900 TPs and all the rest FPs).
\n\nBasically, the more TPs you can get correct, the more sloppy your FPs can be.
\n\n[/quote]
\n\nYes, it's interesting to see where various people are along that curve -- especially people with nearly identical scores.
", "rawMarkdown": "[quote=aldente;138117]\r\n\r\nDo you know if the random split is stratified? \r\n\r\n[/quote]\r\n\r\nI don't think so.\r\n\r\n[quote=Taffey Lewis;138245]\r\n\r\nan mcc score of 0.50 could be obtained with as few as 1,735 predicted \"positives\" (1734 TPs and 1 FP) all the way up to as many as 27,125 predicted \"positives\" (all 6900 TPs and all the rest FPs).\r\n\r\nBasically, the more TPs you can get correct, the more sloppy your FPs can be. \r\n\r\n[/quote]\r\n\r\nYes, it's interesting to see where various people are along that curve -- especially people with nearly identical scores." }, { "id": 138303, "postDate": "2016-10-07T23:48:50.443Z", "content": "@Tilii, good catch. I'll edit my code by stealing your snippet ;)
", "rawMarkdown": "@Tilii, good catch. I'll edit my code by stealing your snippet ;) " }, { "id": 138153, "postDate": "2016-10-07T12:37:12.860Z", "content": "Hi everyone !
\n\nMy public LB score is 0.42704, and my submission has 1915 ones.
\nI feel something strange. train_numeric.csv have 6879 ones.\ntrain/test split may not be random split.
@BreakfastPirate, thank you. Do you know if the random split is stratified? I'm a bit nervous with only smaller amount of 1's in the submission. 6879 out of 1.2M is already very small, and the result of 1's in our prediction is even much smaller.
\n\nRegarding hand-coded, I suspect some people will try to do it, as looks like people started to look back at TalkingData with similar consecutive Ids. Although, folks @0.4+ have confirmed there's no leakage. I was temped to do it as well, but now I realize it's too dangerous just like in Santander.
", "rawMarkdown": "@BreakfastPirate, thank you. Do you know if the random split is stratified? I'm a bit nervous with only smaller amount of 1's in the submission. 6879 out of 1.2M is already very small, and the result of 1's in our prediction is even much smaller. \r\n\r\nRegarding hand-coded, I suspect some people will try to do it, as looks like people started to look back at TalkingData with similar consecutive Ids. Although, folks @0.4+ have confirmed there's no leakage. I was temped to do it as well, but now I realize it's too dangerous just like in Santander. " }, { "id": 138081, "postDate": "2016-10-07T02:49:59.627Z", "content": "Great! I have questions. How is public/private split? How is distribution of true '1's across public/private, is it also 30/70%?
\n\nWith smaller amount of '1's shared above, 1900 or 2800 out of ~6879, is there chance most of submission's '1's are in the public test set, or vice versa (unbalanced)?
\n\nExample, X has submission with 2000 '1's, LB score is low, is it possible because of most of the correct '1's are in the private set? In other words, X will climb the tower at the end? \n(or the opposite.)
\n\nIf so, is it possible for big shake-up at the end? Thank you.
", "rawMarkdown": "Great! I have questions. How is public/private split? How is distribution of true '1's across public/private, is it also 30/70%? \r\n\r\nWith smaller amount of '1's shared above, 1900 or 2800 out of ~6879, is there chance most of submission's '1's are in the public test set, or vice versa (unbalanced)? \r\n\r\nExample, X has submission with 2000 '1's, LB score is low, is it possible because of most of the correct '1's are in the private set? In other words, X will climb the tower at the end? \r\n(or the opposite.)\r\n\r\nIf so, is it possible for big shake-up at the end? Thank you. " }, { "id": 137850, "postDate": "2016-10-05T15:01:09.043Z", "content": "Update: 1926 1's with public LB of about 0.44. So that's about 1606 TPs and 320 FPs.
\n\nSo very similar to David Thaler. It is interesting, Rohan, that you, David, and I have almost the same public LB score (currently 10th, 11th, and 12th places), but you have many more 1's than David and I. It would seem to show that there are multiple approaches to achieving that score. Or maybe we use different thresholds for making the final decision to predict 1 or 0, and there is a plateau on the MCC curve between those two points. Either way, it's interesting.
", "rawMarkdown": "Update: 1926 1's with public LB of about 0.44. So that's about 1606 TPs and 320 FPs. \r\n\r\nSo very similar to David Thaler. It is interesting, Rohan, that you, David, and I have almost the same public LB score (currently 10th, 11th, and 12th places), but you have many more 1's than David and I. It would seem to show that there are multiple approaches to achieving that score. Or maybe we use different thresholds for making the final decision to predict 1 or 0, and there is a plateau on the MCC curve between those two points. Either way, it's interesting." }, { "id": 137783, "postDate": "2016-10-05T02:37:58.667Z", "content": "[quote=NetworkMiner;137782]
\n\n@BreakfastPirate, how are you able to tell the number of TP's and FP's?
\n\n[/quote]
\n\nIt's an estimation. We know the number of rows in the test set. I'm assuming that the proportion of positives in the test set is the same as in the training set, so that gives us an approximation of the number of positives in the test set. Given those numbers, the MCC score, and the definition of MCC, we can reverse engineer the number of TPs, FPs, TNs, FNs.
\n\nTP = ?
\n\nFP = number_of_ones_in_submission - TP
\n\nFN = approximate_number_of_positives_in_test_set - TP
\n\nTN = number_of_rows_in_test_set - TP - FP - FN
\n\nThe use MCC equation to calculate approximate TP, FP, FN, TN (may need to use trial-and-error). II put these equations in a spreadsheet and then plugged in numbers for TP until I got an MCC that matched my public LB score. I realize the public LB score is based on only 30% of the data, but it is intended to be an approximation.
", "rawMarkdown": "[quote=NetworkMiner;137782]\r\n\r\n@BreakfastPirate, how are you able to tell the number of TP's and FP's?\r\n\r\n[/quote]\r\n\r\nIt's an estimation. We know the number of rows in the test set. I'm assuming that the proportion of positives in the test set is the same as in the training set, so that gives us an approximation of the number of positives in the test set. Given those numbers, the MCC score, and the definition of MCC, we can reverse engineer the number of TPs, FPs, TNs, FNs.\r\n\r\nTP = ?\r\n\r\nFP = number_of_ones_in_submission - TP\r\n\r\nFN = approximate_number_of_positives_in_test_set - TP\r\n\r\nTN = number_of_rows_in_test_set - TP - FP - FN\r\n\r\nThe use MCC equation to calculate approximate TP, FP, FN, TN (may need to use trial-and-error). II put these equations in a spreadsheet and then plugged in numbers for TP until I got an MCC that matched my public LB score. I realize the public LB score is based on only 30% of the data, but it is intended to be an approximation.\r\n" }, { "id": 137782, "postDate": "2016-10-05T02:24:28.267Z", "content": "Nevermind, I see that you are reporting based on your CV predictions. The original question was about submissions.
\n\n@BreakfastPirate, how are you able to tell the number of TP's and FP's?
", "rawMarkdown": "Nevermind, I see that you are reporting based on your CV predictions. The original question was about submissions.\r\n\r\n@BreakfastPirate, how are you able to tell the number of TP's and FP's?" }, { "id": 137781, "postDate": "2016-10-05T02:22:14.707Z", "content": "@DavidThaler, \nHow are you able to tell how many TP's, FP's and FN's?
", "rawMarkdown": "@DavidThaler, \r\nHow are you able to tell how many TP's, FP's and FN's? " }, { "id": 138662, "postDate": "2016-10-10T07:11:30.413Z", "rawMarkdown": "", "votes": 1, "isDeleted": true } ], "comments": [ { "id": 137943, "author_name": "Darragh", "author_url": "", "post_date": "2016-10-06T07:17:34.163000", "content": "@ rohan... here you go
\n\n> table(sub[,2])\n\n 0 1 \n1180984 2764 \nY'all are way overthinking this. The range of positives for a given mcc score is a symptom of the mcc metric itself. It is dominated by the number of correct TPs first, and secondly by the # of FPs. You could easily game it out using Excel Solver, it's that easy.
\n\nGiven 1183748 total rows and (an assumed) 6900 true positives, an mcc score of 0.50 could be obtained with as few as 1,735 predicted "positives" (1734 TPs and 1 FP) all the way up to as many as 27,125 predicted "positives" (all 6900 TPs and all the rest FPs).
\n\nBasically, the more TPs you can get correct, the more sloppy your FPs can be. Setting up that simple solver can give you a pretty good hint as to where you are in terms of TPs for a given mcc score and number of positives in your submission.
", "votes": 6, "replies": [] }, { "id": 138088, "author_name": "BreakfastPirate", "author_url": "", "post_date": "2016-10-07T05:01:09.117000", "content": "[quote=aldente;138081]
\n\nGreat! I have questions. How is public/private split? How is distribution of true '1's across public/private, is it also 30/70%?
\n\nWith smaller amount of '1's shared above, 1900 or 2800 out of ~6879, is there chance most of submission's '1's are in the public test set, or vice versa (unbalanced)?
\n\nExample, X has submission with 2000 '1's, LB score is low, is it possible because of most of the correct '1's are in the private set? In other words, X will climb the tower at the end? \n(or the opposite.)
\n\nIf so, is it possible for big shake-up at the end? Thank you.
\n\n[/quote]
\n\nPublic/private split is random. So I would assume that the proportion of 1's is approximately the same in the private part as in the public part, but with only ~6879 1's, it is possible that the proportions might be a little different.
\n\nSo the public LB is based on ~2060 1's (30% of 6879). With that small a number of positive examples, it would certainly be possible to overfit to the public LB. Santander was a similar situation - small number of positive examples. In Santander, there was a big shake-up because people tried out a bunch of hand-coded rules that worked on the training data, and then they kept only the ones that increased their public LB score. But the rules were overfit to the training data and public LB data. I don't see people doing that on this contest (yet). I suspect that people will be tempted to overfit their cutoff-threshold (for converting continuous results to 0's and 1's) to the public LB. But the best way to pick that is to take the weighted average of your CV and LB scores for various cut-off thresholds.
", "votes": 3, "replies": [] }, { "id": 137779, "author_name": "Vopani", "author_url": "", "post_date": "2016-10-05T02:10:51.833000", "content": "I've got 2851 1's that scored ~ 0.44 on the LB. Thats 0.24%.
\n% of 1s in my CV is around the same proportion.
Thank you, @Taffey Lewis.
\n\n@toshi_k, would you like to elaborate more please?
\n\nThis is my Python 2 code to estimate number of TP and FP in the submission file based on its public LB score. (With assumptions explained by BreakfastPirate et.al. above.)
\n\nimport pandas as pd\nimport numpy as np\nfrom scipy.optimize import minimize\ndef mcc(tp, tn, fp, fn):\n sup = tp * tn - fp * fn\n inf = float(tp + fp) * float(tp + fn) * float(tn + fp) * float(tn + fn)\n if inf == 0:\n return 0\n else:\n return float(sup) / np.sqrt(inf)\n\n# fsub = submission file, lbscore = public LB score.\ndef estimate_tp_tn(fsub, lbscore):\n sub = pd.read_csv(fsub)\n numone = np.sum(sub['Response'].values)\n print 'num of one:', numone\n\n def fun(x):\n tp = x[0]\n fp = numone - tp\n fn = 6879 - tp\n tn = 1183748 - tp - fp - fn\n score = mcc(tp, tn, fp, fn)\n return abs(lbscore - score)\n\n res = minimize(fun, [1], method='L-BFGS-B', \n tol=0.00001, options={'maxiter': 100000})\n tp = int(round(res.x[0]))\n fp = numone - tp\n fn = 6879 - tp\n tn = 1183748 - tp - fp - fn\n print 'estimated tp:', tp, 'fp:', fp, 'tn:', tn, 'fn:', fn \n print 'estimated mcc:', lbscore-res.fun, 'lb-score:', lbscore\nExample usage,
\n\nestimate_tp_tn(fsub = './sub1/sub_xgb_v08.csv', lbscore = 0.28226)\nConsole's output is something like,
\n\nnum of one: 2108\nestimated tp: 1109 fp: 1099 tn: 1175770 fn: 5770\nestimated mcc: 0.282259999999 lb-score: 0.28226\n@aldente, nicely done.
\n\nThis was not converging for some of my files until I set the solver explicitly and increased the number of iterations in minimize line:
\n\nres = minimize(fun, [1], method='L-BFGS-B', tol=0.00001, options={'maxiter': 100000})\n@Rohan, you are Top-10 now :P
", "votes": 1, "replies": [] }, { "id": 137855, "author_name": "Vopani", "author_url": "", "post_date": "2016-10-05T15:56:17.833000", "content": "You should be mighty pleased with yourself now!
\n\nHhhmm... I double checked and sure, I have 2851 of those 'interesting 1s'. My previous submission which scored ~ 0.42 on the LB has 2012 1s, and it isn't a drastically different model.
\n\nNot really sure right now what to infer from it. Probably will make me feel better if someone else in Top-10 has 2500+ 1s. Anyone? :-)
", "votes": 1, "replies": [] }, { "id": 137780, "author_name": "David Thaler", "author_url": "", "post_date": "2016-10-05T02:18:58.513000", "content": "I have more or less what you have, Pirate. One of my recent models gets 2000 or so 1's, with 1650 TP's, 350 FP's and 5200 FN's. A different configuration of that model gets more like 1600 1's, 1350 TP's, 250 FP's and 5500 FN's. Both of those are CV predictions.
\n\nI think everyone is predicting fewer positives than there actually are.
", "votes": 2, "replies": [] }, { "id": 138475, "author_name": "happycube", "author_url": "", "post_date": "2016-10-09T02:22:11.140000", "content": ".40886, 1921 positives in file.
", "votes": 0, "replies": [] }, { "id": 138403, "author_name": "PrakashSubedi", "author_url": "", "post_date": "2016-10-08T16:30:21.573000", "content": "TP = 1452\nFP = 626\nTN = 1176242\nFN = 5427
\n\nLB Score -> 0.38867
", "votes": 0, "replies": [] }, { "id": 138323, "author_name": "BreakfastPirate", "author_url": "", "post_date": "2016-10-08T02:37:10.333000", "content": "[quote=aldente;138117]
\n\nDo you know if the random split is stratified?
\n\n[/quote]
\n\nI don't think so.
\n\n[quote=Taffey Lewis;138245]
\n\nan mcc score of 0.50 could be obtained with as few as 1,735 predicted "positives" (1734 TPs and 1 FP) all the way up to as many as 27,125 predicted "positives" (all 6900 TPs and all the rest FPs).
\n\nBasically, the more TPs you can get correct, the more sloppy your FPs can be.
\n\n[/quote]
\n\nYes, it's interesting to see where various people are along that curve -- especially people with nearly identical scores.
", "votes": 0, "replies": [] }, { "id": 138303, "author_name": "aldente", "author_url": "", "post_date": "2016-10-07T23:48:50.443000", "content": "@Tilii, good catch. I'll edit my code by stealing your snippet ;)
", "votes": 0, "replies": [] }, { "id": 138153, "author_name": "toshi_k", "author_url": "", "post_date": "2016-10-07T12:37:12.860000", "content": "Hi everyone !
\n\nMy public LB score is 0.42704, and my submission has 1915 ones.
\nI feel something strange. train_numeric.csv have 6879 ones.\ntrain/test split may not be random split.
@BreakfastPirate, thank you. Do you know if the random split is stratified? I'm a bit nervous with only smaller amount of 1's in the submission. 6879 out of 1.2M is already very small, and the result of 1's in our prediction is even much smaller.
\n\nRegarding hand-coded, I suspect some people will try to do it, as looks like people started to look back at TalkingData with similar consecutive Ids. Although, folks @0.4+ have confirmed there's no leakage. I was temped to do it as well, but now I realize it's too dangerous just like in Santander.
", "votes": 0, "replies": [] }, { "id": 138081, "author_name": "aldente", "author_url": "", "post_date": "2016-10-07T02:49:59.627000", "content": "Great! I have questions. How is public/private split? How is distribution of true '1's across public/private, is it also 30/70%?
\n\nWith smaller amount of '1's shared above, 1900 or 2800 out of ~6879, is there chance most of submission's '1's are in the public test set, or vice versa (unbalanced)?
\n\nExample, X has submission with 2000 '1's, LB score is low, is it possible because of most of the correct '1's are in the private set? In other words, X will climb the tower at the end? \n(or the opposite.)
\n\nIf so, is it possible for big shake-up at the end? Thank you.
", "votes": 0, "replies": [] }, { "id": 137850, "author_name": "BreakfastPirate", "author_url": "", "post_date": "2016-10-05T15:01:09.043000", "content": "Update: 1926 1's with public LB of about 0.44. So that's about 1606 TPs and 320 FPs.
\n\nSo very similar to David Thaler. It is interesting, Rohan, that you, David, and I have almost the same public LB score (currently 10th, 11th, and 12th places), but you have many more 1's than David and I. It would seem to show that there are multiple approaches to achieving that score. Or maybe we use different thresholds for making the final decision to predict 1 or 0, and there is a plateau on the MCC curve between those two points. Either way, it's interesting.
", "votes": 0, "replies": [] }, { "id": 137783, "author_name": "BreakfastPirate", "author_url": "", "post_date": "2016-10-05T02:37:58.667000", "content": "[quote=NetworkMiner;137782]
\n\n@BreakfastPirate, how are you able to tell the number of TP's and FP's?
\n\n[/quote]
\n\nIt's an estimation. We know the number of rows in the test set. I'm assuming that the proportion of positives in the test set is the same as in the training set, so that gives us an approximation of the number of positives in the test set. Given those numbers, the MCC score, and the definition of MCC, we can reverse engineer the number of TPs, FPs, TNs, FNs.
\n\nTP = ?
\n\nFP = number_of_ones_in_submission - TP
\n\nFN = approximate_number_of_positives_in_test_set - TP
\n\nTN = number_of_rows_in_test_set - TP - FP - FN
\n\nThe use MCC equation to calculate approximate TP, FP, FN, TN (may need to use trial-and-error). II put these equations in a spreadsheet and then plugged in numbers for TP until I got an MCC that matched my public LB score. I realize the public LB score is based on only 30% of the data, but it is intended to be an approximation.
", "votes": 0, "replies": [] }, { "id": 137782, "author_name": "NetworkMiner", "author_url": "", "post_date": "2016-10-05T02:24:28.267000", "content": "Nevermind, I see that you are reporting based on your CV predictions. The original question was about submissions.
\n\n@BreakfastPirate, how are you able to tell the number of TP's and FP's?
", "votes": 0, "replies": [] }, { "id": 137781, "author_name": "NetworkMiner", "author_url": "", "post_date": "2016-10-05T02:22:14.707000", "content": "@DavidThaler, \nHow are you able to tell how many TP's, FP's and FN's?
", "votes": 0, "replies": [] }, { "id": 138662, "author_name": "", "author_url": "", "post_date": "2016-10-10T07:11:30.413000", "content": "", "votes": 1, "replies": [] } ], "raw_markdown_by_id": { "137943": "@ rohan... here you go\r\n\r\n > table(sub[,2])\r\n \r\n 0 1 \r\n 1180984 2764 ", "138245": "Y'all are way overthinking this. The range of positives for a given mcc score is a symptom of the mcc metric itself. It is dominated by the number of correct TPs first, and secondly by the # of FPs. You could easily game it out using Excel Solver, it's that easy.\r\n\r\nGiven 1183748 total rows and (an assumed) 6900 true positives, an mcc score of 0.50 could be obtained with as few as 1,735 predicted \"positives\" (1734 TPs and 1 FP) all the way up to as many as 27,125 predicted \"positives\" (all 6900 TPs and all the rest FPs).\r\n\r\nBasically, the more TPs you can get correct, the more sloppy your FPs can be. Setting up that simple solver can give you a pretty good hint as to where you are in terms of TPs for a given mcc score and number of positives in your submission.\r\n\r\n", "138088": "[quote=aldente;138081]\r\n\r\nGreat! I have questions. How is public/private split? How is distribution of true '1's across public/private, is it also 30/70%? \r\n\r\nWith smaller amount of '1's shared above, 1900 or 2800 out of ~6879, is there chance most of submission's '1's are in the public test set, or vice versa (unbalanced)? \r\n\r\nExample, X has submission with 2000 '1's, LB score is low, is it possible because of most of the correct '1's are in the private set? In other words, X will climb the tower at the end? \r\n(or the opposite.)\r\n\r\nIf so, is it possible for big shake-up at the end? Thank you. \r\n\r\n[/quote]\r\n\r\nPublic/private split is random. So I would assume that the proportion of 1's is approximately the same in the private part as in the public part, but with only ~6879 1's, it is possible that the proportions might be a little different.\r\n\r\nSo the public LB is based on ~2060 1's (30% of 6879). With that small a number of positive examples, it would certainly be possible to overfit to the public LB. Santander was a similar situation - small number of positive examples. In Santander, there was a big shake-up because people tried out a bunch of hand-coded rules that worked on the training data, and then they kept only the ones that increased their public LB score. But the rules were overfit to the training data and public LB data. I don't see people doing that on this contest (yet). I suspect that people will be tempted to overfit their cutoff-threshold (for converting continuous results to 0's and 1's) to the public LB. But the best way to pick that is to take the weighted average of your CV and LB scores for various cut-off thresholds.", "137779": "I've got 2851 1's that scored ~ 0.44 on the LB. Thats 0.24%. \r\n% of 1s in my CV is around the same proportion.", "137760": "Anyone willing to share how many 1's are in their submission?\r\n\r\nI've got 1826 with public LB score ~ 0.31. I don't think that's giving away too much information.\r\n\r\nBy my calculation, this means I have roughly in the ballpark of 1120 TP's and about 706 FP's. (That is extrapolating to the whole test set, not just the 30% that the public leaderboard is based on. My CV score is about the same as my public LB score.)\r\n\r\nBut then again, I'm only in the 0.3x club, not the 0.4x club. :-)", "138277": "Thank you, @Taffey Lewis. \r\n\r\n@toshi_k, would you like to elaborate more please? \r\n\r\nThis is my Python 2 code to estimate number of TP and FP in the submission file based on its public LB score. (With assumptions explained by BreakfastPirate et.al. above.) \r\n\r\n import pandas as pd\r\n import numpy as np\r\n from scipy.optimize import minimize\r\n def mcc(tp, tn, fp, fn):\r\n sup = tp * tn - fp * fn\r\n inf = float(tp + fp) * float(tp + fn) * float(tn + fp) * float(tn + fn)\r\n if inf == 0:\r\n return 0\r\n else:\r\n return float(sup) / np.sqrt(inf)\r\n \r\n # fsub = submission file, lbscore = public LB score.\r\n def estimate_tp_tn(fsub, lbscore):\r\n sub = pd.read_csv(fsub)\r\n numone = np.sum(sub['Response'].values)\r\n print 'num of one:', numone\r\n \r\n def fun(x):\r\n tp = x[0]\r\n fp = numone - tp\r\n fn = 6879 - tp\r\n tn = 1183748 - tp - fp - fn\r\n score = mcc(tp, tn, fp, fn)\r\n return abs(lbscore - score)\r\n \r\n res = minimize(fun, [1], method='L-BFGS-B', \r\n tol=0.00001, options={'maxiter': 100000})\r\n tp = int(round(res.x[0]))\r\n fp = numone - tp\r\n fn = 6879 - tp\r\n tn = 1183748 - tp - fp - fn\r\n print 'estimated tp:', tp, 'fp:', fp, 'tn:', tn, 'fn:', fn \r\n print 'estimated mcc:', lbscore-res.fun, 'lb-score:', lbscore\r\n\r\nExample usage,\r\n\r\n estimate_tp_tn(fsub = './sub1/sub_xgb_v08.csv', lbscore = 0.28226)\r\n\r\nConsole's output is something like, \r\n\r\n num of one: 2108\r\n estimated tp: 1109 fp: 1099 tn: 1175770 fn: 5770\r\n estimated mcc: 0.282259999999 lb-score: 0.28226\r\n\r\n", "138299": "@aldente, nicely done.\r\n\r\nThis was not converging for some of my files until I set the solver explicitly and increased the number of iterations in *minimize* line:\r\n\r\n res = minimize(fun, [1], method='L-BFGS-B', tol=0.00001, options={'maxiter': 100000})\r\n\r\n", "137931": "@Rohan, you are Top-10 now :P", "137855": "You should be mighty pleased with yourself now!\r\n\r\nHhhmm... I double checked and sure, I have 2851 of those 'interesting 1s'. My previous submission which scored ~ 0.42 on the LB has 2012 1s, and it isn't a drastically different model.\r\n\r\nNot really sure right now what to infer from it. Probably will make me feel better if someone else in Top-10 has 2500+ 1s. Anyone? :-)", "137780": "I have more or less what you have, Pirate. One of my recent models gets 2000 or so 1's, with 1650 TP's, 350 FP's and 5200 FN's. A different configuration of that model gets more like 1600 1's, 1350 TP's, 250 FP's and 5500 FN's. Both of those are CV predictions. \r\n\r\nI think everyone is predicting fewer positives than there actually are.", "138475": ".40886, 1921 positives in file.", "138403": "TP = 1452\r\nFP = 626\r\nTN = 1176242\r\nFN = 5427\r\n\r\nLB Score -> 0.38867", "138323": "[quote=aldente;138117]\r\n\r\nDo you know if the random split is stratified? \r\n\r\n[/quote]\r\n\r\nI don't think so.\r\n\r\n[quote=Taffey Lewis;138245]\r\n\r\nan mcc score of 0.50 could be obtained with as few as 1,735 predicted \"positives\" (1734 TPs and 1 FP) all the way up to as many as 27,125 predicted \"positives\" (all 6900 TPs and all the rest FPs).\r\n\r\nBasically, the more TPs you can get correct, the more sloppy your FPs can be. \r\n\r\n[/quote]\r\n\r\nYes, it's interesting to see where various people are along that curve -- especially people with nearly identical scores.", "138303": "@Tilii, good catch. I'll edit my code by stealing your snippet ;) ", "138153": "Hi everyone !\r\n\r\nMy public LB score is 0.42704, and my submission has 1915 ones.