![\"image\"](\"http://playagricola.com/Kaggle/sub33.jpg\")
Thank you Kaggle for a fun competition and thank you Kagglers for fun and stimulating discussions/kernels. My final model is a classification hyperplane that uses 15 features:
\n\npred = sigmoid( -0.0661x_16 + 0.178x_33 + -0.0526x_45 + -0.0736x_63 + 0.171x_65 + -0.105x_73 + -0.118x_91 + 0-0.0446x_108 + -0.0984x117 + 0.0462x164 + -0.0513x_189 + 0.112x_199 + -0.0704x_209 + -0.120x_217 + -0.0455x_239 )\nThis hyperplane was found by performing logistic regression with L1-penalty on the 2225 observations in the public test dataset combined with the training dataset using LB probes. It achieves CV 0.914, LB 0.890, and Private score 0.869. The solution would be more accurate if I had two more weeks to allow the LB-probe regression iteration to converge.
\n\nTo find coefficient a_33 (the coefficient in front of x_33) you execute and submit the following code:
var = 33\ntest = pd.read_csv('test.csv')\nsub = pd.read_csv('sample_submission.csv')\nsub['target'] = test[str(var)]\nsub.to_csv('submission'+str(var)+'.csv',index=False)\n
Then the value of a_33 is computed by
a_33 = LB_SCORE_33 β 0.500 = 0.671 β 0.500 = 0.171\nIn the past 10 days, I was only able to compute 50 coefficients. The remaining 250 coefficients have been set to zero. To prevent overfitting, I (1) modify a_k based on the training data and (2) set all a_k with abs(a_k)<0.04 equal to a_k=0 because it was shown here that we cannot be confident that a variable is useful in this situation.
A full explanation of my LB-probing techinque with code and simulation is posted here. I look forward to reading other Kaggler's solutions. Please share.
\n\nUPDATE: A pictorial / mathematical explanation why a_k = AUC - 0.5 is posted here.
Thanks for sharing. I have a question about \"a_33 = LB_SCORE_33 β 0.500 = 0.671 β 0.500 = 0.171\". Why this work? How can we prove that?
", "votes": 5, "replies": [ { "id": 528589, "author_name": "Chris Deotte", "author_url": "", "post_date": "2019-05-08T06:14:01.897000", "content": "Hi John. Since the competition metric is AUC, the actual value of a_33 doesn't matter. We just need the ratio of a_33 divided by each other a_k to be correct. It turns out that the ratio of AUC_33 - 0.5 divided by AUC_k - 0.5 is approximately equal to that ratio. Therefore using AUC_k - 0.5 as a proxy for a_k works.
Thanks a lot~
", "votes": 1, "replies": [] }, { "id": 532886, "author_name": "Chris Deotte", "author_url": "", "post_date": "2019-05-18T01:03:22.920000", "content": "Hi @john159151 , I posted the mathematics explaining why a_k = AUC - 0.5 here.
Great. Thanks
", "votes": 0, "replies": [] } ] }, { "id": 528547, "author_name": "Giba", "author_url": "", "post_date": "2019-05-08T03:59:58.223000", "content": "That's amazing Chris. Definitely LB probing is a skill that all Kagglers must have. Thanks for sharing your solution.
", "votes": 3, "replies": [ { "id": 528596, "author_name": "Chris Deotte", "author_url": "", "post_date": "2019-05-08T06:30:16.100000", "content": "Thanks Giba. This comp encouraged me to explore LB probing techniques. I'm excited with my results. I learned a lot including that creating synthetic data and running simulations is very helpful for optimizing techniques.
", "votes": 1, "replies": [] } ] }, { "id": 535361, "author_name": "Marcos Azvdo", "author_url": "", "post_date": "2019-05-22T18:51:49.290000", "content": "Very Nice!! Congrats!
", "votes": 1, "replies": [] }, { "id": 534724, "author_name": "Rashidul H", "author_url": "", "post_date": "2019-05-21T18:31:08.073000", "content": "Congrats!
", "votes": 1, "replies": [] }, { "id": 533856, "author_name": "mcpongg", "author_url": "", "post_date": "2019-05-20T06:40:00.590000", "content": "Awesome ππ»
", "votes": 1, "replies": [] }, { "id": 531969, "author_name": "Saikat Biswas", "author_url": "", "post_date": "2019-05-16T00:13:22.580000", "content": "Learning a lot from your kernal. Thank you for sharing
", "votes": 1, "replies": [ { "id": 532235, "author_name": "Chris Deotte", "author_url": "", "post_date": "2019-05-16T13:40:46.090000", "content": "Thanks Saikat
", "votes": 0, "replies": [] } ] }, { "id": 531958, "author_name": "Vanessa Martins", "author_url": "", "post_date": "2019-05-15T23:26:12.490000", "content": "congrats!
", "votes": 1, "replies": [] }, { "id": 531931, "author_name": "flych3r", "author_url": "", "post_date": "2019-05-15T20:45:28.003000", "content": "nice
", "votes": 1, "replies": [] }, { "id": 531913, "author_name": "Gregory H Eales", "author_url": "", "post_date": "2019-05-15T19:30:33.613000", "content": "Really cool!
", "votes": 1, "replies": [] }, { "id": 531659, "author_name": "mcpongg", "author_url": "", "post_date": "2019-05-15T09:52:46.043000", "content": "Awesome!
", "votes": 1, "replies": [] }, { "id": 530750, "author_name": "Lesha Bardyshev", "author_url": "", "post_date": "2019-05-13T15:34:26.793000", "content": "cool!
", "votes": 1, "replies": [] }, { "id": 530623, "author_name": "databender", "author_url": "", "post_date": "2019-05-13T09:49:43.063000", "content": "Congratulations @cdeotte π
", "votes": 1, "replies": [] }, { "id": 530398, "author_name": "ankit", "author_url": "", "post_date": "2019-05-12T17:44:46.713000", "content": "If you had more time to compute the rest 250, who knows? :)
", "votes": 1, "replies": [] }, { "id": 529763, "author_name": "Prashali Shinde", "author_url": "", "post_date": "2019-05-10T17:19:04.347000", "content": "VERY GOOD!!
", "votes": 1, "replies": [] }, { "id": 529534, "author_name": "Akhilesh", "author_url": "", "post_date": "2019-05-10T06:30:06.870000", "content": "The idea of just using that 15 dimensional hyperplane is in itself brilliant!!!!
", "votes": 1, "replies": [ { "id": 532233, "author_name": "Chris Deotte", "author_url": "", "post_date": "2019-05-16T13:34:17.077000", "content": "Thank you
", "votes": 0, "replies": [] } ] }, { "id": 529370, "author_name": "Kirill Talalaev", "author_url": "", "post_date": "2019-05-09T18:30:12.910000", "content": "Nice!
", "votes": 1, "replies": [] }, { "id": 529335, "author_name": "Akhilesh", "author_url": "", "post_date": "2019-05-09T17:12:27.203000", "content": "Congratulations Chris. You are taking things one competition at a time and you are doing great!!!
", "votes": 1, "replies": [] }, { "id": 529079, "author_name": "Kha Vo", "author_url": "", "post_date": "2019-05-09T06:47:12.623000", "content": "This competition should be renamed to βPlease overfit IIβ :)
", "votes": 1, "replies": [] }, { "id": 529028, "author_name": "WhiteLord", "author_url": "", "post_date": "2019-05-09T03:35:15.823000", "content": "Congratulations, and thank you for sharing the insight of your solution. ;D
", "votes": 1, "replies": [] }, { "id": 528887, "author_name": "Arthur Paulino", "author_url": "", "post_date": "2019-05-08T21:02:59.963000", "content": "There are some precious jewels here. I'll read everything carefully.
\n\nThank you, @cdeotte!
", "votes": 1, "replies": [ { "id": 528971, "author_name": "Chris Deotte", "author_url": "", "post_date": "2019-05-09T01:10:51.623000", "content": "Thanks Arthur
", "votes": 0, "replies": [] } ] }, { "id": 528651, "author_name": "Parul Garg", "author_url": "", "post_date": "2019-05-08T09:19:54.843000", "content": "Nice share
", "votes": 1, "replies": [] }, { "id": 528643, "author_name": "Pascal Pfeiffer", "author_url": "", "post_date": "2019-05-08T08:43:59.663000", "content": "Congratulations to your 2nd place, Chris.
\n\nAmazing solution. Thank you for sharing!\nThe way you probe the LB to get the coefficients is genious.
", "votes": 1, "replies": [ { "id": 528739, "author_name": "Chris Deotte", "author_url": "", "post_date": "2019-05-08T13:23:28.520000", "content": "Thanks Ilu !
", "votes": 0, "replies": [] } ] }, { "id": 528600, "author_name": "AleksanderMendyk", "author_url": "", "post_date": "2019-05-08T06:33:57.353000", "content": "Congratulations and thank you for sharing
", "votes": 1, "replies": [] }, { "id": 528599, "author_name": "Gilles Vandewiele", "author_url": "", "post_date": "2019-05-08T06:32:34.430000", "content": "Amazing trick! Love the simplicity of it :)
", "votes": 1, "replies": [] }, { "id": 528570, "author_name": "Eric R", "author_url": "", "post_date": "2019-05-08T05:15:20.810000", "content": "Congrats! I'm also curious about the math, why variable-wise probing works.
", "votes": 1, "replies": [ { "id": 528601, "author_name": "Gilles Vandewiele", "author_url": "", "post_date": "2019-05-08T06:34:26.057000", "content": "The variables are normalized between 0 and 1, thus they can be used as \"predictions\". The AUC you get back from the LB gives some valuable information as this indicates the probability that the feature value of a randomly sampled positive datapoint is higher than the feature of a randomly sampled negative datapoint.
", "votes": 1, "replies": [] }, { "id": 528603, "author_name": "Chris Deotte", "author_url": "", "post_date": "2019-05-08T06:42:40.243000", "content": "Hi Eric. A simple answer is: the only important thing in the hyperplane formula is the relative strength of the coefficients compared with the others. The ratio of the coefficients is roughly proportional to the ratio of their AUC - 0.5.
A more lengthy answer is: logistic regression is similar to naive Bayes. When the variables are independent (and uncorrelated), instead of building one model with 300 variables, you can build 300 models of 1 variable each and combine them. Finding the AUC of one variable at a time is like building one model at a time. Each model is logit(p) = (AUC_k - 0.5)*x_k = a_k*x_k and the ensemble of models is logit(p) = a_0*x_0 + a_1*x_1 + .... + a_298*x_298 + a_299*x_299. (We can ignore all the bias terms because the competition metric is AUC).
Since you asked with the word \"math\", I'll say an even lengthier answer :-)
\n\nIf you perform linear regression (which is similar to logistic regression) using matrix algebra, you put all the variables x_0, x_1, ..., x_298, x_299 as 300 columns of a matrix A. Then you put the 300 unknown coefficients in a vector u. And you put all the targets in a vector y. Then you want to solve the matrix equation Au = y. The solution is u = (AA^T)^(-1) * A^T * y. The matrix AA^T is both the correlation and covariance matrix of the x's because all x's have mean 0 and std 1. If the variables are independent with no correlation, then the matrix (AA^T) is a diagonal matrix and it's inverse is diagonal. So each element of u can be found separately as u_k = (A_k,k) ^(-1) * (A^T)_row-k * y. This demonstrates how coefficients in multi-variable regression can be solved one at a time when the variables are independent and uncorrelated.
Hi Eric ( @vivaroma ), above explains why you can calculate each a_k separately. I posted the mathematics explaining why a_k = AUC - 0.5 here.
Congrats Chris! As always your analysis is clearly explained and thoroughly thought through. This takes LB probing to a whole new level!
", "votes": 1, "replies": [ { "id": 528592, "author_name": "Chris Deotte", "author_url": "", "post_date": "2019-05-08T06:19:16.480000", "content": "Thanks Rob. Yeah I had a lot of fun designing LB probing techniques for this competition. I created synthetic data and then tried out tons of LB probing strategies on the synthetic data to see what converges fastest and what doesn't overfit LB. I entered this comp 20 days ago. I used model 3 in my kernel here for 10 days and got up to LB 0.875. Then I discovered model 4 which converges faster and doesn't overfit the public test dataset as much as model 3. In 10 days it got up to LB 0.890.
", "votes": 5, "replies": [] } ] }, { "id": 530289, "author_name": "DONXUX", "author_url": "", "post_date": "2019-05-12T12:14:27.233000", "content": "good
", "votes": 2, "replies": [] }, { "id": 530280, "author_name": "SaiAdityaGarlapati", "author_url": "", "post_date": "2019-05-12T11:38:28.027000", "content": "Great!!
", "votes": 2, "replies": [] }, { "id": 529105, "author_name": "Himanshu Soni", "author_url": "", "post_date": "2019-05-09T07:40:23.033000", "content": "Nice and great effort Chris you have done LB probing very smartly and consistently you moved to get insights of various features, your ideas will help us to get good rank in another competition.
", "votes": 2, "replies": [] }, { "id": 528655, "author_name": "", "author_url": "", "post_date": "2019-05-08T09:37:07.310000", "content": "", "votes": 2, "replies": [ { "id": 529506, "author_name": "", "author_url": "", "post_date": "2019-05-10T04:15:15.283000", "content": "", "votes": 0, "replies": [] } ] }, { "id": 528625, "author_name": "", "author_url": "", "post_date": "2019-05-08T07:41:58.083000", "content": "", "votes": 2, "replies": [ { "id": 529010, "author_name": "", "author_url": "", "post_date": "2019-05-09T02:44:50.260000", "content": "", "votes": 2, "replies": [] }, { "id": 529019, "author_name": "", "author_url": "", "post_date": "2019-05-09T03:10:29.190000", "content": "", "votes": 1, "replies": [] } ] }, { "id": 528620, "author_name": "", "author_url": "", "post_date": "2019-05-08T07:17:05.170000", "content": "", "votes": 2, "replies": [ { "id": 528974, "author_name": "", "author_url": "", "post_date": "2019-05-09T01:27:56.157000", "content": "", "votes": 1, "replies": [] } ] }, { "id": 528535, "author_name": "", "author_url": "", "post_date": "2019-05-08T03:24:31.137000", "content": "", "votes": 2, "replies": [ { "id": 528593, "author_name": "", "author_url": "", "post_date": "2019-05-08T06:20:25.987000", "content": "", "votes": 1, "replies": [] } ] }, { "id": 3191826, "author_name": "", "author_url": "", "post_date": "2025-05-02T07:33:40.847000", "content": "", "votes": 0, "replies": [] }, { "id": 529926, "author_name": "", "author_url": "", "post_date": "2019-05-11T07:20:29.047000", "content": "", "votes": 1, "replies": [] }, { "id": 531972, "author_name": "", "author_url": "", "post_date": "2019-05-16T00:16:26.747000", "content": "", "votes": 1, "replies": [] }, { "id": 531967, "author_name": "", "author_url": "", "post_date": "2019-05-16T00:03:29.600000", "content": "", "votes": 1, "replies": [] }, { "id": 531523, "author_name": "", "author_url": "", "post_date": "2019-05-15T04:14:10.740000", "content": "", "votes": 1, "replies": [] }, { "id": 530422, "author_name": "", "author_url": "", "post_date": "2019-05-12T18:57:43.860000", "content": "", "votes": 1, "replies": [] }, { "id": 529063, "author_name": "", "author_url": "", "post_date": "2019-05-09T05:24:59.747000", "content": "", "votes": 1, "replies": [] }, { "id": 528670, "author_name": "", "author_url": "", "post_date": "2019-05-08T10:29:38.307000", "content": "", "votes": 1, "replies": [] } ], "comment_count": 60, "messages": [ { "id": 528513, "postDate": "2019-05-08T02:00:49.880Z", "content": "Thank you Kaggle for a fun competition and thank you Kagglers for fun and stimulating discussions/kernels. My final model is a classification hyperplane that uses 15 features:
\n\npred = sigmoid( -0.0661x_16 + 0.178x_33 + -0.0526x_45 + -0.0736x_63 + 0.171x_65 + -0.105x_73 + -0.118x_91 + 0-0.0446x_108 + -0.0984x117 + 0.0462x164 + -0.0513x_189 + 0.112x_199 + -0.0704x_209 + -0.120x_217 + -0.0455x_239 )\nThis hyperplane was found by performing logistic regression with L1-penalty on the 2225 observations in the public test dataset combined with the training dataset using LB probes. It achieves CV 0.914, LB 0.890, and Private score 0.869. The solution would be more accurate if I had two more weeks to allow the LB-probe regression iteration to converge.
\n\nTo find coefficient a_33 (the coefficient in front of x_33) you execute and submit the following code:
var = 33\ntest = pd.read_csv('test.csv')\nsub = pd.read_csv('sample_submission.csv')\nsub['target'] = test[str(var)]\nsub.to_csv('submission'+str(var)+'.csv',index=False)\n
Then the value of a_33 is computed by
a_33 = LB_SCORE_33 β 0.500 = 0.671 β 0.500 = 0.171\nIn the past 10 days, I was only able to compute 50 coefficients. The remaining 250 coefficients have been set to zero. To prevent overfitting, I (1) modify a_k based on the training data and (2) set all a_k with abs(a_k)<0.04 equal to a_k=0 because it was shown here that we cannot be confident that a variable is useful in this situation.
A full explanation of my LB-probing techinque with code and simulation is posted here. I look forward to reading other Kaggler's solutions. Please share.
\n\nUPDATE: A pictorial / mathematical explanation why a_k = AUC - 0.5 is posted here.
Thanks for sharing. I have a question about \"a_33 = LB_SCORE_33 β 0.500 = 0.671 β 0.500 = 0.171\". Why this work? How can we prove that?
", "rawMarkdown": "Thanks for sharing. I have a question about \"a_33 = LB_SCORE_33 β 0.500 = 0.671 β 0.500 = 0.171\". Why this work? How can we prove that?", "votes": 5, "replies": [ { "id": 528589, "postDate": "2019-05-08T06:14:01.897Z", "content": "Hi John. Since the competition metric is AUC, the actual value of a_33 doesn't matter. We just need the ratio of a_33 divided by each other a_k to be correct. It turns out that the ratio of AUC_33 - 0.5 divided by AUC_k - 0.5 is approximately equal to that ratio. Therefore using AUC_k - 0.5 as a proxy for a_k works.
Thanks a lot~
", "rawMarkdown": "Thanks a lot~", "votes": 1 }, { "id": 532886, "postDate": "2019-05-18T01:03:22.920Z", "content": "Hi @john159151 , I posted the mathematics explaining why a_k = AUC - 0.5 here.
Great. Thanks
", "rawMarkdown": "Great. Thanks" } ] }, { "id": 528547, "postDate": "2019-05-08T03:59:58.223Z", "content": "That's amazing Chris. Definitely LB probing is a skill that all Kagglers must have. Thanks for sharing your solution.
", "rawMarkdown": "That's amazing Chris. Definitely LB probing is a skill that all Kagglers must have. Thanks for sharing your solution.", "votes": 3, "replies": [ { "id": 528596, "postDate": "2019-05-08T06:30:16.100Z", "content": "Thanks Giba. This comp encouraged me to explore LB probing techniques. I'm excited with my results. I learned a lot including that creating synthetic data and running simulations is very helpful for optimizing techniques.
", "rawMarkdown": "Thanks Giba. This comp encouraged me to explore LB probing techniques. I'm excited with my results. I learned a lot including that creating synthetic data and running simulations is very helpful for optimizing techniques.", "votes": 1 } ] }, { "id": 535361, "postDate": "2019-05-22T18:51:49.290Z", "content": "Very Nice!! Congrats!
", "rawMarkdown": "Very Nice!! Congrats!", "votes": 1 }, { "id": 534724, "postDate": "2019-05-21T18:31:08.073Z", "content": "Congrats!
", "rawMarkdown": "**Congrats!**", "votes": 1 }, { "id": 533856, "postDate": "2019-05-20T06:40:00.590Z", "content": "Awesome ππ»
", "rawMarkdown": "Awesome ππ»", "votes": 1 }, { "id": 531969, "postDate": "2019-05-16T00:13:22.580Z", "content": "Learning a lot from your kernal. Thank you for sharing
", "rawMarkdown": "Learning a lot from your kernal. Thank you for sharing", "votes": 1, "replies": [ { "id": 532235, "postDate": "2019-05-16T13:40:46.090Z", "content": "Thanks Saikat
", "rawMarkdown": "Thanks Saikat" } ] }, { "id": 531958, "postDate": "2019-05-15T23:26:12.490Z", "content": "congrats!
", "rawMarkdown": "congrats!", "votes": 1 }, { "id": 531931, "postDate": "2019-05-15T20:45:28.003Z", "content": "nice
", "rawMarkdown": "nice", "votes": 1 }, { "id": 531913, "postDate": "2019-05-15T19:30:33.613Z", "content": "Really cool!
", "rawMarkdown": "Really cool!", "votes": 1 }, { "id": 531659, "postDate": "2019-05-15T09:52:46.043Z", "content": "Awesome!
", "rawMarkdown": "Awesome!", "votes": 1 }, { "id": 530750, "postDate": "2019-05-13T15:34:26.793Z", "content": "cool!
", "rawMarkdown": "cool!", "votes": 1 }, { "id": 530623, "postDate": "2019-05-13T09:49:43.063Z", "content": "Congratulations @cdeotte π
", "rawMarkdown": "Congratulations @cdeotte π ", "votes": 1 }, { "id": 530398, "postDate": "2019-05-12T17:44:46.713Z", "content": "If you had more time to compute the rest 250, who knows? :)
", "rawMarkdown": "If you had more time to compute the rest 250, who knows? :)", "votes": 1 }, { "id": 529763, "postDate": "2019-05-10T17:19:04.347Z", "content": "VERY GOOD!!
", "rawMarkdown": "VERY GOOD!!", "votes": 1 }, { "id": 529534, "postDate": "2019-05-10T06:30:06.870Z", "content": "The idea of just using that 15 dimensional hyperplane is in itself brilliant!!!!
", "rawMarkdown": "The idea of just using that 15 dimensional hyperplane is in itself brilliant!!!!", "votes": 1, "replies": [ { "id": 532233, "postDate": "2019-05-16T13:34:17.077Z", "content": "Thank you
", "rawMarkdown": "Thank you" } ] }, { "id": 529370, "postDate": "2019-05-09T18:30:12.910Z", "content": "Nice!
", "rawMarkdown": "Nice!", "votes": 1 }, { "id": 529335, "postDate": "2019-05-09T17:12:27.203Z", "content": "Congratulations Chris. You are taking things one competition at a time and you are doing great!!!
", "rawMarkdown": "Congratulations Chris. You are taking things one competition at a time and you are doing great!!!\n", "votes": 1 }, { "id": 529079, "postDate": "2019-05-09T06:47:12.623Z", "content": "This competition should be renamed to βPlease overfit IIβ :)
", "rawMarkdown": "This competition should be renamed to βPlease overfit IIβ :)", "votes": 1 }, { "id": 529028, "postDate": "2019-05-09T03:35:15.823Z", "content": "Congratulations, and thank you for sharing the insight of your solution. ;D
", "rawMarkdown": "Congratulations, and thank you for sharing the insight of your solution. ;D", "votes": 1 }, { "id": 528887, "postDate": "2019-05-08T21:02:59.963Z", "content": "There are some precious jewels here. I'll read everything carefully.
\n\nThank you, @cdeotte!
", "rawMarkdown": "There are some precious jewels here. I'll read everything carefully.\n\nThank you, @cdeotte!", "votes": 1, "replies": [ { "id": 528971, "postDate": "2019-05-09T01:10:51.623Z", "content": "Thanks Arthur
", "rawMarkdown": "Thanks Arthur" } ] }, { "id": 528651, "postDate": "2019-05-08T09:19:54.843Z", "content": "Nice share
", "rawMarkdown": "Nice share", "votes": 1 }, { "id": 528643, "postDate": "2019-05-08T08:43:59.663Z", "content": "Congratulations to your 2nd place, Chris.
\n\nAmazing solution. Thank you for sharing!\nThe way you probe the LB to get the coefficients is genious.
", "rawMarkdown": "Congratulations to your 2nd place, Chris.\n\nAmazing solution. Thank you for sharing!\nThe way you probe the LB to get the coefficients is genious. ", "votes": 1, "replies": [ { "id": 528739, "postDate": "2019-05-08T13:23:28.520Z", "content": "Thanks Ilu !
", "rawMarkdown": "Thanks Ilu !" } ] }, { "id": 528600, "postDate": "2019-05-08T06:33:57.353Z", "content": "Congratulations and thank you for sharing
", "rawMarkdown": "Congratulations and thank you for sharing", "votes": 1 }, { "id": 528599, "postDate": "2019-05-08T06:32:34.430Z", "content": "Amazing trick! Love the simplicity of it :)
", "rawMarkdown": "Amazing trick! Love the simplicity of it :)", "votes": 1 }, { "id": 528570, "postDate": "2019-05-08T05:15:20.810Z", "content": "Congrats! I'm also curious about the math, why variable-wise probing works.
", "rawMarkdown": "Congrats! I'm also curious about the math, why variable-wise probing works.", "votes": 1, "replies": [ { "id": 528601, "postDate": "2019-05-08T06:34:26.057Z", "content": "The variables are normalized between 0 and 1, thus they can be used as \"predictions\". The AUC you get back from the LB gives some valuable information as this indicates the probability that the feature value of a randomly sampled positive datapoint is higher than the feature of a randomly sampled negative datapoint.
", "rawMarkdown": "The variables are normalized between 0 and 1, thus they can be used as \"predictions\". The AUC you get back from the LB gives some valuable information as this indicates the probability that the feature value of a randomly sampled positive datapoint is higher than the feature of a randomly sampled negative datapoint.", "votes": 1 }, { "id": 528603, "postDate": "2019-05-08T06:42:40.243Z", "content": "Hi Eric. A simple answer is: the only important thing in the hyperplane formula is the relative strength of the coefficients compared with the others. The ratio of the coefficients is roughly proportional to the ratio of their AUC - 0.5.
A more lengthy answer is: logistic regression is similar to naive Bayes. When the variables are independent (and uncorrelated), instead of building one model with 300 variables, you can build 300 models of 1 variable each and combine them. Finding the AUC of one variable at a time is like building one model at a time. Each model is logit(p) = (AUC_k - 0.5)*x_k = a_k*x_k and the ensemble of models is logit(p) = a_0*x_0 + a_1*x_1 + .... + a_298*x_298 + a_299*x_299. (We can ignore all the bias terms because the competition metric is AUC).
Since you asked with the word \"math\", I'll say an even lengthier answer :-)
\n\nIf you perform linear regression (which is similar to logistic regression) using matrix algebra, you put all the variables x_0, x_1, ..., x_298, x_299 as 300 columns of a matrix A. Then you put the 300 unknown coefficients in a vector u. And you put all the targets in a vector y. Then you want to solve the matrix equation Au = y. The solution is u = (AA^T)^(-1) * A^T * y. The matrix AA^T is both the correlation and covariance matrix of the x's because all x's have mean 0 and std 1. If the variables are independent with no correlation, then the matrix (AA^T) is a diagonal matrix and it's inverse is diagonal. So each element of u can be found separately as u_k = (A_k,k) ^(-1) * (A^T)_row-k * y. This demonstrates how coefficients in multi-variable regression can be solved one at a time when the variables are independent and uncorrelated.
Hi Eric ( @vivaroma ), above explains why you can calculate each a_k separately. I posted the mathematics explaining why a_k = AUC - 0.5 here.
Congrats Chris! As always your analysis is clearly explained and thoroughly thought through. This takes LB probing to a whole new level!
", "rawMarkdown": "Congrats Chris! As always your analysis is clearly explained and thoroughly thought through. This takes LB probing to a whole new level!", "votes": 1, "replies": [ { "id": 528592, "postDate": "2019-05-08T06:19:16.480Z", "content": "Thanks Rob. Yeah I had a lot of fun designing LB probing techniques for this competition. I created synthetic data and then tried out tons of LB probing strategies on the synthetic data to see what converges fastest and what doesn't overfit LB. I entered this comp 20 days ago. I used model 3 in my kernel here for 10 days and got up to LB 0.875. Then I discovered model 4 which converges faster and doesn't overfit the public test dataset as much as model 3. In 10 days it got up to LB 0.890.
", "rawMarkdown": "Thanks Rob. Yeah I had a lot of fun designing LB probing techniques for this competition. I created synthetic data and then tried out tons of LB probing strategies on the synthetic data to see what converges fastest and what doesn't overfit LB. I entered this comp 20 days ago. I used model 3 in my kernel [here][1] for 10 days and got up to LB 0.875. Then I discovered model 4 which converges faster and doesn't overfit the public test dataset as much as model 3. In 10 days it got up to LB 0.890. \n \n[1]: https://www.kaggle.com/cdeotte/lb-probing-strategies-0-890-2nd-place", "votes": 5 } ] }, { "id": 530289, "postDate": "2019-05-12T12:14:27.233Z", "content": "good
", "rawMarkdown": "good", "votes": 2 }, { "id": 530280, "postDate": "2019-05-12T11:38:28.027Z", "content": "Great!!
", "rawMarkdown": "Great!!", "votes": 2 }, { "id": 529105, "postDate": "2019-05-09T07:40:23.033Z", "content": "Nice and great effort Chris you have done LB probing very smartly and consistently you moved to get insights of various features, your ideas will help us to get good rank in another competition.
", "rawMarkdown": "Nice and great effort Chris you have done LB probing very smartly and consistently you moved to get insights of various features, your ideas will help us to get good rank in another competition. ", "votes": 2 }, { "id": 528655, "postDate": "2019-05-08T09:37:07.310Z", "content": "Awesome work and thanks for sharing your ideas now and throughout this competition!
", "rawMarkdown": "Awesome work and thanks for sharing your ideas now and throughout this competition!", "votes": 2, "replies": [ { "id": 529506, "postDate": "2019-05-10T04:15:15.283Z", "content": "Thanks Anna
", "rawMarkdown": "Thanks Anna" } ] }, { "id": 528625, "postDate": "2019-05-08T07:41:58.083Z", "content": "Congratulations and thanks for your great sharing!\nChris, it seems that you find a hyperplane which is used for classification.\nAnd I have some questions for it, if not mind could you explain it?\n1. Have you tried to do some scalars or transformations to the data? Because I think the data is nearly standard normal distribution I will try to convert it into standard normal distribution. And in fact, after doing transformations, I get increased in CV, LB, and Private, which is considered as a way to denoise in my opinions.\n2. Have you tried to use some other models like SVC support vector classification and LDA naive Bayes that you said in your kernel, which get better or worse performance than LR Logistic regression? And what makes you choose LR not other models.
\n\nAnd thanks again to your outstanding ideas and great sharings since the Santander game!!!
", "rawMarkdown": "Congratulations and thanks for your great sharing!\nChris, it seems that you find a hyperplane which is used for classification.\nAnd I have some questions for it, if not mind could you explain it?\n1. Have you tried to do some scalars or transformations to the data? Because I think the data is **nearly standard normal** distribution I will try to convert it into **standard normal** distribution. And in fact, after doing transformations, I get increased in CV, LB, and Private, which is considered as a way to denoise in my opinions.\n2. Have you tried to use some other models like **SVC support vector classification** and **LDA naive Bayes** that you said in your [kernel](https://www.kaggle.com/cdeotte/lb-probing-strategies-0-890-2nd-place#Conclusion), which get better or worse performance than **LR Logistic regression**? And what makes you choose **LR** not other models.\n\nAnd thanks again to your outstanding ideas and great sharings since the Santander game!!!", "votes": 2, "replies": [ { "id": 529010, "postDate": "2019-05-09T02:44:50.260Z", "content": "Thanks Bowen. (1) I have not tried transforming the data. I believe the data is already standard normal. (2) All three methods LR, SVC, and LDA find 300 a_k and predict with a hyperplane. I say that my method is similar to LR to give people a concrete method to compare with. However using AUC to calculate a_k is approximate and not exact. Therefore my solution is not exactly LR but rather probably a mixture of all three of these.
Thanks for your explanation, Chris.
", "rawMarkdown": "Thanks for your explanation, Chris.", "votes": 1 } ] }, { "id": 528620, "postDate": "2019-05-08T07:17:05.170Z", "content": "Congrats, Chris. so the reason this approach works is because variables are independent? Do you think this would also work in Santander's then?
", "rawMarkdown": "Congrats, Chris. so the reason this approach works is because variables are independent? Do you think this would also work in Santander's then?", "votes": 2, "replies": [ { "id": 528974, "postDate": "2019-05-09T01:27:56.157Z", "content": "Thanks. This would work on Santander, however it wouldn't produce a high LB. This method is only a linear classifier and linear classifiers only scored around 0.850 on Santander. The winning solutions were nonlinear in each variable.
", "rawMarkdown": "Thanks. This would work on Santander, however it wouldn't produce a high LB. This method is only a linear classifier and linear classifiers only scored around 0.850 on Santander. The winning solutions were nonlinear in each variable.", "votes": 1 } ] }, { "id": 528535, "postDate": "2019-05-08T03:24:31.137Z", "content": "Congratulations Chris and thanks for sharing your ideas behind this competition!!!
", "rawMarkdown": "Congratulations Chris and thanks for sharing your ideas behind this competition!!! ", "votes": 2, "replies": [ { "id": 528593, "postDate": "2019-05-08T06:20:25.987Z", "content": "Thanks
", "rawMarkdown": "Thanks", "votes": 1 } ] }, { "id": 3191826, "postDate": "2025-05-02T07:33:40.847Z", "content": "Hello, sorry for commenting such an old solution, but why is the link of the submission in an online agricola webpage? I love agricola
", "rawMarkdown": "Hello, sorry for commenting such an old solution, but why is the link of the submission in an online agricola webpage? I love agricola" }, { "id": 529926, "postDate": "2019-05-11T07:20:29.047Z", "rawMarkdown": "", "votes": 1, "isDeleted": true }, { "id": 531972, "postDate": "2019-05-16T00:16:26.747Z", "content": "Nice! Thanks
", "rawMarkdown": "Nice! Thanks", "votes": 1 }, { "id": 531967, "postDate": "2019-05-16T00:03:29.600Z", "content": "Nice! Thanks
", "rawMarkdown": "Nice! Thanks", "votes": 1 }, { "id": 531523, "postDate": "2019-05-15T04:14:10.740Z", "content": "Very Good!! Thanks for Sharing!!
", "rawMarkdown": "Very Good!! Thanks for Sharing!!", "votes": 1 }, { "id": 530422, "postDate": "2019-05-12T18:57:43.860Z", "content": "Nice. Thanks for sharing this..
", "rawMarkdown": "Nice. Thanks for sharing this..", "votes": 1 }, { "id": 529063, "postDate": "2019-05-09T05:24:59.747Z", "content": "Thank you for sharing this.
", "rawMarkdown": "Thank you for sharing this.", "votes": 1 }, { "id": 528670, "postDate": "2019-05-08T10:29:38.307Z", "content": "Thank you!I've learnt a lot from this.
", "rawMarkdown": "Thank you!I've learnt a lot from this.", "votes": 1 } ], "complete": true, "source": "live", "fetched_at": "2026-10-19T19:12:58Z", "competition_id": 12896, "competition_slug": "dont-overfit-ii", "team_id": 3060501, "solution_url": "/competitions/dont-overfit-ii/writeups/chris-deotte-2nd-place-solution" }