![\"Public](\"https://i.imgur.com/zTmMg73.png\")
First of all, I want to say that I feel incredibly honored to win. The original Don’t Overfit was my first Kaggle competition and coincides with when I discovered machine learning. I remember poring over forum posts to learn how to fit models with the caret package and writing R code on the commuter train on my way to job interviews in Boston, a city where I’d just arrived. On those train rides, I daydreamed about one day having a job where I did machine learning for work.
\n\nI can’t believe it’s been 8 years.
\n\nThe original Don’t Overfit competition was the start of a trajectory that took me to the career I have today. Thank you @salimali for piquing my interest, and thank you to the Kaggle community for all you’ve given to me.
\n\nThis competition, of course, was completely different from the original Don’t Overfit 😁
\n\nIn the original competition, we only had 5 submissions total for the private leaderboard and no leaderboard feedback.
\n\nI believe my critical insight was being among the first to realize leaderboard probing would be key to winning. I started a dedicated effort to submit every individual variable as a single submission. I was able to get the AUC for all 300 variables on the public leaderboard, and as @cdeotte already discussed, used (AUC public) - 0.50 to come up with “coefficients” based on AUC. If Chris had had time for 250 more submissions, he would have had a good chance of beating me. Just using the 300 public LB “coefficients”, with no further processing, gives a private LB score of 0.853, which would have been good enough for 110th place or so.
\n\nI additionally used the AUCs on the training data to add a little bit of extra information to this model, using the following formula: 0.11*(AUC train) + 0.89*(AUC pubic) - 0.5\nThe weight for AUC train is (rows train)/[rows train + rows public]\nThe weight for AUC public is (rows public)/[rows train + rows public]
This gives a private leaderboard score of 0.856, good enough for 60th place or so. (About where I finished in the first iteration of Don’t Overfit!).
\n\nIt also gives the following plot of public AUC vs training AUC, which led me to believe all 300 variables were used to create the target variable. This is also a useful plot to calibrate simulated datasets (more on this later):\n
This methodology overfits the public leaderboard, as there are many small AUCs that should probably be zero. Using Chris’ capping heuristic, we can set every coefficient with abs(cf) < 0.04 = 0, which yields the following equation:\n\n-0.0661*X16 + -0.0448*X29 + 0.1778*X33 + -0.0526*X45 + -0.0736*X63 + 0.171*X65 + -0.059*X70 + -0.1047*X73 + -0.118*X91 + -0.0422*X106 + -0.0446*X108 + -0.0985*X117 + -0.0448*X132 + 0.0462*X164 + -0.0514*X189 + 0.1117*X199 + -0.0704*X209 + -0.1198*X217 + -0.0456*X239\n
This is very similar to Chris’ solution, just with more variables. This gives a private leaderboard score of 0.877, which is good enough for first place.
\n\nI don’t like capping, and I wanted a solution that smoothly shrinks smaller AUCs to zero. For example, variable 239 in the above equation has a coefficient of -0.0456, and there’s a good chance that is really a zero (but I’m not certain)!
\n\nSo what I spent most of my time doing was looking for an equation that would smoothly shrink small coefficients to zero, while leaving large coefficients more or less alone. I dove down a bit of a rabbit hole here, but it was a fun rabbit hole, and I’m interested to see what approaches other people take to solving the same problem.
\n\nThe approach I took was:\n1. Simulate a dataset with 20,000 rows, 300 columns, sampled from the random normal distribution. AKA matrix(rnorm(20000*300), ncol=300) in R. I am 99.99% sure this is how Kaggle simulated the data for this competition.\n2. Apply my probed coefficients to the data. This isn’t perfect, but the coefficients I probed should be pretty representative of the real distribution of coefficients. AKA matrix(rnorm(20000*300), ncol=300) %*% probed_cf in R.\n3. Add a little random noise.\n4. Choose a classification cutoff that gives a target distribution in the training data of 160 0s and 90 1s.
I then fiddled with steps 2-4 until I got (AUC train - 250 rows) vs (AUC public - 1975 rows) plots that looked like the above plot.
\n\nI also simulated “leaderboard probing” to try to get simulated results that had training AUCs of .95 and public AUCs around .92, as those were my results on the real leaderboard.
\n\nOnce I had my simulated data, the problem becomes: predict the “real” coefficients, given the probed coefficients. I played around with many different methodologies for this, and finally ended up using Eureqa Desktop to discover the following equation:\n\npenalized_cf = 0.0320794465661013 + 1.38501844798023*cf - 9.51464300036321*cf^3 - \n 0.0654557703259389*plogis(85.9772069321308*cf)\n\n(Where plogis is the sigmoid function in R)\nHere is a plot of the penalized coefficients vs. the raw coefficients. Note that coefficients near zero get shrunk almost completely to zero, but not entirely:\n![\"Penalized\"](\"https://i.imgur.com/9Wp6v3q.png\")
So to recap, my final submission was:\n\ncf = 0.11*(auc_train) + 0.89*(auc_pubic) - 0.5\npenalized_cf = 0.032 + 1.385*cf - 9.515*cf^3 - 0.065*plogis(85.977*cf)\nsub = X_sub %*% penalized_cf\n
This achieves a private LB score of 0.886 and a public LB score of 0.923
\n\nI’ll post more detailed code (along with the probed AUCs) in some public kernels.
", "comments": [ { "id": 528883, "author_name": "Zach", "author_url": "", "post_date": "2019-05-08T20:51:10.910000", "content": "As a follow up, here are the 300 probed AUCs as a kernel and as a dataset:\nhttps://www.kaggle.com/zachmayer/300-probed-aucs?scriptVersionId=13927278\nhttps://www.kaggle.com/zachmayer/300-probed-aucs-from-dont-overfit-ii
", "votes": 9, "replies": [] }, { "id": 528984, "author_name": "Zach", "author_url": "", "post_date": "2019-05-09T01:39:05.227000", "content": "And here's my code: https://www.kaggle.com/zachmayer/first-place-solution?scriptVersionId=13934694
", "votes": 5, "replies": [] }, { "id": 531407, "author_name": "PBenavides", "author_url": "", "post_date": "2019-05-14T20:42:33.437000", "content": "Congratulations for the win!
", "votes": 1, "replies": [] }, { "id": 529176, "author_name": "Eric R", "author_url": "", "post_date": "2019-05-09T10:53:06.060000", "content": "@zachmayer @cdeotte Another question is how did you realize LB probing was the key to winning. It seems the LB probing would work only if pred=sigmoid(linear combination of variables + noise) and there's risk of wasting precious submissions on individual variable probing.
From the beginning (20 days ago), all my best models were hyperplanes. And the way my LB increased with each submission was mimicking my simulations using synthetic data made from (linear + noise) so I was pretty confident that was how the data was made.
For my first week, I was just randomly rotating my best hyperplane 5 degrees and resubmitting. (Code posted here in model three) I was averaging an increase of 0.001 LB each day. By the end of the week, I had 0.875 LB. Afterward, I had 10 days left. With limited time, I needed a more efficient probing scheme. This is when I devised the AUC probe method.
", "votes": 2, "replies": [] }, { "id": 529353, "author_name": "Zach", "author_url": "", "post_date": "2019-05-09T17:44:34.330000", "content": "For me this was an educated guess. The last competition was a linear model, and nothing I saw in the training data indicated anything other than linear relationships.
", "votes": 2, "replies": [] }, { "id": 529466, "author_name": "Eric R", "author_url": "", "post_date": "2019-05-10T00:45:10.593000", "content": "Thanks for the insights! I spent most of the time probing with non-linearity ideas, and never managed to improve LB because there're none.
", "votes": 1, "replies": [] } ] }, { "id": 529104, "author_name": "Himanshu Soni", "author_url": "", "post_date": "2019-05-09T07:33:07.977000", "content": "Congrats zach!! I am looking forward for your top 1st place solution.
", "votes": 1, "replies": [] }, { "id": 529053, "author_name": "Chris Deotte", "author_url": "", "post_date": "2019-05-09T04:48:52.473000", "content": "Congratulations Zach. That was a great idea using the probed coefficients to generate random data. And then using the random data to find the relationship between the probed AUC and equation coefficients. That gave your model a higher score than mine. Well done!! Thanks for sharing your solution, your code, and probed AUCs.
", "votes": 1, "replies": [ { "id": 529355, "author_name": "Zach", "author_url": "", "post_date": "2019-05-09T17:45:30.023000", "content": "Thank you!
\n\nI will also share my probe results on the private LB, since I have those scores now that the competition is over.
", "votes": 1, "replies": [] } ] }, { "id": 529395, "author_name": "feature-blind", "author_url": "", "post_date": "2019-05-09T19:30:19.213000", "content": "A true masterpiece! Makes me realise how much there's yet to learn. One can tell your victory is definitely not an accident :) \nCongratulations on winning the competition, and on how successful you've turned out to be in your chosen career path. Bravo!
", "votes": 2, "replies": [] }, { "id": 691184, "author_name": "Paul Dawidowicz", "author_url": "", "post_date": "2019-12-09T18:51:58.047000", "content": "Very clever. But it's not a viable methodology when you don't know to what data the model would be applied in the production stage of the Model. If we only could submit 5 attempts, how would this methodology perform? so the accuracy is a function of how many times we can query the model in the future, so it seems to be a form of overfitting.
", "votes": 0, "replies": [] }, { "id": 532936, "author_name": "Massoud Hosseinali", "author_url": "", "post_date": "2019-05-18T04:46:27.057000", "content": "Thanks for posting this inspirational discussion. I'm your 8-years-ago version now, starting my journey.
", "votes": 0, "replies": [] }, { "id": 530211, "author_name": "Aleksandr Larko", "author_url": "", "post_date": "2019-05-12T05:51:48.813000", "content": "Hi! Congratulations for the win! WOW!
", "votes": 0, "replies": [ { "id": 539866, "author_name": "Zach", "author_url": "", "post_date": "2019-05-30T16:17:28.617000", "content": "Thank you!
", "votes": 0, "replies": [] } ] }, { "id": 529928, "author_name": "Eu Jin Lok", "author_url": "", "post_date": "2019-05-11T07:23:51.700000", "content": "Congrats @zachmayer ! You've come a long way, can't believe its 8 years ago... time flies.
", "votes": 0, "replies": [ { "id": 532306, "author_name": "Zach", "author_url": "", "post_date": "2019-05-16T15:57:20.157000", "content": "I know! It's really crazy. It's been a fun ride!
", "votes": 1, "replies": [] } ] }, { "id": 529772, "author_name": "Kailash Negi", "author_url": "", "post_date": "2019-05-10T17:44:39.777000", "content": "Congratulations!!
", "votes": 0, "replies": [] }, { "id": 529612, "author_name": "Prabhjot Singh", "author_url": "", "post_date": "2019-05-10T10:29:20.220000", "content": "Congrats!
", "votes": 0, "replies": [] }, { "id": 529503, "author_name": "Héctor A Iurisci", "author_url": "", "post_date": "2019-05-10T04:00:48.737000", "content": "Congratulations! Thanks for sharing your solution.
", "votes": 0, "replies": [] }, { "id": 529459, "author_name": "Hüseyin Kaya", "author_url": "", "post_date": "2019-05-10T00:21:34.753000", "content": "Congratulations.
", "votes": 0, "replies": [] }, { "id": 529436, "author_name": "Adam Pacławski", "author_url": "", "post_date": "2019-05-09T22:18:07.530000", "content": "Congratulations and thank you for sharing the way you made to solve this problem.
", "votes": 0, "replies": [] }, { "id": 529029, "author_name": "WhiteLord", "author_url": "", "post_date": "2019-05-09T03:36:12.400000", "content": "Than you for sharing :D
", "votes": 0, "replies": [] }, { "id": 529015, "author_name": "Eric R", "author_url": "", "post_date": "2019-05-09T02:54:29.213000", "content": "Thanks for sharing. A lot has changed in the past eight years.
", "votes": 0, "replies": [] }, { "id": 529007, "author_name": "", "author_url": "", "post_date": "2019-05-09T02:42:35.560000", "content": "", "votes": 0, "replies": [] }, { "id": 623240, "author_name": "Saniya Nafees", "author_url": "", "post_date": "2019-09-10T16:16:57.523000", "content": "Extremely helpful, thank you!
", "votes": 0, "replies": [] }, { "id": 529277, "author_name": "Mikaros", "author_url": "", "post_date": "2019-05-09T15:12:35.770000", "content": "Thanks for sharing.
", "votes": 0, "replies": [] } ], "comment_count": 27, "messages": [ { "id": 528878, "postDate": "2019-05-08T20:13:15.693Z", "content": "First of all, I want to say that I feel incredibly honored to win. The original Don’t Overfit was my first Kaggle competition and coincides with when I discovered machine learning. I remember poring over forum posts to learn how to fit models with the caret package and writing R code on the commuter train on my way to job interviews in Boston, a city where I’d just arrived. On those train rides, I daydreamed about one day having a job where I did machine learning for work.
\n\nI can’t believe it’s been 8 years.
\n\nThe original Don’t Overfit competition was the start of a trajectory that took me to the career I have today. Thank you @salimali for piquing my interest, and thank you to the Kaggle community for all you’ve given to me.
\n\nThis competition, of course, was completely different from the original Don’t Overfit 😁
\n\nIn the original competition, we only had 5 submissions total for the private leaderboard and no leaderboard feedback.
\n\nI believe my critical insight was being among the first to realize leaderboard probing would be key to winning. I started a dedicated effort to submit every individual variable as a single submission. I was able to get the AUC for all 300 variables on the public leaderboard, and as @cdeotte already discussed, used (AUC public) - 0.50 to come up with “coefficients” based on AUC. If Chris had had time for 250 more submissions, he would have had a good chance of beating me. Just using the 300 public LB “coefficients”, with no further processing, gives a private LB score of 0.853, which would have been good enough for 110th place or so.
\n\nI additionally used the AUCs on the training data to add a little bit of extra information to this model, using the following formula: 0.11*(AUC train) + 0.89*(AUC pubic) - 0.5\nThe weight for AUC train is (rows train)/[rows train + rows public]\nThe weight for AUC public is (rows public)/[rows train + rows public]
This gives a private leaderboard score of 0.856, good enough for 60th place or so. (About where I finished in the first iteration of Don’t Overfit!).
\n\nIt also gives the following plot of public AUC vs training AUC, which led me to believe all 300 variables were used to create the target variable. This is also a useful plot to calibrate simulated datasets (more on this later):\n
This methodology overfits the public leaderboard, as there are many small AUCs that should probably be zero. Using Chris’ capping heuristic, we can set every coefficient with abs(cf) < 0.04 = 0, which yields the following equation:\n\n-0.0661*X16 + -0.0448*X29 + 0.1778*X33 + -0.0526*X45 + -0.0736*X63 + 0.171*X65 + -0.059*X70 + -0.1047*X73 + -0.118*X91 + -0.0422*X106 + -0.0446*X108 + -0.0985*X117 + -0.0448*X132 + 0.0462*X164 + -0.0514*X189 + 0.1117*X199 + -0.0704*X209 + -0.1198*X217 + -0.0456*X239\n
This is very similar to Chris’ solution, just with more variables. This gives a private leaderboard score of 0.877, which is good enough for first place.
\n\nI don’t like capping, and I wanted a solution that smoothly shrinks smaller AUCs to zero. For example, variable 239 in the above equation has a coefficient of -0.0456, and there’s a good chance that is really a zero (but I’m not certain)!
\n\nSo what I spent most of my time doing was looking for an equation that would smoothly shrink small coefficients to zero, while leaving large coefficients more or less alone. I dove down a bit of a rabbit hole here, but it was a fun rabbit hole, and I’m interested to see what approaches other people take to solving the same problem.
\n\nThe approach I took was:\n1. Simulate a dataset with 20,000 rows, 300 columns, sampled from the random normal distribution. AKA matrix(rnorm(20000*300), ncol=300) in R. I am 99.99% sure this is how Kaggle simulated the data for this competition.\n2. Apply my probed coefficients to the data. This isn’t perfect, but the coefficients I probed should be pretty representative of the real distribution of coefficients. AKA matrix(rnorm(20000*300), ncol=300) %*% probed_cf in R.\n3. Add a little random noise.\n4. Choose a classification cutoff that gives a target distribution in the training data of 160 0s and 90 1s.
I then fiddled with steps 2-4 until I got (AUC train - 250 rows) vs (AUC public - 1975 rows) plots that looked like the above plot.
\n\nI also simulated “leaderboard probing” to try to get simulated results that had training AUCs of .95 and public AUCs around .92, as those were my results on the real leaderboard.
\n\nOnce I had my simulated data, the problem becomes: predict the “real” coefficients, given the probed coefficients. I played around with many different methodologies for this, and finally ended up using Eureqa Desktop to discover the following equation:\n\npenalized_cf = 0.0320794465661013 + 1.38501844798023*cf - 9.51464300036321*cf^3 - \n 0.0654557703259389*plogis(85.9772069321308*cf)\n\n(Where plogis is the sigmoid function in R)\nHere is a plot of the penalized coefficients vs. the raw coefficients. Note that coefficients near zero get shrunk almost completely to zero, but not entirely:\n
So to recap, my final submission was:\n\ncf = 0.11*(auc_train) + 0.89*(auc_pubic) - 0.5\npenalized_cf = 0.032 + 1.385*cf - 9.515*cf^3 - 0.065*plogis(85.977*cf)\nsub = X_sub %*% penalized_cf\n
This achieves a private LB score of 0.886 and a public LB score of 0.923
\n\nI’ll post more detailed code (along with the probed AUCs) in some public kernels.
", "rawMarkdown": "First of all, I want to say that I feel incredibly honored to win. [The original Don’t Overfit](https://www.kaggle.com/c/overfitting/leaderboard) was my first Kaggle competition and coincides with when I discovered machine learning. I remember poring over forum posts to learn how to fit models with the caret package and writing R code on the commuter train on my way to job interviews in Boston, a city where I’d just arrived. On those train rides, I daydreamed about one day having a job where I did machine learning for work.\n\nI can’t believe it’s been 8 years.\n\nThe original Don’t Overfit competition was the start of a trajectory that took me to the career I have today. Thank you @salimali for piquing my interest, and thank you to the Kaggle community for all you’ve given to me.\n\n***\n\nThis competition, of course, was completely different from the original Don’t Overfit 😁 \n\nIn the original competition, we only had 5 submissions total for the private leaderboard and no leaderboard feedback. \n\n I believe my critical insight was being among the first to realize leaderboard probing would be key to winning. I started a dedicated effort to submit every individual variable as a single submission. I was able to get the [AUC for all 300 variables on the public leaderboard](https://www.kaggle.com/zachmayer/300-probed-aucs?scriptVersionId=13927278), and as @cdeotte already [discussed](https://www.kaggle.com/c/dont-overfit-ii/discussion/91683#latest-528739), used (AUC public) - 0.50 to come up with “coefficients” based on AUC. If Chris had had time for 250 more submissions, he would have had a good chance of beating me. Just using the 300 public LB “coefficients”, with no further processing, gives a private LB score of **0.853**, which would have been good enough for 110th place or so. \n\nI additionally used the AUCs on the training data to add a little bit of extra information to this model, using the following formula: `0.11*(AUC train) + 0.89*(AUC pubic) - 0.5`\nThe weight for AUC train is `(rows train)/[rows train + rows public]`\nThe weight for AUC public is `(rows public)/[rows train + rows public]`\n\nThis gives a private leaderboard score of **0.856**, good enough for 60th place or so. (About where I finished in the first iteration of Don’t Overfit!).\n\nIt also gives the following plot of public AUC vs training AUC, which led me to believe all 300 variables were used to create the target variable. This is also a useful plot to calibrate simulated datasets (more on this later):\n\n\nThis methodology overfits the public leaderboard, as there are many small AUCs that should probably be zero. Using Chris’ capping heuristic, we can set every coefficient with abs(cf) < 0.04 = 0, which yields the following equation:\n```\n-0.0661*X16 + -0.0448*X29 + 0.1778*X33 + -0.0526*X45 + -0.0736*X63 + 0.171*X65 + -0.059*X70 + -0.1047*X73 + -0.118*X91 + -0.0422*X106 + -0.0446*X108 + -0.0985*X117 + -0.0448*X132 + 0.0462*X164 + -0.0514*X189 + 0.1117*X199 + -0.0704*X209 + -0.1198*X217 + -0.0456*X239\n```\n\nThis is very similar to Chris’ solution, just with more variables. This gives a private leaderboard score of **0.877**, which is good enough for first place.\n\n***\n\nI don’t like capping, and I wanted a solution that smoothly shrinks smaller AUCs to zero. For example, variable 239 in the above equation has a coefficient of -0.0456, and there’s a good chance that is really a zero (but I’m not certain)!\n\nSo what I spent most of my time doing was looking for an equation that would smoothly shrink small coefficients to zero, while leaving large coefficients more or less alone. I dove down a bit of a rabbit hole here, but it was a fun rabbit hole, and I’m interested to see what approaches other people take to solving the same problem.\n\nThe approach I took was:\n1. Simulate a dataset with 20,000 rows, 300 columns, sampled from the random normal distribution. AKA `matrix(rnorm(20000*300), ncol=300)` in R. I am 99.99% sure this is how Kaggle simulated the data for this competition.\n2. Apply my probed coefficients to the data. This isn’t perfect, but the coefficients I probed should be pretty representative of the real distribution of coefficients. AKA `matrix(rnorm(20000*300), ncol=300) %*% probed_cf` in R.\n3. Add a little random noise.\n4. Choose a classification cutoff that gives a target distribution in the training data of 160 0s and 90 1s.\n\nI then fiddled with steps 2-4 until I got (AUC train - 250 rows) vs (AUC public - 1975 rows) plots that looked like the above plot.\n\nI also simulated “leaderboard probing” to try to get simulated results that had training AUCs of .95 and public AUCs around .92, as those were my results on the real leaderboard.\n\n***\n\nOnce I had my simulated data, the problem becomes: predict the “real” coefficients, given the probed coefficients. I played around with many different methodologies for this, and finally ended up using [Eureqa Desktop](https://www.nutonian.com/products/eureqa-desktop/) to discover the following equation:\n```\npenalized_cf = 0.0320794465661013 + 1.38501844798023*cf - 9.51464300036321*cf^3 - \n 0.0654557703259389*plogis(85.9772069321308*cf)\n```\n(Where plogis is the sigmoid function in R)\nHere is a plot of the penalized coefficients vs. the raw coefficients. Note that coefficients near zero get shrunk almost completely to zero, but not entirely:\n\n\n***\n\nSo to recap, my final submission was:\n```\ncf = 0.11*(auc_train) + 0.89*(auc_pubic) - 0.5\npenalized_cf = 0.032 + 1.385*cf - 9.515*cf^3 - 0.065*plogis(85.977*cf)\nsub = X_sub %*% penalized_cf\n```\n\nThis achieves a private LB score of **0.886** and a public LB score of 0.923\n\nI’ll post more detailed code (along with the probed AUCs) in some public kernels.\n", "votes": 65 }, { "id": 528883, "postDate": "2019-05-08T20:51:10.910Z", "content": "As a follow up, here are the 300 probed AUCs as a kernel and as a dataset:\nhttps://www.kaggle.com/zachmayer/300-probed-aucs?scriptVersionId=13927278\nhttps://www.kaggle.com/zachmayer/300-probed-aucs-from-dont-overfit-ii
", "rawMarkdown": "As a follow up, here are the 300 probed AUCs as a kernel and as a dataset:\nhttps://www.kaggle.com/zachmayer/300-probed-aucs?scriptVersionId=13927278\nhttps://www.kaggle.com/zachmayer/300-probed-aucs-from-dont-overfit-ii", "votes": 9 }, { "id": 528984, "postDate": "2019-05-09T01:39:05.227Z", "content": "And here's my code: https://www.kaggle.com/zachmayer/first-place-solution?scriptVersionId=13934694
", "rawMarkdown": "And here's my code: https://www.kaggle.com/zachmayer/first-place-solution?scriptVersionId=13934694", "votes": 5 }, { "id": 531407, "postDate": "2019-05-14T20:42:33.437Z", "content": "Congratulations for the win!
", "rawMarkdown": "Congratulations for the win!", "votes": 1 }, { "id": 529176, "postDate": "2019-05-09T10:53:06.060Z", "content": "@zachmayer @cdeotte Another question is how did you realize LB probing was the key to winning. It seems the LB probing would work only if pred=sigmoid(linear combination of variables + noise) and there's risk of wasting precious submissions on individual variable probing.
From the beginning (20 days ago), all my best models were hyperplanes. And the way my LB increased with each submission was mimicking my simulations using synthetic data made from (linear + noise) so I was pretty confident that was how the data was made.
For my first week, I was just randomly rotating my best hyperplane 5 degrees and resubmitting. (Code posted here in model three) I was averaging an increase of 0.001 LB each day. By the end of the week, I had 0.875 LB. Afterward, I had 10 days left. With limited time, I needed a more efficient probing scheme. This is when I devised the AUC probe method.
", "rawMarkdown": "From the beginning (20 days ago), all my best models were hyperplanes. And the way my LB increased with each submission was mimicking my simulations using synthetic data made from `(linear + noise)` so I was pretty confident that was how the data was made.\n \nFor my first week, I was just randomly rotating my best hyperplane 5 degrees and resubmitting. (Code posted [here][1] in model three) I was averaging an increase of 0.001 LB each day. By the end of the week, I had 0.875 LB. Afterward, I had 10 days left. With limited time, I needed a more efficient probing scheme. This is when I devised the AUC probe method. \n \n[1]: https://www.kaggle.com/cdeotte/lb-probing-strategies-0-890-2nd-place", "votes": 2 }, { "id": 529353, "postDate": "2019-05-09T17:44:34.330Z", "content": "For me this was an educated guess. The last competition was a linear model, and nothing I saw in the training data indicated anything other than linear relationships.
", "rawMarkdown": "For me this was an educated guess. The last competition was a linear model, and nothing I saw in the training data indicated anything other than linear relationships.", "votes": 2 }, { "id": 529466, "postDate": "2019-05-10T00:45:10.593Z", "content": "Thanks for the insights! I spent most of the time probing with non-linearity ideas, and never managed to improve LB because there're none.
", "rawMarkdown": "Thanks for the insights! I spent most of the time probing with non-linearity ideas, and never managed to improve LB because there're none.", "votes": 1 } ] }, { "id": 529104, "postDate": "2019-05-09T07:33:07.977Z", "content": "Congrats zach!! I am looking forward for your top 1st place solution.
", "rawMarkdown": "Congrats zach!! I am looking forward for your top 1st place solution. ", "votes": 1 }, { "id": 529053, "postDate": "2019-05-09T04:48:52.473Z", "content": "Congratulations Zach. That was a great idea using the probed coefficients to generate random data. And then using the random data to find the relationship between the probed AUC and equation coefficients. That gave your model a higher score than mine. Well done!! Thanks for sharing your solution, your code, and probed AUCs.
", "rawMarkdown": "Congratulations Zach. That was a great idea using the probed coefficients to generate random data. And then using the random data to find the relationship between the probed AUC and equation coefficients. That gave your model a higher score than mine. Well done!! Thanks for sharing your solution, your code, and probed AUCs.", "votes": 1, "replies": [ { "id": 529355, "postDate": "2019-05-09T17:45:30.023Z", "content": "Thank you!
\n\nI will also share my probe results on the private LB, since I have those scores now that the competition is over.
", "rawMarkdown": "Thank you!\n\nI will also share my probe results on the private LB, since I have those scores now that the competition is over.", "votes": 1 } ] }, { "id": 529395, "postDate": "2019-05-09T19:30:19.213Z", "content": "A true masterpiece! Makes me realise how much there's yet to learn. One can tell your victory is definitely not an accident :) \nCongratulations on winning the competition, and on how successful you've turned out to be in your chosen career path. Bravo!
", "rawMarkdown": "A true masterpiece! Makes me realise how much there's yet to learn. One can tell your victory is definitely not an accident :) \nCongratulations on winning the competition, and on how successful you've turned out to be in your chosen career path. Bravo!", "votes": 2 }, { "id": 691184, "postDate": "2019-12-09T18:51:58.047Z", "content": "Very clever. But it's not a viable methodology when you don't know to what data the model would be applied in the production stage of the Model. If we only could submit 5 attempts, how would this methodology perform? so the accuracy is a function of how many times we can query the model in the future, so it seems to be a form of overfitting.
", "rawMarkdown": "Very clever. But it's not a viable methodology when you don't know to what data the model would be applied in the production stage of the Model. If we only could submit 5 attempts, how would this methodology perform? so the accuracy is a function of how many times we can query the model in the future, so it seems to be a form of overfitting." }, { "id": 532936, "postDate": "2019-05-18T04:46:27.057Z", "content": "Thanks for posting this inspirational discussion. I'm your 8-years-ago version now, starting my journey.
", "rawMarkdown": "Thanks for posting this inspirational discussion. I'm your 8-years-ago version now, starting my journey." }, { "id": 530211, "postDate": "2019-05-12T05:51:48.813Z", "content": "Hi! Congratulations for the win! WOW!
", "rawMarkdown": "Hi! Congratulations for the win! WOW!", "replies": [ { "id": 539866, "postDate": "2019-05-30T16:17:28.617Z", "content": "Thank you!
", "rawMarkdown": "Thank you!" } ] }, { "id": 529928, "postDate": "2019-05-11T07:23:51.700Z", "content": "Congrats @zachmayer ! You've come a long way, can't believe its 8 years ago... time flies.
", "rawMarkdown": "Congrats @zachmayer ! You've come a long way, can't believe its 8 years ago... time flies. ", "replies": [ { "id": 532306, "postDate": "2019-05-16T15:57:20.157Z", "content": "I know! It's really crazy. It's been a fun ride!
", "rawMarkdown": "I know! It's really crazy. It's been a fun ride!", "votes": 1 } ] }, { "id": 529772, "postDate": "2019-05-10T17:44:39.777Z", "content": "Congratulations!!
", "rawMarkdown": "Congratulations!!" }, { "id": 529612, "postDate": "2019-05-10T10:29:20.220Z", "content": "Congrats!
", "rawMarkdown": "Congrats!" }, { "id": 529503, "postDate": "2019-05-10T04:00:48.737Z", "content": "Congratulations! Thanks for sharing your solution.
", "rawMarkdown": "Congratulations! Thanks for sharing your solution." }, { "id": 529459, "postDate": "2019-05-10T00:21:34.753Z", "content": "Congratulations.
", "rawMarkdown": "Congratulations." }, { "id": 529436, "postDate": "2019-05-09T22:18:07.530Z", "content": "Congratulations and thank you for sharing the way you made to solve this problem.
", "rawMarkdown": "Congratulations and thank you for sharing the way you made to solve this problem." }, { "id": 529029, "postDate": "2019-05-09T03:36:12.400Z", "content": "Than you for sharing :D
", "rawMarkdown": "Than you for sharing :D " }, { "id": 529015, "postDate": "2019-05-09T02:54:29.213Z", "content": "Thanks for sharing. A lot has changed in the past eight years.
", "rawMarkdown": "Thanks for sharing. A lot has changed in the past eight years." }, { "id": 529007, "postDate": "2019-05-09T02:42:35.560Z", "rawMarkdown": "", "isDeleted": true }, { "id": 623240, "postDate": "2019-09-10T16:16:57.523Z", "content": "Extremely helpful, thank you!
", "rawMarkdown": "Extremely helpful, thank you!" }, { "id": 529277, "postDate": "2019-05-09T15:12:35.770Z", "content": "Thanks for sharing.
", "rawMarkdown": "Thanks for sharing." } ], "complete": true, "source": "live", "fetched_at": "2026-10-19T19:13:00Z", "competition_id": 12896, "competition_slug": "dont-overfit-ii", "team_id": 2738366, "solution_url": "/competitions/dont-overfit-ii/writeups/zach-first-place-solution" }