Hello,
\n\nFirst I would like to thank Kaggle for this interesting challenge, where a regression technic was able to win.\nI would like to thank all players. I would like to thank especially Robert Stockton : his post \"To reach the top ranks...\" helped me a lot, and convinced me to look for simple things and try as simple technics as possible.
\n\nOne year ago, after reading Introduction to Machine Learning with Python (Sarah Guido, Andreas Mûller O'Reilly), I tried my first real challenge (Microsoft Malware Prediction).\nI ended 10th with a blend of two LGBM models. There was a big shake up (I jumped from 170th on the public LB to 10th on private). Was I lucky ? \nBut during the Microsoft Malware Prediction challenge, I learned a very important thing : to trust my CV, and only trust my CV.
\n\nI do not have time to try a lot of challenges and to test many things. At the beginning of this Categorical Challenge, several players wrote Logistic should be a good solution. Great, a challenge for me !\nCramer's V between each feature were very low, so features were independant and Trees technics wouldn't work very well. And https://www.kaggle.com/tunguz/adversicat’>adversarial validation would be useless.
\n\nLike many of you, I began by trying one hot encoders, then target encoders, thermometer encoder, but nothing other than one hot encoders was significant.
\n\nRegression technics love parsimony. After trying the linear relation between target and each ordinal feature like suggested by \"To reach the top ranks...\", and jump ahead with the first 0.8085 score (I was lucky and proud), I tried many things to have a more parcimonous model : I tested linear relations with other nominal features, I tried to make clusters of values of coefficients of a Logistic, I tried to make bins of values, etc. \nI’ve seen there was a symetry in target mean of day values.
\n\nThe difference between to top ranks on AUC was at the 5th decimal, so I needed a way to be sure to make good decisions without using the public LB (I don’t trust public LB – to trust Public LB is just like to trust only one test fold). \nAnd also, I stuck to target leakage ; AUC on my CV grew, but I realised two weeks ago that my public score did not.\nSo I had to find a way to make good decisions and to avoid target leakage too: I read the easiest way to avoid target leakage is cross validation. \nSo I've repeated some cross validations on other folds than my first CV, and repeated again : before to make a decision, if I had the same conclusion with several combinations of 5 folds, the conclusion is good. Else it is not.
\n\nMy two private submissions scored only 0.80839 (yesterday evening at 11 PM UTC) and 0.80845 on the public leaderboard : they both score 0.80282 on private leaderboard.\nYesterday at 11PM UTC, I was not very optimstic. But I saw that many of the best players submited a lot of solutions, so I had a chance, maybe they trusted too much the Public LB instead of their own CV.
\n\nMy best solution on public LB (0.80852) score only 0.80279 on private.
\n\nThe ordinal feature trick to reach the top of the public LB was hidden in one of my post two weeks ago. Sorry ! I believe we all read new notebooks, but we are only a few to read discussions. Here is my solution.
\n\nI had other solutions which score 0.80847 or 0.80845 on public leaderboard : I did not choose them, because I did not do my double ou tripler cross validation on them, or because when I did it, I had not the same conclusion as on my first CV. I didn't trust them.
\n\nI had a lot of fun, I learned a lot during this challenge.
", "comments": [ { "id": 694358, "author_name": "Miyabon", "author_url": "", "post_date": "2019-12-13T13:17:39.973000", "content": "Congrats!🎉 ✨ Nice solution!! Thanks for sharing!!😄 👍
", "votes": 1, "replies": [] }, { "id": 692696, "author_name": "Eric Vos", "author_url": "", "post_date": "2019-12-11T16:23:16.160000", "content": "Nice job !\nThanks for your explanations.
", "votes": 1, "replies": [] }, { "id": 692269, "author_name": "Robert Stockton", "author_url": "", "post_date": "2019-12-11T04:59:27.747000", "content": "Very nice explanation and I agree with all of your conclusions. This really was a perfect competition for reinforcing good CV habits, since we had a lot of reason to believe that everything was indeed randomly distributed with no hidden differences between the train and test sets.
\n\nI'll look forward to reading your solution in detail.
", "votes": 1, "replies": [ { "id": 692293, "author_name": "Advait", "author_url": "", "post_date": "2019-12-11T05:31:12.627000", "content": "> This really was a perfect competition for reinforcing good CV habits
\n\nIndeed! Lesson learnt. This was a really good 'first competition' for me as a new kaggler. My first submission scores a 0.80820 on private set and it has no feature selection or anything. Just ordinal encoding for \"ord\" columns and onehot everything else, LogisticRegression.
", "votes": 1, "replies": [] }, { "id": 692297, "author_name": "aldparis", "author_url": "", "post_date": "2019-12-11T05:42:56.073000", "content": "You were really impressive on the public LB. Welcome to Kaggle !
", "votes": 0, "replies": [] }, { "id": 692306, "author_name": "aldparis", "author_url": "", "post_date": "2019-12-11T05:58:08.923000", "content": "Thank you for you comments ! \nThere was no hidden difference between train and test. But I think there were hidden differences between the first 25% part of the test set, and the 75% last part, as we can see in our AUC on public (0.808) and private (0.803). \nThe better is the generalization of our model, the better will be our score on private test set !
", "votes": 0, "replies": [] }, { "id": 692329, "author_name": "Advait", "author_url": "", "post_date": "2019-12-11T06:35:57.380000", "content": "Thank you! and congratulations!
", "votes": 1, "replies": [] } ] }, { "id": 692228, "author_name": "kailai", "author_url": "", "post_date": "2019-12-11T03:41:59.873000", "content": "How did you choose features? You chose 20 features,right? And chose 15000 features after one hot
", "votes": 1, "replies": [ { "id": 692751, "author_name": "aldparis", "author_url": "", "post_date": "2019-12-11T17:33:28.940000", "content": "I kept all features except bin_0. Bin_0 had a very low Cramer's V with target. 87% of train observations had the same bin_0 value. Target means for the values of bin_0 were very close. Models without bin_0 had better validation AUC than models with.
\n\nOne Hot Encoders create one feature for each value of every original feature you give to the one hot encoder. There was almost 15 000 features after One Hot Encoders.
\n\nI used One Hot Encoders for every feature, except for ordinal features (the big trick - from ord_0 to ord_5) : instead to evaluate a coefficient for each value of each ordinal feature like in one hot encoder, my logistic regression evaluate a unique coefficient for each ordinal, this coefficient is the gradient between the ordinal feature and the logit of target. Regression technics love parsimony : the less are the coefficients to evaluate, the better could be the generalization of our logisitic model.
\n\nBut in this challenge, I don't think anyone find a way to evaluate less than 11 900 coefficients for the nom_9 feature. Only Kaggle new the function (which could have 1 a 2 or more parameters) used to generate nom_9 ; if we were able to find that function and the values of the parameters of that function, our logistic should generalize better. I tried a linear function (only a gradient coefficient instead of 11981 coefficients) with a nom_9 mapped on target means sorted, but it did not work.
\n\nSorry for my bad english !
", "votes": 1, "replies": [] }, { "id": 692823, "author_name": "kailai", "author_url": "", "post_date": "2019-12-11T19:10:18.783000", "content": "Thank you for your friendly response!
", "votes": 1, "replies": [] } ] }, { "id": 692120, "author_name": "Gere", "author_url": "", "post_date": "2019-12-10T23:07:43.980000", "content": "Nice description! 👋 Actually the MinMax scaling was always the OH my Cat notebook, but it was mainly to speed up convergence.\nThere is a lot going on in the solution. What are the essentially needed tricks as compared to plain One-hot and ordinal encoding of appropriate variables?
", "votes": 1, "replies": [ { "id": 692765, "author_name": "aldparis", "author_url": "", "post_date": "2019-12-11T17:53:17.197000", "content": "Sorry, I'm not sure to understand your question. But here is my answer : (a famous dead french politician always answered like that to journalists 😃)\nThe main trick is in the cell 22 of the notebook : ordinal features are only sorted by their target means and given to a StandardScaler before logistic. StandardScaler was better than MixMaxScaler or RobustScaler in this challenge : my 0.80852 on public LB was with MinMaxScaler.
", "votes": 0, "replies": [] }, { "id": 694383, "author_name": "Gere", "author_url": "", "post_date": "2019-12-13T13:51:25.930000", "content": "Oh, would you say that is the main trick? Weird, because I had that (https://www.kaggle.com/superant/oh-my-cat?scriptVersionId=21729216 cell 7), but somehow I did not keep that. Probably my CV wasn't great and it seemed it is not valuable. So, apart from this trick, and OneHot there is no other special treatment going on for your solution?
", "votes": 0, "replies": [] }, { "id": 694499, "author_name": "aldparis", "author_url": "", "post_date": "2019-12-13T17:52:10.703000", "content": "Sorry, I didn't see your cell 7. I believe the mistake is that you split ord_5. With minmaxscaler or standardscaler, I saw by CV that it was not a good choice. It was a good choice with thermometer encoder or one hot encoder, but not with minmaxscaler or standardscaler. \nAnd for day and month, one hot encoder was better than minmaxscaler.\nAnd there is no other special treatment.
\n\nThe main trick is especially on ord_5. Take a look to my cell 20. \nCV jumped from 0.8035762 to 0.8040304 only with ord_5. On the public LB, I jumped from 0.80827 to 0.80852 just by adding ord_5 to ordinal features with a minmaxscaler (at the beginning I used minmaxscaler and later, I user standardscaler).
\n\nGood luck in 2019 Data Science Bowl ! You are impressive. But pay attention to your CV !
", "votes": 1, "replies": [] } ] } ], "comment_count": 14, "messages": [ { "id": 692100, "postDate": "2019-12-10T22:04:23.817Z", "content": "Hello,
\n\nFirst I would like to thank Kaggle for this interesting challenge, where a regression technic was able to win.\nI would like to thank all players. I would like to thank especially Robert Stockton : his post \"To reach the top ranks...\" helped me a lot, and convinced me to look for simple things and try as simple technics as possible.
\n\nOne year ago, after reading Introduction to Machine Learning with Python (Sarah Guido, Andreas Mûller O'Reilly), I tried my first real challenge (Microsoft Malware Prediction).\nI ended 10th with a blend of two LGBM models. There was a big shake up (I jumped from 170th on the public LB to 10th on private). Was I lucky ? \nBut during the Microsoft Malware Prediction challenge, I learned a very important thing : to trust my CV, and only trust my CV.
\n\nI do not have time to try a lot of challenges and to test many things. At the beginning of this Categorical Challenge, several players wrote Logistic should be a good solution. Great, a challenge for me !\nCramer's V between each feature were very low, so features were independant and Trees technics wouldn't work very well. And https://www.kaggle.com/tunguz/adversicat’>adversarial validation would be useless.
\n\nLike many of you, I began by trying one hot encoders, then target encoders, thermometer encoder, but nothing other than one hot encoders was significant.
\n\nRegression technics love parsimony. After trying the linear relation between target and each ordinal feature like suggested by \"To reach the top ranks...\", and jump ahead with the first 0.8085 score (I was lucky and proud), I tried many things to have a more parcimonous model : I tested linear relations with other nominal features, I tried to make clusters of values of coefficients of a Logistic, I tried to make bins of values, etc. \nI’ve seen there was a symetry in target mean of day values.
\n\nThe difference between to top ranks on AUC was at the 5th decimal, so I needed a way to be sure to make good decisions without using the public LB (I don’t trust public LB – to trust Public LB is just like to trust only one test fold). \nAnd also, I stuck to target leakage ; AUC on my CV grew, but I realised two weeks ago that my public score did not.\nSo I had to find a way to make good decisions and to avoid target leakage too: I read the easiest way to avoid target leakage is cross validation. \nSo I've repeated some cross validations on other folds than my first CV, and repeated again : before to make a decision, if I had the same conclusion with several combinations of 5 folds, the conclusion is good. Else it is not.
\n\nMy two private submissions scored only 0.80839 (yesterday evening at 11 PM UTC) and 0.80845 on the public leaderboard : they both score 0.80282 on private leaderboard.\nYesterday at 11PM UTC, I was not very optimstic. But I saw that many of the best players submited a lot of solutions, so I had a chance, maybe they trusted too much the Public LB instead of their own CV.
\n\nMy best solution on public LB (0.80852) score only 0.80279 on private.
\n\nThe ordinal feature trick to reach the top of the public LB was hidden in one of my post two weeks ago. Sorry ! I believe we all read new notebooks, but we are only a few to read discussions. Here is my solution.
\n\nI had other solutions which score 0.80847 or 0.80845 on public leaderboard : I did not choose them, because I did not do my double ou tripler cross validation on them, or because when I did it, I had not the same conclusion as on my first CV. I didn't trust them.
\n\nI had a lot of fun, I learned a lot during this challenge.
", "rawMarkdown": "Hello,\n\nFirst I would like to thank Kaggle for this interesting challenge, where a regression technic was able to win.\nI would like to thank all players. I would like to thank especially Robert Stockton : his post \"To reach the top ranks...\" helped me a lot, and convinced me to look for simple things and try as simple technics as possible.\n\nCongrats!🎉 ✨ Nice solution!! Thanks for sharing!!😄 👍
", "rawMarkdown": "Congrats!🎉 ✨ Nice solution!! Thanks for sharing!!😄 👍 ", "votes": 1 }, { "id": 692696, "postDate": "2019-12-11T16:23:16.160Z", "content": "Nice job !\nThanks for your explanations.
", "rawMarkdown": "Nice job !\nThanks for your explanations.", "votes": 1 }, { "id": 692269, "postDate": "2019-12-11T04:59:27.747Z", "content": "Very nice explanation and I agree with all of your conclusions. This really was a perfect competition for reinforcing good CV habits, since we had a lot of reason to believe that everything was indeed randomly distributed with no hidden differences between the train and test sets.
\n\nI'll look forward to reading your solution in detail.
", "rawMarkdown": "Very nice explanation and I agree with all of your conclusions. This really was a perfect competition for reinforcing good CV habits, since we had a lot of reason to believe that everything was indeed randomly distributed with no hidden differences between the train and test sets.\n\nI'll look forward to reading your solution in detail.\n", "votes": 1, "replies": [ { "id": 692293, "postDate": "2019-12-11T05:31:12.627Z", "content": "> This really was a perfect competition for reinforcing good CV habits
\n\nIndeed! Lesson learnt. This was a really good 'first competition' for me as a new kaggler. My first submission scores a 0.80820 on private set and it has no feature selection or anything. Just ordinal encoding for \"ord\" columns and onehot everything else, LogisticRegression.
", "rawMarkdown": "> This really was a perfect competition for reinforcing good CV habits\n\nIndeed! Lesson learnt. This was a really good 'first competition' for me as a new kaggler. My first submission scores a 0.80820 on private set and it has no feature selection or anything. Just ordinal encoding for \"ord\" columns and onehot everything else, LogisticRegression.", "votes": 1 }, { "id": 692297, "postDate": "2019-12-11T05:42:56.073Z", "content": "You were really impressive on the public LB. Welcome to Kaggle !
", "rawMarkdown": "You were really impressive on the public LB. Welcome to Kaggle !" }, { "id": 692306, "postDate": "2019-12-11T05:58:08.923Z", "content": "Thank you for you comments ! \nThere was no hidden difference between train and test. But I think there were hidden differences between the first 25% part of the test set, and the 75% last part, as we can see in our AUC on public (0.808) and private (0.803). \nThe better is the generalization of our model, the better will be our score on private test set !
", "rawMarkdown": "Thank you for you comments ! \nThere was no hidden difference between train and test. But I think there were hidden differences between the first 25% part of the test set, and the 75% last part, as we can see in our AUC on public (0.808) and private (0.803). \nThe better is the generalization of our model, the better will be our score on private test set !" }, { "id": 692329, "postDate": "2019-12-11T06:35:57.380Z", "content": "Thank you! and congratulations!
", "rawMarkdown": "Thank you! and congratulations! ", "votes": 1 } ] }, { "id": 692228, "postDate": "2019-12-11T03:41:59.873Z", "content": "How did you choose features? You chose 20 features,right? And chose 15000 features after one hot
", "rawMarkdown": "How did you choose features? You chose 20 features,right? And chose 15000 features after one hot ", "votes": 1, "replies": [ { "id": 692751, "postDate": "2019-12-11T17:33:28.940Z", "content": "I kept all features except bin_0. Bin_0 had a very low Cramer's V with target. 87% of train observations had the same bin_0 value. Target means for the values of bin_0 were very close. Models without bin_0 had better validation AUC than models with.
\n\nOne Hot Encoders create one feature for each value of every original feature you give to the one hot encoder. There was almost 15 000 features after One Hot Encoders.
\n\nI used One Hot Encoders for every feature, except for ordinal features (the big trick - from ord_0 to ord_5) : instead to evaluate a coefficient for each value of each ordinal feature like in one hot encoder, my logistic regression evaluate a unique coefficient for each ordinal, this coefficient is the gradient between the ordinal feature and the logit of target. Regression technics love parsimony : the less are the coefficients to evaluate, the better could be the generalization of our logisitic model.
\n\nBut in this challenge, I don't think anyone find a way to evaluate less than 11 900 coefficients for the nom_9 feature. Only Kaggle new the function (which could have 1 a 2 or more parameters) used to generate nom_9 ; if we were able to find that function and the values of the parameters of that function, our logistic should generalize better. I tried a linear function (only a gradient coefficient instead of 11981 coefficients) with a nom_9 mapped on target means sorted, but it did not work.
\n\nSorry for my bad english !
", "rawMarkdown": "I kept all features except bin_0. Bin_0 had a very low Cramer's V with target. 87% of train observations had the same bin_0 value. Target means for the values of bin_0 were very close. Models without bin_0 had better validation AUC than models with.\n\nOne Hot Encoders create one feature for each value of every original feature you give to the one hot encoder. There was almost 15 000 features after One Hot Encoders. \n\nI used One Hot Encoders for every feature, except for ordinal features (the big trick - from ord_0 to ord_5) : instead to evaluate a coefficient for each value of each ordinal feature like in one hot encoder, my logistic regression evaluate a unique coefficient for each ordinal, this coefficient is the gradient between the ordinal feature and the logit of target. Regression technics love parsimony : the less are the coefficients to evaluate, the better could be the generalization of our logisitic model. \n\nBut in this challenge, I don't think anyone find a way to evaluate less than 11 900 coefficients for the nom_9 feature. Only Kaggle new the function (which could have 1 a 2 or more parameters) used to generate nom_9 ; if we were able to find that function and the values of the parameters of that function, our logistic should generalize better. I tried a linear function (only a gradient coefficient instead of 11981 coefficients) with a nom_9 mapped on target means sorted, but it did not work.\n\nSorry for my bad english !", "votes": 1 }, { "id": 692823, "postDate": "2019-12-11T19:10:18.783Z", "content": "Thank you for your friendly response!
", "rawMarkdown": "Thank you for your friendly response!", "votes": 1 } ] }, { "id": 692120, "postDate": "2019-12-10T23:07:43.980Z", "content": "Nice description! 👋 Actually the MinMax scaling was always the OH my Cat notebook, but it was mainly to speed up convergence.\nThere is a lot going on in the solution. What are the essentially needed tricks as compared to plain One-hot and ordinal encoding of appropriate variables?
", "rawMarkdown": "Nice description! 👋 Actually the MinMax scaling was always the OH my Cat notebook, but it was mainly to speed up convergence.\nThere is a lot going on in the solution. What are the essentially needed tricks as compared to plain One-hot and ordinal encoding of appropriate variables?", "votes": 1, "replies": [ { "id": 692765, "postDate": "2019-12-11T17:53:17.197Z", "content": "Sorry, I'm not sure to understand your question. But here is my answer : (a famous dead french politician always answered like that to journalists 😃)\nThe main trick is in the cell 22 of the notebook : ordinal features are only sorted by their target means and given to a StandardScaler before logistic. StandardScaler was better than MixMaxScaler or RobustScaler in this challenge : my 0.80852 on public LB was with MinMaxScaler.
", "rawMarkdown": "Sorry, I'm not sure to understand your question. But here is my answer : (a famous dead french politician always answered like that to journalists 😃)\nThe main trick is in the cell 22 of the notebook : ordinal features are only sorted by their target means and given to a StandardScaler before logistic. StandardScaler was better than MixMaxScaler or RobustScaler in this challenge : my 0.80852 on public LB was with MinMaxScaler." }, { "id": 694383, "postDate": "2019-12-13T13:51:25.930Z", "content": "Oh, would you say that is the main trick? Weird, because I had that (https://www.kaggle.com/superant/oh-my-cat?scriptVersionId=21729216 cell 7), but somehow I did not keep that. Probably my CV wasn't great and it seemed it is not valuable. So, apart from this trick, and OneHot there is no other special treatment going on for your solution?
", "rawMarkdown": "Oh, would you say that is the main trick? Weird, because I had that (https://www.kaggle.com/superant/oh-my-cat?scriptVersionId=21729216 cell 7), but somehow I did not keep that. Probably my CV wasn't great and it seemed it is not valuable. So, apart from this trick, and OneHot there is no other special treatment going on for your solution?" }, { "id": 694499, "postDate": "2019-12-13T17:52:10.703Z", "content": "Sorry, I didn't see your cell 7. I believe the mistake is that you split ord_5. With minmaxscaler or standardscaler, I saw by CV that it was not a good choice. It was a good choice with thermometer encoder or one hot encoder, but not with minmaxscaler or standardscaler. \nAnd for day and month, one hot encoder was better than minmaxscaler.\nAnd there is no other special treatment.
\n\nThe main trick is especially on ord_5. Take a look to my cell 20. \nCV jumped from 0.8035762 to 0.8040304 only with ord_5. On the public LB, I jumped from 0.80827 to 0.80852 just by adding ord_5 to ordinal features with a minmaxscaler (at the beginning I used minmaxscaler and later, I user standardscaler).
\n\nGood luck in 2019 Data Science Bowl ! You are impressive. But pay attention to your CV !
", "rawMarkdown": "Sorry, I didn't see your cell 7. I believe the mistake is that you split ord_5. With minmaxscaler or standardscaler, I saw by CV that it was not a good choice. It was a good choice with thermometer encoder or one hot encoder, but not with minmaxscaler or standardscaler. \nAnd for day and month, one hot encoder was better than minmaxscaler.\nAnd there is no other special treatment.\n\nThe main trick is especially on ord_5. Take a look to my cell 20. \nCV jumped from 0.8035762 to 0.8040304 only with ord_5. On the public LB, I jumped from 0.80827 to 0.80852 just by adding ord_5 to ordinal features with a minmaxscaler (at the beginning I used minmaxscaler and later, I user standardscaler).\n\nGood luck in 2019 Data Science Bowl ! You are impressive. But pay attention to your CV !", "votes": 1 } ] } ], "complete": true, "source": "live", "fetched_at": "2026-10-19T17:33:00Z", "competition_id": 14999, "competition_slug": "cat-in-the-dat", "team_id": 3738438, "solution_url": "/competitions/cat-in-the-dat/writeups/aldparis-2nd-place-solution" }