I only came in 7th, so my approach probably doesn't matter much, but here are the keys to my strategy:\n* Ordinal encoding for the \"ord\" columns, normalized to 0-1 in order to make the coefficients prettier.\n* One-hot encoding for everything else, using sparse (csr) representation so that it fits in memory.\n* LogisticRegression with the 'saga' solver, and a very carefully tuned C value of 0.1172. I used 50 iterations, but the number really didn't matter.\n* Trusting my CV. I really worked it hard in order to get the optimal C value, and the leaderboard improvements always matched the public LB. (It was only \"pretty good\" on the private LB, though. If I'd stuck with an older C value of \"0.133\" I would have been in the money. That's how the shake-up goes.)
\n\nThat's it. Lots of encodings were tried, but nothing beat one-hot. You just had to be willing to accept the thousands of columns that resulted.
", "comments": [ { "id": 691626, "author_name": "kanopri", "author_url": "", "post_date": "2019-12-10T10:37:52.220000", "content": "The biggest difference is that I used the 'liblinear' solver instead.
", "votes": 4, "replies": [] }, { "id": 691413, "author_name": "aldparis", "author_url": "", "post_date": "2019-12-10T05:37:00.870000", "content": "Hi Robert,\nCongratulations and thank you a lot !\nYour approach was the good one.\nYou and I have two differences. First, I never tuned C, I kept until the end the magic 0.123456789 C.\nSecond, like you, I trusted only my CV : but I saw that when I changed the seed to create my 5 folds, I have different results and decisions were not the same : so I did double cross validation, even tripler cross validation. I will explain later.\nThank you for your generous post. I helped to try very simple things like the first trick !\nAlexandre
", "votes": 4, "replies": [ { "id": 692035, "author_name": "Gere", "author_url": "", "post_date": "2019-12-10T19:42:30.987000", "content": "Glad the magic C worked :-D That's sexier than tuning many digits of hyperparameters.
", "votes": 1, "replies": [] }, { "id": 692055, "author_name": "aldparis", "author_url": "", "post_date": "2019-12-10T20:01:20.093000", "content": "Yes ! Thank you for that magic C !
", "votes": 0, "replies": [] }, { "id": 692214, "author_name": "kanopri", "author_url": "", "post_date": "2019-12-11T03:13:31.810000", "content": "Magic C may be the best C value for the perfect model still unknown but does not suit my model.
\n\nAfter the competition ends, I saw this comment: \"It's also for people who want to try out various hyper-parameter optimization algorithms. :-)\", so tuning hyper-parameter seems as important as feature engineering!
", "votes": 0, "replies": [] } ] }, { "id": 691748, "author_name": "KDJ2020", "author_url": "", "post_date": "2019-12-10T13:10:18.643000", "content": "Hi Robert,\nThank you for the interesting competition. You helped me a lot. I only came in 6th but I've just published a 0.80284 private LB solution notebook.
", "votes": 1, "replies": [] }, { "id": 691415, "author_name": "Advait", "author_url": "", "post_date": "2019-12-10T05:44:24.193000", "content": "Exactly the same but with a C value of 0.12. I also removed infrequent values from some \"ord\" and \"nom\" columns which put me to the top of the public LB but didn't work out in the end. I tried many many encodings as well but nothing beat onehot. And thank you for your first post about keeping things simple. Tried LogisticRegression and did not bother using anything else :)
", "votes": 2, "replies": [] }, { "id": 691712, "author_name": "King Prawns", "author_url": "", "post_date": "2019-12-10T12:27:35.413000", "content": "How did you do your CV if you may share?
", "votes": 0, "replies": [ { "id": 692282, "author_name": "Robert Stockton", "author_url": "", "post_date": "2019-12-11T05:16:14.937000", "content": "tl;dr: I used the sklearn \"RepeatedKFold\" CV generator with 3 repetitions of 4 splits.
\n\nOne of the unique properties of this computation is that we had very \"clean\" synthetic data. Even though noise was introduced to keep it challenging, we could have confidence that (1) the order of the training instances didn't matter; and (2) the characteristics of the test set were identical to those of the train set; and (3) the public and private parts of the leaderboard also had identical characteristics. (We didn't have actual proof of these things, but that was the simplest and most likely assumption.)
\n\nThus, there was no need to get fancy with cross-validation. A simple 5-way split made a great starting point, and it didn't even matter whether shuffling was enabled. This is what I used for most of the competition. However, as others have noted, the choice of particular random seed -- and \"don't shuffle\" could be considered just another random seed in this case -- did make a difference in the single major hyper-parameter that we wish to tune. Thus, when I started going to extremes in my tuning, I turned to the \"RepeatedKFold\" class, which automatically creates multiple N-way splits and combines them together. I used a value of 4 splits of 3 folds because it gave more reliable results than KFold(5) without requiring too much additional runtime. I could have gone deeper, but this was sufficient for me to narrow in on a precise enough value to get to the point of diminishing returns -- after all, too many attempts would likely just lead to overfitting on the public LB rather than overfitting to a single CV.
", "votes": 0, "replies": [] } ] }, { "id": 691610, "author_name": "KRKirov", "author_url": "", "post_date": "2019-12-10T10:20:54.307000", "content": "Hey, how much did the ordinal encoding help, compared to applying OHE on all columns?
", "votes": 0, "replies": [ { "id": 691861, "author_name": "aldparis", "author_url": "", "post_date": "2019-12-10T15:21:44.917000", "content": "There was a linear relation between target and each ordinal feature, I will show it later in a notebook. And regression methods love parsimony.
", "votes": 0, "replies": [] }, { "id": 692082, "author_name": "KRKirov", "author_url": "", "post_date": "2019-12-10T20:59:13.797000", "content": "Great, I look forward to seeing your solution.
", "votes": 0, "replies": [] } ] }, { "id": 691627, "author_name": "", "author_url": "", "post_date": "2019-12-10T10:38:10.533000", "content": "", "votes": 0, "replies": [] } ], "comment_count": 13, "messages": [ { "id": 691390, "postDate": "2019-12-10T04:54:40.670Z", "content": "I only came in 7th, so my approach probably doesn't matter much, but here are the keys to my strategy:\n* Ordinal encoding for the \"ord\" columns, normalized to 0-1 in order to make the coefficients prettier.\n* One-hot encoding for everything else, using sparse (csr) representation so that it fits in memory.\n* LogisticRegression with the 'saga' solver, and a very carefully tuned C value of 0.1172. I used 50 iterations, but the number really didn't matter.\n* Trusting my CV. I really worked it hard in order to get the optimal C value, and the leaderboard improvements always matched the public LB. (It was only \"pretty good\" on the private LB, though. If I'd stuck with an older C value of \"0.133\" I would have been in the money. That's how the shake-up goes.)
\n\nThat's it. Lots of encodings were tried, but nothing beat one-hot. You just had to be willing to accept the thousands of columns that resulted.
", "rawMarkdown": "I only came in 7th, so my approach probably doesn't matter much, but here are the keys to my strategy:\n* Ordinal encoding for the \"ord\" columns, normalized to 0-1 in order to make the coefficients prettier.\n* One-hot encoding for everything else, using sparse (csr) representation so that it fits in memory.\n* LogisticRegression with the 'saga' solver, and a very carefully tuned C value of 0.1172. I used 50 iterations, but the number really didn't matter.\n* Trusting my CV. I really worked it hard in order to get the optimal C value, and the leaderboard improvements always matched the public LB. (It was only \"pretty good\" on the private LB, though. If I'd stuck with an older C value of \"0.133\" I would have been in the money. That's how the shake-up goes.) \n\nThat's it. Lots of encodings were tried, but nothing beat one-hot. You just had to be willing to accept the thousands of columns that resulted.\n\n", "votes": 18 }, { "id": 691626, "postDate": "2019-12-10T10:37:52.220Z", "content": "The biggest difference is that I used the 'liblinear' solver instead.
", "rawMarkdown": "The biggest difference is that I used the 'liblinear' solver instead.", "votes": 4 }, { "id": 691413, "postDate": "2019-12-10T05:37:00.870Z", "content": "Hi Robert,\nCongratulations and thank you a lot !\nYour approach was the good one.\nYou and I have two differences. First, I never tuned C, I kept until the end the magic 0.123456789 C.\nSecond, like you, I trusted only my CV : but I saw that when I changed the seed to create my 5 folds, I have different results and decisions were not the same : so I did double cross validation, even tripler cross validation. I will explain later.\nThank you for your generous post. I helped to try very simple things like the first trick !\nAlexandre
", "rawMarkdown": "Hi Robert,\nCongratulations and thank you a lot !\nYour approach was the good one.\nYou and I have two differences. First, I never tuned C, I kept until the end the magic 0.123456789 C.\nSecond, like you, I trusted only my CV : but I saw that when I changed the seed to create my 5 folds, I have different results and decisions were not the same : so I did double cross validation, even tripler cross validation. I will explain later.\nThank you for your generous post. I helped to try very simple things like the first trick !\nAlexandre", "votes": 4, "replies": [ { "id": 692035, "postDate": "2019-12-10T19:42:30.987Z", "content": "Glad the magic C worked :-D That's sexier than tuning many digits of hyperparameters.
", "rawMarkdown": "Glad the magic C worked :-D That's sexier than tuning many digits of hyperparameters.", "votes": 1 }, { "id": 692055, "postDate": "2019-12-10T20:01:20.093Z", "content": "Yes ! Thank you for that magic C !
", "rawMarkdown": "Yes ! Thank you for that magic C !" }, { "id": 692214, "postDate": "2019-12-11T03:13:31.810Z", "content": "Magic C may be the best C value for the perfect model still unknown but does not suit my model.
\n\nAfter the competition ends, I saw this comment: \"It's also for people who want to try out various hyper-parameter optimization algorithms. :-)\", so tuning hyper-parameter seems as important as feature engineering!
", "rawMarkdown": "Magic C may be the best C value for the perfect model still unknown but does not suit my model.\n\nAfter the competition ends, I saw this comment:[ \"It's also for people who want to try out various hyper-parameter optimization algorithms. :-)\"](https://www.kaggle.com/c/cat-in-the-dat/discussion/105713#607779), so tuning hyper-parameter seems as important as feature engineering!" } ] }, { "id": 691748, "postDate": "2019-12-10T13:10:18.643Z", "content": "Hi Robert,\nThank you for the interesting competition. You helped me a lot. I only came in 6th but I've just published a 0.80284 private LB solution notebook.
", "rawMarkdown": "Hi Robert,\nThank you for the interesting competition. You helped me a lot. I only came in 6th but I've just published a 0.80284 private LB solution notebook.", "votes": 1 }, { "id": 691415, "postDate": "2019-12-10T05:44:24.193Z", "content": "Exactly the same but with a C value of 0.12. I also removed infrequent values from some \"ord\" and \"nom\" columns which put me to the top of the public LB but didn't work out in the end. I tried many many encodings as well but nothing beat onehot. And thank you for your first post about keeping things simple. Tried LogisticRegression and did not bother using anything else :)
", "rawMarkdown": "Exactly the same but with a C value of 0.12. I also removed infrequent values from some \"ord\" and \"nom\" columns which put me to the top of the public LB but didn't work out in the end. I tried many many encodings as well but nothing beat onehot. And thank you for your first [post about keeping things simple](https://www.kaggle.com/c/cat-in-the-dat/discussion/113726#latest-687840). Tried LogisticRegression and did not bother using anything else :)", "votes": 2 }, { "id": 691712, "postDate": "2019-12-10T12:27:35.413Z", "content": "How did you do your CV if you may share?
", "rawMarkdown": "How did you do your CV if you may share? ", "replies": [ { "id": 692282, "postDate": "2019-12-11T05:16:14.937Z", "content": "tl;dr: I used the sklearn \"RepeatedKFold\" CV generator with 3 repetitions of 4 splits.
\n\nOne of the unique properties of this computation is that we had very \"clean\" synthetic data. Even though noise was introduced to keep it challenging, we could have confidence that (1) the order of the training instances didn't matter; and (2) the characteristics of the test set were identical to those of the train set; and (3) the public and private parts of the leaderboard also had identical characteristics. (We didn't have actual proof of these things, but that was the simplest and most likely assumption.)
\n\nThus, there was no need to get fancy with cross-validation. A simple 5-way split made a great starting point, and it didn't even matter whether shuffling was enabled. This is what I used for most of the competition. However, as others have noted, the choice of particular random seed -- and \"don't shuffle\" could be considered just another random seed in this case -- did make a difference in the single major hyper-parameter that we wish to tune. Thus, when I started going to extremes in my tuning, I turned to the \"RepeatedKFold\" class, which automatically creates multiple N-way splits and combines them together. I used a value of 4 splits of 3 folds because it gave more reliable results than KFold(5) without requiring too much additional runtime. I could have gone deeper, but this was sufficient for me to narrow in on a precise enough value to get to the point of diminishing returns -- after all, too many attempts would likely just lead to overfitting on the public LB rather than overfitting to a single CV.
", "rawMarkdown": "tl;dr: I used the sklearn \"RepeatedKFold\" CV generator with 3 repetitions of 4 splits.\n\nOne of the unique properties of this computation is that we had very \"clean\" synthetic data. Even though noise was introduced to keep it challenging, we could have confidence that (1) the order of the training instances didn't matter; and (2) the characteristics of the test set were identical to those of the train set; and (3) the public and private parts of the leaderboard also had identical characteristics. (We didn't have actual *proof* of these things, but that was the simplest and most likely assumption.)\n\nThus, there was no need to get fancy with cross-validation. A simple 5-way split made a great starting point, and it didn't even matter whether shuffling was enabled. This is what I used for most of the competition. However, as others have noted, the choice of particular random seed -- and \"don't shuffle\" could be considered just another random seed in this case -- did make a difference in the single major hyper-parameter that we wish to tune. Thus, when I started going to extremes in my tuning, I turned to the \"RepeatedKFold\" class, which automatically creates multiple N-way splits and combines them together. I used a value of 4 splits of 3 folds because it gave more reliable results than KFold(5) without requiring too much additional runtime. I could have gone deeper, but this was sufficient for me to narrow in on a precise enough value to get to the point of diminishing returns -- after all, too many attempts would likely just lead to overfitting on the public LB rather than overfitting to a single CV.\n" } ] }, { "id": 691610, "postDate": "2019-12-10T10:20:54.307Z", "content": "Hey, how much did the ordinal encoding help, compared to applying OHE on all columns?
", "rawMarkdown": "Hey, how much did the ordinal encoding help, compared to applying OHE on all columns?", "replies": [ { "id": 691861, "postDate": "2019-12-10T15:21:44.917Z", "content": "There was a linear relation between target and each ordinal feature, I will show it later in a notebook. And regression methods love parsimony.
", "rawMarkdown": "There was a linear relation between target and each ordinal feature, I will show it later in a notebook. And regression methods love parsimony." }, { "id": 692082, "postDate": "2019-12-10T20:59:13.797Z", "content": "Great, I look forward to seeing your solution.
", "rawMarkdown": "Great, I look forward to seeing your solution." } ] }, { "id": 691627, "postDate": "2019-12-10T10:38:10.533Z", "rawMarkdown": "", "isDeleted": true } ], "complete": true, "source": "live", "fetched_at": "2026-10-19T17:32:49Z", "competition_id": 14999, "competition_slug": "cat-in-the-dat", "team_id": 3731656, "solution_url": "/competitions/cat-in-the-dat/writeups/robert-stockton-post-mortem-what-worked-for-me" }