{
  "id": 140345,
  "title": "Improvements to get score 0.78797",
  "url": "/competitions/cat-in-the-dat-ii/discussion/140345",
  "author_name": "",
  "post_date": "2020-04-01T12:15:37.751324300Z",
  "votes": 15,
  "comment_count": 17,
  "views": 0,
  "content": "<p>Here is what worked for me to get score 0.78797.</p>\n\n<ul>\n<li>Baseline</li>\n</ul>\n\n<p>As a baseline model, I used logistic regression with default parameters and <code>liblinear</code> solver. All features in dataset are one-hot encoded.</p>\n\n<p>CV: 0.78130, private score: 0.78527</p>\n\n<ul>\n<li>Tuning hyperparameters</li>\n</ul>\n\n<p>After hyperparameters optimization, I found the following configuration yields a highest CV score.</p>\n\n<p><code>python\nestimator = LogisticRegression(\n    C=0.049,\n    class_weight={0: 1, 1: 1.42},\n    solver='liblinear',\n    fit_intercept=True,\n    penalty='l2')\n</code></p>\n\n<p>CV: 0.78519, private score: 0.78704</p>\n\n<ul>\n<li>Drop <code>bin_3</code></li>\n</ul>\n\n<p>I dropped <code>bin_3</code> feature, as it seems to be not really important, and keeping\nit in the dataset doesn't improve the score.</p>\n\n<p>CV: 0.78520, private score: 0.78704</p>\n\n<ul>\n<li>Ordinal encoding</li>\n</ul>\n\n<p>I used ordinal encoding for <code>ord_0</code>, <code>ord_1</code>, <code>ord_4</code>, <code>ord_5</code>, approximating \ntarget average with a linear function. For <code>ord_4</code> and <code>ord_5</code>, I removed outliers, \ncategories with small amount of observations, before applying the linear regression. </p>\n\n<p>CV: 0.78582, private score: 0.78727</p>\n\n<ul>\n<li>Grouping <code>nom_6</code></li>\n</ul>\n\n<p>For <code>nom_6</code> feature, I removed all categories which have less than 90 observations (replaced it with <code>nan</code>). Then using target encoding with cross-validation, converted it to numeric and grouped in three groups with <code>qcut</code>.</p>\n\n<p><code>python\nx['nom_6'] = pd.qcut(x['nom_6'], 3, labels=False, duplicates='drop')\n</code></p>\n\n<p>CV: 0.78691, private score: 0.78796</p>\n\n<ul>\n<li>Filtering <code>nom_9</code></li>\n</ul>\n\n<p>For <code>nom_9</code> feature, I removed all categories which have less than 60 observations (replaced it with <code>nan</code>) and combined together categories which have equal target average. </p>\n\n<p>CV: 0.78691, private score: 0.78797</p>\n\n<ul>\n<li>Missing values</li>\n</ul>\n\n<p>For one-hot encoded features (all features except <code>ord_0</code>, <code>ord_1</code>, <code>ord_4</code>, <code>ord_5</code>), \nI replaced missing values with <code>-1</code>. For ordinal encoded features, I replaced it with \nthe target probability, <code>0.18721</code>.  </p>\n\n<ul>\n<li>Results</li>\n</ul>\n\n<p>That's it, though I haven't chosen the best submission for final score and the official\nresults are a bit worse.</p>\n\n<p>Private score 0.78795 (110 place)\nPublic score 0.78669 (22 place)</p>\n\n<p>The solution code is here: <a href=\"https://github.com/viktorsapozhok/cafeen\">https://github.com/viktorsapozhok/cafeen</a></p>",
  "messages": [
    {
      "id": "793929",
      "postDate": "04/01/2020 12:15:37",
      "content": "<p>Here is what worked for me to get score 0.78797.</p>\n\n<ul>\n<li>Baseline</li>\n</ul>\n\n<p>As a baseline model, I used logistic regression with default parameters and <code>liblinear</code> solver. All features in dataset are one-hot encoded.</p>\n\n<p>CV: 0.78130, private score: 0.78527</p>\n\n<ul>\n<li>Tuning hyperparameters</li>\n</ul>\n\n<p>After hyperparameters optimization, I found the following configuration yields a highest CV score.</p>\n\n<p><code>python\nestimator = LogisticRegression(\n    C=0.049,\n    class_weight={0: 1, 1: 1.42},\n    solver='liblinear',\n    fit_intercept=True,\n    penalty='l2')\n</code></p>\n\n<p>CV: 0.78519, private score: 0.78704</p>\n\n<ul>\n<li>Drop <code>bin_3</code></li>\n</ul>\n\n<p>I dropped <code>bin_3</code> feature, as it seems to be not really important, and keeping\nit in the dataset doesn't improve the score.</p>\n\n<p>CV: 0.78520, private score: 0.78704</p>\n\n<ul>\n<li>Ordinal encoding</li>\n</ul>\n\n<p>I used ordinal encoding for <code>ord_0</code>, <code>ord_1</code>, <code>ord_4</code>, <code>ord_5</code>, approximating \ntarget average with a linear function. For <code>ord_4</code> and <code>ord_5</code>, I removed outliers, \ncategories with small amount of observations, before applying the linear regression. </p>\n\n<p>CV: 0.78582, private score: 0.78727</p>\n\n<ul>\n<li>Grouping <code>nom_6</code></li>\n</ul>\n\n<p>For <code>nom_6</code> feature, I removed all categories which have less than 90 observations (replaced it with <code>nan</code>). Then using target encoding with cross-validation, converted it to numeric and grouped in three groups with <code>qcut</code>.</p>\n\n<p><code>python\nx['nom_6'] = pd.qcut(x['nom_6'], 3, labels=False, duplicates='drop')\n</code></p>\n\n<p>CV: 0.78691, private score: 0.78796</p>\n\n<ul>\n<li>Filtering <code>nom_9</code></li>\n</ul>\n\n<p>For <code>nom_9</code> feature, I removed all categories which have less than 60 observations (replaced it with <code>nan</code>) and combined together categories which have equal target average. </p>\n\n<p>CV: 0.78691, private score: 0.78797</p>\n\n<ul>\n<li>Missing values</li>\n</ul>\n\n<p>For one-hot encoded features (all features except <code>ord_0</code>, <code>ord_1</code>, <code>ord_4</code>, <code>ord_5</code>), \nI replaced missing values with <code>-1</code>. For ordinal encoded features, I replaced it with \nthe target probability, <code>0.18721</code>.  </p>\n\n<ul>\n<li>Results</li>\n</ul>\n\n<p>That's it, though I haven't chosen the best submission for final score and the official\nresults are a bit worse.</p>\n\n<p>Private score 0.78795 (110 place)\nPublic score 0.78669 (22 place)</p>\n\n<p>The solution code is here: <a href=\"https://github.com/viktorsapozhok/cafeen\">https://github.com/viktorsapozhok/cafeen</a></p>",
      "rawMarkdown": "Here is what worked for me to get score 0.78797.\n\n- Baseline\n\nAs a baseline model, I used logistic regression with default parameters and `liblinear` solver. All features in dataset are one-hot encoded.\n\nCV: 0.78130, private score: 0.78527\n\n- Tuning hyperparameters\n\nAfter hyperparameters optimization, I found the following configuration yields a highest CV score.\n\n```python\nestimator = LogisticRegression(\n    C=0.049,\n    class_weight={0: 1, 1: 1.42},\n    solver='liblinear',\n    fit_intercept=True,\n    penalty='l2')\n```\n\nCV: 0.78519, private score: 0.78704\n\n- Drop `bin_3`\n\nI dropped `bin_3` feature, as it seems to be not really important, and keeping\nit in the dataset doesn't improve the score.\n\nCV: 0.78520, private score: 0.78704\n\n- Ordinal encoding\n\nI used ordinal encoding for `ord_0`, `ord_1`, `ord_4`, `ord_5`, approximating \ntarget average with a linear function. For `ord_4` and `ord_5`, I removed outliers, \ncategories with small amount of observations, before applying the linear regression. \n\nCV: 0.78582, private score: 0.78727\n\n- Grouping `nom_6`\n\nFor `nom_6` feature, I removed all categories which have less than 90 observations (replaced it with `nan`). Then using target encoding with cross-validation, converted it to numeric and grouped in three groups with `qcut`.\n\n```python\nx['nom_6'] = pd.qcut(x['nom_6'], 3, labels=False, duplicates='drop')\n```\n\nCV: 0.78691, private score: 0.78796\n\n- Filtering `nom_9`\n\nFor `nom_9` feature, I removed all categories which have less than 60 observations (replaced it with `nan`) and combined together categories which have equal target average. \n\nCV: 0.78691, private score: 0.78797\n\n- Missing values\n\nFor one-hot encoded features (all features except `ord_0`, `ord_1`, `ord_4`, `ord_5`), \nI replaced missing values with `-1`. For ordinal encoded features, I replaced it with \nthe target probability, `0.18721`.  \n\n- Results\n\nThat's it, though I haven't chosen the best submission for final score and the official\nresults are a bit worse.\n\nPrivate score 0.78795 (110 place)\nPublic score 0.78669 (22 place)\n\nThe solution code is here: https://github.com/viktorsapozhok/cafeen",
      "votes": null
    },
    {
      "id": "794065",
      "postDate": "04/01/2020 14:25:00",
      "content": "<p>Great job!</p>",
      "rawMarkdown": "Great job!",
      "votes": null
    },
    {
      "id": "794066",
      "postDate": "04/01/2020 14:26:37",
      "content": "<p>nicely done</p>",
      "rawMarkdown": "nicely done",
      "votes": null
    },
    {
      "id": "794527",
      "postDate": "04/01/2020 22:01:43",
      "content": "<p>Hey, thanks for sharing! I have a couple questions:</p>\n\n<ul>\n<li><p>Could you elaborate on how did you do the ordinal encoding? I see that the target mean follows a linear distribution with the sorted categories, but you mention that you applied a linear regression, how?</p></li>\n<li><p>Why didn't you ordinal encode ord_3 as it shows the above mentioned behaviour?</p></li>\n<li><p>How did you come up with the idea of remove categories with less than 90 observations and then group in 3 groups in nom_6?</p></li>\n</ul>\n\n<p>Thanks again!</p>",
      "rawMarkdown": "Hey, thanks for sharing! I have a couple questions:\n\n- Could you elaborate on how did you do the ordinal encoding? I see that the target mean follows a linear distribution with the sorted categories, but you mention that you applied a linear regression, how?\n\n- Why didn't you ordinal encode ord_3 as it shows the above mentioned behaviour?\n\n- How did you come up with the idea of remove categories with less than 90 observations and then group in 3 groups in nom_6?\n\nThanks again!",
      "votes": null
    },
    {
      "id": "794564",
      "postDate": "04/01/2020 22:36:06",
      "content": "<ul>\n<li><p>For ordinal feature, e.g. ord_4, I calculated target mean for every category, saw that the mean is close to linear function and fitted linear regression (statsmodels.OLS), where dependent variable is ord_4 categories enumerated as 1, 2, 3 etc and independent variable is target mean. Missing values in this case I replaced by 0.18721. My guess here was that this is a synthetic dataset and missing values were just randomly chosen. It means that target probability for missing value is the same as target probability in the whole dataset. <a href=\"https://github.com/viktorsapozhok/cafeen/blob/24f624b7299c8099daae26d6d1734c6200d7c8be/cafeen/steps.py#L76\">Here is the function which does linear encoding</a></p></li>\n<li><p>I tried but it didn't give me a boost on CV score. I think its because ord_3 is \"more linear\" than ord_4 or ord_5. And it doesnt matter whether its encoded with target mean or just one-hot. ord_4 and ord_5 in contrast have some categories deviated from linear function and applying linear encoding we smooth such deviations. So there is nothing to \"smooth\" for ord_3 and that is why linear encoding (probably) didn't help.</p></li>\n<li><p>Just tried many different approaches and nothing worked except of it. After I saw that filtering outliers and grouping works, I tuned these numbers using grid search. I am pretty sure there is a better encoding for nom_6, but I just didn't find it.</p></li>\n</ul>",
      "rawMarkdown": "For ordinal feature, e.g. ord\\_4, I calculated target mean for every category, saw that the mean is close to linear function and fitted linear regression (statsmodels.OLS), where dependent variable is ord\\_4 categories enumerated as 1, 2, 3 etc and independent variable is target mean. Missing values in this case I replaced by 0.18721. My guess here was that this is a synthetic dataset and missing values were just randomly chosen. It means that target probability for missing value is the same as target probability in the whole dataset. [Here is the function which does linear encoding](https://github.com/viktorsapozhok/cafeen/blob/24f624b7299c8099daae26d6d1734c6200d7c8be/cafeen/steps.py#L76)\n\n- I tried but it didn't give me a boost on CV score. I think its because ord\\_3 is \"more linear\" than ord\\_4 or ord\\_5. And it doesnt matter whether its encoded with target mean or just one-hot. ord\\_4 and ord\\_5 in contrast have some categories deviated from linear function and applying linear encoding we smooth such deviations. So there is nothing to \"smooth\" for ord\\_3 and that is why linear encoding (probably) didn't help.\n\n- Just tried many different approaches and nothing worked except of it. After I saw that filtering outliers and grouping works, I tuned these numbers using grid search. I am pretty sure there is a better encoding for nom\\_6, but I just didn't find it.",
      "votes": null
    },
    {
      "id": "795167",
      "postDate": "04/02/2020 13:30:38",
      "content": "<p>Okay! I got it, so for ord_0, ord_1, ord_4, ord_5 you actually did something like a mean target encoding but you assigned the mean of the target to each category using the linear fitting. Did you do that in a KFold way or similar to avoid leakage?</p>",
      "rawMarkdown": "Okay! I got it, so for ord_0, ord_1, ord_4, ord_5 you actually did something like a mean target encoding but you assigned the mean of the target to each category using the linear fitting. Did you do that in a KFold way or similar to avoid leakage?",
      "votes": null
    },
    {
      "id": "795182",
      "postDate": "04/02/2020 13:41:27",
      "content": "<p>Right, for ord_4 and ord_5 I removed categories with small amounts of observations before linear fitting, but I dont think it affected significantly the score. Didn't use kfold for it.</p>",
      "rawMarkdown": "Right, for ord\\_4 and ord\\_5 I removed categories with small amounts of observations before linear fitting, but I dont think it affected significantly the score. Didn't use kfold for it.",
      "votes": null
    },
    {
      "id": "795493",
      "postDate": "04/02/2020 18:58:04",
      "content": "<p>S.s.   fake names in the kaggle new app.</p>",
      "rawMarkdown": "S.s.   fake names in the kaggle new app.",
      "votes": null
    },
    {
      "id": "798572",
      "postDate": "04/05/2020 16:47:16",
      "content": "<p>Thanks for sharing your solution!</p>\n\n<p>I have two questions:\n1. What was the standard deviation of your final CV score?\n2. What is the rationale for the validation strategy you chose, why 2-fold splitting?</p>",
      "rawMarkdown": "Thanks for sharing your solution!\n\nI have two questions:\n1. What was the standard deviation of your final CV score?\n2. What is the rationale for the validation strategy you chose, why 2-fold splitting?",
      "votes": null
    },
    {
      "id": "798584",
      "postDate": "04/05/2020 16:58:19",
      "content": "<p>Didn't understand a question about standard deviation. For cross-validation I used 4 iterations, randomly splitting dataset in 2 folds (300 vs 300) in each iteration. CV score is therefore an average score over 4 iterations. Used 2 folds because in this case I got cv score close to public score. When I used more folds, I got score significantly higher than the public score and it confused me a bit.</p>",
      "rawMarkdown": "Didn't understand a question about standard deviation. For cross-validation I used 4 iterations, randomly splitting dataset in 2 folds (300 vs 300) in each iteration. CV score is therefore an average score over 4 iterations. Used 2 folds because in this case I got cv score close to public score. When I used more folds, I got score significantly higher than the public score and it confused me a bit.",
      "votes": null
    },
    {
      "id": "798617",
      "postDate": "04/05/2020 17:28:22",
      "content": "<blockquote>\n  <p>Didn't understand a question about standard deviation.</p>\n</blockquote>\n\n<p>Well, if you can calculate the mean of 4 values you can also get the standard deviation of this set of values. In my case, and in some published kernels, the standard deviation was about 0.0015. The question is actually about the variance of your model. You did a big effort to remove outliers and feature levels with a small number of observations and I'm wondering whether your model gave more consistent predictions.</p>\n\n<blockquote>\n  <p>When I used more folds, I got score significantly higher than the public score and it confused me a bit.</p>\n</blockquote>\n\n<p>I hoped there was some more profound reason. Anyway, thanks for the reply!</p>",
      "rawMarkdown": "&gt; Didn't understand a question about standard deviation.\n\nWell, if you can calculate the mean of 4 values you can also get the standard deviation of this set of values. In my case, and in some published kernels, the standard deviation was about 0.0015. The question is actually about the variance of your model. You did a big effort to remove outliers and feature levels with a small number of observations and I'm wondering whether your model gave more consistent predictions.\n\n&gt; When I used more folds, I got score significantly higher than the public score and it confused me a bit.\n\nI hoped there was some more profound reason. Anyway, thanks for the reply!",
      "votes": null
    },
    {
      "id": "798630",
      "postDate": "04/05/2020 17:44:49",
      "content": "<p>cv scores in my best submission are: 0.78825, 0.78604, 0.78682, 0.786878\nstdev: 0.000795</p>",
      "rawMarkdown": "cv scores in my best submission are: 0.78825, 0.78604, 0.78682, 0.786878\nstdev: 0.000795",
      "votes": null
    },
    {
      "id": "798648",
      "postDate": "04/05/2020 17:54:06",
      "content": "<p>So the std is roughly 2 times smaller. Great!\nThanks once again!</p>",
      "rawMarkdown": "So the std is roughly 2 times smaller. Great!\nThanks once again!",
      "votes": null
    },
    {
      "id": "799174",
      "postDate": "04/06/2020 08:49:10",
      "content": "<p>I've got another couple of related questions:\n1. I realized that the score on the validation set alone doesn't give the full picture. What were the mean score and its std on the training sets for your best model? It must be barely larger than the validation score.\n2. You said that your validation score significantly increased when you used more than two folds. Could you give me some numbers?\nI will explain to you why I'm asking this question.  The target encoding of 3 nominal features  nom_5, nom_6, and nom_9 gave the largest increase in my CV score (roughly, from 0.786 to 0.788). Similar to what you did, I have also tried different ways to filter the values in these features with a few observations but inevitably got the decrease of both the training and the validation scores. That puzzled me at that time. Your solution made me think that perhaps the absolute value of my CV score was not fair because of the inevitable target leak produced by the target encoding and that filtering the outliers was reducing the leakage. Now I have doubts again. I don't want to go through all the experiments that I have done, but maybe your results will give me some clues.</p>",
      "rawMarkdown": "I've got another couple of related questions:\n1. I realized that the score on the validation set alone doesn't give the full picture. What were the mean score and its std on the training sets for your best model? It must be barely larger than the validation score.\n2. You said that your validation score significantly increased when you used more than two folds. Could you give me some numbers?\nI will explain to you why I'm asking this question.  The target encoding of 3 nominal features  nom_5, nom_6, and nom_9 gave the largest increase in my CV score (roughly, from 0.786 to 0.788). Similar to what you did, I have also tried different ways to filter the values in these features with a few observations but inevitably got the decrease of both the training and the validation scores. That puzzled me at that time. Your solution made me think that perhaps the absolute value of my CV score was not fair because of the inevitable target leak produced by the target encoding and that filtering the outliers was reducing the leakage. Now I have doubts again. I don't want to go through all the experiments that I have done, but maybe your results will give me some clues.",
      "votes": null
    },
    {
      "id": "799183",
      "postDate": "04/06/2020 08:56:36",
      "content": "<p>But I dont use target encoding for these features, I one-hot encoded them. Before one-hot encoding I grouped nom_6 into three categories (based on target encoding with cv) and replaced outliers in nom_9 by nans (without using any target encoding). And I did nothing with nom_5, only one-hot encoding</p>",
      "rawMarkdown": "But I dont use target encoding for these features, I one-hot encoded them. Before one-hot encoding I grouped nom\\_6 into three categories (based on target encoding with cv) and replaced outliers in nom\\_9 by nans (without using any target encoding). And I did nothing with nom\\_5, only one-hot encoding",
      "votes": null
    },
    {
      "id": "799215",
      "postDate": "04/06/2020 09:14:45",
      "content": "<p>Yes, I understand it well and that is why I'm asking my question.\nYou shouldn't have target leakage and your CV scores are consistently lower than the scores on the LB (well, the private score is higher than the CV score, but I can admit it). Initially, that made me think that my CV score of 0.788... was an over-optimistic (due to the target leakage) estimation. It was a satisfactory explanation (with some hand-waving) until you mentioned that with another CV strategy you got higher CV scores. That's why I'd like to have more information.</p>",
      "rawMarkdown": "Yes, I understand it well and that is why I'm asking my question.\nYou shouldn't have target leakage and your CV scores are consistently lower than the scores on the LB (well, the private score is higher than the CV score, but I can admit it). Initially, that made me think that my CV score of 0.788... was an over-optimistic (due to the target leakage) estimation. It was a satisfactory explanation (with some hand-waving) until you mentioned that with another CV strategy you got higher CV scores. That's why I'd like to have more information.",
      "votes": null
    },
    {
      "id": "809953",
      "postDate": "04/16/2020 16:05:55",
      "content": "<p>I think you mixed up the words independent and dependent variable in your description. Looking at your code I think that the enumerated categories are the independent values. The independent variables are also the ones that you use as an input to your model to estimate the dependent value.</p>\n\n<p>Also as <a href=\"/delai50\">@delai50</a> mentioned, I think a more appropriate description is \"modified\" mean target encoding. As far is I can see this isn't ordinal encoding. I think you could call it \"modified\" ordinal encoding, if you use really use the means as independent variable, then linear fit it and input the category mean to get a number which is close to the original enumerated value.</p>",
      "rawMarkdown": "I think you mixed up the words independent and dependent variable in your description. Looking at your code I think that the enumerated categories are the independent values. The independent variables are also the ones that you use as an input to your model to estimate the dependent value.\n\nAlso as @delai50 mentioned, I think a more appropriate description is \"modified\" mean target encoding. As far is I can see this isn't ordinal encoding. I think you could call it \"modified\" ordinal encoding, if you use really use the means as independent variable, then linear fit it and input the category mean to get a number which is close to the original enumerated value.",
      "votes": null
    },
    {
      "id": "810007",
      "postDate": "04/16/2020 16:51:09",
      "content": "<p>you're right, off course target mean is dependent variable</p>",
      "rawMarkdown": "you're right, off course target mean is dependent variable",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 794065,
      "author_name": "kailai",
      "author_url": "",
      "post_date": "04/01/2020 14:25:00",
      "content": "<p>Great job!</p>",
      "votes": null,
      "replies": [
        {
          "id": 795493,
          "author_name": "solyoh21",
          "author_url": "",
          "post_date": "04/02/2020 18:58:04",
          "content": "<p>S.s.   fake names in the kaggle new app.</p>",
          "votes": null,
          "replies": []
        }
      ]
    },
    {
      "id": 794066,
      "author_name": "puremath86",
      "author_url": "",
      "post_date": "04/01/2020 14:26:37",
      "content": "<p>nicely done</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 794527,
      "author_name": "delai50",
      "author_url": "",
      "post_date": "04/01/2020 22:01:43",
      "content": "<p>Hey, thanks for sharing! I have a couple questions:</p>\n\n<ul>\n<li><p>Could you elaborate on how did you do the ordinal encoding? I see that the target mean follows a linear distribution with the sorted categories, but you mention that you applied a linear regression, how?</p></li>\n<li><p>Why didn't you ordinal encode ord_3 as it shows the above mentioned behaviour?</p></li>\n<li><p>How did you come up with the idea of remove categories with less than 90 observations and then group in 3 groups in nom_6?</p></li>\n</ul>\n\n<p>Thanks again!</p>",
      "votes": null,
      "replies": [
        {
          "id": 794564,
          "author_name": "vktrspzhk",
          "author_url": "",
          "post_date": "04/01/2020 22:36:06",
          "content": "<ul>\n<li><p>For ordinal feature, e.g. ord_4, I calculated target mean for every category, saw that the mean is close to linear function and fitted linear regression (statsmodels.OLS), where dependent variable is ord_4 categories enumerated as 1, 2, 3 etc and independent variable is target mean. Missing values in this case I replaced by 0.18721. My guess here was that this is a synthetic dataset and missing values were just randomly chosen. It means that target probability for missing value is the same as target probability in the whole dataset. <a href=\"https://github.com/viktorsapozhok/cafeen/blob/24f624b7299c8099daae26d6d1734c6200d7c8be/cafeen/steps.py#L76\">Here is the function which does linear encoding</a></p></li>\n<li><p>I tried but it didn't give me a boost on CV score. I think its because ord_3 is \"more linear\" than ord_4 or ord_5. And it doesnt matter whether its encoded with target mean or just one-hot. ord_4 and ord_5 in contrast have some categories deviated from linear function and applying linear encoding we smooth such deviations. So there is nothing to \"smooth\" for ord_3 and that is why linear encoding (probably) didn't help.</p></li>\n<li><p>Just tried many different approaches and nothing worked except of it. After I saw that filtering outliers and grouping works, I tuned these numbers using grid search. I am pretty sure there is a better encoding for nom_6, but I just didn't find it.</p></li>\n</ul>",
          "votes": null,
          "replies": []
        },
        {
          "id": 795167,
          "author_name": "delai50",
          "author_url": "",
          "post_date": "04/02/2020 13:30:38",
          "content": "<p>Okay! I got it, so for ord_0, ord_1, ord_4, ord_5 you actually did something like a mean target encoding but you assigned the mean of the target to each category using the linear fitting. Did you do that in a KFold way or similar to avoid leakage?</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 795182,
          "author_name": "vktrspzhk",
          "author_url": "",
          "post_date": "04/02/2020 13:41:27",
          "content": "<p>Right, for ord_4 and ord_5 I removed categories with small amounts of observations before linear fitting, but I dont think it affected significantly the score. Didn't use kfold for it.</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 809953,
          "author_name": "towadroid",
          "author_url": "",
          "post_date": "04/16/2020 16:05:55",
          "content": "<p>I think you mixed up the words independent and dependent variable in your description. Looking at your code I think that the enumerated categories are the independent values. The independent variables are also the ones that you use as an input to your model to estimate the dependent value.</p>\n\n<p>Also as <a href=\"/delai50\">@delai50</a> mentioned, I think a more appropriate description is \"modified\" mean target encoding. As far is I can see this isn't ordinal encoding. I think you could call it \"modified\" ordinal encoding, if you use really use the means as independent variable, then linear fit it and input the category mean to get a number which is close to the original enumerated value.</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 810007,
          "author_name": "vktrspzhk",
          "author_url": "",
          "post_date": "04/16/2020 16:51:09",
          "content": "<p>you're right, off course target mean is dependent variable</p>",
          "votes": null,
          "replies": []
        }
      ]
    },
    {
      "id": 798572,
      "author_name": "vcerchez",
      "author_url": "",
      "post_date": "04/05/2020 16:47:16",
      "content": "<p>Thanks for sharing your solution!</p>\n\n<p>I have two questions:\n1. What was the standard deviation of your final CV score?\n2. What is the rationale for the validation strategy you chose, why 2-fold splitting?</p>",
      "votes": null,
      "replies": [
        {
          "id": 798584,
          "author_name": "vktrspzhk",
          "author_url": "",
          "post_date": "04/05/2020 16:58:19",
          "content": "<p>Didn't understand a question about standard deviation. For cross-validation I used 4 iterations, randomly splitting dataset in 2 folds (300 vs 300) in each iteration. CV score is therefore an average score over 4 iterations. Used 2 folds because in this case I got cv score close to public score. When I used more folds, I got score significantly higher than the public score and it confused me a bit.</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 798617,
          "author_name": "vcerchez",
          "author_url": "",
          "post_date": "04/05/2020 17:28:22",
          "content": "<blockquote>\n  <p>Didn't understand a question about standard deviation.</p>\n</blockquote>\n\n<p>Well, if you can calculate the mean of 4 values you can also get the standard deviation of this set of values. In my case, and in some published kernels, the standard deviation was about 0.0015. The question is actually about the variance of your model. You did a big effort to remove outliers and feature levels with a small number of observations and I'm wondering whether your model gave more consistent predictions.</p>\n\n<blockquote>\n  <p>When I used more folds, I got score significantly higher than the public score and it confused me a bit.</p>\n</blockquote>\n\n<p>I hoped there was some more profound reason. Anyway, thanks for the reply!</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 798630,
          "author_name": "vktrspzhk",
          "author_url": "",
          "post_date": "04/05/2020 17:44:49",
          "content": "<p>cv scores in my best submission are: 0.78825, 0.78604, 0.78682, 0.786878\nstdev: 0.000795</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 798648,
          "author_name": "vcerchez",
          "author_url": "",
          "post_date": "04/05/2020 17:54:06",
          "content": "<p>So the std is roughly 2 times smaller. Great!\nThanks once again!</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 799174,
          "author_name": "vcerchez",
          "author_url": "",
          "post_date": "04/06/2020 08:49:10",
          "content": "<p>I've got another couple of related questions:\n1. I realized that the score on the validation set alone doesn't give the full picture. What were the mean score and its std on the training sets for your best model? It must be barely larger than the validation score.\n2. You said that your validation score significantly increased when you used more than two folds. Could you give me some numbers?\nI will explain to you why I'm asking this question.  The target encoding of 3 nominal features  nom_5, nom_6, and nom_9 gave the largest increase in my CV score (roughly, from 0.786 to 0.788). Similar to what you did, I have also tried different ways to filter the values in these features with a few observations but inevitably got the decrease of both the training and the validation scores. That puzzled me at that time. Your solution made me think that perhaps the absolute value of my CV score was not fair because of the inevitable target leak produced by the target encoding and that filtering the outliers was reducing the leakage. Now I have doubts again. I don't want to go through all the experiments that I have done, but maybe your results will give me some clues.</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 799183,
          "author_name": "vktrspzhk",
          "author_url": "",
          "post_date": "04/06/2020 08:56:36",
          "content": "<p>But I dont use target encoding for these features, I one-hot encoded them. Before one-hot encoding I grouped nom_6 into three categories (based on target encoding with cv) and replaced outliers in nom_9 by nans (without using any target encoding). And I did nothing with nom_5, only one-hot encoding</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 799215,
          "author_name": "vcerchez",
          "author_url": "",
          "post_date": "04/06/2020 09:14:45",
          "content": "<p>Yes, I understand it well and that is why I'm asking my question.\nYou shouldn't have target leakage and your CV scores are consistently lower than the scores on the LB (well, the private score is higher than the CV score, but I can admit it). Initially, that made me think that my CV score of 0.788... was an over-optimistic (due to the target leakage) estimation. It was a satisfactory explanation (with some hand-waving) until you mentioned that with another CV strategy you got higher CV scores. That's why I'd like to have more information.</p>",
          "votes": null,
          "replies": []
        }
      ]
    }
  ],
  "raw_markdown_by_id": {
    "793929": "Here is what worked for me to get score 0.78797.\n\n- Baseline\n\nAs a baseline model, I used logistic regression with default parameters and `liblinear` solver. All features in dataset are one-hot encoded.\n\nCV: 0.78130, private score: 0.78527\n\n- Tuning hyperparameters\n\nAfter hyperparameters optimization, I found the following configuration yields a highest CV score.\n\n```python\nestimator = LogisticRegression(\n    C=0.049,\n    class_weight={0: 1, 1: 1.42},\n    solver='liblinear',\n    fit_intercept=True,\n    penalty='l2')\n```\n\nCV: 0.78519, private score: 0.78704\n\n- Drop `bin_3`\n\nI dropped `bin_3` feature, as it seems to be not really important, and keeping\nit in the dataset doesn't improve the score.\n\nCV: 0.78520, private score: 0.78704\n\n- Ordinal encoding\n\nI used ordinal encoding for `ord_0`, `ord_1`, `ord_4`, `ord_5`, approximating \ntarget average with a linear function. For `ord_4` and `ord_5`, I removed outliers, \ncategories with small amount of observations, before applying the linear regression. \n\nCV: 0.78582, private score: 0.78727\n\n- Grouping `nom_6`\n\nFor `nom_6` feature, I removed all categories which have less than 90 observations (replaced it with `nan`). Then using target encoding with cross-validation, converted it to numeric and grouped in three groups with `qcut`.\n\n```python\nx['nom_6'] = pd.qcut(x['nom_6'], 3, labels=False, duplicates='drop')\n```\n\nCV: 0.78691, private score: 0.78796\n\n- Filtering `nom_9`\n\nFor `nom_9` feature, I removed all categories which have less than 60 observations (replaced it with `nan`) and combined together categories which have equal target average. \n\nCV: 0.78691, private score: 0.78797\n\n- Missing values\n\nFor one-hot encoded features (all features except `ord_0`, `ord_1`, `ord_4`, `ord_5`), \nI replaced missing values with `-1`. For ordinal encoded features, I replaced it with \nthe target probability, `0.18721`.  \n\n- Results\n\nThat's it, though I haven't chosen the best submission for final score and the official\nresults are a bit worse.\n\nPrivate score 0.78795 (110 place)\nPublic score 0.78669 (22 place)\n\nThe solution code is here: https://github.com/viktorsapozhok/cafeen",
    "794065": "Great job!",
    "794066": "nicely done",
    "794527": "Hey, thanks for sharing! I have a couple questions:\n\n- Could you elaborate on how did you do the ordinal encoding? I see that the target mean follows a linear distribution with the sorted categories, but you mention that you applied a linear regression, how?\n\n- Why didn't you ordinal encode ord_3 as it shows the above mentioned behaviour?\n\n- How did you come up with the idea of remove categories with less than 90 observations and then group in 3 groups in nom_6?\n\nThanks again!",
    "794564": "For ordinal feature, e.g. ord\\_4, I calculated target mean for every category, saw that the mean is close to linear function and fitted linear regression (statsmodels.OLS), where dependent variable is ord\\_4 categories enumerated as 1, 2, 3 etc and independent variable is target mean. Missing values in this case I replaced by 0.18721. My guess here was that this is a synthetic dataset and missing values were just randomly chosen. It means that target probability for missing value is the same as target probability in the whole dataset. [Here is the function which does linear encoding](https://github.com/viktorsapozhok/cafeen/blob/24f624b7299c8099daae26d6d1734c6200d7c8be/cafeen/steps.py#L76)\n\n- I tried but it didn't give me a boost on CV score. I think its because ord\\_3 is \"more linear\" than ord\\_4 or ord\\_5. And it doesnt matter whether its encoded with target mean or just one-hot. ord\\_4 and ord\\_5 in contrast have some categories deviated from linear function and applying linear encoding we smooth such deviations. So there is nothing to \"smooth\" for ord\\_3 and that is why linear encoding (probably) didn't help.\n\n- Just tried many different approaches and nothing worked except of it. After I saw that filtering outliers and grouping works, I tuned these numbers using grid search. I am pretty sure there is a better encoding for nom\\_6, but I just didn't find it.",
    "795167": "Okay! I got it, so for ord_0, ord_1, ord_4, ord_5 you actually did something like a mean target encoding but you assigned the mean of the target to each category using the linear fitting. Did you do that in a KFold way or similar to avoid leakage?",
    "795182": "Right, for ord\\_4 and ord\\_5 I removed categories with small amounts of observations before linear fitting, but I dont think it affected significantly the score. Didn't use kfold for it.",
    "795493": "S.s.   fake names in the kaggle new app.",
    "798572": "Thanks for sharing your solution!\n\nI have two questions:\n1. What was the standard deviation of your final CV score?\n2. What is the rationale for the validation strategy you chose, why 2-fold splitting?",
    "798584": "Didn't understand a question about standard deviation. For cross-validation I used 4 iterations, randomly splitting dataset in 2 folds (300 vs 300) in each iteration. CV score is therefore an average score over 4 iterations. Used 2 folds because in this case I got cv score close to public score. When I used more folds, I got score significantly higher than the public score and it confused me a bit.",
    "798617": "&gt; Didn't understand a question about standard deviation.\n\nWell, if you can calculate the mean of 4 values you can also get the standard deviation of this set of values. In my case, and in some published kernels, the standard deviation was about 0.0015. The question is actually about the variance of your model. You did a big effort to remove outliers and feature levels with a small number of observations and I'm wondering whether your model gave more consistent predictions.\n\n&gt; When I used more folds, I got score significantly higher than the public score and it confused me a bit.\n\nI hoped there was some more profound reason. Anyway, thanks for the reply!",
    "798630": "cv scores in my best submission are: 0.78825, 0.78604, 0.78682, 0.786878\nstdev: 0.000795",
    "798648": "So the std is roughly 2 times smaller. Great!\nThanks once again!",
    "799174": "I've got another couple of related questions:\n1. I realized that the score on the validation set alone doesn't give the full picture. What were the mean score and its std on the training sets for your best model? It must be barely larger than the validation score.\n2. You said that your validation score significantly increased when you used more than two folds. Could you give me some numbers?\nI will explain to you why I'm asking this question.  The target encoding of 3 nominal features  nom_5, nom_6, and nom_9 gave the largest increase in my CV score (roughly, from 0.786 to 0.788). Similar to what you did, I have also tried different ways to filter the values in these features with a few observations but inevitably got the decrease of both the training and the validation scores. That puzzled me at that time. Your solution made me think that perhaps the absolute value of my CV score was not fair because of the inevitable target leak produced by the target encoding and that filtering the outliers was reducing the leakage. Now I have doubts again. I don't want to go through all the experiments that I have done, but maybe your results will give me some clues.",
    "799183": "But I dont use target encoding for these features, I one-hot encoded them. Before one-hot encoding I grouped nom\\_6 into three categories (based on target encoding with cv) and replaced outliers in nom\\_9 by nans (without using any target encoding). And I did nothing with nom\\_5, only one-hot encoding",
    "799215": "Yes, I understand it well and that is why I'm asking my question.\nYou shouldn't have target leakage and your CV scores are consistently lower than the scores on the LB (well, the private score is higher than the CV score, but I can admit it). Initially, that made me think that my CV score of 0.788... was an over-optimistic (due to the target leakage) estimation. It was a satisfactory explanation (with some hand-waving) until you mentioned that with another CV strategy you got higher CV scores. That's why I'd like to have more information.",
    "809953": "I think you mixed up the words independent and dependent variable in your description. Looking at your code I think that the enumerated categories are the independent values. The independent variables are also the ones that you use as an input to your model to estimate the dependent value.\n\nAlso as @delai50 mentioned, I think a more appropriate description is \"modified\" mean target encoding. As far is I can see this isn't ordinal encoding. I think you could call it \"modified\" ordinal encoding, if you use really use the means as independent variable, then linear fit it and input the category mean to get a number which is close to the original enumerated value.",
    "810007": "you're right, off course target mean is dependent variable"
  },
  "source": "meta"
}