{
  "id": 68177,
  "title": "When do I use the mean, frequncy encoding at user level",
  "url": "/competitions/ga-customer-revenue-prediction/discussion/68177",
  "author_name": "",
  "post_date": "2018-10-10T06:11:44.518023100Z",
  "votes": 2,
  "comment_count": 1,
  "views": 0,
  "content": "<p>I'm solving this competiton using user level prediction.</p>\n\n<p>For now, I have a concern about the right use of mean and frequncy encoding.</p>\n\n<p>To use user level prediction, we need to do groupby with fullVisitorId.</p>\n\n<p>Until now, I've do the below procedure.\n(1)\nMean and frequency encoding for categorical feature\n-&gt; groupby fullVisitorId using mean()\n-&gt; prediction</p>\n\n<p>But, after looking some samples, some samples have float category in specific feature\n because ,I don't know why, they have various category in the same feature.</p>\n\n<p>So, I generated new category after  groupby using mean().</p>\n\n<p>For now, I set the new procedure shown in below.</p>\n\n<p>(2)\nLabel encoding -&gt; groupby fullVisitorId -&gt; choose most frequent category\n-&gt; mean and frequency encoding -&gt; prediction</p>\n\n<p>I want to know which one is correct way.</p>\n\n<p>Please, help me!</p>",
  "messages": [
    {
      "id": "401469",
      "postDate": "10/10/2018 06:11:44",
      "content": "<p>I'm solving this competiton using user level prediction.</p>\n\n<p>For now, I have a concern about the right use of mean and frequncy encoding.</p>\n\n<p>To use user level prediction, we need to do groupby with fullVisitorId.</p>\n\n<p>Until now, I've do the below procedure.\n(1)\nMean and frequency encoding for categorical feature\n-&gt; groupby fullVisitorId using mean()\n-&gt; prediction</p>\n\n<p>But, after looking some samples, some samples have float category in specific feature\n because ,I don't know why, they have various category in the same feature.</p>\n\n<p>So, I generated new category after  groupby using mean().</p>\n\n<p>For now, I set the new procedure shown in below.</p>\n\n<p>(2)\nLabel encoding -&gt; groupby fullVisitorId -&gt; choose most frequent category\n-&gt; mean and frequency encoding -&gt; prediction</p>\n\n<p>I want to know which one is correct way.</p>\n\n<p>Please, help me!</p>",
      "rawMarkdown": "I'm solving this competiton using user level prediction.\n\nFor now, I have a concern about the right use of mean and frequncy encoding.\n\nTo use user level prediction, we need to do groupby with fullVisitorId.\n\nUntil now, I've do the below procedure.\n(1)\nMean and frequency encoding for categorical feature\n-&gt; groupby fullVisitorId using mean()\n-&gt; prediction\n\n\nBut, after looking some samples, some samples have float category in specific feature\n because ,I don't know why, they have various category in the same feature.\n\nSo, I generated new category after  groupby using mean().\n\nFor now, I set the new procedure shown in below.\n\n(2)\nLabel encoding -&gt; groupby fullVisitorId -&gt; choose most frequent category\n-&gt; mean and frequency encoding -&gt; prediction\n\nI want to know which one is correct way.\n\nPlease, help me!",
      "votes": null
    },
    {
      "id": "401788",
      "postDate": "10/10/2018 17:54:59",
      "content": "<p>Hi YouHan, I think the second one makes more sense. You lose some information but at least this information makes sense. I'll take an example from the doc of LabelEncoder (I slightly changed it):\n<a href=\"http://scikit-learn.org/stable/modules/generated/sklearn.preprocessing.LabelEncoder.html\">http://scikit-learn.org/stable/modules/generated/sklearn.preprocessing.LabelEncoder.html</a></p>\n\n<pre><code>&gt;&gt;&gt; le = preprocessing.LabelEncoder()\n&gt;&gt;&gt; le.fit([\"paris\", \"paris\", \"tokyo\", \"amsterdam\"])\nLabelEncoder()\n&gt;&gt;&gt; le.transform([\"tokyo\", \"tokyo\", \"paris\",\"amsterdam\"]) \narray([2, 2, 1, 3]...)\n</code></pre>\n\n<p>If you take method (1), if the user A connected from Amsterdam and Paris you'd have two values : 3 and 1. If you take the mean() after grouping, you'd use 3+1/2 =2...  so the model will think the user is from Tokyo.</p>\n\n<p>Hope this helps.</p>",
      "rawMarkdown": "Hi YouHan, I think the second one makes more sense. You lose some information but at least this information makes sense. I'll take an example from the doc of LabelEncoder (I slightly changed it):\nhttp://scikit-learn.org/stable/modules/generated/sklearn.preprocessing.LabelEncoder.html\n\n    &gt;&gt;&gt; le = preprocessing.LabelEncoder()\n    &gt;&gt;&gt; le.fit([\"paris\", \"paris\", \"tokyo\", \"amsterdam\"])\n    LabelEncoder()\n    &gt;&gt;&gt; le.transform([\"tokyo\", \"tokyo\", \"paris\",\"amsterdam\"]) \n    array([2, 2, 1, 3]...)\n\nIf you take method (1), if the user A connected from Amsterdam and Paris you'd have two values : 3 and 1. If you take the mean() after grouping, you'd use 3+1/2 =2...  so the model will think the user is from Tokyo.\n\nHope this helps.",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 401788,
      "author_name": "areveillon",
      "author_url": "",
      "post_date": "10/10/2018 17:54:59",
      "content": "<p>Hi YouHan, I think the second one makes more sense. You lose some information but at least this information makes sense. I'll take an example from the doc of LabelEncoder (I slightly changed it):\n<a href=\"http://scikit-learn.org/stable/modules/generated/sklearn.preprocessing.LabelEncoder.html\">http://scikit-learn.org/stable/modules/generated/sklearn.preprocessing.LabelEncoder.html</a></p>\n\n<pre><code>&gt;&gt;&gt; le = preprocessing.LabelEncoder()\n&gt;&gt;&gt; le.fit([\"paris\", \"paris\", \"tokyo\", \"amsterdam\"])\nLabelEncoder()\n&gt;&gt;&gt; le.transform([\"tokyo\", \"tokyo\", \"paris\",\"amsterdam\"]) \narray([2, 2, 1, 3]...)\n</code></pre>\n\n<p>If you take method (1), if the user A connected from Amsterdam and Paris you'd have two values : 3 and 1. If you take the mean() after grouping, you'd use 3+1/2 =2...  so the model will think the user is from Tokyo.</p>\n\n<p>Hope this helps.</p>",
      "votes": null,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "401469": "I'm solving this competiton using user level prediction.\n\nFor now, I have a concern about the right use of mean and frequncy encoding.\n\nTo use user level prediction, we need to do groupby with fullVisitorId.\n\nUntil now, I've do the below procedure.\n(1)\nMean and frequency encoding for categorical feature\n-&gt; groupby fullVisitorId using mean()\n-&gt; prediction\n\n\nBut, after looking some samples, some samples have float category in specific feature\n because ,I don't know why, they have various category in the same feature.\n\nSo, I generated new category after  groupby using mean().\n\nFor now, I set the new procedure shown in below.\n\n(2)\nLabel encoding -&gt; groupby fullVisitorId -&gt; choose most frequent category\n-&gt; mean and frequency encoding -&gt; prediction\n\nI want to know which one is correct way.\n\nPlease, help me!",
    "401788": "Hi YouHan, I think the second one makes more sense. You lose some information but at least this information makes sense. I'll take an example from the doc of LabelEncoder (I slightly changed it):\nhttp://scikit-learn.org/stable/modules/generated/sklearn.preprocessing.LabelEncoder.html\n\n    &gt;&gt;&gt; le = preprocessing.LabelEncoder()\n    &gt;&gt;&gt; le.fit([\"paris\", \"paris\", \"tokyo\", \"amsterdam\"])\n    LabelEncoder()\n    &gt;&gt;&gt; le.transform([\"tokyo\", \"tokyo\", \"paris\",\"amsterdam\"]) \n    array([2, 2, 1, 3]...)\n\nIf you take method (1), if the user A connected from Amsterdam and Paris you'd have two values : 3 and 1. If you take the mean() after grouping, you'd use 3+1/2 =2...  so the model will think the user is from Tokyo.\n\nHope this helps."
  },
  "source": "meta"
}