{
  "id": 11209,
  "title": "variety has too many values",
  "url": "/competitions/avazu-ctr-prediction/discussion/11209",
  "author_name": "",
  "post_date": "2014-12-11T16:20:44.943Z",
  "votes": null,
  "comment_count": 7,
  "views": 1553,
  "content": "<p>Varieties such as site_id, site_domain,app_id,app_domain have too many values, how to deal with them.</p>",
  "messages": [
    {
      "id": "60010",
      "postDate": "12/11/2014 16:20:44",
      "content": "<p>Varieties such as site_id, site_domain,app_id,app_domain have too many values, how to deal with them.</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "60012",
      "postDate": "12/11/2014 16:22:53",
      "content": "<p>they are ok actually, compared to device id and device ip. you can always count them and throw away infrequent values.</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "60013",
      "postDate": "12/11/2014 16:28:36",
      "content": "<p>Is that means i should delete the record that device ip infrequent, or i can use frequent value replace it.</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "60016",
      "postDate": "12/11/2014 16:32:17",
      "content": "<p>[quote=Dating Data;60013]</p>\n<p>Is that means i should delete the record that device ip infrequent, or i can use frequent value replace it.</p>\n<p>[/quote]</p>\n<p>it's better to replace&nbsp;all infrequent values with&nbsp;one dummy value rather than delete the records.</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "60020",
      "postDate": "12/11/2014 16:42:21",
      "content": "<p>Can you tell more about dummy value, a value that i create causally just to avoid deleting record or others.&nbsp;I'm really a freshman.</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "60021",
      "postDate": "12/11/2014 16:50:46",
      "content": "<p>[quote=Dating Data;60020]</p>\n<p>Can you tell more about dummy value, a value that i create causally just to avoid deleting record or others.&nbsp;I'm really a freshman.</p>\n<p>[/quote]</p>\n<p>Sure, the code is something like the following, sorry for my bad English, I don't know how to say it clearer in words..</p>\n<p>import pandas as pd<br>import numpy as np<br>train=pd.read_csv('../data/train')<br>feature=train.columns.values[2:] # ignore id and click<br>for i in feature:<br>&nbsp; &nbsp; &nbsp;for j in np.unique(train[i]):<br>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp;&nbsp;print i,j,'count',train[train[i]==j].shape[0]</p>\n<p>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; if &nbsp;train[train[i]==j].shape[0]&lt;10: # let's say 10 is the threshold</p>\n<p>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;train[train[i]==j][i]='dummy' &nbsp; # replace it with any value you want</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "60023",
      "postDate": "12/11/2014 16:56:53",
      "content": "<p>Thank you very much. My English is also bad.</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "60078",
      "postDate": "12/12/2014 05:47:01",
      "content": "<p>Sorry,i have a question again.I have run the code you give me, but it takes too much time, since i just use 3 million records. Is there some ways to solve it. Besides,&nbsp;when you deal with such a big data, what do you always do to save time and memory.</p>",
      "rawMarkdown": "",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 60012,
      "author_name": "jiweiliu",
      "author_url": "",
      "post_date": "12/11/2014 16:22:53",
      "content": "<p>they are ok actually, compared to device id and device ip. you can always count them and throw away infrequent values.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 60013,
      "author_name": "datingdata",
      "author_url": "",
      "post_date": "12/11/2014 16:28:36",
      "content": "<p>Is that means i should delete the record that device ip infrequent, or i can use frequent value replace it.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 60016,
      "author_name": "jiweiliu",
      "author_url": "",
      "post_date": "12/11/2014 16:32:17",
      "content": "<p>[quote=Dating Data;60013]</p>\n<p>Is that means i should delete the record that device ip infrequent, or i can use frequent value replace it.</p>\n<p>[/quote]</p>\n<p>it's better to replace&nbsp;all infrequent values with&nbsp;one dummy value rather than delete the records.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 60020,
      "author_name": "datingdata",
      "author_url": "",
      "post_date": "12/11/2014 16:42:21",
      "content": "<p>Can you tell more about dummy value, a value that i create causally just to avoid deleting record or others.&nbsp;I'm really a freshman.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 60021,
      "author_name": "jiweiliu",
      "author_url": "",
      "post_date": "12/11/2014 16:50:46",
      "content": "<p>[quote=Dating Data;60020]</p>\n<p>Can you tell more about dummy value, a value that i create causally just to avoid deleting record or others.&nbsp;I'm really a freshman.</p>\n<p>[/quote]</p>\n<p>Sure, the code is something like the following, sorry for my bad English, I don't know how to say it clearer in words..</p>\n<p>import pandas as pd<br>import numpy as np<br>train=pd.read_csv('../data/train')<br>feature=train.columns.values[2:] # ignore id and click<br>for i in feature:<br>&nbsp; &nbsp; &nbsp;for j in np.unique(train[i]):<br>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp;&nbsp;print i,j,'count',train[train[i]==j].shape[0]</p>\n<p>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; if &nbsp;train[train[i]==j].shape[0]&lt;10: # let's say 10 is the threshold</p>\n<p>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;train[train[i]==j][i]='dummy' &nbsp; # replace it with any value you want</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 60023,
      "author_name": "datingdata",
      "author_url": "",
      "post_date": "12/11/2014 16:56:53",
      "content": "<p>Thank you very much. My English is also bad.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 60078,
      "author_name": "datingdata",
      "author_url": "",
      "post_date": "12/12/2014 05:47:01",
      "content": "<p>Sorry,i have a question again.I have run the code you give me, but it takes too much time, since i just use 3 million records. Is there some ways to solve it. Besides,&nbsp;when you deal with such a big data, what do you always do to save time and memory.</p>",
      "votes": null,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "60010": "",
    "60012": "",
    "60013": "",
    "60016": "",
    "60020": "",
    "60021": "",
    "60023": "",
    "60078": ""
  },
  "source": "meta"
}