{
  "id": 11331,
  "title": "Feature vector dimension",
  "url": "/competitions/avazu-ctr-prediction/discussion/11331",
  "author_name": "",
  "post_date": "2014-12-23T22:56:55.570Z",
  "votes": 8,
  "comment_count": 2,
  "views": 1639,
  "content": "<p>FYI., here are # of unique&nbsp;values for each feature in the training set. A&nbsp;feature vector built using&nbsp;each of the unique values would be 9,449,205&nbsp;dimensional. To improve&nbsp;performance of scanning this vector, &quot;feature hashing&quot; will be very useful.</p>\n<p>c1: 7<br>banner_pos: 7<br>site_id: 4737<br>site_domain: 7745<br>site_category: 26<br>app_id: 8552<br>app_domain: 559<br>app_category: 36<br>device_id: 2686408<br>device_ip: 6729486<br>device_model: 8251<br>device_type: 5<br>device_conn_type: 4<br>c14: 2626<br>c15: 8<br>c16: 9<br>c17: 435<br>c18: 4<br>c19: 68<br>c20: 172<br>c21: 60</p>",
  "messages": [
    {
      "id": "61790",
      "postDate": "12/23/2014 22:56:55",
      "content": "<p>FYI., here are # of unique&nbsp;values for each feature in the training set. A&nbsp;feature vector built using&nbsp;each of the unique values would be 9,449,205&nbsp;dimensional. To improve&nbsp;performance of scanning this vector, &quot;feature hashing&quot; will be very useful.</p>\n<p>c1: 7<br>banner_pos: 7<br>site_id: 4737<br>site_domain: 7745<br>site_category: 26<br>app_id: 8552<br>app_domain: 559<br>app_category: 36<br>device_id: 2686408<br>device_ip: 6729486<br>device_model: 8251<br>device_type: 5<br>device_conn_type: 4<br>c14: 2626<br>c15: 8<br>c16: 9<br>c17: 435<br>c18: 4<br>c19: 68<br>c20: 172<br>c21: 60</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "61836",
      "postDate": "12/24/2014 18:36:28",
      "content": "<p>Many of these feature+value pairs are of pretty limited utility since the value never appears in the test data and the features are mostly categorical. I don't have the numbers on hand right now, but you can reduce this 9 mill by only looking at train/test shared feature+values.</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "61842",
      "postDate": "12/24/2014 20:35:43",
      "content": "<p>I don't think it's a good idea to only consider features &amp; values at the intersection on train + test data.&nbsp;</p>\n<p>device_id and device_ip contribute vast majority of the features (over 9 mill)&nbsp;but&nbsp;I doubt if these&nbsp;features can greatly help a model's prediction accuracy. Removing them from the feature vector (agree, there is some loss of data) will greatly decrease the feature vector size.</p>\n<p>I'll compare model that uses these features against one that doesn't and see how prediction accuracy varies.</p>",
      "rawMarkdown": "",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 61836,
      "author_name": "liamhuber",
      "author_url": "",
      "post_date": "12/24/2014 18:36:28",
      "content": "<p>Many of these feature+value pairs are of pretty limited utility since the value never appears in the test data and the features are mostly categorical. I don't have the numbers on hand right now, but you can reduce this 9 mill by only looking at train/test shared feature+values.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 61842,
      "author_name": "hogwild",
      "author_url": "",
      "post_date": "12/24/2014 20:35:43",
      "content": "<p>I don't think it's a good idea to only consider features &amp; values at the intersection on train + test data.&nbsp;</p>\n<p>device_id and device_ip contribute vast majority of the features (over 9 mill)&nbsp;but&nbsp;I doubt if these&nbsp;features can greatly help a model's prediction accuracy. Removing them from the feature vector (agree, there is some loss of data) will greatly decrease the feature vector size.</p>\n<p>I'll compare model that uses these features against one that doesn't and see how prediction accuracy varies.</p>",
      "votes": null,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "61790": "",
    "61836": "",
    "61842": ""
  },
  "source": "meta"
}