{
  "id": 10793,
  "title": "Strategy for Counting Instances",
  "url": "/competitions/avazu-ctr-prediction/discussion/10793",
  "author_name": "",
  "post_date": "2014-10-29T21:41:26.477Z",
  "votes": 1,
  "comment_count": 4,
  "views": 1465,
  "content": "<p>Curious how people are approaching the EDA on this file - w/o loading into main memory? Specifically,the approach used to counting the number of unique instances of a given feature value, determining how many of each feature value resides in the training data (to determine which values are 'rare') etc.</p>\n\n<p>Is the typical strategy to read in chunks of data at a time and compile counts in a dictionary?</p>",
  "messages": [
    {
      "id": "57020",
      "postDate": "10/29/2014 21:41:26",
      "content": "<p>Curious how people are approaching the EDA on this file - w/o loading into main memory? Specifically,the approach used to counting the number of unique instances of a given feature value, determining how many of each feature value resides in the training data (to determine which values are 'rare') etc.</p>\n\n<p>Is the typical strategy to read in chunks of data at a time and compile counts in a dictionary?</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "57042",
      "postDate": "10/30/2014 11:23:13",
      "content": "<p>I just did a single line by line pass of the thing and output a bunch of files listing each unique key for each feature. Even though the training file in its entirety is huge, the dictionaries fit into main memory pretty easily (the worst of it is the device_ip column with about 6 million different values). The whole thing took about 5 minutes to run, but now I can just reference those files if I need that information in the future.</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "57050",
      "postDate": "10/30/2014 16:14:47",
      "content": "<p>Other option would be to load data into an SQL database.</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "57059",
      "postDate": "10/30/2014 18:20:55",
      "content": "<p>Read them in chunks, count instances per chunk and merge them into the 'counts-so-far'. It used to be called 'keep a running tally', but now they call it 'single-threaded map-reduce'.</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "57060",
      "postDate": "10/30/2014 18:25:25",
      "content": "<p>Well, I have been able to count everything in memory using R and the data.table package, but I enjoy a PC with 16GB of RAM. My R session went up to 12GB. Once the data was loaded (quite fast indeed, a few minutes) and prepared a bit, queries were pretty fast.</p>",
      "rawMarkdown": "",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 57042,
      "author_name": "ngutten",
      "author_url": "",
      "post_date": "10/30/2014 11:23:13",
      "content": "<p>I just did a single line by line pass of the thing and output a bunch of files listing each unique key for each feature. Even though the training file in its entirety is huge, the dictionaries fit into main memory pretty easily (the worst of it is the device_ip column with about 6 million different values). The whole thing took about 5 minutes to run, but now I can just reference those files if I need that information in the future.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 57050,
      "author_name": "alexandreyc",
      "author_url": "",
      "post_date": "10/30/2014 16:14:47",
      "content": "<p>Other option would be to load data into an SQL database.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 57059,
      "author_name": "byang1",
      "author_url": "",
      "post_date": "10/30/2014 18:20:55",
      "content": "<p>Read them in chunks, count instances per chunk and merge them into the 'counts-so-far'. It used to be called 'keep a running tally', but now they call it 'single-threaded map-reduce'.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 57060,
      "author_name": "martely",
      "author_url": "",
      "post_date": "10/30/2014 18:25:25",
      "content": "<p>Well, I have been able to count everything in memory using R and the data.table package, but I enjoy a PC with 16GB of RAM. My R session went up to 12GB. Once the data was loaded (quite fast indeed, a few minutes) and prepared a bit, queries were pretty fast.</p>",
      "votes": null,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "57020": "",
    "57042": "",
    "57050": "",
    "57059": "",
    "57060": ""
  },
  "source": "meta"
}