{
  "id": 23058,
  "title": "This dataset have too many columns for HDF5. What else to use?",
  "url": "/competitions/bosch-production-line-performance/discussion/23058",
  "author_name": "",
  "post_date": "2016-08-23T20:30:16.470Z",
  "votes": 3,
  "comment_count": 5,
  "views": 1382,
  "content": "<p>I almost always convert initial csv files to pandas HDF5 database which allows for a much quicker load time (something like an order of magnitude quicker). But it appears that this particular dataset has way too many columns for an HDF5 file, which has a limitation of 64k for the length of all column names (Really? 64k in 2016? This is not Comodore 64 we're talking about! This is not even a big data for gods sake!). So the question is what to replace HDF5 with? sql and csv are much slower and less efficient. Split table in parts? Not very elegant and will require several reads and concatenates. Any ideas? Thanks!</p>",
  "messages": [
    {
      "id": "132246",
      "postDate": "08/23/2016 20:30:16",
      "content": "<p>I almost always convert initial csv files to pandas HDF5 database which allows for a much quicker load time (something like an order of magnitude quicker). But it appears that this particular dataset has way too many columns for an HDF5 file, which has a limitation of 64k for the length of all column names (Really? 64k in 2016? This is not Comodore 64 we're talking about! This is not even a big data for gods sake!). So the question is what to replace HDF5 with? sql and csv are much slower and less efficient. Split table in parts? Not very elegant and will require several reads and concatenates. Any ideas? Thanks!</p>",
      "rawMarkdown": "I almost always convert initial csv files to pandas HDF5 database which allows for a much quicker load time (something like an order of magnitude quicker). But it appears that this particular dataset has way too many columns for an HDF5 file, which has a limitation of 64k for the length of all column names (Really? 64k in 2016? This is not Comodore 64 we're talking about! This is not even a big data for gods sake!). So the question is what to replace HDF5 with? sql and csv are much slower and less efficient. Split table in parts? Not very elegant and will require several reads and concatenates. Any ideas? Thanks!",
      "votes": null
    },
    {
      "id": "132279",
      "postDate": "08/24/2016 08:00:20",
      "content": "<p>Try apache spark</p>",
      "rawMarkdown": "Try apache spark",
      "votes": null
    },
    {
      "id": "132344",
      "postDate": "08/24/2016 17:57:15",
      "content": "<p>Mmmm... to me, Apache Spark is a totally different workflow compared to pandas.\nI would like to be able to save and load pandas dataframes quickly in a binary format so that sparsity of data is preserved (not the case with csv).</p>",
      "rawMarkdown": "Mmmm... to me, Apache Spark is a totally different workflow compared to pandas.\r\nI would like to be able to save and load pandas dataframes quickly in a binary format so that sparsity of data is preserved (not the case with csv).",
      "votes": null
    },
    {
      "id": "132348",
      "postDate": "08/24/2016 18:29:51",
      "content": "<p>I put the column names in a separate dataset, use numeric types for categorical labels vs. strings,  and use numerical column names without problems when using HDF5.</p>",
      "rawMarkdown": "I put the column names in a separate dataset, use numeric types for categorical labels vs. strings,  and use numerical column names without problems when using HDF5.",
      "votes": null
    },
    {
      "id": "132353",
      "postDate": "08/24/2016 18:55:30",
      "content": "<p>That's an interesting idea!\nDid you convert categorical variables to pandas categoricals? Sparse?\nWith Categoricals pandas gives me a warning that &gt;512 columns will give performance issues with HDF5.\nAnd I can't convert categoricals to sparse as it throws an error...</p>",
      "rawMarkdown": "That's an interesting idea!\r\nDid you convert categorical variables to pandas categoricals? Sparse?\r\nWith Categoricals pandas gives me a warning that >512 columns will give performance issues with HDF5.\r\nAnd I can't convert categoricals to sparse as it throws an error...",
      "votes": null
    },
    {
      "id": "132357",
      "postDate": "08/24/2016 20:22:25",
      "content": "<p>I'm using h5py and line-by-line parsing of the CSV.  I create a dense array with sentinel 0-values as 'missing' and the category label modulo a small prime as the feature.  E.g. T145-&gt;145-&gt;145 % 37-&gt;34 as the feature.  I'm probably throwing something important away, though, because it looks like some of these might be bitmasks or special states.  </p>\n\n<p>My model then learns 3-dimensional embeddings from these features.  It seems to work reasonably well, but I'm not terribly high up the leaderboard either.</p>\n\n<p>I haven't noticed performance problems, but I'm not using any chunking or other advanced features.  The data ends up fitting entirely inside of the computer's block cache too so your mileage may vary.</p>",
      "rawMarkdown": "I'm using h5py and line-by-line parsing of the CSV.  I create a dense array with sentinel 0-values as 'missing' and the category label modulo a small prime as the feature.  E.g. T145->145->145 % 37->34 as the feature.  I'm probably throwing something important away, though, because it looks like some of these might be bitmasks or special states.  \r\n\r\nMy model then learns 3-dimensional embeddings from these features.  It seems to work reasonably well, but I'm not terribly high up the leaderboard either.\r\n\r\nI haven't noticed performance problems, but I'm not using any chunking or other advanced features.  The data ends up fitting entirely inside of the computer's block cache too so your mileage may vary.",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 132279,
      "author_name": "",
      "author_url": "",
      "post_date": "08/24/2016 08:00:20",
      "content": "<p>Try apache spark</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 132344,
      "author_name": "sakvaua",
      "author_url": "",
      "post_date": "08/24/2016 17:57:15",
      "content": "<p>Mmmm... to me, Apache Spark is a totally different workflow compared to pandas.\nI would like to be able to save and load pandas dataframes quickly in a binary format so that sparsity of data is preserved (not the case with csv).</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 132348,
      "author_name": "mgranberry",
      "author_url": "",
      "post_date": "08/24/2016 18:29:51",
      "content": "<p>I put the column names in a separate dataset, use numeric types for categorical labels vs. strings,  and use numerical column names without problems when using HDF5.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 132353,
      "author_name": "sakvaua",
      "author_url": "",
      "post_date": "08/24/2016 18:55:30",
      "content": "<p>That's an interesting idea!\nDid you convert categorical variables to pandas categoricals? Sparse?\nWith Categoricals pandas gives me a warning that &gt;512 columns will give performance issues with HDF5.\nAnd I can't convert categoricals to sparse as it throws an error...</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 132357,
      "author_name": "mgranberry",
      "author_url": "",
      "post_date": "08/24/2016 20:22:25",
      "content": "<p>I'm using h5py and line-by-line parsing of the CSV.  I create a dense array with sentinel 0-values as 'missing' and the category label modulo a small prime as the feature.  E.g. T145-&gt;145-&gt;145 % 37-&gt;34 as the feature.  I'm probably throwing something important away, though, because it looks like some of these might be bitmasks or special states.  </p>\n\n<p>My model then learns 3-dimensional embeddings from these features.  It seems to work reasonably well, but I'm not terribly high up the leaderboard either.</p>\n\n<p>I haven't noticed performance problems, but I'm not using any chunking or other advanced features.  The data ends up fitting entirely inside of the computer's block cache too so your mileage may vary.</p>",
      "votes": null,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "132246": "I almost always convert initial csv files to pandas HDF5 database which allows for a much quicker load time (something like an order of magnitude quicker). But it appears that this particular dataset has way too many columns for an HDF5 file, which has a limitation of 64k for the length of all column names (Really? 64k in 2016? This is not Comodore 64 we're talking about! This is not even a big data for gods sake!). So the question is what to replace HDF5 with? sql and csv are much slower and less efficient. Split table in parts? Not very elegant and will require several reads and concatenates. Any ideas? Thanks!",
    "132279": "Try apache spark",
    "132344": "Mmmm... to me, Apache Spark is a totally different workflow compared to pandas.\r\nI would like to be able to save and load pandas dataframes quickly in a binary format so that sparsity of data is preserved (not the case with csv).",
    "132348": "I put the column names in a separate dataset, use numeric types for categorical labels vs. strings,  and use numerical column names without problems when using HDF5.",
    "132353": "That's an interesting idea!\r\nDid you convert categorical variables to pandas categoricals? Sparse?\r\nWith Categoricals pandas gives me a warning that >512 columns will give performance issues with HDF5.\r\nAnd I can't convert categoricals to sparse as it throws an error...",
    "132357": "I'm using h5py and line-by-line parsing of the CSV.  I create a dense array with sentinel 0-values as 'missing' and the category label modulo a small prime as the feature.  E.g. T145->145->145 % 37->34 as the feature.  I'm probably throwing something important away, though, because it looks like some of these might be bitmasks or special states.  \r\n\r\nMy model then learns 3-dimensional embeddings from these features.  It seems to work reasonably well, but I'm not terribly high up the leaderboard either.\r\n\r\nI haven't noticed performance problems, but I'm not using any chunking or other advanced features.  The data ends up fitting entirely inside of the computer's block cache too so your mileage may vary."
  },
  "source": "meta"
}