{
  "id": 5073,
  "title": "Fast access for netCDF4?",
  "url": "/competitions/ams-2014-solar-energy-prediction-contest/discussion/5073",
  "author_name": "",
  "post_date": "2013-07-11T23:24:42.847Z",
  "votes": null,
  "comment_count": 4,
  "views": 2170,
  "content": "<p>Hi! I haven't used netCDF4 before. I'm doing it like this in Python:</p>\r\n<p>import netCDF4</p>\r\n<p>data = netCDF4.Dataset('apcp_sfc_latlon_subset_19940101_20071231.nc')</p>\r\n<p>data.variables['Total_precipitation'][0][0][0][0][0]</p>\r\n<p>That will give me one of the values, which is great. But it's super slow to access things this way. Untenably slow. Like nearly half a second. Ridiculous.</p>\r\n<p>Is there a good way to get quicker random access to these guys? Is there something simple that will pull it all into memory for me or something like that? Are people loading everything into a database to achieve this purpose? Other thoughts?</p>",
  "messages": [
    {
      "id": "27038",
      "postDate": "07/11/2013 23:24:42",
      "content": "<p>Hi! I haven't used netCDF4 before. I'm doing it like this in Python:</p>\r\n<p>import netCDF4</p>\r\n<p>data = netCDF4.Dataset('apcp_sfc_latlon_subset_19940101_20071231.nc')</p>\r\n<p>data.variables['Total_precipitation'][0][0][0][0][0]</p>\r\n<p>That will give me one of the values, which is great. But it's super slow to access things this way. Untenably slow. Like nearly half a second. Ridiculous.</p>\r\n<p>Is there a good way to get quicker random access to these guys? Is there something simple that will pull it all into memory for me or something like that? Are people loading everything into a database to achieve this purpose? Other thoughts?</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "27041",
      "postDate": "07/12/2013 00:48:11",
      "content": "<p>Ah okay, never mind; it's easy to copy into memory the standard way with [:]...</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "27135",
      "postDate": "07/13/2013 15:46:59",
      "content": "<p>Are you able to fit all 15 big array into python memory?</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "27236",
      "postDate": "07/15/2013 15:17:55",
      "content": "<p>Yup. I've only been using the training ones, but it hasn't been an issue.</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "27544",
      "postDate": "07/23/2013 11:45:32",
      "content": "<p>The problem is that the file has only an one big chunk of commpressed data. Therefore each time you need to acces one element, needs to inflate the whole dataset.</p>\n<p>Regards</p>\n<p>I.M.</p>",
      "rawMarkdown": "",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 27041,
      "author_name": "ajschumacher",
      "author_url": "",
      "post_date": "07/12/2013 00:48:11",
      "content": "<p>Ah okay, never mind; it's easy to copy into memory the standard way with [:]...</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 27135,
      "author_name": "herimanitra",
      "author_url": "",
      "post_date": "07/13/2013 15:46:59",
      "content": "<p>Are you able to fit all 15 big array into python memory?</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 27236,
      "author_name": "ajschumacher",
      "author_url": "",
      "post_date": "07/15/2013 15:17:55",
      "content": "<p>Yup. I've only been using the training ones, but it hasn't been an issue.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 27544,
      "author_name": "inigomontoya0",
      "author_url": "",
      "post_date": "07/23/2013 11:45:32",
      "content": "<p>The problem is that the file has only an one big chunk of commpressed data. Therefore each time you need to acces one element, needs to inflate the whole dataset.</p>\n<p>Regards</p>\n<p>I.M.</p>",
      "votes": null,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "27038": "",
    "27041": "",
    "27135": "",
    "27236": "",
    "27544": ""
  },
  "source": "meta"
}