{
  "id": 5526,
  "title": "Read NETCDF files using C API",
  "url": "/competitions/ams-2014-solar-energy-prediction-contest/discussion/5526",
  "author_name": "",
  "post_date": "2013-08-24T19:55:46.610Z",
  "votes": null,
  "comment_count": 1,
  "views": 892,
  "content": "<p>Hi,</p>\n<p>Did anyone tried reading training data with C API functions? I am reading record by record from each dataset, by calling&nbsp;nc_get_vara_float with fixed time and ensemble, and with fhour, lat, lon spanning full extents (5, 9, 16). Each function call reads 3kb of contigues data, and these chunks are read consequently.</p>\n<p>The performance is surprisingy very low; it will take half a day to read all the files this way. The problem is present on both Win64 and Linux.</p>\n<p>Thanks for any ideas,</p>\n<p>Max.</p>\n<p>&nbsp;</p>",
  "messages": [
    {
      "id": "29395",
      "postDate": "08/24/2013 19:55:46",
      "content": "<p>Hi,</p>\n<p>Did anyone tried reading training data with C API functions? I am reading record by record from each dataset, by calling&nbsp;nc_get_vara_float with fixed time and ensemble, and with fhour, lat, lon spanning full extents (5, 9, 16). Each function call reads 3kb of contigues data, and these chunks are read consequently.</p>\n<p>The performance is surprisingy very low; it will take half a day to read all the files this way. The problem is present on both Win64 and Linux.</p>\n<p>Thanks for any ideas,</p>\n<p>Max.</p>\n<p>&nbsp;</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "29432",
      "postDate": "08/25/2013 09:30:11",
      "content": "<p>Okay, fixed that with loading the whole variable into the memory. 2GB required for all the vars. It is not a scalable solution, so I am still interested in what is wrong with&nbsp;nc_get_vara_float.</p>\n<p>Best reagrds,</p>\n<p>Max.</p>",
      "rawMarkdown": "",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 29432,
      "author_name": "milakov",
      "author_url": "",
      "post_date": "08/25/2013 09:30:11",
      "content": "<p>Okay, fixed that with loading the whole variable into the memory. 2GB required for all the vars. It is not a scalable solution, so I am still interested in what is wrong with&nbsp;nc_get_vara_float.</p>\n<p>Best reagrds,</p>\n<p>Max.</p>",
      "votes": null,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "29395": "",
    "29432": ""
  },
  "source": "meta"
}