{
  "id": 113207,
  "title": "Large data workflow",
  "url": "/competitions/ashrae-energy-prediction/discussion/113207",
  "author_name": "maka",
  "post_date": "2019-10-17T18:44:10.499000",
  "votes": 2,
  "comment_count": 0,
  "views": 0,
  "content": "<p>Hi, how are you dealing with generating features and such, given the large size of the data?</p>\n\n<p>I ended up with a workflow of doing some basic datetime manipulations and feature-generation in pandas for the individual files. Then using dask to do joins. Dataset is then saved in a hdf5 file.</p>\n\n<p>Seems to work for now. What is your workflow like?</p>",
  "messages": [
    {
      "id": 651649,
      "postDate": "2019-10-17T18:44:10.500Z",
      "content": "<p>Hi, how are you dealing with generating features and such, given the large size of the data?</p>\n\n<p>I ended up with a workflow of doing some basic datetime manipulations and feature-generation in pandas for the individual files. Then using dask to do joins. Dataset is then saved in a hdf5 file.</p>\n\n<p>Seems to work for now. What is your workflow like?</p>",
      "rawMarkdown": "Hi, how are you dealing with generating features and such, given the large size of the data?\n\nI ended up with a workflow of doing some basic datetime manipulations and feature-generation in pandas for the individual files. Then using dask to do joins. Dataset is then saved in a hdf5 file.\n\nSeems to work for now. What is your workflow like?\n\n",
      "votes": 2
    }
  ],
  "comments": [],
  "raw_markdown_by_id": {
    "651649": "Hi, how are you dealing with generating features and such, given the large size of the data?\n\nI ended up with a workflow of doing some basic datetime manipulations and feature-generation in pandas for the individual files. Then using dask to do joins. Dataset is then saved in a hdf5 file.\n\nSeems to work for now. What is your workflow like?\n\n"
  }
}