{
  "id": 113660,
  "title": "What is the best way to storage files?",
  "url": "/competitions/ashrae-energy-prediction/discussion/113660",
  "author_name": "",
  "post_date": "2019-10-21T07:47:11.274525200Z",
  "votes": 1,
  "comment_count": 4,
  "views": 0,
  "content": "<p>Since the data that we are using is large:\ntrain (20216100, 16)\ntest (41697600, 16)</p>\n\n<p>Is there an optimal way to save the files, keeping the formats and the headers?</p>",
  "messages": [
    {
      "id": "653945",
      "postDate": "10/21/2019 07:47:11",
      "content": "<p>Since the data that we are using is large:\ntrain (20216100, 16)\ntest (41697600, 16)</p>\n\n<p>Is there an optimal way to save the files, keeping the formats and the headers?</p>",
      "rawMarkdown": "Since the data that we are using is large:\ntrain (20216100, 16)\ntest (41697600, 16)\n\nIs there an optimal way to save the files, keeping the formats and the headers?",
      "votes": null
    },
    {
      "id": "654015",
      "postDate": "10/21/2019 10:01:06",
      "content": "<p>I did some experiments and find <strong>parquet is superior both in speed and disk space</strong>.</p>\n\n<p><code>python\nimport pandas as pd\ndf = pd.read_csv('../input/train.csv.zip')\ndf.to_parquet('../data/train.parquet')\npd.read_parquet('../data/train.parquet')\n</code></p>\n\n<p>File size:\n<code>\ntrain.csv 648M,\ntrain.csv.zip 167M,\ntrain.parquet 116M.\n</code></p>\n\n<p>Reading parquet takes 829 ms ± 131 (hard disk not SSD), while original csv.zip takes 6.79 s ± 4.19 ms; 8 times faster.</p>\n\n<p>Other binary formats have similar reading speed, but the size varies:</p>\n\n<p><code>text\npickle 559M,\nHDF5 2.7G.\n</code></p>\n\n<p>Using compression with HDF5 (complevel=4) reduces size to 714M, but slows down reading speed to 2.71 s ± 3.52 ms.</p>\n\n<p>If you have sufficient disk space, any binary format (parquet, pickle, HDF5, ...) is fine. For SSD, pickle might be faster, but I heard pickle + pandas sometimes have problems when the versions change.​</p>",
      "rawMarkdown": "I did some experiments and find **parquet is superior both in speed and disk space**.\n\n```python\nimport pandas as pd\ndf = pd.read_csv('../input/train.csv.zip')\ndf.to_parquet('../data/train.parquet')\npd.read_parquet('../data/train.parquet')\n```\n\nFile size:\n```\ntrain.csv 648M,\ntrain.csv.zip 167M,\ntrain.parquet 116M.\n```\n\nReading parquet takes 829 ms ± 131 (hard disk not SSD), while original csv.zip takes 6.79 s ± 4.19 ms; 8 times faster.\n\nOther binary formats have similar reading speed, but the size varies:\n\n```text\npickle 559M,\nHDF5 2.7G.\n```\n\nUsing compression with HDF5 (complevel=4) reduces size to 714M, but slows down reading speed to 2.71 s ± 3.52 ms.\n\n\nIf you have sufficient disk space, any binary format (parquet, pickle, HDF5, ...) is fine. For SSD, pickle might be faster, but I heard pickle + pandas sometimes have problems when the versions change.​",
      "votes": null
    },
    {
      "id": "654165",
      "postDate": "10/21/2019 14:39:56",
      "content": "<p>+1</p>",
      "rawMarkdown": "1",
      "votes": null
    },
    {
      "id": "654227",
      "postDate": "10/21/2019 15:56:57",
      "content": "<p>+1</p>",
      "rawMarkdown": "1",
      "votes": null
    },
    {
      "id": "654303",
      "postDate": "10/21/2019 17:22:08",
      "content": "<p>Thanks!! After giving it a try hdf5 seems to be working better.</p>",
      "rawMarkdown": "Thanks!! After giving it a try hdf5 seems to be working better.",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 654015,
      "author_name": "junkoda",
      "author_url": "",
      "post_date": "10/21/2019 10:01:06",
      "content": "<p>I did some experiments and find <strong>parquet is superior both in speed and disk space</strong>.</p>\n\n<p><code>python\nimport pandas as pd\ndf = pd.read_csv('../input/train.csv.zip')\ndf.to_parquet('../data/train.parquet')\npd.read_parquet('../data/train.parquet')\n</code></p>\n\n<p>File size:\n<code>\ntrain.csv 648M,\ntrain.csv.zip 167M,\ntrain.parquet 116M.\n</code></p>\n\n<p>Reading parquet takes 829 ms ± 131 (hard disk not SSD), while original csv.zip takes 6.79 s ± 4.19 ms; 8 times faster.</p>\n\n<p>Other binary formats have similar reading speed, but the size varies:</p>\n\n<p><code>text\npickle 559M,\nHDF5 2.7G.\n</code></p>\n\n<p>Using compression with HDF5 (complevel=4) reduces size to 714M, but slows down reading speed to 2.71 s ± 3.52 ms.</p>\n\n<p>If you have sufficient disk space, any binary format (parquet, pickle, HDF5, ...) is fine. For SSD, pickle might be faster, but I heard pickle + pandas sometimes have problems when the versions change.​</p>",
      "votes": null,
      "replies": [
        {
          "id": 654303,
          "author_name": "cmougan",
          "author_url": "",
          "post_date": "10/21/2019 17:22:08",
          "content": "<p>Thanks!! After giving it a try hdf5 seems to be working better.</p>",
          "votes": null,
          "replies": []
        }
      ]
    },
    {
      "id": 654165,
      "author_name": "alexfaraday",
      "author_url": "",
      "post_date": "10/21/2019 14:39:56",
      "content": "<p>+1</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 654227,
      "author_name": "kaishasan",
      "author_url": "",
      "post_date": "10/21/2019 15:56:57",
      "content": "<p>+1</p>",
      "votes": null,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "653945": "Since the data that we are using is large:\ntrain (20216100, 16)\ntest (41697600, 16)\n\nIs there an optimal way to save the files, keeping the formats and the headers?",
    "654015": "I did some experiments and find **parquet is superior both in speed and disk space**.\n\n```python\nimport pandas as pd\ndf = pd.read_csv('../input/train.csv.zip')\ndf.to_parquet('../data/train.parquet')\npd.read_parquet('../data/train.parquet')\n```\n\nFile size:\n```\ntrain.csv 648M,\ntrain.csv.zip 167M,\ntrain.parquet 116M.\n```\n\nReading parquet takes 829 ms ± 131 (hard disk not SSD), while original csv.zip takes 6.79 s ± 4.19 ms; 8 times faster.\n\nOther binary formats have similar reading speed, but the size varies:\n\n```text\npickle 559M,\nHDF5 2.7G.\n```\n\nUsing compression with HDF5 (complevel=4) reduces size to 714M, but slows down reading speed to 2.71 s ± 3.52 ms.\n\n\nIf you have sufficient disk space, any binary format (parquet, pickle, HDF5, ...) is fine. For SSD, pickle might be faster, but I heard pickle + pandas sometimes have problems when the versions change.​",
    "654165": "1",
    "654227": "1",
    "654303": "Thanks!! After giving it a try hdf5 seems to be working better."
  },
  "source": "meta"
}