{
  "id": 21873,
  "title": "fastest way to write the csv",
  "url": "/competitions/facebook-v-predicting-check-ins/discussion/21873",
  "author_name": "",
  "post_date": "2016-06-24T18:03:16.293Z",
  "votes": null,
  "comment_count": 7,
  "views": 696,
  "content": "<p>Pandas dataframe is a nice way to write the csv file (attached).\nBut can we beat that?</p>\n\n<p>Using a list is a very smart way...</p>\n\n<p>df = pd.DataFrame(preds, dtype=str, columns=['c1', 'c2', 'c3'])</p>\n\n<p>ds_sub = df.c1.str.cat([df.c2, df.c3], sep=' ')</p>\n\n<p>ds_sub.name = 'place_id'</p>\n\n<p>ds_sub.to_csv('submission_sample.csv', index=True, header=True, index_label='row_id')</p>\n\n<p>But can we beat this?\nUsing less memory...</p>",
  "messages": [
    {
      "id": "125020",
      "postDate": "06/24/2016 18:03:16",
      "content": "<p>Pandas dataframe is a nice way to write the csv file (attached).\nBut can we beat that?</p>\n\n<p>Using a list is a very smart way...</p>\n\n<p>df = pd.DataFrame(preds, dtype=str, columns=['c1', 'c2', 'c3'])</p>\n\n<p>ds_sub = df.c1.str.cat([df.c2, df.c3], sep=' ')</p>\n\n<p>ds_sub.name = 'place_id'</p>\n\n<p>ds_sub.to_csv('submission_sample.csv', index=True, header=True, index_label='row_id')</p>\n\n<p>But can we beat this?\nUsing less memory...</p>",
      "rawMarkdown": "Pandas dataframe is a nice way to write the csv file (attached).\r\nBut can we beat that?\r\n\r\nUsing a list is a very smart way...\r\n\r\ndf = pd.DataFrame(preds, dtype=str, columns=['c1', 'c2', 'c3'])\r\n\r\nds_sub = df.c1.str.cat([df.c2, df.c3], sep=' ')\r\n\r\nds_sub.name = 'place_id'\r\n\r\nds_sub.to_csv('submission_sample.csv', index=True, header=True, index_label='row_id')\r\n\r\nBut can we beat this?\r\nUsing less memory...",
      "votes": null
    },
    {
      "id": "125027",
      "postDate": "06/24/2016 19:39:10",
      "content": "<p>I did some testing on this, and I haven't found a faster way. Pandas is really good. </p>",
      "rawMarkdown": "I did some testing on this, and I haven't found a faster way. Pandas is really good.",
      "votes": null
    },
    {
      "id": "125037",
      "postDate": "06/24/2016 21:01:24",
      "content": "<p>@Michael Hartman: Thank you for your kind reply. Really appreciated.</p>",
      "rawMarkdown": "Michael Hartman: Thank you for your kind reply. Really appreciated.",
      "votes": null
    },
    {
      "id": "125047",
      "postDate": "06/25/2016 01:56:32",
      "content": "<p>I found reading and writing like this very fast (almost instantaneously). Hope it helps</p>",
      "rawMarkdown": "I found reading and writing like this very fast (almost instantaneously). Hope it helps",
      "votes": null
    },
    {
      "id": "125100",
      "postDate": "06/26/2016 02:50:41",
      "content": "<p>It seems possible to save 4 seconds... with a simple code... using less memory...\nSee:\n<a href=\"https://www.kaggle.com/drarfc/facebook-v-predicting-check-ins/fastest-way-to-write-the-csv\">https://www.kaggle.com/drarfc/facebook-v-predicting-check-ins/fastest-way-to-write-the-csv</a></p>",
      "rawMarkdown": "It seems possible to save 4 seconds... with a simple code... using less memory...\r\nSee:\r\nhttps://www.kaggle.com/drarfc/facebook-v-predicting-check-ins/fastest-way-to-write-the-csv",
      "votes": null
    },
    {
      "id": "125309",
      "postDate": "06/28/2016 14:23:52",
      "content": "<p>I have tested a new code and it is possible to save 20 seconds, cutting the writing time by half:</p>\n\n<pre><code>out = open('sample_submission.csv', &quot;w&quot;)\n\nout.write(&quot;row_id,place_id\\n&quot;)\n\nrows = ['']*8607230\n\nfor num in range(0,8607230):\n\n    rows[num]='%d,%d %d %d\\n' % (num,preds[num,0],preds[num,1],preds[num,2])\n\nout.writelines(rows)\n\nout.close()\n</code></pre>",
      "rawMarkdown": "I have tested a new code and it is possible to save 20 seconds, cutting the writing time by half:\r\n\r\n    out = open('sample_submission.csv', \"w\")\r\n\r\n    out.write(\"row_id,place_id\\n\")\r\n\r\n    rows = ['']*8607230\r\n\r\n    for num in range(0,8607230):\r\n\r\n        rows[num]='%d,%d %d %d\\n' % (num,preds[num,0],preds[num,1],preds[num,2])\r\n\r\n    out.writelines(rows)\r\n\r\n    out.close()",
      "votes": null
    },
    {
      "id": "125335",
      "postDate": "06/28/2016 18:05:58",
      "content": "<p>Seriously guys, save 4-20 seconds on 10-30 hours model calculation? That's against python code of conduct! I have to report you to the Elders of the Python!</p>",
      "rawMarkdown": "Seriously guys, save 4-20 seconds on 10-30 hours model calculation? That's against python code of conduct! I have to report you to the Elders of the Python!",
      "votes": null
    },
    {
      "id": "125353",
      "postDate": "06/28/2016 21:07:07",
      "content": "<p>:) Seriously DennisSakva, If the competition is about the hardware with 10-30 hours model calculation and 32GB Memory or more... you may be correct.\nIf you have hardware restrictions...  1200 seconds and 8GB memory... in this case 20 seconds matter. Do you agree?</p>",
      "rawMarkdown": ":) Seriously DennisSakva, If the competition is about the hardware with 10-30 hours model calculation and 32GB Memory or more... you may be correct.\r\nIf you have hardware restrictions...  1200 seconds and 8GB memory... in this case 20 seconds matter. Do you agree?",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 125027,
      "author_name": "zeroblue",
      "author_url": "",
      "post_date": "06/24/2016 19:39:10",
      "content": "<p>I did some testing on this, and I haven't found a faster way. Pandas is really good. </p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 125037,
      "author_name": "drarfc",
      "author_url": "",
      "post_date": "06/24/2016 21:01:24",
      "content": "<p>@Michael Hartman: Thank you for your kind reply. Really appreciated.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 125047,
      "author_name": "ascending",
      "author_url": "",
      "post_date": "06/25/2016 01:56:32",
      "content": "<p>I found reading and writing like this very fast (almost instantaneously). Hope it helps</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 125100,
      "author_name": "drarfc",
      "author_url": "",
      "post_date": "06/26/2016 02:50:41",
      "content": "<p>It seems possible to save 4 seconds... with a simple code... using less memory...\nSee:\n<a href=\"https://www.kaggle.com/drarfc/facebook-v-predicting-check-ins/fastest-way-to-write-the-csv\">https://www.kaggle.com/drarfc/facebook-v-predicting-check-ins/fastest-way-to-write-the-csv</a></p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 125309,
      "author_name": "drarfc",
      "author_url": "",
      "post_date": "06/28/2016 14:23:52",
      "content": "<p>I have tested a new code and it is possible to save 20 seconds, cutting the writing time by half:</p>\n\n<pre><code>out = open('sample_submission.csv', &quot;w&quot;)\n\nout.write(&quot;row_id,place_id\\n&quot;)\n\nrows = ['']*8607230\n\nfor num in range(0,8607230):\n\n    rows[num]='%d,%d %d %d\\n' % (num,preds[num,0],preds[num,1],preds[num,2])\n\nout.writelines(rows)\n\nout.close()\n</code></pre>",
      "votes": null,
      "replies": []
    },
    {
      "id": 125335,
      "author_name": "sakvaua",
      "author_url": "",
      "post_date": "06/28/2016 18:05:58",
      "content": "<p>Seriously guys, save 4-20 seconds on 10-30 hours model calculation? That's against python code of conduct! I have to report you to the Elders of the Python!</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 125353,
      "author_name": "drarfc",
      "author_url": "",
      "post_date": "06/28/2016 21:07:07",
      "content": "<p>:) Seriously DennisSakva, If the competition is about the hardware with 10-30 hours model calculation and 32GB Memory or more... you may be correct.\nIf you have hardware restrictions...  1200 seconds and 8GB memory... in this case 20 seconds matter. Do you agree?</p>",
      "votes": null,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "125020": "Pandas dataframe is a nice way to write the csv file (attached).\r\nBut can we beat that?\r\n\r\nUsing a list is a very smart way...\r\n\r\ndf = pd.DataFrame(preds, dtype=str, columns=['c1', 'c2', 'c3'])\r\n\r\nds_sub = df.c1.str.cat([df.c2, df.c3], sep=' ')\r\n\r\nds_sub.name = 'place_id'\r\n\r\nds_sub.to_csv('submission_sample.csv', index=True, header=True, index_label='row_id')\r\n\r\nBut can we beat this?\r\nUsing less memory...",
    "125027": "I did some testing on this, and I haven't found a faster way. Pandas is really good.",
    "125037": "Michael Hartman: Thank you for your kind reply. Really appreciated.",
    "125047": "I found reading and writing like this very fast (almost instantaneously). Hope it helps",
    "125100": "It seems possible to save 4 seconds... with a simple code... using less memory...\r\nSee:\r\nhttps://www.kaggle.com/drarfc/facebook-v-predicting-check-ins/fastest-way-to-write-the-csv",
    "125309": "I have tested a new code and it is possible to save 20 seconds, cutting the writing time by half:\r\n\r\n    out = open('sample_submission.csv', \"w\")\r\n\r\n    out.write(\"row_id,place_id\\n\")\r\n\r\n    rows = ['']*8607230\r\n\r\n    for num in range(0,8607230):\r\n\r\n        rows[num]='%d,%d %d %d\\n' % (num,preds[num,0],preds[num,1],preds[num,2])\r\n\r\n    out.writelines(rows)\r\n\r\n    out.close()",
    "125335": "Seriously guys, save 4-20 seconds on 10-30 hours model calculation? That's against python code of conduct! I have to report you to the Elders of the Python!",
    "125353": ":) Seriously DennisSakva, If the competition is about the hardware with 10-30 hours model calculation and 32GB Memory or more... you may be correct.\r\nIf you have hardware restrictions...  1200 seconds and 8GB memory... in this case 20 seconds matter. Do you agree?"
  },
  "source": "meta"
}