{
  "id": 114121,
  "title": "Submission in rar fails?",
  "url": "/competitions/ashrae-energy-prediction/discussion/114121",
  "author_name": "",
  "post_date": "2019-10-24T12:08:49.168279300Z",
  "votes": 1,
  "comment_count": 3,
  "views": 0,
  "content": "<p>Hi all,</p>\n\n<p>Sorry if this is a stupid question, but I am quite new to this. I tried to make a submission via a rar file (since the csv is so large). But when I tried to score the submssion I got an error message:\n Failed\n ERROR: Column 'Rar! 6s'was not expected (Line 1, Column 1)\n ERROR: Required column 'row_id' could not be found\n ERROR: Required column 'meter_reading' could not be found</p>\n\n<p>When I subsequently submitted the original CSV file, everything worked just fine.</p>\n\n<p>Is there an extra step I should take when submitting a rar file? \nOr can I better use ZIP files?</p>\n\n<p>Regards,</p>\n\n<p>Ellen</p>",
  "messages": [
    {
      "id": "656570",
      "postDate": "10/24/2019 12:08:49",
      "content": "<p>Hi all,</p>\n\n<p>Sorry if this is a stupid question, but I am quite new to this. I tried to make a submission via a rar file (since the csv is so large). But when I tried to score the submssion I got an error message:\n Failed\n ERROR: Column 'Rar! 6s'was not expected (Line 1, Column 1)\n ERROR: Required column 'row_id' could not be found\n ERROR: Required column 'meter_reading' could not be found</p>\n\n<p>When I subsequently submitted the original CSV file, everything worked just fine.</p>\n\n<p>Is there an extra step I should take when submitting a rar file? \nOr can I better use ZIP files?</p>\n\n<p>Regards,</p>\n\n<p>Ellen</p>",
      "rawMarkdown": "Hi all,\n\nSorry if this is a stupid question, but I am quite new to this. I tried to make a submission via a rar file (since the csv is so large). But when I tried to score the submssion I got an error message:\n Failed\n ERROR: Column 'Rar! 6s'was not expected (Line 1, Column 1)\n ERROR: Required column 'row_id' could not be found\n ERROR: Required column 'meter_reading' could not be found\n\nWhen I subsequently submitted the original CSV file, everything worked just fine.\n\nIs there an extra step I should take when submitting a rar file? \nOr can I better use ZIP files?\n\nRegards,\n\nEllen",
      "votes": null
    },
    {
      "id": "656573",
      "postDate": "10/24/2019 12:11:44",
      "content": "<p><code>sub.to_csv(FULL_PATH</code> + 'subs/submission_0.1.gz', index = False, compression= 'gzip')</p>",
      "rawMarkdown": "`sub.to_csv(FULL_PATH` + 'subs/submission_0.1.gz', index = False, compression= 'gzip')",
      "votes": null
    },
    {
      "id": "656764",
      "postDate": "10/24/2019 15:38:40",
      "content": "<p>Yes, as Oleg said, writing zip or gz file is very easy with python + pandas, and I recommend that you use those formats, too. You just write .zip or .gz at the end of your file name in pandas DataFrame.to_csv() then the library automatically creates a compressed file. You can submit the compressed file as usual.</p>\n\n<p>RAR or 7z file could be smaller than zip/gz, but I didn't see a dramatic difference when I applied to train.csv.</p>\n\n<p><code>\ntrain.csv.zip  120M\ntrain.rar        110M\ntrain.7z          80M\n</code></p>\n\n<p>I got the same error with rar file using Ubuntu, while 7z worked.</p>\n\n<p><code>\nsudo apt install p7zip-full p7zip-rar\n7z a submission.7z submission.csv\n</code></p>\n\n<p>Summary: Use zip or gz for simplicity. You can use 7z if file size is crucial. I don't know how rar file should be created, maybe depends on the software for compression.</p>",
      "rawMarkdown": "Yes, as Oleg said, writing zip or gz file is very easy with python + pandas, and I recommend that you use those formats, too. You just write .zip or .gz at the end of your file name in pandas DataFrame.to_csv() then the library automatically creates a compressed file. You can submit the compressed file as usual.\n\nRAR or 7z file could be smaller than zip/gz, but I didn't see a dramatic difference when I applied to train.csv.\n\n```\ntrain.csv.zip  120M\ntrain.rar        110M\ntrain.7z          80M\n```\n\nI got the same error with rar file using Ubuntu, while 7z worked.\n\n```\nsudo apt install p7zip-full p7zip-rar\n7z a submission.7z submission.csv\n```\n\nSummary: Use zip or gz for simplicity. You can use 7z if file size is crucial. I don't know how rar file should be created, maybe depends on the software for compression.",
      "votes": null
    },
    {
      "id": "657480",
      "postDate": "10/25/2019 07:15:02",
      "content": "<p>Creating a gz-file worked! Thanks for your help.</p>",
      "rawMarkdown": "Creating a gz-file worked! Thanks for your help.",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 656573,
      "author_name": "oleg90777",
      "author_url": "",
      "post_date": "10/24/2019 12:11:44",
      "content": "<p><code>sub.to_csv(FULL_PATH</code> + 'subs/submission_0.1.gz', index = False, compression= 'gzip')</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 656764,
      "author_name": "junkoda",
      "author_url": "",
      "post_date": "10/24/2019 15:38:40",
      "content": "<p>Yes, as Oleg said, writing zip or gz file is very easy with python + pandas, and I recommend that you use those formats, too. You just write .zip or .gz at the end of your file name in pandas DataFrame.to_csv() then the library automatically creates a compressed file. You can submit the compressed file as usual.</p>\n\n<p>RAR or 7z file could be smaller than zip/gz, but I didn't see a dramatic difference when I applied to train.csv.</p>\n\n<p><code>\ntrain.csv.zip  120M\ntrain.rar        110M\ntrain.7z          80M\n</code></p>\n\n<p>I got the same error with rar file using Ubuntu, while 7z worked.</p>\n\n<p><code>\nsudo apt install p7zip-full p7zip-rar\n7z a submission.7z submission.csv\n</code></p>\n\n<p>Summary: Use zip or gz for simplicity. You can use 7z if file size is crucial. I don't know how rar file should be created, maybe depends on the software for compression.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 657480,
      "author_name": "ellenderidder",
      "author_url": "",
      "post_date": "10/25/2019 07:15:02",
      "content": "<p>Creating a gz-file worked! Thanks for your help.</p>",
      "votes": null,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "656570": "Hi all,\n\nSorry if this is a stupid question, but I am quite new to this. I tried to make a submission via a rar file (since the csv is so large). But when I tried to score the submssion I got an error message:\n Failed\n ERROR: Column 'Rar! 6s'was not expected (Line 1, Column 1)\n ERROR: Required column 'row_id' could not be found\n ERROR: Required column 'meter_reading' could not be found\n\nWhen I subsequently submitted the original CSV file, everything worked just fine.\n\nIs there an extra step I should take when submitting a rar file? \nOr can I better use ZIP files?\n\nRegards,\n\nEllen",
    "656573": "`sub.to_csv(FULL_PATH` + 'subs/submission_0.1.gz', index = False, compression= 'gzip')",
    "656764": "Yes, as Oleg said, writing zip or gz file is very easy with python + pandas, and I recommend that you use those formats, too. You just write .zip or .gz at the end of your file name in pandas DataFrame.to_csv() then the library automatically creates a compressed file. You can submit the compressed file as usual.\n\nRAR or 7z file could be smaller than zip/gz, but I didn't see a dramatic difference when I applied to train.csv.\n\n```\ntrain.csv.zip  120M\ntrain.rar        110M\ntrain.7z          80M\n```\n\nI got the same error with rar file using Ubuntu, while 7z worked.\n\n```\nsudo apt install p7zip-full p7zip-rar\n7z a submission.7z submission.csv\n```\n\nSummary: Use zip or gz for simplicity. You can use 7z if file size is crucial. I don't know how rar file should be created, maybe depends on the software for compression.",
    "657480": "Creating a gz-file worked! Thanks for your help."
  },
  "source": "meta"
}