{
  "id": 25123,
  "title": "Submission file not getting stored in csv",
  "url": "/competitions/bosch-production-line-performance/discussion/25123",
  "author_name": "",
  "post_date": "2016-11-05T16:21:21.490Z",
  "votes": null,
  "comment_count": 6,
  "views": 404,
  "content": "<p>I am trying to store the results in csv file for submission.  R is not able to write the complete file due to huge size. I also manually tried to store the results in different files and then tried to combine it into a single file without any success. Anyone has seen this problem or any suggestion how to resolve this.</p>",
  "messages": [
    {
      "id": "142939",
      "postDate": "11/05/2016 16:21:21",
      "content": "<p>I am trying to store the results in csv file for submission.  R is not able to write the complete file due to huge size. I also manually tried to store the results in different files and then tried to combine it into a single file without any success. Anyone has seen this problem or any suggestion how to resolve this.</p>",
      "rawMarkdown": "I am trying to store the results in csv file for submission.  R is not able to write the complete file due to huge size. I also manually tried to store the results in different files and then tried to combine it into a single file without any success. Anyone has seen this problem or any suggestion how to resolve this.",
      "votes": null
    },
    {
      "id": "142944",
      "postDate": "11/05/2016 16:49:10",
      "content": "<p>R should not have trouble writing a submission file around 1.1 Million rows with two columns. Not that big.\ncan you post the command line that you are using to write the submission file?</p>\n\n<p>Also include the output of:</p>\n\n<p>str(response)\ndim(response)</p>\n\n<p>where &quot;response&quot; is the matrix/dataframe you are trying to output.</p>\n\n<p>Good luck,</p>",
      "rawMarkdown": "R should not have trouble writing a submission file around 1.1 Million rows with two columns. Not that big.\r\ncan you post the command line that you are using to write the submission file?\r\n\r\nAlso include the output of:\r\n\r\nstr(response)\r\ndim(response)\r\n\r\nwhere \"response\" is the matrix/dataframe you are trying to output.\r\n\r\nGood luck,",
      "votes": null
    },
    {
      "id": "142955",
      "postDate": "11/05/2016 18:00:48",
      "content": "<p>This is the response object definition I am using:\ndf &lt;- data.frame(&quot;Id&quot; = integer(7), &quot;Response&quot; = numeric(1), row.names= NULL, stringsAsFactors=FALSE)\nThe file write statement:\nwrite.csv(df,file = &quot;my_solution.csv&quot;, row.names = FALSE)</p>\n\n<p>str(df)\n'data.frame':   1183748 obs. of  2 variables:\n $ test_id                 : Factor w/ 300000 levels &quot;1&quot;,&quot;10&quot;,&quot;100&quot;,..: 300000 1 55658 111021 222374 288880 2 11093 27755 38997 ...\n $ as.numeric.pred....0.03.: num  0 0 0 0 0 0 0 0 0 0 ...</p>\n\n<p>dim(df)\n[1] 1183748       2</p>\n\n<p>Getting a message: File not loaded completely.</p>",
      "rawMarkdown": "This is the response object definition I am using:\r\ndf <- data.frame(\"Id\" = integer(7), \"Response\" = numeric(1), row.names= NULL, stringsAsFactors=FALSE)\r\nThe file write statement:\r\nwrite.csv(df,file = \"my_solution.csv\", row.names = FALSE)\r\n\r\n str(df)\r\n'data.frame':\t1183748 obs. of  2 variables:\r\n $ test_id                 : Factor w/ 300000 levels \"1\",\"10\",\"100\",..: 300000 1 55658 111021 222374 288880 2 11093 27755 38997 ...\r\n $ as.numeric.pred....0.03.: num  0 0 0 0 0 0 0 0 0 0 ...\r\n\r\ndim(df)\r\n[1] 1183748       2\r\n\r\nGetting a message: File not loaded completely.",
      "votes": null
    },
    {
      "id": "142956",
      "postDate": "11/05/2016 18:10:26",
      "content": "<p>On my submission variable, I get:</p>\n\n<blockquote>\n  <p>str(out1)\n  'data.frame':   1183748 obs. of  2 variables:\n   $ Id      : num  1 2 3 5 8 10 12 15 17 19 ...\n $ Response: num  0.00278 0.00133 0.00226 0.00348 0.00314 ...</p>\n</blockquote>\n\n<p>You have test_id as a &quot;Factor&quot;. Maybe that is the problem? Did you change &quot;test.csv&quot; into factors at some point, and you included &quot;Id&quot; as one of the factors?</p>",
      "rawMarkdown": "On my submission variable, I get:\r\n\r\n> str(out1)\r\n'data.frame':   1183748 obs. of  2 variables:\r\n $ Id      : num  1 2 3 5 8 10 12 15 17 19 ...\r\n $ Response: num  0.00278 0.00133 0.00226 0.00348 0.00314 ...\r\n\r\nYou have test_id as a \"Factor\". Maybe that is the problem? Did you change \"test.csv\" into factors at some point, and you included \"Id\" as one of the factors?",
      "votes": null
    },
    {
      "id": "142976",
      "postDate": "11/05/2016 19:41:45",
      "content": "<p>The maximum  number of rows successfully stored are upto: 1048576. Looking for a solution how to store the rest of the rows in this single file. </p>",
      "rawMarkdown": "The maximum  number of rows successfully stored are upto: 1048576. Looking for a solution how to store the rest of the rows in this single file.",
      "votes": null
    },
    {
      "id": "142977",
      "postDate": "11/05/2016 19:57:40",
      "content": "<p>that number is also the maximum rows in an excel spreadsheet. are you doing any processing in excel?</p>",
      "rawMarkdown": "that number is also the maximum rows in an excel spreadsheet. are you doing any processing in excel?",
      "votes": null
    },
    {
      "id": "143010",
      "postDate": "11/06/2016 04:32:12",
      "content": "<p>My submissions were having some issues, so wanted to check the files. I am able to open the file using Notepad++ and found the issue. Now it is resolved and I am able to submit it. Thanks!</p>",
      "rawMarkdown": "My submissions were having some issues, so wanted to check the files. I am able to open the file using Notepad++ and found the issue. Now it is resolved and I am able to submit it. Thanks!",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 142944,
      "author_name": "richardepstein",
      "author_url": "",
      "post_date": "11/05/2016 16:49:10",
      "content": "<p>R should not have trouble writing a submission file around 1.1 Million rows with two columns. Not that big.\ncan you post the command line that you are using to write the submission file?</p>\n\n<p>Also include the output of:</p>\n\n<p>str(response)\ndim(response)</p>\n\n<p>where &quot;response&quot; is the matrix/dataframe you are trying to output.</p>\n\n<p>Good luck,</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 142955,
      "author_name": "triparc",
      "author_url": "",
      "post_date": "11/05/2016 18:00:48",
      "content": "<p>This is the response object definition I am using:\ndf &lt;- data.frame(&quot;Id&quot; = integer(7), &quot;Response&quot; = numeric(1), row.names= NULL, stringsAsFactors=FALSE)\nThe file write statement:\nwrite.csv(df,file = &quot;my_solution.csv&quot;, row.names = FALSE)</p>\n\n<p>str(df)\n'data.frame':   1183748 obs. of  2 variables:\n $ test_id                 : Factor w/ 300000 levels &quot;1&quot;,&quot;10&quot;,&quot;100&quot;,..: 300000 1 55658 111021 222374 288880 2 11093 27755 38997 ...\n $ as.numeric.pred....0.03.: num  0 0 0 0 0 0 0 0 0 0 ...</p>\n\n<p>dim(df)\n[1] 1183748       2</p>\n\n<p>Getting a message: File not loaded completely.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 142956,
      "author_name": "richardepstein",
      "author_url": "",
      "post_date": "11/05/2016 18:10:26",
      "content": "<p>On my submission variable, I get:</p>\n\n<blockquote>\n  <p>str(out1)\n  'data.frame':   1183748 obs. of  2 variables:\n   $ Id      : num  1 2 3 5 8 10 12 15 17 19 ...\n $ Response: num  0.00278 0.00133 0.00226 0.00348 0.00314 ...</p>\n</blockquote>\n\n<p>You have test_id as a &quot;Factor&quot;. Maybe that is the problem? Did you change &quot;test.csv&quot; into factors at some point, and you included &quot;Id&quot; as one of the factors?</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 142976,
      "author_name": "triparc",
      "author_url": "",
      "post_date": "11/05/2016 19:41:45",
      "content": "<p>The maximum  number of rows successfully stored are upto: 1048576. Looking for a solution how to store the rest of the rows in this single file. </p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 142977,
      "author_name": "richardepstein",
      "author_url": "",
      "post_date": "11/05/2016 19:57:40",
      "content": "<p>that number is also the maximum rows in an excel spreadsheet. are you doing any processing in excel?</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 143010,
      "author_name": "triparc",
      "author_url": "",
      "post_date": "11/06/2016 04:32:12",
      "content": "<p>My submissions were having some issues, so wanted to check the files. I am able to open the file using Notepad++ and found the issue. Now it is resolved and I am able to submit it. Thanks!</p>",
      "votes": null,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "142939": "I am trying to store the results in csv file for submission.  R is not able to write the complete file due to huge size. I also manually tried to store the results in different files and then tried to combine it into a single file without any success. Anyone has seen this problem or any suggestion how to resolve this.",
    "142944": "R should not have trouble writing a submission file around 1.1 Million rows with two columns. Not that big.\r\ncan you post the command line that you are using to write the submission file?\r\n\r\nAlso include the output of:\r\n\r\nstr(response)\r\ndim(response)\r\n\r\nwhere \"response\" is the matrix/dataframe you are trying to output.\r\n\r\nGood luck,",
    "142955": "This is the response object definition I am using:\r\ndf <- data.frame(\"Id\" = integer(7), \"Response\" = numeric(1), row.names= NULL, stringsAsFactors=FALSE)\r\nThe file write statement:\r\nwrite.csv(df,file = \"my_solution.csv\", row.names = FALSE)\r\n\r\n str(df)\r\n'data.frame':\t1183748 obs. of  2 variables:\r\n $ test_id                 : Factor w/ 300000 levels \"1\",\"10\",\"100\",..: 300000 1 55658 111021 222374 288880 2 11093 27755 38997 ...\r\n $ as.numeric.pred....0.03.: num  0 0 0 0 0 0 0 0 0 0 ...\r\n\r\ndim(df)\r\n[1] 1183748       2\r\n\r\nGetting a message: File not loaded completely.",
    "142956": "On my submission variable, I get:\r\n\r\n> str(out1)\r\n'data.frame':   1183748 obs. of  2 variables:\r\n $ Id      : num  1 2 3 5 8 10 12 15 17 19 ...\r\n $ Response: num  0.00278 0.00133 0.00226 0.00348 0.00314 ...\r\n\r\nYou have test_id as a \"Factor\". Maybe that is the problem? Did you change \"test.csv\" into factors at some point, and you included \"Id\" as one of the factors?",
    "142976": "The maximum  number of rows successfully stored are upto: 1048576. Looking for a solution how to store the rest of the rows in this single file.",
    "142977": "that number is also the maximum rows in an excel spreadsheet. are you doing any processing in excel?",
    "143010": "My submissions were having some issues, so wanted to check the files. I am able to open the file using Notepad++ and found the issue. Now it is resolved and I am able to submit it. Thanks!"
  },
  "source": "meta"
}