{
  "id": 117671,
  "title": "Cannot Submit - My submission Dataframe will not save properly",
  "url": "/competitions/ashrae-energy-prediction/discussion/117671",
  "author_name": "",
  "post_date": "2019-11-17T03:38:13.552461600Z",
  "votes": 1,
  "comment_count": 2,
  "views": 0,
  "content": "<p>The shape of my submission dataframe is (41697600, 2), with no missing (NA) values - df.isna().sum().sum() = 0. The columns are both floats and the prediction meter readings are rounded to 3 decimal places. When I've been saving as a CSV  - df.to_csv('path') , if  then submit is, or reopen is, over half the rows are missing. I've been doing this on Google Cloud. Any ideas anyone? Will let you know when solved. </p>\n\n<p>The latest shape after saving is (17739140, 2) - where are the rows gone?</p>",
  "messages": [
    {
      "id": "674770",
      "postDate": "11/17/2019 03:38:13",
      "content": "<p>The shape of my submission dataframe is (41697600, 2), with no missing (NA) values - df.isna().sum().sum() = 0. The columns are both floats and the prediction meter readings are rounded to 3 decimal places. When I've been saving as a CSV  - df.to_csv('path') , if  then submit is, or reopen is, over half the rows are missing. I've been doing this on Google Cloud. Any ideas anyone? Will let you know when solved. </p>\n\n<p>The latest shape after saving is (17739140, 2) - where are the rows gone?</p>",
      "rawMarkdown": "The shape of my submission dataframe is (41697600, 2), with no missing (NA) values - df.isna().sum().sum() = 0. The columns are both floats and the prediction meter readings are rounded to 3 decimal places. When I've been saving as a CSV  - df.to_csv('path') , if  then submit is, or reopen is, over half the rows are missing. I've been doing this on Google Cloud. Any ideas anyone? Will let you know when solved. \n\nThe latest shape after saving is (17739140, 2) - where are the rows gone?",
      "votes": null
    },
    {
      "id": "674784",
      "postDate": "11/17/2019 04:23:38",
      "content": "<p>Update - This was the fault of the Jupyter set-up on google cloud. When I saved the predictions on coloabs - it worked. </p>",
      "rawMarkdown": "Update - This was the fault of the Jupyter set-up on google cloud. When I saved the predictions on coloabs - it worked.",
      "votes": null
    },
    {
      "id": "674813",
      "postDate": "11/17/2019 05:40:05",
      "content": "<p>Update - Bad news - I'm going about my business saving my processed data - now i'm paranoid this isn't even saving properly. Need to investigate correct procedures for saving on google cloud, without destroying data</p>",
      "rawMarkdown": "Update - Bad news - I'm going about my business saving my processed data - now i'm paranoid this isn't even saving properly. Need to investigate correct procedures for saving on google cloud, without destroying data",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 674784,
      "author_name": "antonp23",
      "author_url": "",
      "post_date": "11/17/2019 04:23:38",
      "content": "<p>Update - This was the fault of the Jupyter set-up on google cloud. When I saved the predictions on coloabs - it worked. </p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 674813,
      "author_name": "antonp23",
      "author_url": "",
      "post_date": "11/17/2019 05:40:05",
      "content": "<p>Update - Bad news - I'm going about my business saving my processed data - now i'm paranoid this isn't even saving properly. Need to investigate correct procedures for saving on google cloud, without destroying data</p>",
      "votes": null,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "674770": "The shape of my submission dataframe is (41697600, 2), with no missing (NA) values - df.isna().sum().sum() = 0. The columns are both floats and the prediction meter readings are rounded to 3 decimal places. When I've been saving as a CSV  - df.to_csv('path') , if  then submit is, or reopen is, over half the rows are missing. I've been doing this on Google Cloud. Any ideas anyone? Will let you know when solved. \n\nThe latest shape after saving is (17739140, 2) - where are the rows gone?",
    "674784": "Update - This was the fault of the Jupyter set-up on google cloud. When I saved the predictions on coloabs - it worked.",
    "674813": "Update - Bad news - I'm going about my business saving my processed data - now i'm paranoid this isn't even saving properly. Need to investigate correct procedures for saving on google cloud, without destroying data"
  },
  "source": "meta"
}