{
  "id": 72761,
  "title": "Loading the entire Training data into the kernel",
  "url": "/competitions/ga-customer-revenue-prediction/discussion/72761",
  "author_name": "Hadi",
  "post_date": "2018-11-27T01:35:47.865000",
  "votes": 0,
  "comment_count": 1,
  "views": 0,
  "content": "<p>I am unable to load the entire training set in either one shot or chunks at a time using pandas csv_read method. The kernel hangs, I can however load a subset of the training data, example: 50,000 rows at a time, however that pose an issue for the pre-processing of the data and feature engineering using the full training set,</p>\n\n<p>To overcome the issue, I have tried to use the bigquery helper and follow the instructions in the following kernels to query and process the dataset using bigquery, However i do get permission issues when I follow/fork any of these kernels:\n<a href=\"https://www.kaggle.com/juliaelliott/ga-bigquery-starter-kernel\">https://www.kaggle.com/juliaelliott/ga-bigquery-starter-kernel</a>\n<a href=\"https://www.kaggle.com/vanausloos/ga-bigquery-starter-kernel\">https://www.kaggle.com/vanausloos/ga-bigquery-starter-kernel</a>\n<a href=\"https://www.kaggle.com/bparesh/updated-eda-using-google-bigquery-sql\">https://www.kaggle.com/bparesh/updated-eda-using-google-bigquery-sql</a></p>\n\n<p>To overcome that, I ended up downloading the training set CSV, and I have created a private bigquery dataset using that data, but I am currently unable to access that through my Kaggle kernel as it seems to me like we are not able to access private bigquery dataset instances from kaggle's kernels or at least i don't seem know how to to do that properly,</p>\n\n<p>I am putting this out there in case anyone has figured out how to process the entire training dataset within a kaggle kernel using Bigquery or not.</p>\n\n<p>Thanks\nHadi </p>",
  "messages": [
    {
      "id": 428256,
      "postDate": "2018-11-27T01:35:47.867Z",
      "content": "<p>I am unable to load the entire training set in either one shot or chunks at a time using pandas csv_read method. The kernel hangs, I can however load a subset of the training data, example: 50,000 rows at a time, however that pose an issue for the pre-processing of the data and feature engineering using the full training set,</p>\n\n<p>To overcome the issue, I have tried to use the bigquery helper and follow the instructions in the following kernels to query and process the dataset using bigquery, However i do get permission issues when I follow/fork any of these kernels:\n<a href=\"https://www.kaggle.com/juliaelliott/ga-bigquery-starter-kernel\">https://www.kaggle.com/juliaelliott/ga-bigquery-starter-kernel</a>\n<a href=\"https://www.kaggle.com/vanausloos/ga-bigquery-starter-kernel\">https://www.kaggle.com/vanausloos/ga-bigquery-starter-kernel</a>\n<a href=\"https://www.kaggle.com/bparesh/updated-eda-using-google-bigquery-sql\">https://www.kaggle.com/bparesh/updated-eda-using-google-bigquery-sql</a></p>\n\n<p>To overcome that, I ended up downloading the training set CSV, and I have created a private bigquery dataset using that data, but I am currently unable to access that through my Kaggle kernel as it seems to me like we are not able to access private bigquery dataset instances from kaggle's kernels or at least i don't seem know how to to do that properly,</p>\n\n<p>I am putting this out there in case anyone has figured out how to process the entire training dataset within a kaggle kernel using Bigquery or not.</p>\n\n<p>Thanks\nHadi </p>",
      "rawMarkdown": "I am unable to load the entire training set in either one shot or chunks at a time using pandas csv_read method. The kernel hangs, I can however load a subset of the training data, example: 50,000 rows at a time, however that pose an issue for the pre-processing of the data and feature engineering using the full training set,\n\nTo overcome the issue, I have tried to use the bigquery helper and follow the instructions in the following kernels to query and process the dataset using bigquery, However i do get permission issues when I follow/fork any of these kernels:\nhttps://www.kaggle.com/juliaelliott/ga-bigquery-starter-kernel\nhttps://www.kaggle.com/vanausloos/ga-bigquery-starter-kernel\nhttps://www.kaggle.com/bparesh/updated-eda-using-google-bigquery-sql\n\nTo overcome that, I ended up downloading the training set CSV, and I have created a private bigquery dataset using that data, but I am currently unable to access that through my Kaggle kernel as it seems to me like we are not able to access private bigquery dataset instances from kaggle's kernels or at least i don't seem know how to to do that properly,\n\nI am putting this out there in case anyone has figured out how to process the entire training dataset within a kaggle kernel using Bigquery or not.\n\nThanks\nHadi "
    },
    {
      "id": 428338,
      "postDate": "2018-11-27T05:31:58.953Z",
      "rawMarkdown": "",
      "isDeleted": true
    }
  ],
  "comments": [
    {
      "id": 428338,
      "author_name": "",
      "author_url": "",
      "post_date": "2018-11-27T05:31:58.953000",
      "content": "",
      "votes": 0,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "428256": "I am unable to load the entire training set in either one shot or chunks at a time using pandas csv_read method. The kernel hangs, I can however load a subset of the training data, example: 50,000 rows at a time, however that pose an issue for the pre-processing of the data and feature engineering using the full training set,\n\nTo overcome the issue, I have tried to use the bigquery helper and follow the instructions in the following kernels to query and process the dataset using bigquery, However i do get permission issues when I follow/fork any of these kernels:\nhttps://www.kaggle.com/juliaelliott/ga-bigquery-starter-kernel\nhttps://www.kaggle.com/vanausloos/ga-bigquery-starter-kernel\nhttps://www.kaggle.com/bparesh/updated-eda-using-google-bigquery-sql\n\nTo overcome that, I ended up downloading the training set CSV, and I have created a private bigquery dataset using that data, but I am currently unable to access that through my Kaggle kernel as it seems to me like we are not able to access private bigquery dataset instances from kaggle's kernels or at least i don't seem know how to to do that properly,\n\nI am putting this out there in case anyone has figured out how to process the entire training dataset within a kaggle kernel using Bigquery or not.\n\nThanks\nHadi ",
    "428338": ""
  }
}