{
  "id": 72424,
  "title": "Kernel keeps on dying, any suggestions",
  "url": "/competitions/ga-customer-revenue-prediction/discussion/72424",
  "author_name": "",
  "post_date": "2018-11-23T05:51:28.226503200Z",
  "votes": null,
  "comment_count": 3,
  "views": 0,
  "content": "<p>Kernel dies within minutes of trying to upload train data. Any suggestions on making it work.</p>\n\n<p>def load_df(csv_path='../input/train_v2.csv', nrows=None):\n    JSON_COLUMNS = ['device', 'geoNetwork', 'totals', 'trafficSource']</p>\n\n<pre><code>df = pd.read_csv(csv_path, \n                 converters={column: json.loads for column in JSON_COLUMNS}, \n                 dtype={'fullVisitorId': 'str'}, # Important!!\n                 nrows=nrows)\n\nfor column in JSON_COLUMNS:\n    column_as_df = json_normalize(df[column])\n    column_as_df.columns = [f\"{column}.{subcolumn}\" for subcolumn in column_as_df.columns]\n    df = df.drop(column, axis=1).merge(column_as_df, right_index=True, left_index=True)\nprint(f\"Loaded {os.path.basename(csv_path)}. Shape: {df.shape}\")\nreturn df\n</code></pre>\n\n<p>train_df = load_df()</p>",
  "messages": [
    {
      "id": "426351",
      "postDate": "11/23/2018 05:51:28",
      "content": "<p>Kernel dies within minutes of trying to upload train data. Any suggestions on making it work.</p>\n\n<p>def load_df(csv_path='../input/train_v2.csv', nrows=None):\n    JSON_COLUMNS = ['device', 'geoNetwork', 'totals', 'trafficSource']</p>\n\n<pre><code>df = pd.read_csv(csv_path, \n                 converters={column: json.loads for column in JSON_COLUMNS}, \n                 dtype={'fullVisitorId': 'str'}, # Important!!\n                 nrows=nrows)\n\nfor column in JSON_COLUMNS:\n    column_as_df = json_normalize(df[column])\n    column_as_df.columns = [f\"{column}.{subcolumn}\" for subcolumn in column_as_df.columns]\n    df = df.drop(column, axis=1).merge(column_as_df, right_index=True, left_index=True)\nprint(f\"Loaded {os.path.basename(csv_path)}. Shape: {df.shape}\")\nreturn df\n</code></pre>\n\n<p>train_df = load_df()</p>",
      "rawMarkdown": "Kernel dies within minutes of trying to upload train data. Any suggestions on making it work.\n\ndef load_df(csv_path='../input/train_v2.csv', nrows=None):\n    JSON_COLUMNS = ['device', 'geoNetwork', 'totals', 'trafficSource']\n    \n    df = pd.read_csv(csv_path, \n                     converters={column: json.loads for column in JSON_COLUMNS}, \n                     dtype={'fullVisitorId': 'str'}, # Important!!\n                     nrows=nrows)\n    \n    for column in JSON_COLUMNS:\n        column_as_df = json_normalize(df[column])\n        column_as_df.columns = [f\"{column}.{subcolumn}\" for subcolumn in column_as_df.columns]\n        df = df.drop(column, axis=1).merge(column_as_df, right_index=True, left_index=True)\n    print(f\"Loaded {os.path.basename(csv_path)}. Shape: {df.shape}\")\n    return df\n\ntrain_df = load_df()",
      "votes": null
    },
    {
      "id": "426480",
      "postDate": "11/23/2018 10:26:09",
      "content": "<p>Hello Sachin,</p>\n\n<p>I would suggest looking at this topic : <a href=\"https://www.kaggle.com/c/ga-customer-revenue-prediction/discussion/71419#420373\">https://www.kaggle.com/c/ga-customer-revenue-prediction/discussion/71419#420373</a></p>\n\n<p>You won't be able to loead the full dataframe because it is too big. You should specify your nrows to begin working with the CSVs</p>",
      "rawMarkdown": "Hello Sachin,\n\nI would suggest looking at this topic : https://www.kaggle.com/c/ga-customer-revenue-prediction/discussion/71419#420373\n\nYou won't be able to loead the full dataframe because it is too big. You should specify your nrows to begin working with the CSVs",
      "votes": null
    },
    {
      "id": "426553",
      "postDate": "11/23/2018 12:41:08",
      "content": "<p>See this kernel on how to do it using dask, which allows to process data in chunks without reading the whole dataframe in memory:\n<a href=\"https://www.kaggle.com/mlisovyi/bigdata-dask-pandas-flat-json-trim-data-upd\">https://www.kaggle.com/mlisovyi/bigdata-dask-pandas-flat-json-trim-data-upd</a></p>",
      "rawMarkdown": "See this kernel on how to do it using dask, which allows to process data in chunks without reading the whole dataframe in memory:\nhttps://www.kaggle.com/mlisovyi/bigdata-dask-pandas-flat-json-trim-data-upd",
      "votes": null
    },
    {
      "id": "426996",
      "postDate": "11/24/2018 10:20:30",
      "content": "<p>Hey Thanks Assous, \nI will  try and will let you know after my tryouts.</p>",
      "rawMarkdown": "Hey Thanks Assous, \nI will  try and will let you know after my tryouts.",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 426480,
      "author_name": "xtelima",
      "author_url": "",
      "post_date": "11/23/2018 10:26:09",
      "content": "<p>Hello Sachin,</p>\n\n<p>I would suggest looking at this topic : <a href=\"https://www.kaggle.com/c/ga-customer-revenue-prediction/discussion/71419#420373\">https://www.kaggle.com/c/ga-customer-revenue-prediction/discussion/71419#420373</a></p>\n\n<p>You won't be able to loead the full dataframe because it is too big. You should specify your nrows to begin working with the CSVs</p>",
      "votes": null,
      "replies": [
        {
          "id": 426996,
          "author_name": "sachinjchorge",
          "author_url": "",
          "post_date": "11/24/2018 10:20:30",
          "content": "<p>Hey Thanks Assous, \nI will  try and will let you know after my tryouts.</p>",
          "votes": null,
          "replies": []
        }
      ]
    },
    {
      "id": 426553,
      "author_name": "mlisovyi",
      "author_url": "",
      "post_date": "11/23/2018 12:41:08",
      "content": "<p>See this kernel on how to do it using dask, which allows to process data in chunks without reading the whole dataframe in memory:\n<a href=\"https://www.kaggle.com/mlisovyi/bigdata-dask-pandas-flat-json-trim-data-upd\">https://www.kaggle.com/mlisovyi/bigdata-dask-pandas-flat-json-trim-data-upd</a></p>",
      "votes": null,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "426351": "Kernel dies within minutes of trying to upload train data. Any suggestions on making it work.\n\ndef load_df(csv_path='../input/train_v2.csv', nrows=None):\n    JSON_COLUMNS = ['device', 'geoNetwork', 'totals', 'trafficSource']\n    \n    df = pd.read_csv(csv_path, \n                     converters={column: json.loads for column in JSON_COLUMNS}, \n                     dtype={'fullVisitorId': 'str'}, # Important!!\n                     nrows=nrows)\n    \n    for column in JSON_COLUMNS:\n        column_as_df = json_normalize(df[column])\n        column_as_df.columns = [f\"{column}.{subcolumn}\" for subcolumn in column_as_df.columns]\n        df = df.drop(column, axis=1).merge(column_as_df, right_index=True, left_index=True)\n    print(f\"Loaded {os.path.basename(csv_path)}. Shape: {df.shape}\")\n    return df\n\ntrain_df = load_df()",
    "426480": "Hello Sachin,\n\nI would suggest looking at this topic : https://www.kaggle.com/c/ga-customer-revenue-prediction/discussion/71419#420373\n\nYou won't be able to loead the full dataframe because it is too big. You should specify your nrows to begin working with the CSVs",
    "426553": "See this kernel on how to do it using dask, which allows to process data in chunks without reading the whole dataframe in memory:\nhttps://www.kaggle.com/mlisovyi/bigdata-dask-pandas-flat-json-trim-data-upd",
    "426996": "Hey Thanks Assous, \nI will  try and will let you know after my tryouts."
  },
  "source": "meta"
}