{
  "id": 71145,
  "title": "json_normalize in Dask?",
  "url": "/competitions/ga-customer-revenue-prediction/discussion/71145",
  "author_name": "",
  "post_date": "2018-11-10T15:49:56.154934500Z",
  "votes": 1,
  "comment_count": 2,
  "views": 0,
  "content": "<p>Hi guys, I'm a beginner in Python for Data Science. I tried reading the training data using pandas on the Kaggle kernel and locally on my PC but both the times it was not successful and my machine froze! I followed the same steps as some of the top kernels posted here but I could not manage to read the csv file. Is this just a memory requirement issue ? \nMy PC specs: 8 GB RAM, core i5 7th gen.</p>\n\n<p>Anyway, I tried figuring out a solution to the above problem and came across Dask. I was successfully able to read and view the contents, however I could not find a function similar to pandas json_normalize so now I have no way to flatten the records contained in the Dask Dataframe. Any of you who have come across such a problem, please share your solutions.</p>\n\n<p>Any suggestions or feedback is highly appreciated.</p>\n\n<p>Thanks.</p>",
  "messages": [
    {
      "id": "418785",
      "postDate": "11/10/2018 15:49:56",
      "content": "<p>Hi guys, I'm a beginner in Python for Data Science. I tried reading the training data using pandas on the Kaggle kernel and locally on my PC but both the times it was not successful and my machine froze! I followed the same steps as some of the top kernels posted here but I could not manage to read the csv file. Is this just a memory requirement issue ? \nMy PC specs: 8 GB RAM, core i5 7th gen.</p>\n\n<p>Anyway, I tried figuring out a solution to the above problem and came across Dask. I was successfully able to read and view the contents, however I could not find a function similar to pandas json_normalize so now I have no way to flatten the records contained in the Dask Dataframe. Any of you who have come across such a problem, please share your solutions.</p>\n\n<p>Any suggestions or feedback is highly appreciated.</p>\n\n<p>Thanks.</p>",
      "rawMarkdown": "Hi guys, I'm a beginner in Python for Data Science. I tried reading the training data using pandas on the Kaggle kernel and locally on my PC but both the times it was not successful and my machine froze! I followed the same steps as some of the top kernels posted here but I could not manage to read the csv file. Is this just a memory requirement issue ? \nMy PC specs: 8 GB RAM, core i5 7th gen.\n\nAnyway, I tried figuring out a solution to the above problem and came across Dask. I was successfully able to read and view the contents, however I could not find a function similar to pandas json_normalize so now I have no way to flatten the records contained in the Dask Dataframe. Any of you who have come across such a problem, please share your solutions.\n\nAny suggestions or feedback is highly appreciated.\n\nThanks.",
      "votes": null
    },
    {
      "id": "426560",
      "postDate": "11/23/2018 12:54:14",
      "content": "<p>I've been working on a solution in dask since a while. See this kernel:\n<a href=\"https://www.kaggle.com/mlisovyi/bigdata-dask-pandas-flat-json-trim-data-upd\">https://www.kaggle.com/mlisovyi/bigdata-dask-pandas-flat-json-trim-data-upd</a>\nI had to implement a custom dictionary parsing instead of using <code>json_normalize</code> is dask and they do not have a native implementation that would allow it same way as in pandas</p>",
      "rawMarkdown": "I've been working on a solution in dask since a while. See this kernel:\nhttps://www.kaggle.com/mlisovyi/bigdata-dask-pandas-flat-json-trim-data-upd\nI had to implement a custom dictionary parsing instead of using `json_normalize` is dask and they do not have a native implementation that would allow it same way as in pandas",
      "votes": null
    },
    {
      "id": "428046",
      "postDate": "11/26/2018 16:55:42",
      "content": "<p>Thanks a lot!</p>",
      "rawMarkdown": "Thanks a lot!",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 426560,
      "author_name": "mlisovyi",
      "author_url": "",
      "post_date": "11/23/2018 12:54:14",
      "content": "<p>I've been working on a solution in dask since a while. See this kernel:\n<a href=\"https://www.kaggle.com/mlisovyi/bigdata-dask-pandas-flat-json-trim-data-upd\">https://www.kaggle.com/mlisovyi/bigdata-dask-pandas-flat-json-trim-data-upd</a>\nI had to implement a custom dictionary parsing instead of using <code>json_normalize</code> is dask and they do not have a native implementation that would allow it same way as in pandas</p>",
      "votes": null,
      "replies": [
        {
          "id": 428046,
          "author_name": "ap1495",
          "author_url": "",
          "post_date": "11/26/2018 16:55:42",
          "content": "<p>Thanks a lot!</p>",
          "votes": null,
          "replies": []
        }
      ]
    }
  ],
  "raw_markdown_by_id": {
    "418785": "Hi guys, I'm a beginner in Python for Data Science. I tried reading the training data using pandas on the Kaggle kernel and locally on my PC but both the times it was not successful and my machine froze! I followed the same steps as some of the top kernels posted here but I could not manage to read the csv file. Is this just a memory requirement issue ? \nMy PC specs: 8 GB RAM, core i5 7th gen.\n\nAnyway, I tried figuring out a solution to the above problem and came across Dask. I was successfully able to read and view the contents, however I could not find a function similar to pandas json_normalize so now I have no way to flatten the records contained in the Dask Dataframe. Any of you who have come across such a problem, please share your solutions.\n\nAny suggestions or feedback is highly appreciated.\n\nThanks.",
    "426560": "I've been working on a solution in dask since a while. See this kernel:\nhttps://www.kaggle.com/mlisovyi/bigdata-dask-pandas-flat-json-trim-data-upd\nI had to implement a custom dictionary parsing instead of using `json_normalize` is dask and they do not have a native implementation that would allow it same way as in pandas",
    "428046": "Thanks a lot!"
  },
  "source": "meta"
}