{
  "id": 437887,
  "title": "Is pandas not as efficient as pyspark in reading parquet files?",
  "url": "/competitions/child-mind-institute-detect-sleep-states/discussion/437887",
  "author_name": "",
  "post_date": "2023-09-08T15:00:36.332845600Z",
  "votes": null,
  "comment_count": 3,
  "views": 0,
  "content": "<p>As a file over 900 megabytes, is it a bit too big for pandas?</p>",
  "messages": [
    {
      "id": "2429365",
      "postDate": "09/08/2023 15:00:36",
      "content": "<p>As a file over 900 megabytes, is it a bit too big for pandas?</p>",
      "rawMarkdown": "As a file over 900 megabytes, is it a bit too big for pandas?",
      "votes": null
    },
    {
      "id": "2429505",
      "postDate": "09/08/2023 16:30:24",
      "content": "<p>I tryed polars and is faster than pandas (as expected)</p>",
      "rawMarkdown": "I tryed polars and is faster than pandas (as expected)",
      "votes": null
    },
    {
      "id": "2429539",
      "postDate": "09/08/2023 16:46:55",
      "content": "<p>I read that PySpark is better in handling large data as compared to pandas, not used yet though.</p>",
      "rawMarkdown": "I read that PySpark is better in handling large data as compared to pandas, not used yet though.",
      "votes": null
    },
    {
      "id": "2429622",
      "postDate": "09/08/2023 17:21:40",
      "content": "<p>Used Polars, its much faster.</p>",
      "rawMarkdown": "Used Polars, its much faster.",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 2429505,
      "author_name": "yantxx",
      "author_url": "",
      "post_date": "09/08/2023 16:30:24",
      "content": "<p>I tryed polars and is faster than pandas (as expected)</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 2429539,
      "author_name": "ddosad",
      "author_url": "",
      "post_date": "09/08/2023 16:46:55",
      "content": "<p>I read that PySpark is better in handling large data as compared to pandas, not used yet though.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 2429622,
      "author_name": "chensixian",
      "author_url": "",
      "post_date": "09/08/2023 17:21:40",
      "content": "<p>Used Polars, its much faster.</p>",
      "votes": null,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "2429365": "As a file over 900 megabytes, is it a bit too big for pandas?",
    "2429505": "I tryed polars and is faster than pandas (as expected)",
    "2429539": "I read that PySpark is better in handling large data as compared to pandas, not used yet though.",
    "2429622": "Used Polars, its much faster."
  },
  "source": "meta"
}