{
  "id": 98187,
  "title": "Parquet can't handle float16.",
  "url": "/competitions/champs-scalar-coupling/discussion/98187",
  "author_name": "Yassine Alouini",
  "post_date": "2019-07-01T20:27:29.447000",
  "votes": 1,
  "comment_count": 0,
  "views": 0,
  "content": "<p>If, similar to what I have experienced, you had an issue converting a processed <code>DataFrame</code> to a <a href=\"https://fr.wikipedia.org/wiki/Apache_Parquet\"><strong>Parquet</strong></a> format due to <a href=\"https://github.com/apache/arrow/issues/2691\">halffloat</a> error message, here is short snippet on how to do it:</p>\n\n<p><code>\nfor col in df.select_dtypes(include=['float16']).columns:\n   df[col] = df[col].astype('float32')\ndf.to_parquet('processed.parquet')\n</code>\nNotice that these \"halffloat\" numbers appeared after using the <a href=\"https://www.kaggle.com/rinnqd/reduce-memory-usage\">reduce memory usage</a> infamous function. </p>\n\n<p>I hope this helps. Please let me know if you have a better solution of course. :)</p>",
  "messages": [
    {
      "id": 566119,
      "postDate": "2019-07-01T20:27:29.447Z",
      "content": "<p>If, similar to what I have experienced, you had an issue converting a processed <code>DataFrame</code> to a <a href=\"https://fr.wikipedia.org/wiki/Apache_Parquet\"><strong>Parquet</strong></a> format due to <a href=\"https://github.com/apache/arrow/issues/2691\">halffloat</a> error message, here is short snippet on how to do it:</p>\n\n<p><code>\nfor col in df.select_dtypes(include=['float16']).columns:\n   df[col] = df[col].astype('float32')\ndf.to_parquet('processed.parquet')\n</code>\nNotice that these \"halffloat\" numbers appeared after using the <a href=\"https://www.kaggle.com/rinnqd/reduce-memory-usage\">reduce memory usage</a> infamous function. </p>\n\n<p>I hope this helps. Please let me know if you have a better solution of course. :)</p>",
      "rawMarkdown": "If, similar to what I have experienced, you had an issue converting a processed `DataFrame` to a [**Parquet**](https://fr.wikipedia.org/wiki/Apache_Parquet) format due to [halffloat](https://github.com/apache/arrow/issues/2691) error message, here is short snippet on how to do it:\n\n```\nfor col in df.select_dtypes(include=['float16']).columns:\n   df[col] = df[col].astype('float32')\ndf.to_parquet('processed.parquet')\n```\nNotice that these \"halffloat\" numbers appeared after using the [reduce memory usage](https://www.kaggle.com/rinnqd/reduce-memory-usage) infamous function. \n\nI hope this helps. Please let me know if you have a better solution of course. :)",
      "votes": 1
    }
  ],
  "comments": [],
  "raw_markdown_by_id": {
    "566119": "If, similar to what I have experienced, you had an issue converting a processed `DataFrame` to a [**Parquet**](https://fr.wikipedia.org/wiki/Apache_Parquet) format due to [halffloat](https://github.com/apache/arrow/issues/2691) error message, here is short snippet on how to do it:\n\n```\nfor col in df.select_dtypes(include=['float16']).columns:\n   df[col] = df[col].astype('float32')\ndf.to_parquet('processed.parquet')\n```\nNotice that these \"halffloat\" numbers appeared after using the [reduce memory usage](https://www.kaggle.com/rinnqd/reduce-memory-usage) infamous function. \n\nI hope this helps. Please let me know if you have a better solution of course. :)"
  }
}