{
  "id": 664981,
  "title": "BF16 vs FP32:Full Precision significantly outperforming BF16",
  "url": "/competitions/deep-past-initiative-machine-translation/discussion/664981",
  "author_name": "",
  "post_date": "2025-12-29T11:21:56.636641Z",
  "votes": 12,
  "comment_count": 4,
  "views": 0,
  "content": "<p>Significant performance gap between BF16 and FP32 training.</p>\n<p>Despite keeping all hyperparameters, seeds, and environment settings identical, the evaluation metrics and Leaderboard (LB) scores are night and day.</p>\n<p>BF16 also significantly hallucinates during inference</p>\n<table>\n<thead>\n<tr>\n<th>Metric</th>\n<th>BF16 (Brain Float 16)</th>\n<th>FP32 (Full Precision)</th>\n</tr>\n</thead>\n<tbody>\n<tr>\n<td>LB Score</td>\n<td>6.0</td>\n<td>18.5</td>\n</tr>\n<tr>\n<td>Final Eval Loss</td>\n<td>~0.81</td>\n<td>~0.62</td>\n</tr>\n<tr>\n<td>Eval chrf</td>\n<td>~3.5</td>\n<td>~4.2</td>\n</tr>\n</tbody>\n</table>\n<p><img src=\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F9193114%2F29d577c848341018d084a871ee625538%2FScreenshot%202025-12-29%20164503.png?generation=1767007266503014&amp;alt=media\" alt=\"\"></p>",
  "messages": [
    {
      "id": "3383090",
      "postDate": "12/29/2025 11:21:56",
      "content": "<p>Significant performance gap between BF16 and FP32 training.</p>\n<p>Despite keeping all hyperparameters, seeds, and environment settings identical, the evaluation metrics and Leaderboard (LB) scores are night and day.</p>\n<p>BF16 also significantly hallucinates during inference</p>\n<table>\n<thead>\n<tr>\n<th>Metric</th>\n<th>BF16 (Brain Float 16)</th>\n<th>FP32 (Full Precision)</th>\n</tr>\n</thead>\n<tbody>\n<tr>\n<td>LB Score</td>\n<td>6.0</td>\n<td>18.5</td>\n</tr>\n<tr>\n<td>Final Eval Loss</td>\n<td>~0.81</td>\n<td>~0.62</td>\n</tr>\n<tr>\n<td>Eval chrf</td>\n<td>~3.5</td>\n<td>~4.2</td>\n</tr>\n</tbody>\n</table>\n<p><img src=\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F9193114%2F29d577c848341018d084a871ee625538%2FScreenshot%202025-12-29%20164503.png?generation=1767007266503014&amp;alt=media\" alt=\"\"></p>",
      "rawMarkdown": "Significant performance gap between BF16 and FP32 training.\n\nDespite keeping all hyperparameters, seeds, and environment settings identical, the evaluation metrics and Leaderboard (LB) scores are night and day.\n\nBF16 also significantly hallucinates during inference\n\n| Metric | BF16 (Brain Float 16) | FP32 (Full Precision) |\n| :--- | :---- | :--- |\n| LB Score | 6.0 | 18.5 |\n| Final Eval Loss | ~0.81 | ~0.62 |\n| Eval chrf |~3.5 | ~4.2 |\n\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F9193114%2F29d577c848341018d084a871ee625538%2FScreenshot%202025-12-29%20164503.png?generation=1767007266503014&alt=media)",
      "votes": null
    },
    {
      "id": "3383092",
      "postDate": "12/29/2025 11:27:33",
      "content": "<p>AFAIK P100 and T4 doesnt have support for bf16 if you train your models on Kaggle. It might be mapping to the native float16 under the hood, which may cause degradation in performance.</p>",
      "rawMarkdown": "AFAIK P100 and T4 doesnt have support for bf16 if you train your models on Kaggle. It might be mapping to the native float16 under the hood, which may cause degradation in performance.",
      "votes": null
    },
    {
      "id": "3383095",
      "postDate": "12/29/2025 11:44:35",
      "content": "<p>Forgot to mention this was L4</p>",
      "rawMarkdown": "Forgot to mention this was L4",
      "votes": null
    },
    {
      "id": "3383261",
      "postDate": "12/29/2025 17:48:40",
      "content": "<p><a href=\"https://www.kaggle.com/hrithik22122\" target=\"_blank\">@hrithik22122</a> \nWhich subset of the data was used for the eval set's loss and chrf you mentioned here?</p>",
      "rawMarkdown": "hrithik22122 \nWhich subset of the data was used for the eval set's loss and chrf you mentioned here?",
      "votes": null
    },
    {
      "id": "3383263",
      "postDate": "12/29/2025 18:08:07",
      "content": "<p>The same val set on the starter notebook | train byt5 small </p>",
      "rawMarkdown": "The same val set on the starter notebook | train byt5 small",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 3383092,
      "author_name": "nlztrk",
      "author_url": "",
      "post_date": "12/29/2025 11:27:33",
      "content": "<p>AFAIK P100 and T4 doesnt have support for bf16 if you train your models on Kaggle. It might be mapping to the native float16 under the hood, which may cause degradation in performance.</p>",
      "votes": null,
      "replies": [
        {
          "id": 3383095,
          "author_name": "hrithik22122",
          "author_url": "",
          "post_date": "12/29/2025 11:44:35",
          "content": "<p>Forgot to mention this was L4</p>",
          "votes": null,
          "replies": []
        }
      ]
    },
    {
      "id": 3383261,
      "author_name": "",
      "author_url": "",
      "post_date": "12/29/2025 17:48:40",
      "content": "<p><a href=\"https://www.kaggle.com/hrithik22122\" target=\"_blank\">@hrithik22122</a> \nWhich subset of the data was used for the eval set's loss and chrf you mentioned here?</p>",
      "votes": null,
      "replies": [
        {
          "id": 3383263,
          "author_name": "hrithik22122",
          "author_url": "",
          "post_date": "12/29/2025 18:08:07",
          "content": "<p>The same val set on the starter notebook | train byt5 small </p>",
          "votes": null,
          "replies": []
        }
      ]
    }
  ],
  "raw_markdown_by_id": {
    "3383090": "Significant performance gap between BF16 and FP32 training.\n\nDespite keeping all hyperparameters, seeds, and environment settings identical, the evaluation metrics and Leaderboard (LB) scores are night and day.\n\nBF16 also significantly hallucinates during inference\n\n| Metric | BF16 (Brain Float 16) | FP32 (Full Precision) |\n| :--- | :---- | :--- |\n| LB Score | 6.0 | 18.5 |\n| Final Eval Loss | ~0.81 | ~0.62 |\n| Eval chrf |~3.5 | ~4.2 |\n\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F9193114%2F29d577c848341018d084a871ee625538%2FScreenshot%202025-12-29%20164503.png?generation=1767007266503014&alt=media)",
    "3383092": "AFAIK P100 and T4 doesnt have support for bf16 if you train your models on Kaggle. It might be mapping to the native float16 under the hood, which may cause degradation in performance.",
    "3383095": "Forgot to mention this was L4",
    "3383261": "hrithik22122 \nWhich subset of the data was used for the eval set's loss and chrf you mentioned here?",
    "3383263": "The same val set on the starter notebook | train byt5 small"
  },
  "source": "meta"
}