For simplicity, we only analyze the macro F1.
\n$$F1=\\frac{2TP}{(TP+FN)+(TP+FP)}\\approx \\frac{2\\sum yp_\\theta(x)}{\\sum y+p_\\theta(x)}$$
\nwhere p(x) is the probability of the positive class, and y is the label of x (0 or 1).
\n$$-\\nabla_{\\theta}\\frac{2 y p_{\\theta}(x)}{y + p_{\\theta}(x)}$$
\n$$=-2\\frac{y \\nabla_{\\theta}p_{\\theta}(x)}{y + p_{\\theta}(x)} + 2\\frac{y p_{\\theta}(x)\\nabla_{\\theta}p_{\\theta}(x)}{\\left(y + p_{\\theta}(x)\\right)^2}$$
\n$$=-\\frac{2\\big(y-\\frac{\\text{F1}}{2}\\big)\\nabla_{\\theta}p_{\\theta}(x)}{\\big[y + p_{\\theta}(x)\\big]} $$
We can use \\(p\\approx log(p)+1,-p \\approx log(1-p)\\) (This approach imposes a greater penalty on misclassified samples) then:
\n$$-\\big(y-\\frac{\\text{F1}}{2}\\big)\\nabla_{\\theta}p_{\\theta}(x)$$
$$=0+-y*\\big(1-\\frac{\\text{F1}}{2}\\big)\\nabla_{\\theta}log(p_{\\theta}(x))$$
\n$$-(1-y)*\\big(\\frac{\\text{F1}}{2}\\big)\\nabla_{\\theta}log(1-p_{\\theta}(x))$$
\nThis is equivalent to weighted cross-entropy.
\nWe can extend this to the multi-class case.
My code is as follows. Using this custom loss function increases my CV ~ 0.002. Although it's a small increase, I believe there's still room for improvement. Here, I only considered macro F1, but we could also incorporate binary F1.
\nI also look forward to hearing more thoughts from everyone on this topic.
\n tensorflow tf\n tensorflow.keras backend K\n\n (tf.keras.losses.Loss):\n ():\n ().__init__(name=name)\n .class_weights = class_weights\n\n ():\n cce = tf.keras.losses.categorical_crossentropy(y_true, y_pred, from_logits=,label_smoothing=)\n\n weighted_cce = cce * K.(y_true * .class_weights, axis=-)\n K.mean(weighted_cce)\n tensorflow.keras.callbacks Callback\n sklearn.metrics f1_score\n numpy np\n\n ():\n (): \n (F1WeightsUpdater, ).__init__()\n .class_weights_var = class_weights_var\n .num_classes = num_classes\n .validation_data = validation_data \n\n ():\n \n X_val, y_val_cat = .validation_data[:] \n\n y_pred_proba = .model.predict(X_val, verbose=)\n\n y_val_labels = np.argmax(y_val_cat, axis=)\n y_pred_labels = np.argmax(y_pred_proba, axis=)\n\n f1_scores = f1_score(y_val_labels, y_pred_labels, labels=(.num_classes), average=)\n new_weights = -f1_scores/ \n .class_weights_var.assign(new_weights)\n\ninitial_weights = tf.Variable([] * , dtype=tf.float32, trainable=)\nweighted_loss = UpdatableWeightedLoss(initial_weights)\nf1_updater_callback = F1WeightsUpdater(\n initial_weights,\n ,\n validation_data=(X_val, y_val) \n)\nmodel.(loss=weighted_loss)\nmodel.fit(callbacks=[f1_updater_callback])\nReference：
\nhttps://kexue.fm/archives/7708
For simplicity, we only analyze the macro F1.
\n$$F1=\\frac{2TP}{(TP+FN)+(TP+FP)}\\approx \\frac{2\\sum yp_\\theta(x)}{\\sum y+p_\\theta(x)}$$
\nwhere p(x) is the probability of the positive class, and y is the label of x (0 or 1).
\n$$-\\nabla_{\\theta}\\frac{2 y p_{\\theta}(x)}{y + p_{\\theta}(x)}$$
\n$$=-2\\frac{y \\nabla_{\\theta}p_{\\theta}(x)}{y + p_{\\theta}(x)} + 2\\frac{y p_{\\theta}(x)\\nabla_{\\theta}p_{\\theta}(x)}{\\left(y + p_{\\theta}(x)\\right)^2}$$
\n$$=-\\frac{2\\big(y-\\frac{\\text{F1}}{2}\\big)\\nabla_{\\theta}p_{\\theta}(x)}{\\big[y + p_{\\theta}(x)\\big]} $$
We can use \\(p\\approx log(p)+1,-p \\approx log(1-p)\\) (This approach imposes a greater penalty on misclassified samples) then:
\n$$-\\big(y-\\frac{\\text{F1}}{2}\\big)\\nabla_{\\theta}p_{\\theta}(x)$$
$$=0+-y*\\big(1-\\frac{\\text{F1}}{2}\\big)\\nabla_{\\theta}log(p_{\\theta}(x))$$
\n$$-(1-y)*\\big(\\frac{\\text{F1}}{2}\\big)\\nabla_{\\theta}log(1-p_{\\theta}(x))$$
\nThis is equivalent to weighted cross-entropy.
\nWe can extend this to the multi-class case.
My code is as follows. Using this custom loss function increases my CV ~ 0.002. Although it's a small increase, I believe there's still room for improvement. Here, I only considered macro F1, but we could also incorporate binary F1.
\nI also look forward to hearing more thoughts from everyone on this topic.
\n tensorflow tf\n tensorflow.keras backend K\n\n (tf.keras.losses.Loss):\n ():\n ().__init__(name=name)\n .class_weights = class_weights\n\n ():\n cce = tf.keras.losses.categorical_crossentropy(y_true, y_pred, from_logits=,label_smoothing=)\n\n weighted_cce = cce * K.(y_true * .class_weights, axis=-)\n K.mean(weighted_cce)\n tensorflow.keras.callbacks Callback\n sklearn.metrics f1_score\n numpy np\n\n ():\n (): \n (F1WeightsUpdater, ).__init__()\n .class_weights_var = class_weights_var\n .num_classes = num_classes\n .validation_data = validation_data \n\n ():\n \n X_val, y_val_cat = .validation_data[:] \n\n y_pred_proba = .model.predict(X_val, verbose=)\n\n y_val_labels = np.argmax(y_val_cat, axis=)\n y_pred_labels = np.argmax(y_pred_proba, axis=)\n\n f1_scores = f1_score(y_val_labels, y_pred_labels, labels=(.num_classes), average=)\n new_weights = -f1_scores/ \n .class_weights_var.assign(new_weights)\n\ninitial_weights = tf.Variable([] * , dtype=tf.float32, trainable=)\nweighted_loss = UpdatableWeightedLoss(initial_weights)\nf1_updater_callback = F1WeightsUpdater(\n initial_weights,\n ,\n validation_data=(X_val, y_val) \n)\nmodel.(loss=weighted_loss)\nmodel.fit(callbacks=[f1_updater_callback])\nReference：
\nhttps://kexue.fm/archives/7708
Seems like a good approach and you got incrementally better results on your CV, but doesn't it cause indirectly overfit on your validation sets? Did you also see improvements on LB?
", "rawMarkdown": "Seems like a good approach and you got incrementally better results on your CV, but doesn't it cause indirectly overfit on your validation sets? Did you also see improvements on LB?", "votes": null }, { "id": "3225496", "postDate": "06/16/2025 13:38:11", "content": "I haven't made a submission on my KFOLD model with this loss yet.
\nBut if using the train_test_split method widely used in public notebooks, unfortunately, the public LB score will drop.
\nI think your statement is also reasonable. The right practice should be adjusting the weight on the training set, instead of using the information from the validation set as in the current code.
I applied this to my current training pipeline, but the CV score dropped by approximately 0.005. So far, every attempt to replace CE loss—including adding a smoothing rate—has been unsuccessful for me.
", "rawMarkdown": "I applied this to my current training pipeline, but the CV score dropped by approximately 0.005. So far, every attempt to replace CE loss—including adding a smoothing rate—has been unsuccessful for me.", "votes": null }, { "id": "3225791", "postDate": "06/16/2025 21:02:30", "content": "Same here, and I tried a lot so far. Thanks for sharing! :)
", "rawMarkdown": "Same here, and I tried a lot so far. Thanks for sharing! :)", "votes": null } ], "comments": [ { "id": 3225336, "author_name": "nlztrk", "author_url": "", "post_date": "06/16/2025 09:18:22", "content": "Seems like a good approach and you got incrementally better results on your CV, but doesn't it cause indirectly overfit on your validation sets? Did you also see improvements on LB?
", "votes": null, "replies": [ { "id": 3225496, "author_name": "act18l", "author_url": "", "post_date": "06/16/2025 13:38:11", "content": "I haven't made a submission on my KFOLD model with this loss yet.
\nBut if using the train_test_split method widely used in public notebooks, unfortunately, the public LB score will drop.
\nI think your statement is also reasonable. The right practice should be adjusting the weight on the training set, instead of using the information from the validation set as in the current code.
I applied this to my current training pipeline, but the CV score dropped by approximately 0.005. So far, every attempt to replace CE loss—including adding a smoothing rate—has been unsuccessful for me.
", "votes": null, "replies": [ { "id": 3225791, "author_name": "janbrederecke", "author_url": "", "post_date": "06/16/2025 21:02:30", "content": "Same here, and I tried a lot so far. Thanks for sharing! :)
", "votes": null, "replies": [] } ] } ], "raw_markdown_by_id": { "3225199": "For simplicity, we only analyze the macro F1.\n\n$$F1=\\frac{2TP}{(TP+FN)+(TP+FP)}\\approx \\frac{2\\sum yp_\\theta(x)}{\\sum y+p_\\theta(x)}$$\n\nwhere p(x) is the probability of the positive class, and y is the label of x (0 or 1).\n\n$$-\\nabla_{\\theta}\\frac{2 y p_{\\theta}(x)}{y + p_{\\theta}(x)}$$\n\n$$=-2\\frac{y \\nabla_{\\theta}p_{\\theta}(x)}{y + p_{\\theta}(x)} + 2\\frac{y p_{\\theta}(x)\\nabla_{\\theta}p_{\\theta}(x)}{\\left(y + p_{\\theta}(x)\\right)^2}$$\n$$=-\\frac{2\\big(y-\\frac{\\text{F1}}{2}\\big)\\nabla_{\\theta}p_{\\theta}(x)}{\\big[y + p_{\\theta}(x)\\big]} $$\n\nWe can use \\\\(p\\approx log(p)+1,-p \\approx log(1-p)\\\\) (This approach imposes a greater penalty on misclassified samples) then:\n$$-\\big(y-\\frac{\\text{F1}}{2}\\big)\\nabla_{\\theta}p_{\\theta}(x)$$\n\n$$=0+-y*\\big(1-\\frac{\\text{F1}}{2}\\big)\\nabla_{\\theta}log(p_{\\theta}(x))$$\n\n$$-(1-y)*\\big(\\frac{\\text{F1}}{2}\\big)\\nabla_{\\theta}log(1-p_{\\theta}(x))$$\n\nThis is equivalent to weighted cross-entropy. \nWe can extend this to the multi-class case.\n\nMy code is as follows. Using this custom loss function increases my CV ~ 0.002. Although it's a small increase, I believe there's still room for improvement. Here, I only considered macro F1, but we could also incorporate binary F1.\n\nI also look forward to hearing more thoughts from everyone on this topic.\n\n\n```python\nimport tensorflow as tf\nfrom tensorflow.keras import backend as K\n\nclass UpdatableWeightedLoss(tf.keras.losses.Loss):\n def __init__(self, class_weights, name='updatable_weighted_loss'):\n super().__init__(name=name)\n self.class_weights = class_weights\n\n def call(self, y_true, y_pred):\n cce = tf.keras.losses.categorical_crossentropy(y_true, y_pred, from_logits=False,label_smoothing=0.1)\n \n weighted_cce = cce * K.sum(y_true * self.class_weights, axis=-1)\n return K.mean(weighted_cce)\nfrom tensorflow.keras.callbacks import Callback\nfrom sklearn.metrics import f1_score\nimport numpy as np\n\nclass F1WeightsUpdater(Callback):\n def __init__(self, class_weights_var, num_classes, validation_data): \n super(F1WeightsUpdater, self).__init__()\n self.class_weights_var = class_weights_var\n self.num_classes = num_classes\n self.validation_data = validation_data \n\n def on_epoch_end(self, epoch, logs=None):\n # Use the stored validation data instead of self.model.validation_data\n X_val, y_val_cat = self.validation_data[0:2] \n\n y_pred_proba = self.model.predict(X_val, verbose=0)\n \n y_val_labels = np.argmax(y_val_cat, axis=1)\n y_pred_labels = np.argmax(y_pred_proba, axis=1)\n\n f1_scores = f1_score(y_val_labels, y_pred_labels, labels=range(self.num_classes), average=None)\n new_weights = 1-f1_scores/2 \n self.class_weights_var.assign(new_weights)\n \ninitial_weights = tf.Variable([1.0] * 18, dtype=tf.float32, trainable=False)\nweighted_loss = UpdatableWeightedLoss(initial_weights)\nf1_updater_callback = F1WeightsUpdater(\n initial_weights,\n 18,\n validation_data=(X_val, y_val) \n)\nmodel.compile(loss=weighted_loss)\nmodel.fit(callbacks=[f1_updater_callback])\n```\n\n**Reference：**\nhttps://kexue.fm/archives/7708", "3225336": "Seems like a good approach and you got incrementally better results on your CV, but doesn't it cause indirectly overfit on your validation sets? Did you also see improvements on LB?", "3225496": "I haven't made a submission on my KFOLD model with this loss yet.\nBut if using the `train_test_split` method widely used in public notebooks, unfortunately, the public LB score will drop.\nI think your statement is also reasonable. The right practice should be adjusting the weight on the training set, instead of using the information from the validation set as in the current code.", "3225746": "I applied this to my current training pipeline, but the CV score dropped by approximately 0.005. So far, every attempt to replace CE loss—including adding a smoothing rate—has been unsuccessful for me.", "3225791": "Same here, and I tried a lot so far. Thanks for sharing! :)" }, "source": "meta" }