I managed to build a custom tf loss for weighted correlation. The code of the loss is rather straightforward (see below), but how to use it with weights a bit less. I show how to do so in this NN baseline notebook. I might have missed another public notebook but I believe this is the first public NN with an adaptated loss and a decent atempt at feature engineering that actually submit.
\nThe notebook heavily rely on my previous Online Feature engineering work for feature design / CV design / submission (see notebook here: https://www.kaggle.com/lucasmorin/on-line-feature-engineering).
\nOn convergence of the NNs: without ton of noise the nn usually overfit widly (performance on train way above the test performance). For the moment the convergence is rather poor: +/-1% inside a fold. Very different results across folds.
\nThe notebook include two feature importance solutions (from @cdeotte work in the Google Ventilator Pressure Prediction Challenge) based on shuffling features. The solution can be rather slow so it is possible to deactivate it easily. I tought this would be an aproriate feature importance in this noisy setup. A similar feature importance trough SHAP approximation of Shapley values. A feature importance plot for a nice visualisation of importance across folds (from @nyanp currently winning solution from Optiver volatility forecasting competition).
\nI've tried some additional tricks without much results (normalisation of targets, post processing, removing the worst models from fold 4. Not really working.
\nFeel free to give your toughts about this below (and don't forget to upvote the notebook if you find it usefull).
\nCustom TF-Keras loss:
\ndef wcorr_loss(y_true, y_pred, w):\n x = tf.cast(y_true, tf.float32)\n y = tf.cast(y_pred, tf.float32)\n w = tf.cast(w, tf.float32)\n wmx = K.sum(tf.multiply(x, w)) / K.sum(w)\n wmy = K.sum(tf.multiply(y, w)) / K.sum(w)\n xm, ym = x-wmx, y-wmy\n tfwcovxy = K.sum( tf.multiply(tf.multiply(xm, w), ym)) / K.sum(w)\n tfwcovxx = K.sum( tf.multiply(tf.multiply(xm, w), xm)) / K.sum(w)\n tfwcovyy = K.sum( tf.multiply(tf.multiply(ym, w), ym)) / K.sum(w)\n r = tfwcovxy / K.sqrt(tf.multiply(tfwcovxx , tfwcovyy))\n r = K.maximum(K.minimum(r, 1.0), -1.0)\n return - r\n\ndef wcorr_fn(w):\n def wcorr_eval(y_true, y_pred):\n return wcorr_loss(y_true, y_pred, w)\n**
", "messages": [ { "id": "1643943", "postDate": "01/09/2022 21:43:31", "content": "I managed to build a custom tf loss for weighted correlation. The code of the loss is rather straightforward (see below), but how to use it with weights a bit less. I show how to do so in this NN baseline notebook. I might have missed another public notebook but I believe this is the first public NN with an adaptated loss and a decent atempt at feature engineering that actually submit.
\nThe notebook heavily rely on my previous Online Feature engineering work for feature design / CV design / submission (see notebook here: https://www.kaggle.com/lucasmorin/on-line-feature-engineering).
\nOn convergence of the NNs: without ton of noise the nn usually overfit widly (performance on train way above the test performance). For the moment the convergence is rather poor: +/-1% inside a fold. Very different results across folds.
\nThe notebook include two feature importance solutions (from @cdeotte work in the Google Ventilator Pressure Prediction Challenge) based on shuffling features. The solution can be rather slow so it is possible to deactivate it easily. I tought this would be an aproriate feature importance in this noisy setup. A similar feature importance trough SHAP approximation of Shapley values. A feature importance plot for a nice visualisation of importance across folds (from @nyanp currently winning solution from Optiver volatility forecasting competition).
\nI've tried some additional tricks without much results (normalisation of targets, post processing, removing the worst models from fold 4. Not really working.
\nFeel free to give your toughts about this below (and don't forget to upvote the notebook if you find it usefull).
\nCustom TF-Keras loss:
\ndef wcorr_loss(y_true, y_pred, w):\n x = tf.cast(y_true, tf.float32)\n y = tf.cast(y_pred, tf.float32)\n w = tf.cast(w, tf.float32)\n wmx = K.sum(tf.multiply(x, w)) / K.sum(w)\n wmy = K.sum(tf.multiply(y, w)) / K.sum(w)\n xm, ym = x-wmx, y-wmy\n tfwcovxy = K.sum( tf.multiply(tf.multiply(xm, w), ym)) / K.sum(w)\n tfwcovxx = K.sum( tf.multiply(tf.multiply(xm, w), xm)) / K.sum(w)\n tfwcovyy = K.sum( tf.multiply(tf.multiply(ym, w), ym)) / K.sum(w)\n r = tfwcovxy / K.sqrt(tf.multiply(tfwcovxx , tfwcovyy))\n r = K.maximum(K.minimum(r, 1.0), -1.0)\n return - r\n\ndef wcorr_fn(w):\n def wcorr_eval(y_true, y_pred):\n return wcorr_loss(y_true, y_pred, w)\n**
", "rawMarkdown": "I managed to build a custom tf loss for weighted correlation. The code of the loss is rather straightforward (see below), but how to use it with weights a bit less. I show how to do so in this **[NN baseline notebook]( https://www.kaggle.com/lucasmorin/online-fe-nn-loss-shap )**. I might have missed another public notebook but I believe this is the first public NN with an adaptated loss and a decent atempt at feature engineering that actually submit. \n\nThe notebook heavily rely on my previous Online Feature engineering work for feature design / CV design / submission (see notebook here: https://www.kaggle.com/lucasmorin/on-line-feature-engineering).\n\n**On convergence of the NNs**: without ton of noise the nn usually overfit widly (performance on train way above the test performance). For the moment the convergence is rather poor: +/-1% inside a fold. Very different results across folds. \n\n**The notebook include two feature importance solutions** (from @cdeotte work in the Google Ventilator Pressure Prediction Challenge) based on shuffling features. The solution can be rather slow so it is possible to deactivate it easily. I tought this would be an aproriate feature importance in this noisy setup. A similar feature importance trough SHAP approximation of Shapley values. A feature importance plot for a nice visualisation of importance across folds (from @nyanp currently winning solution from Optiver volatility forecasting competition).\n \nI've tried some **additional tricks** without much results (normalisation of targets, post processing, removing the worst models from fold 4. Not really working.\n\nFeel free to give your toughts about this below (and don't forget to upvote the notebook if you find it usefull).\n\n---\n\nCustom TF-Keras loss:\n\n```\ndef wcorr_loss(y_true, y_pred, w):\n x = tf.cast(y_true, tf.float32)\n y = tf.cast(y_pred, tf.float32)\n w = tf.cast(w, tf.float32)\n wmx = K.sum(tf.multiply(x, w)) / K.sum(w)\n wmy = K.sum(tf.multiply(y, w)) / K.sum(w)\n xm, ym = x-wmx, y-wmy\n tfwcovxy = K.sum( tf.multiply(tf.multiply(xm, w), ym)) / K.sum(w)\n tfwcovxx = K.sum( tf.multiply(tf.multiply(xm, w), xm)) / K.sum(w)\n tfwcovyy = K.sum( tf.multiply(tf.multiply(ym, w), ym)) / K.sum(w)\n r = tfwcovxy / K.sqrt(tf.multiply(tfwcovxx , tfwcovyy))\n r = K.maximum(K.minimum(r, 1.0), -1.0)\n return - r\n\ndef wcorr_fn(w):\n def wcorr_eval(y_true, y_pred):\n return wcorr_loss(y_true, y_pred, w)\n```\n**", "votes": null }, { "id": "1646659", "postDate": "01/11/2022 22:45:18", "content": "Thanks for the loss function. This helped my NN converge better. I also utilize MAE loss for each coin which might be an additional trick that helps your NN do better.
", "rawMarkdown": "Thanks for the loss function. This helped my NN converge better. I also utilize MAE loss for each coin which might be an additional trick that helps your NN do better.", "votes": null }, { "id": "1647084", "postDate": "01/12/2022 09:49:15", "content": "When I trained directly with the weighted correlation, I did not get very good results. I also found that the predicted values are very \"off\" because of the normalisation (not that it matters for this competition) but it just annoyed me. So I tried training first with MAE then once it reaches a local optima, I switch the loss function to weighted corr / mae. This ended up attaining much better weighted corr. You can try to see if it works for you. Loss function code below.
\n`
def tf_weighted_correlation(a, b, weights):\n\n w = np.ravel(weights)\n sum_w = np.sum(w)\n mean_a = tf.reduce_sum(a * w, axis=-1) / sum_w\n mean_b = tf.reduce_sum(b * w, axis=-1) / sum_w \n var_a = tf.reduce_sum(w[np.newaxis,...] * tf.pow(a - tf.expand_dims(mean_a, axis=-1), 2), axis=-1) / sum_w\n var_b = tf.reduce_sum(w[np.newaxis,...] * tf.pow(b - tf.expand_dims(mean_b, axis=-1), 2), axis=-1) / sum_w\n\n cov = tf.reduce_sum((a * b * w[np.newaxis,...]), axis=-1) / sum_w - mean_a * mean_b \n corr = cov / tf.sqrt(var_a * var_b) \n\n return -tf.reduce_mean(corr)\n\ndef corr_loss(y_true, y_pred): \n weights = df_asset_details['Weight'].to_numpy()\n return tf_weighted_correlation(y_true, y_pred, weights)\n\ndef corr_mae_loss(y_true, y_pred):\n neg_corr = corr_loss(y_true, y_pred)\n mae = tf.keras.losses.MeanAbsoluteError()\n mae_loss = mae(y_true, y_pred)\n return neg_corr / mae_loss\n`
", "rawMarkdown": "When I trained directly with the weighted correlation, I did not get very good results. I also found that the predicted values are very \"off\" because of the normalisation (not that it matters for this competition) but it just annoyed me. So I tried training first with MAE then once it reaches a local optima, I switch the loss function to weighted corr / mae. This ended up attaining much better weighted corr. You can try to see if it works for you. Loss function code below.\n`\n\n def tf_weighted_correlation(a, b, weights):\n\n w = np.ravel(weights)\n sum_w = np.sum(w)\n mean_a = tf.reduce_sum(a * w, axis=-1) / sum_w\n mean_b = tf.reduce_sum(b * w, axis=-1) / sum_w \n var_a = tf.reduce_sum(w[np.newaxis,...] * tf.pow(a - tf.expand_dims(mean_a, axis=-1), 2), axis=-1) / sum_w\n var_b = tf.reduce_sum(w[np.newaxis,...] * tf.pow(b - tf.expand_dims(mean_b, axis=-1), 2), axis=-1) / sum_w\n\n cov = tf.reduce_sum((a * b * w[np.newaxis,...]), axis=-1) / sum_w - mean_a * mean_b \n corr = cov / tf.sqrt(var_a * var_b) \n\n return -tf.reduce_mean(corr)\n\n def corr_loss(y_true, y_pred): \n weights = df_asset_details['Weight'].to_numpy()\n return tf_weighted_correlation(y_true, y_pred, weights)\n\n def corr_mae_loss(y_true, y_pred):\n neg_corr = corr_loss(y_true, y_pred)\n mae = tf.keras.losses.MeanAbsoluteError()\n mae_loss = mae(y_true, y_pred)\n return neg_corr / mae_loss\n`", "votes": null } ], "comments": [ { "id": 1646659, "author_name": "petersk20", "author_url": "", "post_date": "01/11/2022 22:45:18", "content": "Thanks for the loss function. This helped my NN converge better. I also utilize MAE loss for each coin which might be an additional trick that helps your NN do better.
", "votes": null, "replies": [] }, { "id": 1647084, "author_name": "luchungi", "author_url": "", "post_date": "01/12/2022 09:49:15", "content": "When I trained directly with the weighted correlation, I did not get very good results. I also found that the predicted values are very \"off\" because of the normalisation (not that it matters for this competition) but it just annoyed me. So I tried training first with MAE then once it reaches a local optima, I switch the loss function to weighted corr / mae. This ended up attaining much better weighted corr. You can try to see if it works for you. Loss function code below.
\n`
def tf_weighted_correlation(a, b, weights):\n\n w = np.ravel(weights)\n sum_w = np.sum(w)\n mean_a = tf.reduce_sum(a * w, axis=-1) / sum_w\n mean_b = tf.reduce_sum(b * w, axis=-1) / sum_w \n var_a = tf.reduce_sum(w[np.newaxis,...] * tf.pow(a - tf.expand_dims(mean_a, axis=-1), 2), axis=-1) / sum_w\n var_b = tf.reduce_sum(w[np.newaxis,...] * tf.pow(b - tf.expand_dims(mean_b, axis=-1), 2), axis=-1) / sum_w\n\n cov = tf.reduce_sum((a * b * w[np.newaxis,...]), axis=-1) / sum_w - mean_a * mean_b \n corr = cov / tf.sqrt(var_a * var_b) \n\n return -tf.reduce_mean(corr)\n\ndef corr_loss(y_true, y_pred): \n weights = df_asset_details['Weight'].to_numpy()\n return tf_weighted_correlation(y_true, y_pred, weights)\n\ndef corr_mae_loss(y_true, y_pred):\n neg_corr = corr_loss(y_true, y_pred)\n mae = tf.keras.losses.MeanAbsoluteError()\n mae_loss = mae(y_true, y_pred)\n return neg_corr / mae_loss\n`
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "1643943": "I managed to build a custom tf loss for weighted correlation. The code of the loss is rather straightforward (see below), but how to use it with weights a bit less. I show how to do so in this **[NN baseline notebook]( https://www.kaggle.com/lucasmorin/online-fe-nn-loss-shap )**. I might have missed another public notebook but I believe this is the first public NN with an adaptated loss and a decent atempt at feature engineering that actually submit. \n\nThe notebook heavily rely on my previous Online Feature engineering work for feature design / CV design / submission (see notebook here: https://www.kaggle.com/lucasmorin/on-line-feature-engineering).\n\n**On convergence of the NNs**: without ton of noise the nn usually overfit widly (performance on train way above the test performance). For the moment the convergence is rather poor: +/-1% inside a fold. Very different results across folds. \n\n**The notebook include two feature importance solutions** (from @cdeotte work in the Google Ventilator Pressure Prediction Challenge) based on shuffling features. The solution can be rather slow so it is possible to deactivate it easily. I tought this would be an aproriate feature importance in this noisy setup. A similar feature importance trough SHAP approximation of Shapley values. A feature importance plot for a nice visualisation of importance across folds (from @nyanp currently winning solution from Optiver volatility forecasting competition).\n \nI've tried some **additional tricks** without much results (normalisation of targets, post processing, removing the worst models from fold 4. Not really working.\n\nFeel free to give your toughts about this below (and don't forget to upvote the notebook if you find it usefull).\n\n---\n\nCustom TF-Keras loss:\n\n```\ndef wcorr_loss(y_true, y_pred, w):\n x = tf.cast(y_true, tf.float32)\n y = tf.cast(y_pred, tf.float32)\n w = tf.cast(w, tf.float32)\n wmx = K.sum(tf.multiply(x, w)) / K.sum(w)\n wmy = K.sum(tf.multiply(y, w)) / K.sum(w)\n xm, ym = x-wmx, y-wmy\n tfwcovxy = K.sum( tf.multiply(tf.multiply(xm, w), ym)) / K.sum(w)\n tfwcovxx = K.sum( tf.multiply(tf.multiply(xm, w), xm)) / K.sum(w)\n tfwcovyy = K.sum( tf.multiply(tf.multiply(ym, w), ym)) / K.sum(w)\n r = tfwcovxy / K.sqrt(tf.multiply(tfwcovxx , tfwcovyy))\n r = K.maximum(K.minimum(r, 1.0), -1.0)\n return - r\n\ndef wcorr_fn(w):\n def wcorr_eval(y_true, y_pred):\n return wcorr_loss(y_true, y_pred, w)\n```\n**", "1646659": "Thanks for the loss function. This helped my NN converge better. I also utilize MAE loss for each coin which might be an additional trick that helps your NN do better.", "1647084": "When I trained directly with the weighted correlation, I did not get very good results. I also found that the predicted values are very \"off\" because of the normalisation (not that it matters for this competition) but it just annoyed me. So I tried training first with MAE then once it reaches a local optima, I switch the loss function to weighted corr / mae. This ended up attaining much better weighted corr. You can try to see if it works for you. Loss function code below.\n`\n\n def tf_weighted_correlation(a, b, weights):\n\n w = np.ravel(weights)\n sum_w = np.sum(w)\n mean_a = tf.reduce_sum(a * w, axis=-1) / sum_w\n mean_b = tf.reduce_sum(b * w, axis=-1) / sum_w \n var_a = tf.reduce_sum(w[np.newaxis,...] * tf.pow(a - tf.expand_dims(mean_a, axis=-1), 2), axis=-1) / sum_w\n var_b = tf.reduce_sum(w[np.newaxis,...] * tf.pow(b - tf.expand_dims(mean_b, axis=-1), 2), axis=-1) / sum_w\n\n cov = tf.reduce_sum((a * b * w[np.newaxis,...]), axis=-1) / sum_w - mean_a * mean_b \n corr = cov / tf.sqrt(var_a * var_b) \n\n return -tf.reduce_mean(corr)\n\n def corr_loss(y_true, y_pred): \n weights = df_asset_details['Weight'].to_numpy()\n return tf_weighted_correlation(y_true, y_pred, weights)\n\n def corr_mae_loss(y_true, y_pred):\n neg_corr = corr_loss(y_true, y_pred)\n mae = tf.keras.losses.MeanAbsoluteError()\n mae_loss = mae(y_true, y_pred)\n return neg_corr / mae_loss\n`" }, "source": "meta" }