{ "id": 93870, "title": "what's your CV Metric Score ?", "url": "/competitions/champs-scalar-coupling/discussion/93870", "author_name": "", "post_date": "2019-05-30T17:52:56.421334800Z", "votes": 3, "comment_count": 14, "views": 0, "content": "

I fixed competition metric by @abhishek with this code by @pestipeti

\n\n

np.log(p.max([metrics.mean_absolute_error(y_true, y_pred), 1e-9]))\ncheck: leak or not

\n\n

My question is: using this fixed metric, what's your best CV score? I just want to make sure that I'm using properly the metric :) For example, with a simple model I obtained this

\n\n

metric(train, oof) = \n0.652302120507013\n

\n\n

Thank you all in advance

", "messages": [ { "id": "539919", "postDate": "05/30/2019 17:52:56", "content": "

I fixed competition metric by @abhishek with this code by @pestipeti

\n\n

np.log(p.max([metrics.mean_absolute_error(y_true, y_pred), 1e-9]))\ncheck: leak or not

\n\n

My question is: using this fixed metric, what's your best CV score? I just want to make sure that I'm using properly the metric :) For example, with a simple model I obtained this

\n\n

metric(train, oof) = \n0.652302120507013\n

\n\n

Thank you all in advance

", "rawMarkdown": "I fixed [competition metric](https://www.kaggle.com/abhishek/competition-metric) by @abhishek with this code by @pestipeti \n\n`np.log(p.max([metrics.mean_absolute_error(y_true, y_pred), 1e-9]))`\ncheck: [leak or not](https://www.kaggle.com/c/champs-scalar-coupling/discussion/93861#latest-539888)\n\nMy question is: using this fixed metric, what's your best CV score? I just want to make sure that I'm using properly the metric :) For example, with a simple model I obtained this\n\n`metric(train, oof) = \n0.652302120507013\n`\n\nThank you all in advance", "votes": null }, { "id": "539921", "postDate": "05/30/2019 17:59:11", "content": "

i just use the function i shared as described on the evaluation page. my cv and lb are very close to each other.

", "rawMarkdown": "i just use the function i shared as described on the evaluation page. my cv and lb are very close to each other.", "votes": null }, { "id": "539925", "postDate": "05/30/2019 18:06:12", "content": "

It does not matter that much. Only when you have a perfect score in any group. In that case the mae (for that group) would equals to 0 and you can't calculate log(0). that is why kaggle added a floor of 1e-9. IMHO nobody will achieve a perfect score..

", "rawMarkdown": "It does not matter that much. Only when you have a perfect score in any group. In that case the mae (for that group) would equals to 0 and you can't calculate log(0). that is why kaggle added a floor of 1e-9. IMHO nobody will achieve a perfect score..", "votes": null }, { "id": "539929", "postDate": "05/30/2019 18:12:34", "content": "

@abhishek your script is perfectly fine, I wanted to check the score of the 'perfect' prediction by this call

\n\n

metric(train, train['scalar_coupling_constant'].values)

\n\n

and it crashed. Nobody will have a perfect score, so technically it has a bug, but does not matter.

", "rawMarkdown": "abhishek your script is perfectly fine, I wanted to check the score of the 'perfect' prediction by this call\n\n`metric(train, train['scalar_coupling_constant'].values)`\n\nand it crashed. Nobody will have a perfect score, so technically it has a bug, but does not matter.", "votes": null }, { "id": "539944", "postDate": "05/30/2019 18:33:27", "content": "

Thank you guys, I checked it and LB = 0.626 CV metric=0.652302120~ 👍 \nI just wanted to make sure ;)

", "rawMarkdown": "Thank you guys, I checked it and LB = 0.626 CV metric=0.652302120~ 👍 \nI just wanted to make sure ;)", "votes": null }, { "id": "540839", "postDate": "06/01/2019 08:06:57", "content": "

In one of my recent models i got a CV score of 0.38 and LB of 1.16 WTF!!!!!!!!!

", "rawMarkdown": "In one of my recent models i got a CV score of 0.38 and LB of 1.16 WTF!!!!!!!!!", "votes": null }, { "id": "540856", "postDate": "06/01/2019 08:35:39", "content": "

It happened to me too.

", "rawMarkdown": "It happened to me too.", "votes": null }, { "id": "540879", "postDate": "06/01/2019 09:33:37", "content": "

what did u do wrong? it happened to me when i included @artgor ’s features in my model.

", "rawMarkdown": "what did u do wrong? it happened to me when i included @artgor ’s features in my model.", "votes": null }, { "id": "540884", "postDate": "06/01/2019 09:48:59", "content": "

This is quite strange

", "rawMarkdown": "This is quite strange", "votes": null }, { "id": "540891", "postDate": "06/01/2019 10:02:52", "content": "

yes, it is.

", "rawMarkdown": "yes, it is.", "votes": null }, { "id": "540897", "postDate": "06/01/2019 10:12:54", "content": "

its because im stupid.

", "rawMarkdown": "its because im stupid.", "votes": null }, { "id": "540898", "postDate": "06/01/2019 10:14:53", "content": "

Lol, I didn't try those features

", "rawMarkdown": "Lol, I didn't try those features", "votes": null }, { "id": "541365", "postDate": "06/02/2019 10:08:19", "content": "

CV/LB are back on track

", "rawMarkdown": "CV/LB are back on track", "votes": null }, { "id": "543155", "postDate": "06/04/2019 11:01:44", "content": "

CV mean score: -0.2723, std: 0.0021.\nPublic Score: -0.407

", "rawMarkdown": "CV mean score: -0.2723, std: 0.0021.\nPublic Score: -0.407", "votes": null }, { "id": "543157", "postDate": "06/04/2019 11:02:16", "content": "

At last I realized that these features were quite useless :)

", "rawMarkdown": "At last I realized that these features were quite useless :)", "votes": null } ], "comments": [ { "id": 539921, "author_name": "abhishek", "author_url": "", "post_date": "05/30/2019 17:59:11", "content": "

i just use the function i shared as described on the evaluation page. my cv and lb are very close to each other.

", "votes": null, "replies": [ { "id": 539929, "author_name": "pestipeti", "author_url": "", "post_date": "05/30/2019 18:12:34", "content": "

@abhishek your script is perfectly fine, I wanted to check the score of the 'perfect' prediction by this call

\n\n

metric(train, train['scalar_coupling_constant'].values)

\n\n

and it crashed. Nobody will have a perfect score, so technically it has a bug, but does not matter.

", "votes": null, "replies": [] } ] }, { "id": 539925, "author_name": "pestipeti", "author_url": "", "post_date": "05/30/2019 18:06:12", "content": "

It does not matter that much. Only when you have a perfect score in any group. In that case the mae (for that group) would equals to 0 and you can't calculate log(0). that is why kaggle added a floor of 1e-9. IMHO nobody will achieve a perfect score..

", "votes": null, "replies": [] }, { "id": 539944, "author_name": "jesucristo", "author_url": "", "post_date": "05/30/2019 18:33:27", "content": "

Thank you guys, I checked it and LB = 0.626 CV metric=0.652302120~ 👍 \nI just wanted to make sure ;)

", "votes": null, "replies": [] }, { "id": 540839, "author_name": "abhishek", "author_url": "", "post_date": "06/01/2019 08:06:57", "content": "

In one of my recent models i got a CV score of 0.38 and LB of 1.16 WTF!!!!!!!!!

", "votes": null, "replies": [ { "id": 540856, "author_name": "adilztn", "author_url": "", "post_date": "06/01/2019 08:35:39", "content": "

It happened to me too.

", "votes": null, "replies": [] }, { "id": 540879, "author_name": "abhishek", "author_url": "", "post_date": "06/01/2019 09:33:37", "content": "

what did u do wrong? it happened to me when i included @artgor ’s features in my model.

", "votes": null, "replies": [] }, { "id": 540884, "author_name": "artgor", "author_url": "", "post_date": "06/01/2019 09:48:59", "content": "

This is quite strange

", "votes": null, "replies": [] }, { "id": 540891, "author_name": "abhishek", "author_url": "", "post_date": "06/01/2019 10:02:52", "content": "

yes, it is.

", "votes": null, "replies": [] }, { "id": 540897, "author_name": "abhishek", "author_url": "", "post_date": "06/01/2019 10:12:54", "content": "

its because im stupid.

", "votes": null, "replies": [] }, { "id": 540898, "author_name": "jesucristo", "author_url": "", "post_date": "06/01/2019 10:14:53", "content": "

Lol, I didn't try those features

", "votes": null, "replies": [] }, { "id": 541365, "author_name": "abhishek", "author_url": "", "post_date": "06/02/2019 10:08:19", "content": "

CV/LB are back on track

", "votes": null, "replies": [] }, { "id": 543157, "author_name": "artgor", "author_url": "", "post_date": "06/04/2019 11:02:16", "content": "

At last I realized that these features were quite useless :)

", "votes": null, "replies": [] } ] }, { "id": 543155, "author_name": "artgor", "author_url": "", "post_date": "06/04/2019 11:01:44", "content": "

CV mean score: -0.2723, std: 0.0021.\nPublic Score: -0.407

", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "539919": "I fixed [competition metric](https://www.kaggle.com/abhishek/competition-metric) by @abhishek with this code by @pestipeti \n\n`np.log(p.max([metrics.mean_absolute_error(y_true, y_pred), 1e-9]))`\ncheck: [leak or not](https://www.kaggle.com/c/champs-scalar-coupling/discussion/93861#latest-539888)\n\nMy question is: using this fixed metric, what's your best CV score? I just want to make sure that I'm using properly the metric :) For example, with a simple model I obtained this\n\n`metric(train, oof) = \n0.652302120507013\n`\n\nThank you all in advance", "539921": "i just use the function i shared as described on the evaluation page. my cv and lb are very close to each other.", "539925": "It does not matter that much. Only when you have a perfect score in any group. In that case the mae (for that group) would equals to 0 and you can't calculate log(0). that is why kaggle added a floor of 1e-9. IMHO nobody will achieve a perfect score..", "539929": "abhishek your script is perfectly fine, I wanted to check the score of the 'perfect' prediction by this call\n\n`metric(train, train['scalar_coupling_constant'].values)`\n\nand it crashed. Nobody will have a perfect score, so technically it has a bug, but does not matter.", "539944": "Thank you guys, I checked it and LB = 0.626 CV metric=0.652302120~ 👍 \nI just wanted to make sure ;)", "540839": "In one of my recent models i got a CV score of 0.38 and LB of 1.16 WTF!!!!!!!!!", "540856": "It happened to me too.", "540879": "what did u do wrong? it happened to me when i included @artgor ’s features in my model.", "540884": "This is quite strange", "540891": "yes, it is.", "540897": "its because im stupid.", "540898": "Lol, I didn't try those features", "541365": "CV/LB are back on track", "543155": "CV mean score: -0.2723, std: 0.0021.\nPublic Score: -0.407", "543157": "At last I realized that these features were quite useless :)" }, "source": "meta" }