I'm sorry for bothering you. I found I'm wrong. Thank you.
\n~~I tried some LB proving and the results suggested that LB scoring is wrong. The detail is shown here.
\nI think Beta is not 0.5 but about 1.2.
\n@philculliton, @julialane Please verify this problem.
If really Beta=0.5, it must be so easy to make better submission than the string-matching baseline model because the FN labels are lesser penalized. However, very few teams achieved this actually. If Beta is bigger, the FNs will heavily be penalized and models exploring the unseen datasets can not beat the baseline. This strongly discourages competitors to make models that aggressively search unseen datasets. It ruins the meaning of this competition the host intends to.~~
I'm sorry for bothering you. I found I'm wrong. Thank you.
\n~~I tried some LB proving and the results suggested that LB scoring is wrong. The detail is shown here.
\nI think Beta is not 0.5 but about 1.2.
\n@philculliton, @julialane Please verify this problem.
If really Beta=0.5, it must be so easy to make better submission than the string-matching baseline model because the FN labels are lesser penalized. However, very few teams achieved this actually. If Beta is bigger, the FNs will heavily be penalized and models exploring the unseen datasets can not beat the baseline. This strongly discourages competitors to make models that aggressively search unseen datasets. It ruins the meaning of this competition the host intends to.~~
Hi! Beta is definitely 0.5. FN labels are less penalized than FPs in that context.
I think it's working as intended.
\n\n\nIf really Beta=0.5, it must be so easy to make better submission than the string-matching baseline model because the FN labels are lesser penalized
\n
I don't think this is a valid assumption, it depends on the model. In my experience ones that generalize will give out a lot false positives. On the other hand, safe methods like string matching has almost zero false positives and which is why it's so hard to beat.
\n\n\nIf Beta is bigger, the FNs will heavily be penalized and models exploring the unseen datasets can not beat the baseline.
\n
This too, should it be the other way around? If Beta is bigger, the FNs will heavily be penalized and more conservative models will be the one that cannot beat the baseline.
", "rawMarkdown": "I think it's working as intended.\n> If really Beta=0.5, it must be so easy to make better submission than the string-matching baseline model because the FN labels are lesser penalized\n\nI don't think this is a valid assumption, it depends on the model. In my experience ones that generalize will give out a lot false positives. On the other hand, safe methods like string matching has almost zero false positives and which is why it's so hard to beat.\n\n> If Beta is bigger, the FNs will heavily be penalized and models exploring the unseen datasets can not beat the baseline. \n\nThis too, should it be the other way around? If Beta is bigger, the FNs will heavily be penalized and more conservative models will be the one that cannot beat the baseline.", "votes": null }, { "id": "1280405", "postDate": "04/21/2021 22:51:38", "content": "@philculliton Thank you for verifying. I checked my calculations and I found that I mistook FP for FN as the target for applying Beta. I'm really sorry for bothering you.
", "rawMarkdown": "philculliton Thank you for verifying. I checked my calculations and I found that I mistook FP for FN as the target for applying Beta. I'm really sorry for bothering you.", "votes": null }, { "id": "1280409", "postDate": "04/21/2021 22:53:47", "content": "@suicaokhoailang Thank you for pointing it out. I found that I'm wrong as described in my reply above.
", "rawMarkdown": "suicaokhoailang Thank you for pointing it out. I found that I'm wrong as described in my reply above.", "votes": null } ], "comments": [ { "id": 1280208, "author_name": "philculliton", "author_url": "", "post_date": "04/21/2021 17:43:32", "content": "Hi! Beta is definitely 0.5. FN labels are less penalized than FPs in that context.
@philculliton Thank you for verifying. I checked my calculations and I found that I mistook FP for FN as the target for applying Beta. I'm really sorry for bothering you.
", "votes": null, "replies": [] } ] }, { "id": 1280251, "author_name": "suicaokhoailang", "author_url": "", "post_date": "04/21/2021 18:32:35", "content": "I think it's working as intended.
\n\n\nIf really Beta=0.5, it must be so easy to make better submission than the string-matching baseline model because the FN labels are lesser penalized
\n
I don't think this is a valid assumption, it depends on the model. In my experience ones that generalize will give out a lot false positives. On the other hand, safe methods like string matching has almost zero false positives and which is why it's so hard to beat.
\n\n\nIf Beta is bigger, the FNs will heavily be penalized and models exploring the unseen datasets can not beat the baseline.
\n
This too, should it be the other way around? If Beta is bigger, the FNs will heavily be penalized and more conservative models will be the one that cannot beat the baseline.
", "votes": null, "replies": [ { "id": 1280409, "author_name": "osciiart", "author_url": "", "post_date": "04/21/2021 22:53:47", "content": "@suicaokhoailang Thank you for pointing it out. I found that I'm wrong as described in my reply above.
", "votes": null, "replies": [] } ] } ], "raw_markdown_by_id": { "1279874": "I'm sorry for bothering you. I found I'm wrong. Thank you.\n~~I tried some LB proving and the results suggested that LB scoring is wrong. The detail is shown [here](https://www.kaggle.com/osciiart/lb-probing-2-i-doubt-lb-scoring-is-correct).\nI think `Beta` is not **0.5** but about **1.2**. \n@philculliton, @julialane Please verify this problem.\n\nIf really `Beta=0.5`, it must be so easy to make better submission than the string-matching baseline model because the FN labels are lesser penalized. However, very few teams achieved this actually. If `Beta` is bigger, the FNs will heavily be penalized and models exploring the unseen datasets can not beat the baseline. This strongly discourages competitors to make models that aggressively search unseen datasets. It ruins the meaning of this competition the host intends to.~~", "1280208": "Hi! `Beta` is definitely `0.5`. FN labels are less penalized than FPs in that context.", "1280251": "I think it's working as intended.\n> If really Beta=0.5, it must be so easy to make better submission than the string-matching baseline model because the FN labels are lesser penalized\n\nI don't think this is a valid assumption, it depends on the model. In my experience ones that generalize will give out a lot false positives. On the other hand, safe methods like string matching has almost zero false positives and which is why it's so hard to beat.\n\n> If Beta is bigger, the FNs will heavily be penalized and models exploring the unseen datasets can not beat the baseline. \n\nThis too, should it be the other way around? If Beta is bigger, the FNs will heavily be penalized and more conservative models will be the one that cannot beat the baseline.", "1280405": "philculliton Thank you for verifying. I checked my calculations and I found that I mistook FP for FN as the target for applying Beta. I'm really sorry for bothering you.", "1280409": "suicaokhoailang Thank you for pointing it out. I found that I'm wrong as described in my reply above." }, "source": "meta" }