As the leaderboard metric (Jaccard index) is directly linked to precision and recall, one might choose to find the optimal combination of precision and recall that would yield the highest Jaccard on the LB. The problem with this is that the score on the leaderboard will not be representative of the 75% recall constraint.
\nOne could then take less risk in predictions (lower recall and fewer token predicted) to maximize its Jaccard index.
\nHow to ensure fairness here ?
", "messages": [ { "id": "3304337", "postDate": "10/20/2025 10:29:27", "content": "As the leaderboard metric (Jaccard index) is directly linked to precision and recall, one might choose to find the optimal combination of precision and recall that would yield the highest Jaccard on the LB. The problem with this is that the score on the leaderboard will not be representative of the 75% recall constraint.
\nOne could then take less risk in predictions (lower recall and fewer token predicted) to maximize its Jaccard index.
\nHow to ensure fairness here ?
", "rawMarkdown": "As the leaderboard metric (Jaccard index) is directly linked to precision and recall, one might choose to find the optimal combination of precision and recall that would yield the highest Jaccard on the LB. The problem with this is that the score on the leaderboard will not be representative of the 75% recall constraint.\n\nOne could then take less risk in predictions (lower recall and fewer token predicted) to maximize its Jaccard index.\n\nHow to ensure fairness here ?", "votes": null }, { "id": "3309873", "postDate": "11/01/2025 16:25:06", "content": "The recall constraint is not enforced in leaderboard. i tried to submit all 1 and it was scored successfully.
", "rawMarkdown": "The recall constraint is not enforced in leaderboard. i tried to submit all 1 and it was scored successfully.", "votes": null }, { "id": "3309876", "postDate": "11/01/2025 16:30:27", "content": "i am seeing big mismatch in cv and Lb. In my local validation i got score of 0.01 but got Lb score of 0.12. \nDo you also have big mismatch?
", "rawMarkdown": "i am seeing big mismatch in cv and Lb. In my local validation i got score of 0.01 but got Lb score of 0.12. \nDo you also have big mismatch?", "votes": null }, { "id": "3309932", "postDate": "11/01/2025 18:20:30", "content": "Yes I experienced the same thing. I did many submissions in order to linearly align my cv with LB.\nThat said, I think the discrepancy between your 0.01 cv and 0.12 lb is too large, you should further investigate your data
", "rawMarkdown": "Yes I experienced the same thing. I did many submissions in order to linearly align my cv with LB.\nThat said, I think the discrepancy between your 0.01 cv and 0.12 lb is too large, you should further investigate your data", "votes": null }, { "id": "3312057", "postDate": "11/06/2025 10:38:14", "content": "", "rawMarkdown": "twainayar", "votes": null }, { "id": "3313665", "postDate": "11/09/2025 15:50:23", "content": "@twainayar can we have confirmation that submission that does not comply with the >75% recall constraint will be discarded from private leaderboard ? This is important for us to select the proper submissions, thank you.
", "rawMarkdown": "twainayar can we have confirmation that submission that does not comply with the >75% recall constraint will be discarded from private leaderboard ? This is important for us to select the proper submissions, thank you.", "votes": null } ], "comments": [ { "id": 3309873, "author_name": "meetbabariya", "author_url": "", "post_date": "11/01/2025 16:25:06", "content": "The recall constraint is not enforced in leaderboard. i tried to submit all 1 and it was scored successfully.
", "votes": null, "replies": [] }, { "id": 3309876, "author_name": "meetbabariya", "author_url": "", "post_date": "11/01/2025 16:30:27", "content": "i am seeing big mismatch in cv and Lb. In my local validation i got score of 0.01 but got Lb score of 0.12. \nDo you also have big mismatch?
", "votes": null, "replies": [ { "id": 3309932, "author_name": "yannfb", "author_url": "", "post_date": "11/01/2025 18:20:30", "content": "Yes I experienced the same thing. I did many submissions in order to linearly align my cv with LB.\nThat said, I think the discrepancy between your 0.01 cv and 0.12 lb is too large, you should further investigate your data
", "votes": null, "replies": [] } ] }, { "id": 3312057, "author_name": "yannfb", "author_url": "", "post_date": "11/06/2025 10:38:14", "content": "", "votes": null, "replies": [ { "id": 3313665, "author_name": "juliencst", "author_url": "", "post_date": "11/09/2025 15:50:23", "content": "@twainayar can we have confirmation that submission that does not comply with the >75% recall constraint will be discarded from private leaderboard ? This is important for us to select the proper submissions, thank you.
", "votes": null, "replies": [] } ] } ], "raw_markdown_by_id": { "3304337": "As the leaderboard metric (Jaccard index) is directly linked to precision and recall, one might choose to find the optimal combination of precision and recall that would yield the highest Jaccard on the LB. The problem with this is that the score on the leaderboard will not be representative of the 75% recall constraint.\n\nOne could then take less risk in predictions (lower recall and fewer token predicted) to maximize its Jaccard index.\n\nHow to ensure fairness here ?", "3309873": "The recall constraint is not enforced in leaderboard. i tried to submit all 1 and it was scored successfully.", "3309876": "i am seeing big mismatch in cv and Lb. In my local validation i got score of 0.01 but got Lb score of 0.12. \nDo you also have big mismatch?", "3309932": "Yes I experienced the same thing. I did many submissions in order to linearly align my cv with LB.\nThat said, I think the discrepancy between your 0.01 cv and 0.12 lb is too large, you should further investigate your data", "3312057": "twainayar", "3313665": "twainayar can we have confirmation that submission that does not comply with the >75% recall constraint will be discarded from private leaderboard ? This is important for us to select the proper submissions, thank you." }, "source": "meta" }