Updated the math thanks to @cpmpml
\nI understand the metrics correctly, this is what it does:
\nPlease correct me if I'm wrong so I can fix the math below.
\nAs you may have noticed, a part of public test data (960 samples) came from, or at least share the same set of citations that came from the train set. By a simple string matching one could get ~0.7 on the leader board.
\nFor the sake of simplicity, let's assume that all the samples matched got a perfect score and each sample only contains one ground truth, which means there's no point in predicting them again, and we only care about the rest to which the matching method output an empty prediction.
\nSo how many of the 960 samples were perfectly matched?
\nAs we assumed, FP = 0, everything matched is correct:
1.25*TP/(1.25*TP + 0.25 * FN + FP) = 0.7 and TP + FN = 960 → TP = 305\nWhich means we have 655 samples left to predict with our models.
\nLet's say we output one prediction for each sample, and 10% of them were correct:
TP' = 305 + 66, FP' = 589, FN' = 589\nThe score on this subset alone is of course 0.1. The global score is, however:
\nF1 = 1.25*TP'/(1.25*TP' + 0.25 * FN' + FP') = 0.386 → worse than the baseline\nThis is a very rough estimation but the point is as you try to predict the rest of the data, you are punished by false positives, and you can only beat the baseline when the score on the rest of the data reach about 0.34. And oh boy it's hard to get that number.
\nI think that explains why the leaderboard seems dead at the moment, it's not, the bar is just unnecessarily way too high to beat.
Updated the math thanks to @cpmpml
\nI understand the metrics correctly, this is what it does:
\nPlease correct me if I'm wrong so I can fix the math below.
\nAs you may have noticed, a part of public test data (960 samples) came from, or at least share the same set of citations that came from the train set. By a simple string matching one could get ~0.7 on the leader board.
\nFor the sake of simplicity, let's assume that all the samples matched got a perfect score and each sample only contains one ground truth, which means there's no point in predicting them again, and we only care about the rest to which the matching method output an empty prediction.
\nSo how many of the 960 samples were perfectly matched?
\nAs we assumed, FP = 0, everything matched is correct:
1.25*TP/(1.25*TP + 0.25 * FN + FP) = 0.7 and TP + FN = 960 → TP = 305\nWhich means we have 655 samples left to predict with our models.
\nLet's say we output one prediction for each sample, and 10% of them were correct:
TP' = 305 + 66, FP' = 589, FN' = 589\nThe score on this subset alone is of course 0.1. The global score is, however:
\nF1 = 1.25*TP'/(1.25*TP' + 0.25 * FN' + FP') = 0.386 → worse than the baseline\nThis is a very rough estimation but the point is as you try to predict the rest of the data, you are punished by false positives, and you can only beat the baseline when the score on the rest of the data reach about 0.34. And oh boy it's hard to get that number.
\nI think that explains why the leaderboard seems dead at the moment, it's not, the bar is just unnecessarily way too high to beat.
Hi! Thanks for your observations. String matching is a flawed baseline, but I think it creates a realistic idea of the low bar for \"beating\" the public leaderboard. I agree that starting off with string matching will lead to less leaderboard movement, but I also think when people start to break beyond roughly 0.7, it will be exciting.
\nNote that there are a large number of labels that are unmatched by train across all samples in public test, so there's definitely room for movement.
", "rawMarkdown": "Hi! Thanks for your observations. String matching is a flawed baseline, but I think it creates a realistic idea of the low bar for \"beating\" the public leaderboard. I agree that starting off with string matching will lead to less leaderboard movement, but I also think when people start to break beyond roughly 0.7, it will be exciting.\n\nNote that there are a large number of labels that are unmatched by train across all samples in public test, so there's definitely room for movement.", "votes": 1, "replies": [ { "id": 1256135, "postDate": "2021-03-29T15:12:04.663Z", "content": "Thank you for your reply. I still don't think that this competition needs an artificial baseline though. Within a few days we all realized how hard this is and the \"real\" score for now I think will be in the 0.1 - 0.2 range, which clearer shows the room for improvement.
", "rawMarkdown": "Thank you for your reply. I still don't think that this competition needs an artificial baseline though. Within a few days we all realized how hard this is and the \"real\" score for now I think will be in the 0.1 - 0.2 range, which clearer shows the room for improvement.", "votes": 3 }, { "id": 1256236, "postDate": "2021-03-29T17:19:50.693Z", "content": "Agreed. The 0.7 is less an artificial baseline and more of a consequence of including some training crossover in the public test (which, with F0.5 scoring, gets inflated precision-wise).
My current LB is a full Deep Learning approach, it's still really far from the baseline but I should check whether combining both helps. I still have a lot of room for improvement though.
", "rawMarkdown": "My current LB is a full Deep Learning approach, it's still really far from the baseline but I should check whether combining both helps. I still have a lot of room for improvement though.", "replies": [ { "id": 1257205, "postDate": "2021-03-30T16:07:06.693Z", "content": "I think that's a very good score according to my validation, I'm submitting my first real sub right now :D
", "rawMarkdown": "I think that's a very good score according to my validation, I'm submitting my first real sub right now :D", "votes": 1 } ] }, { "id": 1256850, "postDate": "2021-03-30T09:53:02.793Z", "content": "Your definition of final metric is wrong, see https://en.wikipedia.org/wiki/F-score
\nWith beta = 0.5 it should be:
\n1.25 * TP/( 1.25 * TP + 0.25 * FN + FP)
", "rawMarkdown": "Your definition of final metric is wrong, see https://en.wikipedia.org/wiki/F-score\n\nWith beta = 0.5 it should be:\n\n1.25 * TP/( 1.25 * TP + 0.25 * FN + FP)", "replies": [ { "id": 1256878, "postDate": "2021-03-30T10:25:36.947Z", "content": "I see, so the punishment for false positives is even worse
", "rawMarkdown": "I see, so the punishment for false positives is even worse " } ] }, { "id": 1257842, "postDate": "2021-03-31T06:25:52.833Z", "rawMarkdown": "", "votes": -2, "isDeleted": true } ], "comments": [ { "id": 1256014, "author_name": "Phil Culliton", "author_url": "", "post_date": "2021-03-29T13:15:22.013000", "content": "Hi! Thanks for your observations. String matching is a flawed baseline, but I think it creates a realistic idea of the low bar for \"beating\" the public leaderboard. I agree that starting off with string matching will lead to less leaderboard movement, but I also think when people start to break beyond roughly 0.7, it will be exciting.
\nNote that there are a large number of labels that are unmatched by train across all samples in public test, so there's definitely room for movement.
", "votes": 1, "replies": [ { "id": 1256135, "author_name": "Khoi Nguyen", "author_url": "", "post_date": "2021-03-29T15:12:04.663000", "content": "Thank you for your reply. I still don't think that this competition needs an artificial baseline though. Within a few days we all realized how hard this is and the \"real\" score for now I think will be in the 0.1 - 0.2 range, which clearer shows the room for improvement.
", "votes": 3, "replies": [] }, { "id": 1256236, "author_name": "Phil Culliton", "author_url": "", "post_date": "2021-03-29T17:19:50.693000", "content": "Agreed. The 0.7 is less an artificial baseline and more of a consequence of including some training crossover in the public test (which, with F0.5 scoring, gets inflated precision-wise).
My current LB is a full Deep Learning approach, it's still really far from the baseline but I should check whether combining both helps. I still have a lot of room for improvement though.
", "votes": 0, "replies": [ { "id": 1257205, "author_name": "Khoi Nguyen", "author_url": "", "post_date": "2021-03-30T16:07:06.693000", "content": "I think that's a very good score according to my validation, I'm submitting my first real sub right now :D
", "votes": 1, "replies": [] } ] }, { "id": 1256850, "author_name": "CPMP", "author_url": "", "post_date": "2021-03-30T09:53:02.793000", "content": "Your definition of final metric is wrong, see https://en.wikipedia.org/wiki/F-score
\nWith beta = 0.5 it should be:
\n1.25 * TP/( 1.25 * TP + 0.25 * FN + FP)
", "votes": 0, "replies": [ { "id": 1256878, "author_name": "Khoi Nguyen", "author_url": "", "post_date": "2021-03-30T10:25:36.947000", "content": "I see, so the punishment for false positives is even worse
", "votes": 0, "replies": [] } ] }, { "id": 1257842, "author_name": "", "author_url": "", "post_date": "2021-03-31T06:25:52.833000", "content": "", "votes": -2, "replies": [] } ], "raw_markdown_by_id": { "1255241": "Updated the math thanks to @cpmpml \n\nI understand the metrics correctly, this is what it does:\n- Let TP = 0, FN = 0, FP = 0\n- For each sample in the test set, count tp, fn and fp\n- TP += tp, FN += fn, FP += fp\n- Final metric = 1.25\\*TP/(1.25\\*TP + 0.25 * FN + FP)\n\nPlease correct me if I'm wrong so I can fix the math below.\n\nAs you may have noticed, a part of public test data (960 samples) came from, or at least share the same set of citations that came from the train set. By a simple string matching one could get ~0.7 on the leader board.\nFor the sake of simplicity, let's assume that all the samples matched got a perfect score and each sample only contains one ground truth, which means there's no point in predicting them again, and we only care about the rest to which the matching method output an empty prediction.\nSo how many of the 960 samples were perfectly matched?\nAs we assumed, FP = 0, everything matched is correct:\n\n```\n1.25*TP/(1.25*TP + 0.25 * FN + FP) = 0.7 and TP + FN = 960 → TP = 305\n\n```\nWhich means we have 655 samples left to predict with our models.\nLet's say we output one prediction for each sample, and 10% of them were correct:\n```\nTP' = 305 + 66, FP' = 589, FN' = 589\n\n```The score on this subset alone is of course 0.1. The global score is, however:\n```\nF1 = 1.25*TP'/(1.25*TP' + 0.25 * FN' + FP') = 0.386 → worse than the baseline\n\n```\nThis is a very rough estimation but the point is as you try to predict the rest of the data, you are punished by false positives, and you can only beat the baseline when the score on the rest of the data reach about 0.34. And oh boy it's hard to get that number.\nI think that explains why the leaderboard seems dead at the moment, it's not, the bar is just unnecessarily way too high to beat. ", "1256014": "Hi! Thanks for your observations. String matching is a flawed baseline, but I think it creates a realistic idea of the low bar for \"beating\" the public leaderboard. I agree that starting off with string matching will lead to less leaderboard movement, but I also think when people start to break beyond roughly 0.7, it will be exciting.\n\nNote that there are a large number of labels that are unmatched by train across all samples in public test, so there's definitely room for movement.", "1257080": "My current LB is a full Deep Learning approach, it's still really far from the baseline but I should check whether combining both helps. I still have a lot of room for improvement though.", "1256850": "Your definition of final metric is wrong, see https://en.wikipedia.org/wiki/F-score\n\nWith beta = 0.5 it should be:\n\n1.25 * TP/( 1.25 * TP + 0.25 * FN + FP)", "1257842": "" } }