{ "id": 243116, "title": "a[nother] metric question - a short and concrete testcase", "url": "/competitions/coleridgeinitiative-show-us-the-data/discussion/243116", "author_name": "GUNER", "post_date": "2021-06-01T07:58:43.648000", "votes": 2, "comment_count": 4, "views": 0, "content": "

Assume a publication with a subtext denoting a dataset as (a single mention in the document and there is no other dataset that is mentioned):
\n'NCES common core of data'
\nin scenario a: I predicted 'common core of data'
\nin scenario b: I predicted 'NCES common core of data'
\nin scenario c: I made two predictions 'common core of data' | 'NCES common core of data'

\n

My assumption is that a and b would be valid TP's (both with a jaccard score above .5)
\nIn c: it will count as one TP and one FP (as the second prediction would not be matched to this single ground truth). Am I correct?

", "messages": [ { "id": 1330992, "postDate": "2021-06-01T07:58:43.650Z", "content": "

Assume a publication with a subtext denoting a dataset as (a single mention in the document and there is no other dataset that is mentioned):
\n'NCES common core of data'
\nin scenario a: I predicted 'common core of data'
\nin scenario b: I predicted 'NCES common core of data'
\nin scenario c: I made two predictions 'common core of data' | 'NCES common core of data'

\n

My assumption is that a and b would be valid TP's (both with a jaccard score above .5)
\nIn c: it will count as one TP and one FP (as the second prediction would not be matched to this single ground truth). Am I correct?

", "rawMarkdown": "Assume a publication with a subtext denoting a dataset as (**a single mention in the document and there is no other dataset that is mentioned)**: \n*'NCES common core of data'*\nin scenario a: I predicted 'common core of data'\nin scenario b: I predicted 'NCES common core of data'\nin scenario c: I made two predictions 'common core of data' | 'NCES common core of data'\n\nMy assumption is that **a and b would be valid TP's** (both with a jaccard score above .5)\n**In c: it will count as one TP and one FP** (as the second prediction would not be matched to this single ground truth). Am I correct?", "votes": 2 }, { "id": 1331087, "postDate": "2021-06-01T09:06:54.480Z", "content": "

Thx Shota, I guess there is only one exception and that is: if Host decided that there are two GT in this example, although only one mention. In that case, case c would prevail over the others.
\nI have to say, some of my tryouts yielded results supporting this later argument.

", "rawMarkdown": "Thx Shota, I guess there is only one exception and that is: if Host decided that there are two GT in this example, although only one mention. In that case, case c would prevail over the others.\nI have to say, some of my tryouts yielded results supporting this later argument.", "replies": [ { "id": 1331093, "postDate": "2021-06-01T09:11:03.743Z", "content": "

I also think one prediction can only be matched with one ground truth based on the results of the following little experiment.

\n

I replaced the labels of prediction with the ones that had the most known labels with a jaccard score of 0.5 or higher.
\n(After the replacement, I made the labels unique to remove the same label.)
\nIf a prediction matched more than one ground truth, my score should remain the same.
\nBut It dropped drastically.

", "rawMarkdown": "I also think one prediction can only be matched with one ground truth based on the results of the following little experiment.\n\nI replaced the labels of prediction with the ones that had the most known labels with a jaccard score of 0.5 or higher.\n(After the replacement, I made the labels unique to remove the same label.)\nIf a prediction matched more than one ground truth, my score should remain the same.\nBut It dropped drastically." } ] }, { "id": 1331017, "postDate": "2021-06-01T08:17:29.763Z", "content": "

Yes, you’re correct!

", "rawMarkdown": "Yes, you’re correct!" }, { "id": 1331020, "postDate": "2021-06-01T08:18:10.493Z", "rawMarkdown": "", "isDeleted": true } ], "comments": [ { "id": 1331087, "author_name": "GUNER", "author_url": "", "post_date": "2021-06-01T09:06:54.480000", "content": "

Thx Shota, I guess there is only one exception and that is: if Host decided that there are two GT in this example, although only one mention. In that case, case c would prevail over the others.
\nI have to say, some of my tryouts yielded results supporting this later argument.

", "votes": 0, "replies": [ { "id": 1331093, "author_name": "Shota", "author_url": "", "post_date": "2021-06-01T09:11:03.743000", "content": "

I also think one prediction can only be matched with one ground truth based on the results of the following little experiment.

\n

I replaced the labels of prediction with the ones that had the most known labels with a jaccard score of 0.5 or higher.
\n(After the replacement, I made the labels unique to remove the same label.)
\nIf a prediction matched more than one ground truth, my score should remain the same.
\nBut It dropped drastically.

", "votes": 0, "replies": [] } ] }, { "id": 1331017, "author_name": "Shota", "author_url": "", "post_date": "2021-06-01T08:17:29.763000", "content": "

Yes, you’re correct!

", "votes": 0, "replies": [] }, { "id": 1331020, "author_name": "", "author_url": "", "post_date": "2021-06-01T08:18:10.493000", "content": "", "votes": 0, "replies": [] } ], "raw_markdown_by_id": { "1330992": "Assume a publication with a subtext denoting a dataset as (**a single mention in the document and there is no other dataset that is mentioned)**: \n*'NCES common core of data'*\nin scenario a: I predicted 'common core of data'\nin scenario b: I predicted 'NCES common core of data'\nin scenario c: I made two predictions 'common core of data' | 'NCES common core of data'\n\nMy assumption is that **a and b would be valid TP's** (both with a jaccard score above .5)\n**In c: it will count as one TP and one FP** (as the second prediction would not be matched to this single ground truth). Am I correct?", "1331087": "Thx Shota, I guess there is only one exception and that is: if Host decided that there are two GT in this example, although only one mention. In that case, case c would prevail over the others.\nI have to say, some of my tryouts yielded results supporting this later argument.", "1331017": "Yes, you’re correct!", "1331020": "" } }