{ "id": 228337, "title": "More train data issues", "url": "/competitions/coleridgeinitiative-show-us-the-data/discussion/228337", "author_name": "", "post_date": "2021-03-24T08:15:01.430652Z", "votes": 54, "comment_count": 30, "views": 0, "content": "

This code

\n
train = pd.read_csv('../input/train.csv')\nalzheimer_train = train[train.dataset_title.str.startswith('Alzheimer')]\nprint(alzheimer_train.dataset_title.unique())\nprint(alzheimer_train.cleaned_label.unique())\n
\n

prints

\n
[\"Alzheimer's Disease Neuroimaging Initiative (ADNI)\"]\n['alzheimers disease neuroimaging initiative' 'adni'\n 'alzheimer s disease neuroimaging initiative adni ']\n
\n

It means that the same dataset appears with three different possible cleaned labels. I would expect a single cleaned label per dataset. In this case, which is the correct answer among these:

\n
'alzheimers disease neuroimaging initiative',\n'adni',\n 'alzheimer s disease neuroimaging initiative adni '\n
\n

I only checked the most frequent dataset but this doesn't look good to me. Also, note how the last one has a trailing empty space.

", "messages": [ { "id": "1250709", "postDate": "03/24/2021 08:15:01", "content": "

This code

\n
train = pd.read_csv('../input/train.csv')\nalzheimer_train = train[train.dataset_title.str.startswith('Alzheimer')]\nprint(alzheimer_train.dataset_title.unique())\nprint(alzheimer_train.cleaned_label.unique())\n
\n

prints

\n
[\"Alzheimer's Disease Neuroimaging Initiative (ADNI)\"]\n['alzheimers disease neuroimaging initiative' 'adni'\n 'alzheimer s disease neuroimaging initiative adni ']\n
\n

It means that the same dataset appears with three different possible cleaned labels. I would expect a single cleaned label per dataset. In this case, which is the correct answer among these:

\n
'alzheimers disease neuroimaging initiative',\n'adni',\n 'alzheimer s disease neuroimaging initiative adni '\n
\n

I only checked the most frequent dataset but this doesn't look good to me. Also, note how the last one has a trailing empty space.

", "rawMarkdown": "This code\n\n```\ntrain = pd.read_csv('../input/train.csv')\nalzheimer_train = train[train.dataset_title.str.startswith('Alzheimer')]\nprint(alzheimer_train.dataset_title.unique())\nprint(alzheimer_train.cleaned_label.unique())\n\n```\nprints\n\n```\n[\"Alzheimer's Disease Neuroimaging Initiative (ADNI)\"]\n['alzheimers disease neuroimaging initiative' 'adni'\n 'alzheimer s disease neuroimaging initiative adni ']\n\n```\nIt means that the same dataset appears with three different possible cleaned labels. I would expect a single cleaned label per dataset. In this case, which is the correct answer among these:\n\n```\n'alzheimers disease neuroimaging initiative',\n'adni',\n 'alzheimer s disease neuroimaging initiative adni '\n\n```\nI only checked the most frequent dataset but this doesn't look good to me. Also, note how the last one has a trailing empty space.", "votes": null }, { "id": "1250761", "postDate": "03/24/2021 09:00:38", "content": "

I have updated my train data problems post asking why the same dataset can have multiple cleaned labels?

\n

This is not a problem as per Evaluation page which mentions that \"Multiple predictions are delineated with a pipe (|) character in the submission file\" and gives below example:

\n\n

where it looks like all the (multiple) cleaned labels are referring to the same dataset viz \"national space objects location dataset\"
\n(whereas we were expecting that multiple labels should correspond to multiple datasets, not the same dataset)

", "rawMarkdown": "I have updated my [train data problems](https://www.kaggle.com/c/coleridgeinitiative-show-us-the-data/discussion/228322) post asking why the same dataset can have multiple cleaned labels?\n\nThis is not a problem as per [Evaluation page](https://www.kaggle.com/c/coleridgeinitiative-show-us-the-data/overview/evaluation) which mentions that \"Multiple predictions are delineated with a pipe (|) character in the submission file\" and gives below example:\n- `01e4e08c-ffea-45a7-adde-6a0c0ad755fc,space location data|national space objects|national space dataset`\n\nwhere it looks like all the (multiple) cleaned labels are referring to the same dataset viz \"national space objects location dataset\"\n(whereas we were expecting that multiple labels should correspond to multiple datasets, not the same dataset)", "votes": null }, { "id": "1250774", "postDate": "03/24/2021 09:11:03", "content": "
\n

where all the (multiple) cleaned labels are referring to the same dataset viz \"national space objects location dataset\"

\n
\n

where do you see that? It is not in the evaluation page.

", "rawMarkdown": "> where all the (multiple) cleaned labels are referring to the same dataset viz \"national space objects location dataset\"\n\nwhere do you see that? It is not in the evaluation page.", "votes": null }, { "id": "1250784", "postDate": "03/24/2021 09:18:47", "content": "", "rawMarkdown": "I believe that this evaluation page example row contains dummy(fake) dataset names, and was pointing out that it gives misleading information to the participants because the labels appear to refer to the same dataset\n\n- In any case, we already know that the competition train data contains [multiple labels for the same dataset](https://www.kaggle.com/c/coleridgeinitiative-show-us-the-data/discussion/228322)", "votes": null }, { "id": "1250801", "postDate": "03/24/2021 09:29:27", "content": "

@sirishks I think you're referring to predictions, it actually makes sense to expect a single cleaned label per dataset

", "rawMarkdown": "sirishks I think you're referring to predictions, it actually makes sense to expect a single cleaned label per dataset", "votes": null }, { "id": "1251342", "postDate": "03/24/2021 17:47:49", "content": "

The data page says 'Your predictions will be short excerpts from the publications that appear to note a dataset. Predictions that more accurately match the precise words used to identify the dataset within the publication will score higher.'

\n

I assume in the training data it is quite likely that excerpts from different publications will differ, even when they refer to the same dataset, and this will result in different labels.

", "rawMarkdown": "The [data page](https://www.kaggle.com/c/coleridgeinitiative-show-us-the-data/data) says 'Your predictions will be **short excerpts from the publications** that appear to note a dataset. Predictions that more accurately match the precise words used to identify the dataset within the publication will score higher.'\n\nI assume in the training data it is quite likely that excerpts from different publications will differ, even when they refer to the same dataset, and this will result in different labels.", "votes": null }, { "id": "1251467", "postDate": "03/24/2021 19:53:49", "content": "

Hi @cpmpml - good questions. There are definitely multiple labels for a given dataset, as each label is a contextual mention in a document. So which answer is correct would depend on the document in question. The idea is less to match labels against document text, and more to figure out that a chunk of text represents a dataset, and extract that.

\n

Trailing, leading, and extra spaces are ignored in the metric.

", "rawMarkdown": "Hi @cpmpml - good questions. There are definitely multiple labels for a given dataset, as each label is a contextual mention in a document. So which answer is correct would depend on the document in question. The idea is less to match labels against document text, and more to figure out that a chunk of text represents a dataset, and extract that.\n\nTrailing, leading, and extra spaces are ignored in the metric.", "votes": null }, { "id": "1252302", "postDate": "03/25/2021 15:11:43", "content": "

Phil, thanks for the answers.

", "rawMarkdown": "Phil, thanks for the answers.", "votes": null }, { "id": "1252494", "postDate": "03/25/2021 18:08:10", "content": "

@philculliton I'm afraid something is still not right. Here is an example with this dataset title:
\n'Aging Integrated Database (AGID)'

\n

There are two publications with it in train.csv

\n

One of them is
\n'71d14baa-31df-4e80-9c18-af7516be766c'

\n

The dataset is referred to as 'Aging Integrated Database (AGID)' in the text.

\n

yet, in train.csv the dataset label is
\n'Aging Integrated Database'

\n

This could be fine maybe, but for this other publication with the same way to refer to the dataset :
\n'93c27c16-9356-4b97-9785-9795ad9bb609'

\n

train.csv contains two dataset label
\n'Aging Integrated Database (AGID)' and 'Aging Integrated Database'

\n

Said differently, There are no occurrences of 'Aging Integrated Database' that are not occurrences of 'Aging Integrated Database (AGID)'

\n

The fact that one pub has two different labels looks like noise.

\n

I checked other cases and they all look noisy too.

\n

I really think there would be more value if we were asked to predict uniquely defined datasets rather than semi random labels.

", "rawMarkdown": "philculliton I'm afraid something is still not right. Here is an example with this dataset title:\n'Aging Integrated Database (AGID)'\n\nThere are two publications with it in train.csv\n\nOne of them is \n'71d14baa-31df-4e80-9c18-af7516be766c'\n\nThe dataset is referred to as 'Aging Integrated Database (AGID)' in the text.\n\nyet, in train.csv the dataset label is\n'Aging Integrated Database'\n\nThis could be fine maybe, but for this other publication with the same way to refer to the dataset :\n'93c27c16-9356-4b97-9785-9795ad9bb609'\n\ntrain.csv contains two dataset label\n'Aging Integrated Database (AGID)' and 'Aging Integrated Database'\n\nSaid differently, There are no occurrences of 'Aging Integrated Database' that are not occurrences of 'Aging Integrated Database (AGID)'\n\nThe fact that one pub has two different labels looks like noise.\n\nI checked other cases and they all look noisy too. \n\nI really think there would be more value if we were asked to predict uniquely defined datasets rather than semi random labels.", "votes": null }, { "id": "1252523", "postDate": "03/25/2021 18:38:32", "content": "

@cmpml Just reinforcing @philculliton comment. It is very standard to reference multiple datasets in empirical science publications. In fact, this is one of the measures that the agencies are interested in - understanding how their dataset fit into the empirical ecosystem. So dataset coreferences in a publication is a very useful outcome measure

", "rawMarkdown": "cmpml Just reinforcing @philculliton comment. It is very standard to reference multiple datasets in empirical science publications. In fact, this is one of the measures that the agencies are interested in - understanding how their dataset fit into the empirical ecosystem. So dataset coreferences in a publication is a very useful outcome measure", "votes": null }, { "id": "1252975", "postDate": "03/26/2021 08:55:45", "content": "

@julialane I m not discussing referencing several datasets in a given publication. I am discussing referencing the same dataset with different labels in a given publication, or across publications. Let me reproduce below a case with this dataset title:
\n'Aging Integrated Database (AGID)'

\n

There are two publications with it in train.csv

\n

One of them is
\n'71d14baa-31df-4e80-9c18-af7516be766c'

\n

The dataset is referred to as 'Aging Integrated Database (AGID)' in the text.

\n

yet, in train.csv the dataset label is
\n'Aging Integrated Database'

\n

This could be fine maybe, but for this other publication with the same way to refer to the dataset :
\n'93c27c16-9356-4b97-9785-9795ad9bb609'

\n

train.csv contains two dataset label
\n'Aging Integrated Database (AGID)' and 'Aging Integrated Database'

\n

Said differently, There are no occurrences of 'Aging Integrated Database' that are not occurrences of 'Aging Integrated Database (AGID)'

\n

How can we decide if the publicaiton has one or two labels when this string appears:
\nAging Integrated Database (AGID)'

\n

It looks like random choice at this point.

\n

I checked other cases and they all look random too.

\n

I really think there would be more value if we were asked to predict uniquely defined datasets rather than semi random labels.

", "rawMarkdown": "julialane I m not discussing referencing several datasets in a given publication. I am discussing referencing the same dataset with different labels in a given publication, or across publications. Let me reproduce below a case with this dataset title:\n'Aging Integrated Database (AGID)'\n\nThere are two publications with it in train.csv\n\nOne of them is\n'71d14baa-31df-4e80-9c18-af7516be766c'\n\nThe dataset is referred to as 'Aging Integrated Database (AGID)' in the text.\n\nyet, in train.csv the dataset label is\n'Aging Integrated Database'\n\nThis could be fine maybe, but for this other publication with the same way to refer to the dataset :\n'93c27c16-9356-4b97-9785-9795ad9bb609'\n\ntrain.csv contains two dataset label\n'Aging Integrated Database (AGID)' and 'Aging Integrated Database'\n\nSaid differently, There are no occurrences of 'Aging Integrated Database' that are not occurrences of 'Aging Integrated Database (AGID)'\n\nHow can we decide if the publicaiton has one or two labels when this string appears:\nAging Integrated Database (AGID)'\n\nIt looks like random choice at this point.\n\nI checked other cases and they all look random too.\n\nI really think there would be more value if we were asked to predict uniquely defined datasets rather than semi random labels.", "votes": null }, { "id": "1253112", "postDate": "03/26/2021 11:23:41", "content": "

@cpmpml Thanks for a really good question. You are correct that the same dataset can be referred to with different names. That’s a feature of the problem. For example, the dataset that I was instrumental in creating – the Longitudinal Employer-Household Dynamics data - can also be called LEHD data or the Longitudinal Employer-Household Dynamics (LEHD). Even within a publication, the researcher might start by saying Longitudinal Employer-Household Dynamics (LEHD) data, and then later just call it LEHD.
\nThe goal of the competition is to understand the context surrounding the datasets (which is why we internally refer to the competition as \"rich context\"). That approach will help identify how researchers refer to newly emerging datasets as well – for example the many different names that are applied to the emerging numbers of COVID datasets.

\n

Thanks again for the opportunity to clarify. Happy to keep discussing, because we're thrilled by the interest and response.

", "rawMarkdown": "cpmpml Thanks for a really good question. You are correct that the same dataset can be referred to with different names. That’s a feature of the problem. For example, the dataset that I was instrumental in creating – the Longitudinal Employer-Household Dynamics data - can also be called LEHD data or the Longitudinal Employer-Household Dynamics (LEHD). Even within a publication, the researcher might start by saying Longitudinal Employer-Household Dynamics (LEHD) data, and then later just call it LEHD. \nThe goal of the competition is to understand the context surrounding the datasets (which is why we internally refer to the competition as \"rich context\"). That approach will help identify how researchers refer to newly emerging datasets as well – for example the many different names that are applied to the emerging numbers of COVID datasets.\n\nThanks again for the opportunity to clarify. Happy to keep discussing, because we're thrilled by the interest and response.", "votes": null }, { "id": "1253166", "postDate": "03/26/2021 12:30:44", "content": "

Thanks for the answer. I think I get that. But in the example I gave, I don't see how the context can help decide between these two labels:

\n

'Aging Integrated Database'
\n'Aging Integrated Database (AGID)'

\n

given the latter is the only one appearing in the publications. I mean, there are not occurrences of the first one that is not followed by ' (AGID)'

", "rawMarkdown": "Thanks for the answer. I think I get that. But in the example I gave, I don't see how the context can help decide between these two labels:\n\n'Aging Integrated Database'\n'Aging Integrated Database (AGID)'\n\ngiven the latter is the only one appearing in the publications. I mean, there are not occurrences of the first one that is not followed by ' (AGID)'", "votes": null }, { "id": "1253175", "postDate": "03/26/2021 12:42:41", "content": "

@cpmpml I think the point is that both names are referring to a dataset, so can be coded accordingly. We don't want to encourage the creation of a giant lookup table, but, rather, identification of the semantic context within which a dataset has been called out. @philculliton can maybe explain better than I.

", "rawMarkdown": "cpmpml I think the point is that both names are referring to a dataset, so can be coded accordingly. We don't want to encourage the creation of a giant lookup table, but, rather, identification of the semantic context within which a dataset has been called out. @philculliton can maybe explain better than I.", "votes": null }, { "id": "1253186", "postDate": "03/26/2021 13:00:10", "content": "

@julialane Could you please answer my questions also where I have identified more serious cases where the labels are incorrect (Problem 5) and correct answers get \"0\" scores (Problem 3).

", "rawMarkdown": "julialane Could you please answer my questions also where I have identified more serious cases where [the labels are incorrect (Problem 5) and correct answers get \"0\" scores (Problem 3)](https://www.kaggle.com/c/coleridgeinitiative-show-us-the-data/discussion/228322#1251493).", "votes": null }, { "id": "1253196", "postDate": "03/26/2021 13:12:47", "content": "

This is a great explanation, @julialane. The task is to identify the semantic context of a dataset's mentions. The test data contains a very large number of datasets, contexts, and publications that are entirely unseen in the training set. A model will need to generalize to the task to do well.

\n

@sirishks - I will be answering your follow-up questions in your thread.

", "rawMarkdown": "This is a great explanation, @julialane. The task is to identify the semantic context of a dataset's mentions. The test data contains a very large number of datasets, contexts, and publications that are entirely unseen in the training set. A model will need to generalize to the task to do well.\n\n@sirishks - I will be answering your follow-up questions in your thread.", "votes": null }, { "id": "1253254", "postDate": "03/26/2021 14:25:39", "content": "

@philculliton @julialane

\n

Did you check the two publications I mention? I doubt it.

\n

In both publication the dataset is referred to by this:
\n'Aging Integrated Database (AGID)'

\n

It is NEVER referred by only this:
\n'Aging Integrated Database'

\n

Yet the latter is a ground truth, but only for one of the two publications. I challenge you to find a difference in context that would explain why the latter is selected for one of the two publications.

", "rawMarkdown": "philculliton @julialane \n\nDid you check the two publications I mention? I doubt it.\n\nIn both publication the dataset is referred to by this:\n'Aging Integrated Database (AGID)'\n\nIt is NEVER referred by only this:\n'Aging Integrated Database'\n\nYet the latter is a ground truth, but only for one of the two publications. I challenge you to find a difference in context that would explain why the latter is selected for one of the two publications.", "votes": null }, { "id": "1253260", "postDate": "03/26/2021 14:35:23", "content": "

@cpmpml - thanks.

\n

Yep, I did check them! The host did as well. The training labels are noisy and they're intended to show that many forms of \"mention\" can be correctly applied to a dataset. For instance, datasets like AGID may be mentioned elsewhere without the \"(AGID)\", so learning simply to match that unique string is insufficient to solve the task.

\n

As you note, AGID may not appear without (AGID) in the training documents, but we're attempting to indicate a pretty broad universe of label differences with a relatively small label set in train. It's meant to help extrapolate.

", "rawMarkdown": "cpmpml - thanks.\n\nYep, I did check them! The host did as well. The training labels are noisy and they're intended to show that many forms of \"mention\" can be correctly applied to a dataset. For instance, datasets like AGID may be mentioned elsewhere without the \"(AGID)\", so learning simply to match that unique string is insufficient to solve the task.\n\nAs you note, AGID may not appear without (AGID) in the training documents, but we're attempting to indicate a pretty broad universe of label differences with a relatively small label set in train. It's meant to help extrapolate.", "votes": null }, { "id": "1253270", "postDate": "03/26/2021 14:47:35", "content": "

Good that you checked.

\n

Are you saying you introduced wrong labels in train to \"help extrapolate\"?

", "rawMarkdown": "Good that you checked.\n\nAre you saying you introduced wrong labels in train to \"help extrapolate\"?", "votes": null }, { "id": "1253560", "postDate": "03/26/2021 21:32:36", "content": "

@CPMP
\nprevious competitions have had noise and we dealt with it, for example https://www.kaggle.com/c/text-normalization-challenge-english-language

\n

usually noise is created because labels were created by their own internal model, or human error

\n

we can use label smoothing and other techniques to fight against.

\n

By the way, the way the metric is set up, anything with Jaccard Score > 0.5 is counted as a True Positive, so if you were to guess \"Aging Integrated Database\" twice it would be rewarded as two true positives.

\n

They picked 0.5 because they probably know there is noise, so this helps to fight against. although i do agree it is not ideal

", "rawMarkdown": "CPMP\nprevious competitions have had noise and we dealt with it, for example https://www.kaggle.com/c/text-normalization-challenge-english-language\n\nusually noise is created because labels were created by their own internal model, or human error\n\nwe can use label smoothing and other techniques to fight against.\n\nBy the way, the way the metric is set up, anything with Jaccard Score > 0.5 is counted as a True Positive, so if you were to guess \"Aging Integrated Database\" twice it would be rewarded as two true positives.\n\nThey picked 0.5 because they probably know there is noise, so this helps to fight against. although i do agree it is not ideal", "votes": null }, { "id": "1254084", "postDate": "03/27/2021 10:36:37", "content": "
\n

if you were to guess \"Aging Integrated Database\" twice it would be rewarded as two true positives.

\n
\n

If it appears only once in the ground truth then one of them wold appear as a false positive. That's why I am bothered.

\n

This assumes the evaluation tab content is accurate.

\n

Not having a non ambiguous description of what we have to predict will lead to massive LB probing to reverse engineer how labels were constructed. Is this a useful use of our time really?

\n

Anyway, I asked questions, let's see if I get answers.

", "rawMarkdown": "> if you were to guess \"Aging Integrated Database\" twice it would be rewarded as two true positives.\n\nIf it appears only once in the ground truth then one of them wold appear as a false positive. That's why I am bothered. \n\nThis assumes the evaluation tab content is accurate.\n\nNot having a non ambiguous description of what we have to predict will lead to massive LB probing to reverse engineer how labels were constructed. Is this a useful use of our time really?\n\nAnyway, I asked questions, let's see if I get answers.", "votes": null }, { "id": "1254086", "postDate": "03/27/2021 10:37:50", "content": "
\n

anything with Jaccard Score > 0.5

\n
\n

\n

Edit: Code is fixed now.

", "rawMarkdown": "> anything with Jaccard Score > 0.5 \n\n~~Which Jaccard score? The code in evaluation tab does not compute a Jaccard score...~~\n\nEdit: Code is fixed now.", "votes": null }, { "id": "1255308", "postDate": "03/28/2021 17:05:29", "content": "

We're having a similar discussion here on a code example of score computation: https://www.kaggle.com/tungmphung/coleridge-initiative-local-score-computation. Perhaps it would be easier to show some nuances of the score calculation on examples.

", "rawMarkdown": "We're having a similar discussion here on a code example of score computation: https://www.kaggle.com/tungmphung/coleridge-initiative-local-score-computation. Perhaps it would be easier to show some nuances of the score calculation on examples.", "votes": null }, { "id": "1267002", "postDate": "04/08/2021 08:29:20", "content": "

You said \"If it appears only once in the ground truth then one of them wold appear as a false positive. That's why I am bothered.\"

\n

However, seems that since words intersection of both you labels will likely be very close in this case and both will be counted as TP (since likely more than Jaccard 0.5 score). Please what are your thoughts on that, do I miss smth?

", "rawMarkdown": "You said \"If it appears only once in the ground truth then one of them wold appear as a false positive. That's why I am bothered.\"\n\nHowever, seems that since words intersection of both you labels will likely be very close in this case and both will be counted as TP (since likely more than Jaccard 0.5 score). Please what are your thoughts on that, do I miss smth?", "votes": null }, { "id": "1267656", "postDate": "04/08/2021 17:17:11", "content": "

Yes that's true (if we have two publications and our two predictions match with a Jaccard score of at least 0.5 the two GT).

\n

But what about 'alzheimers disease neuroimaging initiative' and 'adni'? Their intersection is 0.

", "rawMarkdown": "Yes that's true (if we have two publications and our two predictions match with a Jaccard score of at least 0.5 the two GT).\n\nBut what about 'alzheimers disease neuroimaging initiative' and 'adni'? Their intersection is 0.", "votes": null }, { "id": "1269354", "postDate": "04/10/2021 13:19:59", "content": "

There are many such examples. If we are expected to predict all forms of a dataset (some examples are with / without plural) appearing in the text and get penalized for not doing so, that just sucks.

\n

What is the point of needing 'Aging Integrated Database (AGID)' AND 'Aging Integrated Database' (when the latter does not appear) in the example above? Or is this what we need to deal with as 'noise'?

", "rawMarkdown": "There are many such examples. If we are expected to predict all forms of a dataset (some examples are with / without plural) appearing in the text and get penalized for not doing so, that just sucks.\n\nWhat is the point of needing 'Aging Integrated Database (AGID)' AND 'Aging Integrated Database' (when the latter does not appear) in the example above? Or is this what we need to deal with as 'noise'?", "votes": null }, { "id": "1269399", "postDate": "04/10/2021 14:01:11", "content": "

If one predicts correct dataset label, but misses plural form, there is no penalty for that, as Jac score is still higher than 0.5.

\n

Looks like that competition is about predicting pieces of texts containing datasets titles

", "rawMarkdown": "If one predicts correct dataset label, but misses plural form, there is no penalty for that, as Jac score is still higher than 0.5. \n\nLooks like that competition is about predicting pieces of texts containing datasets titles", "votes": null }, { "id": "1269738", "postDate": "04/10/2021 21:22:28", "content": "

Does that mean that a match such as this could end up being submitted?

\n

Longitudinal Employer Household Dynamics data|Longitudinal Employer-Household Dynamics data |LEHD data|LEHD data|LEHD data|LEHD data|Longitudinal Employer Household Dynamics data

\n

or in this case should we just clean it to submit the unique values?. Or is the same?

", "rawMarkdown": "Does that mean that a match such as this could end up being submitted?\n\nLongitudinal Employer Household Dynamics data|Longitudinal Employer-Household Dynamics data |LEHD data|LEHD data|LEHD data|LEHD data|Longitudinal Employer Household Dynamics data\n\nor in this case should we just clean it to submit the unique values?. Or is the same?", "votes": null }, { "id": "1275465", "postDate": "04/16/2021 10:23:10", "content": "

@julialane could you please answer @verracodeguacas question? I understand we need to extract the mentions corresponding to the datasets. But there can be many duplicates (e.g. LEHD data mentioned many times and in the same way in the same publication). Should we keep only the unique values? From a preliminary test, submitting all the mentions (containing duplicates) achieved a much lower score than submitting only the unique values. But this seems in contrast with what you write.

\n
\n

The goal of the competition is to understand the context surrounding the datasets.

\n
\n

A clarification on this would be much appreciated. Thanks!

", "rawMarkdown": "julialane could you please answer @verracodeguacas question? I understand we need to extract the mentions corresponding to the datasets. But there can be many duplicates (e.g. `LEHD data` mentioned many times and in the same way in the same publication). Should we keep only the unique values? From a preliminary test, submitting all the mentions (containing duplicates) achieved a much lower score than submitting only the unique values. But this seems in contrast with what you write.\n> The goal of the competition is to understand the context surrounding the datasets.\n\nA clarification on this would be much appreciated. Thanks!", "votes": null }, { "id": "1275941", "postDate": "04/16/2021 21:20:26", "content": "

@eguidotti @verracodeguadas I believe @philculliton has already answered this extensively. He's on vacation today; am sure he'll respond next week when he's back

", "rawMarkdown": "eguidotti @verracodeguadas I believe @philculliton has already answered this extensively. He's on vacation today; am sure he'll respond next week when he's back", "votes": null }, { "id": "1275945", "postDate": "04/16/2021 21:37:35", "content": "

Thanks @julialane for your reply. Can you please provide the link/quote to @philculliton extensive answer? I've been reading all threads and couldn't find it. But I'm rather new to kaggle and maybe I'm missing something. The question boils down to: should we keep duplicated mentions in the prediction string or keep only the unique mentions? Thanks!

", "rawMarkdown": "Thanks @julialane for your reply. Can you please provide the link/quote to @philculliton extensive answer? I've been reading all threads and couldn't find it. But I'm rather new to kaggle and maybe I'm missing something. The question boils down to: should we keep duplicated mentions in the prediction string or keep only the unique mentions? Thanks!", "votes": null } ], "comments": [ { "id": 1250761, "author_name": "sirishks", "author_url": "", "post_date": "03/24/2021 09:00:38", "content": "

I have updated my train data problems post asking why the same dataset can have multiple cleaned labels?

\n

This is not a problem as per Evaluation page which mentions that \"Multiple predictions are delineated with a pipe (|) character in the submission file\" and gives below example:

\n\n

where it looks like all the (multiple) cleaned labels are referring to the same dataset viz \"national space objects location dataset\"
\n(whereas we were expecting that multiple labels should correspond to multiple datasets, not the same dataset)

", "votes": null, "replies": [ { "id": 1250774, "author_name": "cpmpml", "author_url": "", "post_date": "03/24/2021 09:11:03", "content": "
\n

where all the (multiple) cleaned labels are referring to the same dataset viz \"national space objects location dataset\"

\n
\n

where do you see that? It is not in the evaluation page.

", "votes": null, "replies": [] }, { "id": 1250784, "author_name": "sirishks", "author_url": "", "post_date": "03/24/2021 09:18:47", "content": "", "votes": null, "replies": [] }, { "id": 1250801, "author_name": "arka47", "author_url": "", "post_date": "03/24/2021 09:29:27", "content": "

@sirishks I think you're referring to predictions, it actually makes sense to expect a single cleaned label per dataset

", "votes": null, "replies": [] }, { "id": 1251342, "author_name": "rai555", "author_url": "", "post_date": "03/24/2021 17:47:49", "content": "

The data page says 'Your predictions will be short excerpts from the publications that appear to note a dataset. Predictions that more accurately match the precise words used to identify the dataset within the publication will score higher.'

\n

I assume in the training data it is quite likely that excerpts from different publications will differ, even when they refer to the same dataset, and this will result in different labels.

", "votes": null, "replies": [] } ] }, { "id": 1251467, "author_name": "philculliton", "author_url": "", "post_date": "03/24/2021 19:53:49", "content": "

Hi @cpmpml - good questions. There are definitely multiple labels for a given dataset, as each label is a contextual mention in a document. So which answer is correct would depend on the document in question. The idea is less to match labels against document text, and more to figure out that a chunk of text represents a dataset, and extract that.

\n

Trailing, leading, and extra spaces are ignored in the metric.

", "votes": null, "replies": [ { "id": 1252302, "author_name": "cpmpml", "author_url": "", "post_date": "03/25/2021 15:11:43", "content": "

Phil, thanks for the answers.

", "votes": null, "replies": [] }, { "id": 1252494, "author_name": "cpmpml", "author_url": "", "post_date": "03/25/2021 18:08:10", "content": "

@philculliton I'm afraid something is still not right. Here is an example with this dataset title:
\n'Aging Integrated Database (AGID)'

\n

There are two publications with it in train.csv

\n

One of them is
\n'71d14baa-31df-4e80-9c18-af7516be766c'

\n

The dataset is referred to as 'Aging Integrated Database (AGID)' in the text.

\n

yet, in train.csv the dataset label is
\n'Aging Integrated Database'

\n

This could be fine maybe, but for this other publication with the same way to refer to the dataset :
\n'93c27c16-9356-4b97-9785-9795ad9bb609'

\n

train.csv contains two dataset label
\n'Aging Integrated Database (AGID)' and 'Aging Integrated Database'

\n

Said differently, There are no occurrences of 'Aging Integrated Database' that are not occurrences of 'Aging Integrated Database (AGID)'

\n

The fact that one pub has two different labels looks like noise.

\n

I checked other cases and they all look noisy too.

\n

I really think there would be more value if we were asked to predict uniquely defined datasets rather than semi random labels.

", "votes": null, "replies": [] }, { "id": 1269354, "author_name": "rohanrao", "author_url": "", "post_date": "04/10/2021 13:19:59", "content": "

There are many such examples. If we are expected to predict all forms of a dataset (some examples are with / without plural) appearing in the text and get penalized for not doing so, that just sucks.

\n

What is the point of needing 'Aging Integrated Database (AGID)' AND 'Aging Integrated Database' (when the latter does not appear) in the example above? Or is this what we need to deal with as 'noise'?

", "votes": null, "replies": [] }, { "id": 1269399, "author_name": "dmitriydubovskiy", "author_url": "", "post_date": "04/10/2021 14:01:11", "content": "

If one predicts correct dataset label, but misses plural form, there is no penalty for that, as Jac score is still higher than 0.5.

\n

Looks like that competition is about predicting pieces of texts containing datasets titles

", "votes": null, "replies": [] } ] }, { "id": 1252523, "author_name": "julialane", "author_url": "", "post_date": "03/25/2021 18:38:32", "content": "

@cmpml Just reinforcing @philculliton comment. It is very standard to reference multiple datasets in empirical science publications. In fact, this is one of the measures that the agencies are interested in - understanding how their dataset fit into the empirical ecosystem. So dataset coreferences in a publication is a very useful outcome measure

", "votes": null, "replies": [ { "id": 1252975, "author_name": "cpmpml", "author_url": "", "post_date": "03/26/2021 08:55:45", "content": "

@julialane I m not discussing referencing several datasets in a given publication. I am discussing referencing the same dataset with different labels in a given publication, or across publications. Let me reproduce below a case with this dataset title:
\n'Aging Integrated Database (AGID)'

\n

There are two publications with it in train.csv

\n

One of them is
\n'71d14baa-31df-4e80-9c18-af7516be766c'

\n

The dataset is referred to as 'Aging Integrated Database (AGID)' in the text.

\n

yet, in train.csv the dataset label is
\n'Aging Integrated Database'

\n

This could be fine maybe, but for this other publication with the same way to refer to the dataset :
\n'93c27c16-9356-4b97-9785-9795ad9bb609'

\n

train.csv contains two dataset label
\n'Aging Integrated Database (AGID)' and 'Aging Integrated Database'

\n

Said differently, There are no occurrences of 'Aging Integrated Database' that are not occurrences of 'Aging Integrated Database (AGID)'

\n

How can we decide if the publicaiton has one or two labels when this string appears:
\nAging Integrated Database (AGID)'

\n

It looks like random choice at this point.

\n

I checked other cases and they all look random too.

\n

I really think there would be more value if we were asked to predict uniquely defined datasets rather than semi random labels.

", "votes": null, "replies": [] } ] }, { "id": 1253112, "author_name": "julialane", "author_url": "", "post_date": "03/26/2021 11:23:41", "content": "

@cpmpml Thanks for a really good question. You are correct that the same dataset can be referred to with different names. That’s a feature of the problem. For example, the dataset that I was instrumental in creating – the Longitudinal Employer-Household Dynamics data - can also be called LEHD data or the Longitudinal Employer-Household Dynamics (LEHD). Even within a publication, the researcher might start by saying Longitudinal Employer-Household Dynamics (LEHD) data, and then later just call it LEHD.
\nThe goal of the competition is to understand the context surrounding the datasets (which is why we internally refer to the competition as \"rich context\"). That approach will help identify how researchers refer to newly emerging datasets as well – for example the many different names that are applied to the emerging numbers of COVID datasets.

\n

Thanks again for the opportunity to clarify. Happy to keep discussing, because we're thrilled by the interest and response.

", "votes": null, "replies": [ { "id": 1253166, "author_name": "cpmpml", "author_url": "", "post_date": "03/26/2021 12:30:44", "content": "

Thanks for the answer. I think I get that. But in the example I gave, I don't see how the context can help decide between these two labels:

\n

'Aging Integrated Database'
\n'Aging Integrated Database (AGID)'

\n

given the latter is the only one appearing in the publications. I mean, there are not occurrences of the first one that is not followed by ' (AGID)'

", "votes": null, "replies": [] }, { "id": 1269738, "author_name": "verracodeguacas", "author_url": "", "post_date": "04/10/2021 21:22:28", "content": "

Does that mean that a match such as this could end up being submitted?

\n

Longitudinal Employer Household Dynamics data|Longitudinal Employer-Household Dynamics data |LEHD data|LEHD data|LEHD data|LEHD data|Longitudinal Employer Household Dynamics data

\n

or in this case should we just clean it to submit the unique values?. Or is the same?

", "votes": null, "replies": [] }, { "id": 1275465, "author_name": "eguidotti", "author_url": "", "post_date": "04/16/2021 10:23:10", "content": "

@julialane could you please answer @verracodeguacas question? I understand we need to extract the mentions corresponding to the datasets. But there can be many duplicates (e.g. LEHD data mentioned many times and in the same way in the same publication). Should we keep only the unique values? From a preliminary test, submitting all the mentions (containing duplicates) achieved a much lower score than submitting only the unique values. But this seems in contrast with what you write.

\n
\n

The goal of the competition is to understand the context surrounding the datasets.

\n
\n

A clarification on this would be much appreciated. Thanks!

", "votes": null, "replies": [] } ] }, { "id": 1253175, "author_name": "julialane", "author_url": "", "post_date": "03/26/2021 12:42:41", "content": "

@cpmpml I think the point is that both names are referring to a dataset, so can be coded accordingly. We don't want to encourage the creation of a giant lookup table, but, rather, identification of the semantic context within which a dataset has been called out. @philculliton can maybe explain better than I.

", "votes": null, "replies": [ { "id": 1253186, "author_name": "sirishks", "author_url": "", "post_date": "03/26/2021 13:00:10", "content": "

@julialane Could you please answer my questions also where I have identified more serious cases where the labels are incorrect (Problem 5) and correct answers get \"0\" scores (Problem 3).

", "votes": null, "replies": [] }, { "id": 1253196, "author_name": "philculliton", "author_url": "", "post_date": "03/26/2021 13:12:47", "content": "

This is a great explanation, @julialane. The task is to identify the semantic context of a dataset's mentions. The test data contains a very large number of datasets, contexts, and publications that are entirely unseen in the training set. A model will need to generalize to the task to do well.

\n

@sirishks - I will be answering your follow-up questions in your thread.

", "votes": null, "replies": [] }, { "id": 1253254, "author_name": "cpmpml", "author_url": "", "post_date": "03/26/2021 14:25:39", "content": "

@philculliton @julialane

\n

Did you check the two publications I mention? I doubt it.

\n

In both publication the dataset is referred to by this:
\n'Aging Integrated Database (AGID)'

\n

It is NEVER referred by only this:
\n'Aging Integrated Database'

\n

Yet the latter is a ground truth, but only for one of the two publications. I challenge you to find a difference in context that would explain why the latter is selected for one of the two publications.

", "votes": null, "replies": [] }, { "id": 1253260, "author_name": "philculliton", "author_url": "", "post_date": "03/26/2021 14:35:23", "content": "

@cpmpml - thanks.

\n

Yep, I did check them! The host did as well. The training labels are noisy and they're intended to show that many forms of \"mention\" can be correctly applied to a dataset. For instance, datasets like AGID may be mentioned elsewhere without the \"(AGID)\", so learning simply to match that unique string is insufficient to solve the task.

\n

As you note, AGID may not appear without (AGID) in the training documents, but we're attempting to indicate a pretty broad universe of label differences with a relatively small label set in train. It's meant to help extrapolate.

", "votes": null, "replies": [] }, { "id": 1253270, "author_name": "cpmpml", "author_url": "", "post_date": "03/26/2021 14:47:35", "content": "

Good that you checked.

\n

Are you saying you introduced wrong labels in train to \"help extrapolate\"?

", "votes": null, "replies": [] }, { "id": 1253560, "author_name": "returnofsputnik", "author_url": "", "post_date": "03/26/2021 21:32:36", "content": "

@CPMP
\nprevious competitions have had noise and we dealt with it, for example https://www.kaggle.com/c/text-normalization-challenge-english-language

\n

usually noise is created because labels were created by their own internal model, or human error

\n

we can use label smoothing and other techniques to fight against.

\n

By the way, the way the metric is set up, anything with Jaccard Score > 0.5 is counted as a True Positive, so if you were to guess \"Aging Integrated Database\" twice it would be rewarded as two true positives.

\n

They picked 0.5 because they probably know there is noise, so this helps to fight against. although i do agree it is not ideal

", "votes": null, "replies": [] }, { "id": 1254084, "author_name": "cpmpml", "author_url": "", "post_date": "03/27/2021 10:36:37", "content": "
\n

if you were to guess \"Aging Integrated Database\" twice it would be rewarded as two true positives.

\n
\n

If it appears only once in the ground truth then one of them wold appear as a false positive. That's why I am bothered.

\n

This assumes the evaluation tab content is accurate.

\n

Not having a non ambiguous description of what we have to predict will lead to massive LB probing to reverse engineer how labels were constructed. Is this a useful use of our time really?

\n

Anyway, I asked questions, let's see if I get answers.

", "votes": null, "replies": [ { "id": 1255308, "author_name": "piotrekga", "author_url": "", "post_date": "03/28/2021 17:05:29", "content": "

We're having a similar discussion here on a code example of score computation: https://www.kaggle.com/tungmphung/coleridge-initiative-local-score-computation. Perhaps it would be easier to show some nuances of the score calculation on examples.

", "votes": null, "replies": [] } ] }, { "id": 1254086, "author_name": "cpmpml", "author_url": "", "post_date": "03/27/2021 10:37:50", "content": "
\n

anything with Jaccard Score > 0.5

\n
\n

\n

Edit: Code is fixed now.

", "votes": null, "replies": [] }, { "id": 1267002, "author_name": "dmitriydubovskiy", "author_url": "", "post_date": "04/08/2021 08:29:20", "content": "

You said \"If it appears only once in the ground truth then one of them wold appear as a false positive. That's why I am bothered.\"

\n

However, seems that since words intersection of both you labels will likely be very close in this case and both will be counted as TP (since likely more than Jaccard 0.5 score). Please what are your thoughts on that, do I miss smth?

", "votes": null, "replies": [] }, { "id": 1267656, "author_name": "hannes82", "author_url": "", "post_date": "04/08/2021 17:17:11", "content": "

Yes that's true (if we have two publications and our two predictions match with a Jaccard score of at least 0.5 the two GT).

\n

But what about 'alzheimers disease neuroimaging initiative' and 'adni'? Their intersection is 0.

", "votes": null, "replies": [] } ] }, { "id": 1275941, "author_name": "julialane", "author_url": "", "post_date": "04/16/2021 21:20:26", "content": "

@eguidotti @verracodeguadas I believe @philculliton has already answered this extensively. He's on vacation today; am sure he'll respond next week when he's back

", "votes": null, "replies": [ { "id": 1275945, "author_name": "eguidotti", "author_url": "", "post_date": "04/16/2021 21:37:35", "content": "

Thanks @julialane for your reply. Can you please provide the link/quote to @philculliton extensive answer? I've been reading all threads and couldn't find it. But I'm rather new to kaggle and maybe I'm missing something. The question boils down to: should we keep duplicated mentions in the prediction string or keep only the unique mentions? Thanks!

", "votes": null, "replies": [] } ] } ], "raw_markdown_by_id": { "1250709": "This code\n\n```\ntrain = pd.read_csv('../input/train.csv')\nalzheimer_train = train[train.dataset_title.str.startswith('Alzheimer')]\nprint(alzheimer_train.dataset_title.unique())\nprint(alzheimer_train.cleaned_label.unique())\n\n```\nprints\n\n```\n[\"Alzheimer's Disease Neuroimaging Initiative (ADNI)\"]\n['alzheimers disease neuroimaging initiative' 'adni'\n 'alzheimer s disease neuroimaging initiative adni ']\n\n```\nIt means that the same dataset appears with three different possible cleaned labels. I would expect a single cleaned label per dataset. In this case, which is the correct answer among these:\n\n```\n'alzheimers disease neuroimaging initiative',\n'adni',\n 'alzheimer s disease neuroimaging initiative adni '\n\n```\nI only checked the most frequent dataset but this doesn't look good to me. Also, note how the last one has a trailing empty space.", "1250761": "I have updated my [train data problems](https://www.kaggle.com/c/coleridgeinitiative-show-us-the-data/discussion/228322) post asking why the same dataset can have multiple cleaned labels?\n\nThis is not a problem as per [Evaluation page](https://www.kaggle.com/c/coleridgeinitiative-show-us-the-data/overview/evaluation) which mentions that \"Multiple predictions are delineated with a pipe (|) character in the submission file\" and gives below example:\n- `01e4e08c-ffea-45a7-adde-6a0c0ad755fc,space location data|national space objects|national space dataset`\n\nwhere it looks like all the (multiple) cleaned labels are referring to the same dataset viz \"national space objects location dataset\"\n(whereas we were expecting that multiple labels should correspond to multiple datasets, not the same dataset)", "1250774": "> where all the (multiple) cleaned labels are referring to the same dataset viz \"national space objects location dataset\"\n\nwhere do you see that? It is not in the evaluation page.", "1250784": "I believe that this evaluation page example row contains dummy(fake) dataset names, and was pointing out that it gives misleading information to the participants because the labels appear to refer to the same dataset\n\n- In any case, we already know that the competition train data contains [multiple labels for the same dataset](https://www.kaggle.com/c/coleridgeinitiative-show-us-the-data/discussion/228322)", "1250801": "sirishks I think you're referring to predictions, it actually makes sense to expect a single cleaned label per dataset", "1251342": "The [data page](https://www.kaggle.com/c/coleridgeinitiative-show-us-the-data/data) says 'Your predictions will be **short excerpts from the publications** that appear to note a dataset. Predictions that more accurately match the precise words used to identify the dataset within the publication will score higher.'\n\nI assume in the training data it is quite likely that excerpts from different publications will differ, even when they refer to the same dataset, and this will result in different labels.", "1251467": "Hi @cpmpml - good questions. There are definitely multiple labels for a given dataset, as each label is a contextual mention in a document. So which answer is correct would depend on the document in question. The idea is less to match labels against document text, and more to figure out that a chunk of text represents a dataset, and extract that.\n\nTrailing, leading, and extra spaces are ignored in the metric.", "1252302": "Phil, thanks for the answers.", "1252494": "philculliton I'm afraid something is still not right. Here is an example with this dataset title:\n'Aging Integrated Database (AGID)'\n\nThere are two publications with it in train.csv\n\nOne of them is \n'71d14baa-31df-4e80-9c18-af7516be766c'\n\nThe dataset is referred to as 'Aging Integrated Database (AGID)' in the text.\n\nyet, in train.csv the dataset label is\n'Aging Integrated Database'\n\nThis could be fine maybe, but for this other publication with the same way to refer to the dataset :\n'93c27c16-9356-4b97-9785-9795ad9bb609'\n\ntrain.csv contains two dataset label\n'Aging Integrated Database (AGID)' and 'Aging Integrated Database'\n\nSaid differently, There are no occurrences of 'Aging Integrated Database' that are not occurrences of 'Aging Integrated Database (AGID)'\n\nThe fact that one pub has two different labels looks like noise.\n\nI checked other cases and they all look noisy too. \n\nI really think there would be more value if we were asked to predict uniquely defined datasets rather than semi random labels.", "1252523": "cmpml Just reinforcing @philculliton comment. It is very standard to reference multiple datasets in empirical science publications. In fact, this is one of the measures that the agencies are interested in - understanding how their dataset fit into the empirical ecosystem. So dataset coreferences in a publication is a very useful outcome measure", "1252975": "julialane I m not discussing referencing several datasets in a given publication. I am discussing referencing the same dataset with different labels in a given publication, or across publications. Let me reproduce below a case with this dataset title:\n'Aging Integrated Database (AGID)'\n\nThere are two publications with it in train.csv\n\nOne of them is\n'71d14baa-31df-4e80-9c18-af7516be766c'\n\nThe dataset is referred to as 'Aging Integrated Database (AGID)' in the text.\n\nyet, in train.csv the dataset label is\n'Aging Integrated Database'\n\nThis could be fine maybe, but for this other publication with the same way to refer to the dataset :\n'93c27c16-9356-4b97-9785-9795ad9bb609'\n\ntrain.csv contains two dataset label\n'Aging Integrated Database (AGID)' and 'Aging Integrated Database'\n\nSaid differently, There are no occurrences of 'Aging Integrated Database' that are not occurrences of 'Aging Integrated Database (AGID)'\n\nHow can we decide if the publicaiton has one or two labels when this string appears:\nAging Integrated Database (AGID)'\n\nIt looks like random choice at this point.\n\nI checked other cases and they all look random too.\n\nI really think there would be more value if we were asked to predict uniquely defined datasets rather than semi random labels.", "1253112": "cpmpml Thanks for a really good question. You are correct that the same dataset can be referred to with different names. That’s a feature of the problem. For example, the dataset that I was instrumental in creating – the Longitudinal Employer-Household Dynamics data - can also be called LEHD data or the Longitudinal Employer-Household Dynamics (LEHD). Even within a publication, the researcher might start by saying Longitudinal Employer-Household Dynamics (LEHD) data, and then later just call it LEHD. \nThe goal of the competition is to understand the context surrounding the datasets (which is why we internally refer to the competition as \"rich context\"). That approach will help identify how researchers refer to newly emerging datasets as well – for example the many different names that are applied to the emerging numbers of COVID datasets.\n\nThanks again for the opportunity to clarify. Happy to keep discussing, because we're thrilled by the interest and response.", "1253166": "Thanks for the answer. I think I get that. But in the example I gave, I don't see how the context can help decide between these two labels:\n\n'Aging Integrated Database'\n'Aging Integrated Database (AGID)'\n\ngiven the latter is the only one appearing in the publications. I mean, there are not occurrences of the first one that is not followed by ' (AGID)'", "1253175": "cpmpml I think the point is that both names are referring to a dataset, so can be coded accordingly. We don't want to encourage the creation of a giant lookup table, but, rather, identification of the semantic context within which a dataset has been called out. @philculliton can maybe explain better than I.", "1253186": "julialane Could you please answer my questions also where I have identified more serious cases where [the labels are incorrect (Problem 5) and correct answers get \"0\" scores (Problem 3)](https://www.kaggle.com/c/coleridgeinitiative-show-us-the-data/discussion/228322#1251493).", "1253196": "This is a great explanation, @julialane. The task is to identify the semantic context of a dataset's mentions. The test data contains a very large number of datasets, contexts, and publications that are entirely unseen in the training set. A model will need to generalize to the task to do well.\n\n@sirishks - I will be answering your follow-up questions in your thread.", "1253254": "philculliton @julialane \n\nDid you check the two publications I mention? I doubt it.\n\nIn both publication the dataset is referred to by this:\n'Aging Integrated Database (AGID)'\n\nIt is NEVER referred by only this:\n'Aging Integrated Database'\n\nYet the latter is a ground truth, but only for one of the two publications. I challenge you to find a difference in context that would explain why the latter is selected for one of the two publications.", "1253260": "cpmpml - thanks.\n\nYep, I did check them! The host did as well. The training labels are noisy and they're intended to show that many forms of \"mention\" can be correctly applied to a dataset. For instance, datasets like AGID may be mentioned elsewhere without the \"(AGID)\", so learning simply to match that unique string is insufficient to solve the task.\n\nAs you note, AGID may not appear without (AGID) in the training documents, but we're attempting to indicate a pretty broad universe of label differences with a relatively small label set in train. It's meant to help extrapolate.", "1253270": "Good that you checked.\n\nAre you saying you introduced wrong labels in train to \"help extrapolate\"?", "1253560": "CPMP\nprevious competitions have had noise and we dealt with it, for example https://www.kaggle.com/c/text-normalization-challenge-english-language\n\nusually noise is created because labels were created by their own internal model, or human error\n\nwe can use label smoothing and other techniques to fight against.\n\nBy the way, the way the metric is set up, anything with Jaccard Score > 0.5 is counted as a True Positive, so if you were to guess \"Aging Integrated Database\" twice it would be rewarded as two true positives.\n\nThey picked 0.5 because they probably know there is noise, so this helps to fight against. although i do agree it is not ideal", "1254084": "> if you were to guess \"Aging Integrated Database\" twice it would be rewarded as two true positives.\n\nIf it appears only once in the ground truth then one of them wold appear as a false positive. That's why I am bothered. \n\nThis assumes the evaluation tab content is accurate.\n\nNot having a non ambiguous description of what we have to predict will lead to massive LB probing to reverse engineer how labels were constructed. Is this a useful use of our time really?\n\nAnyway, I asked questions, let's see if I get answers.", "1254086": "> anything with Jaccard Score > 0.5 \n\n~~Which Jaccard score? The code in evaluation tab does not compute a Jaccard score...~~\n\nEdit: Code is fixed now.", "1255308": "We're having a similar discussion here on a code example of score computation: https://www.kaggle.com/tungmphung/coleridge-initiative-local-score-computation. Perhaps it would be easier to show some nuances of the score calculation on examples.", "1267002": "You said \"If it appears only once in the ground truth then one of them wold appear as a false positive. That's why I am bothered.\"\n\nHowever, seems that since words intersection of both you labels will likely be very close in this case and both will be counted as TP (since likely more than Jaccard 0.5 score). Please what are your thoughts on that, do I miss smth?", "1267656": "Yes that's true (if we have two publications and our two predictions match with a Jaccard score of at least 0.5 the two GT).\n\nBut what about 'alzheimers disease neuroimaging initiative' and 'adni'? Their intersection is 0.", "1269354": "There are many such examples. If we are expected to predict all forms of a dataset (some examples are with / without plural) appearing in the text and get penalized for not doing so, that just sucks.\n\nWhat is the point of needing 'Aging Integrated Database (AGID)' AND 'Aging Integrated Database' (when the latter does not appear) in the example above? Or is this what we need to deal with as 'noise'?", "1269399": "If one predicts correct dataset label, but misses plural form, there is no penalty for that, as Jac score is still higher than 0.5. \n\nLooks like that competition is about predicting pieces of texts containing datasets titles", "1269738": "Does that mean that a match such as this could end up being submitted?\n\nLongitudinal Employer Household Dynamics data|Longitudinal Employer-Household Dynamics data |LEHD data|LEHD data|LEHD data|LEHD data|Longitudinal Employer Household Dynamics data\n\nor in this case should we just clean it to submit the unique values?. Or is the same?", "1275465": "julialane could you please answer @verracodeguacas question? I understand we need to extract the mentions corresponding to the datasets. But there can be many duplicates (e.g. `LEHD data` mentioned many times and in the same way in the same publication). Should we keep only the unique values? From a preliminary test, submitting all the mentions (containing duplicates) achieved a much lower score than submitting only the unique values. But this seems in contrast with what you write.\n> The goal of the competition is to understand the context surrounding the datasets.\n\nA clarification on this would be much appreciated. Thanks!", "1275941": "eguidotti @verracodeguadas I believe @philculliton has already answered this extensively. He's on vacation today; am sure he'll respond next week when he's back", "1275945": "Thanks @julialane for your reply. Can you please provide the link/quote to @philculliton extensive answer? I've been reading all threads and couldn't find it. But I'm rather new to kaggle and maybe I'm missing something. The question boils down to: should we keep duplicated mentions in the prediction string or keep only the unique mentions? Thanks!" }, "source": "meta" }