Hi,,
\nEach mention of a dataset in the 'private' test set is matched with one and only one Ground Truth Labels (GT), am I correct?
If that is not the case, there will be 'artifical' FN's, as I dont think neither baseline methods nor 'ML algorithms' intend to create more than one prediction for any single mention,,
\nI believe.
(example: if a substring of the publication is 'NCES common core of data' and that is a dataset you want us to find, you create only one GT for this mention. There may be other GT based on the same dataset onlyif those other mentions cannot be assigned to the same GT.
\nPlease note that this is not 'directly' a question on how to calculate score (it is resolved thanks to Shota), but a question on how 'you' decide the GT(which ofcourse affect everything).
\nPS: could you also verify what you mean for competition deadline as in: For transparency, the host requests that participants post links to external data sources here as you use them, no later than the competition deadline specified..
\n 'competition deadline' is ….(a or b)
\na. Entry Deadline .. June 15, 2021
\nb. Submission Deadline.. June 22, 2021
Thank you in advance,
", "messages": [ { "id": 1332807, "postDate": "2021-06-02T10:11:45.273Z", "content": "Hi,,
\nEach mention of a dataset in the 'private' test set is matched with one and only one Ground Truth Labels (GT), am I correct?
If that is not the case, there will be 'artifical' FN's, as I dont think neither baseline methods nor 'ML algorithms' intend to create more than one prediction for any single mention,,
\nI believe.
(example: if a substring of the publication is 'NCES common core of data' and that is a dataset you want us to find, you create only one GT for this mention. There may be other GT based on the same dataset onlyif those other mentions cannot be assigned to the same GT.
\nPlease note that this is not 'directly' a question on how to calculate score (it is resolved thanks to Shota), but a question on how 'you' decide the GT(which ofcourse affect everything).
\nPS: could you also verify what you mean for competition deadline as in: For transparency, the host requests that participants post links to external data sources here as you use them, no later than the competition deadline specified..
\n 'competition deadline' is ….(a or b)
\na. Entry Deadline .. June 15, 2021
\nb. Submission Deadline.. June 22, 2021
Thank you in advance,
", "rawMarkdown": "Hi,,\nEach mention of a dataset in the 'private' test set is matched with one and only one Ground Truth Labels (GT), am I correct? \n\nIf that is not the case, there will be 'artifical' FN's, as I dont think neither baseline methods nor 'ML algorithms' intend to create more than one prediction for any single mention,, \nI believe. \n\n(example: if a substring of the publication is 'NCES common core of data' and that is a dataset you want us to find, you create only one GT for this mention. There may be other GT based on the same dataset onlyif those other mentions cannot be assigned to the same GT.\n\nPlease note that this is not 'directly' a question on how to calculate score (it is resolved thanks to Shota), but a question on how 'you' decide the GT(which ofcourse affect everything).\n\nPS: could you also verify what you mean for competition deadline as in: *For transparency, the host requests that participants post links to external data sources here as you use them, no later than the competition deadline specified..*\n 'competition deadline' is ….(a or b)\na. Entry Deadline .. June 15, 2021\nb. Submission Deadline.. June 22, 2021\n\nThank you in advance,\n", "votes": 2 }, { "id": 1336375, "postDate": "2021-06-04T20:43:43.013Z", "content": "Hi all
\nGreat question. The root issue is that authors refer to the same dataset in different ways. Even the same author in a single publication can refer to the same dataset differently. In the labelling we tried to capture this behavior by creating aliases so that the terms all point to the same underlying data entity. While some noisiness and messiness is unavoidable (as we have repeatedly noted), we worked really hard to avoid that in the test set.
\nAlso - just a reminder from yesterday - we are thrilled to announce that we will be hosting the Show US the Data Conference in mid October in Washington, D.C., where we will highlight the results of this competition. As you know, the goal of the competition has been to help US federal agencies use state-of-the-art methods to quickly discover how their data are being used. Winners of this competition, as well as strong contributors, will have the opportunity to present solutions to a broader audience of federal agencies, publishers, and researchers. We welcome the Kaggle community to contribute your knowledge and expertise to these exciting efforts.
\nWe are looking to incorporate the results of the winning models in some initial tools, like data usage scorecards that summarize usage for government datasets, and Data Inventories that provide lists of agency data assets and their use. We are also looking for collaborators on industry papers to extend the research of the work done on this competition.
\nPlease fill out this form below if you want us to keep you informed on these efforts. We will only use the information here for the purposes described above.
\nThank you again for your hard work and collaboration!
Maybe you should publish 1 file from the test data containing all the labels, it would help a lot
", "rawMarkdown": "Maybe you should publish 1 file from the test data containing all the labels, it would help a lot", "votes": 1 }, { "id": 1336894, "postDate": "2021-06-05T09:13:41.297Z", "rawMarkdown": "", "isDeleted": true }, { "id": 1341164, "postDate": "2021-06-08T13:23:44.220Z", "content": "I asked for something similar in https://www.kaggle.com/c/coleridgeinitiative-show-us-the-data/discussion/235822
\nI am still waiting for an answer that was promised about a month ago. But actually this type of question has been ignored in many discussions. At this point, my assumption is that labeling is somewhat random and inconsistent which might be the cause for this avoidance.
\nThe lack of an example, that shows what perfect labeling looks like, is really frustrating and made me stop working on this competition.
", "rawMarkdown": "I asked for something similar in https://www.kaggle.com/c/coleridgeinitiative-show-us-the-data/discussion/235822\n\nI am still waiting for an answer that was promised about a month ago. But actually this type of question has been ignored in many discussions. At this point, my assumption is that labeling is somewhat random and inconsistent which might be the cause for this avoidance.\n\nThe lack of an example, that shows what perfect labeling looks like, is really frustrating and made me stop working on this competition." } ] }, { "id": 1335874, "postDate": "2021-06-04T13:42:36.080Z", "content": "This is a huge problem. For example, in the first document there are two labels 'national education longitudinal study' and 'education longitudinal study' in the markup, but there is only mention of 'national education longitudinal study', so it is not clear how we should predict both cases
", "rawMarkdown": "This is a huge problem. For example, in the first document there are two labels 'national education longitudinal study' and 'education longitudinal study' in the markup, but there is only mention of 'national education longitudinal study', so it is not clear how we should predict both cases", "replies": [ { "id": 1335941, "postDate": "2021-06-04T14:26:14.533Z", "content": "I checked related earlier discussion(s) but it looks like those did not produce an outcome.
\nhttps://www.kaggle.com/c/coleridgeinitiative-show-us-the-data/discussion/236972
\n
The thing is, 'having one label per mention' is not a cosmetic or standardization-only issue. So deciding on one or the other label as the Ground Truth (shorter or longer version of the 'mention') is not the big issue. At the end you will have a TP in both situations as long as you predict one of those.
\nThe big thing is: having a 'non-consistent' double labeling (yes this double-labeling is not 'consistent' all along) will make the scoring simply 'false'.
\nGood thing is that 'this issue' does not require another long round of relabeling session and could be solved with a simple script running on the test set.
\nI can actually volenteer to run it on the 'test' set, no problem 😄
I agree with you. The first document mentions 'national education longitudinal study' three times. In a month of participation, I still don't know whether we should predict three 'national education longitudinal studies' or one. And why there is both 'national education longitudinal study' and 'education longitudinal study' in the markup
", "rawMarkdown": "I agree with you. The first document mentions 'national education longitudinal study' three times. In a month of participation, I still don't know whether we should predict three 'national education longitudinal studies' or one. And why there is both 'national education longitudinal study' and 'education longitudinal study' in the markup" } ] }, { "id": 1334262, "postDate": "2021-06-03T11:18:29.980Z", "content": "I guess the,
\nbottomline is, as indicated in various early discusions, if private test set includes double labeling (creating more than one ground truth label for one mention), as is the case in various publications in train and public test set (it is not hard to check that), this competition might lose [some of] its sense, because as highlighted above, the aim of an algorithm should not be trying to guess when a specific mention could be associated with two GTs and when not.
There is still time, and I don't see any other way… But since this is such a big problem, most probably, as a freshman, I am missing something,…
\nI hope we will hear from host/kaggle/Masters soon…as I know I am not the only one :-)
", "rawMarkdown": "I guess the,\nbottomline is, as indicated in various early discusions, if private test set includes double labeling (creating more than one ground truth label for one mention), as is the case in various publications in train and public test set (it is not hard to check that), this competition might lose [some of] its sense, because as highlighted above, the aim of an algorithm should not be trying to guess when a specific mention could be associated with two GTs and when not. \n\nThere is still time, and I don't see any other way... But since this is such a big problem, most probably, as a freshman, I am missing something,...\n\nI hope we will hear from host/kaggle/Masters soon...as I know I am not the only one :-)\n\n\n\n", "replies": [ { "id": 1335823, "postDate": "2021-06-04T13:10:14.090Z", "content": "I guess uploading attachements to kaggle in discussions is currently disabled….
\nBelow the link to the visual on the problem, if anyone wants to shed a light.
\nhttps://drive.google.com/file/d/1X807ANiWXvjxOfbC0_RIeG_O36X-rqAt/view?usp=sharing
\n
Hi all
\nGreat question. The root issue is that authors refer to the same dataset in different ways. Even the same author in a single publication can refer to the same dataset differently. In the labelling we tried to capture this behavior by creating aliases so that the terms all point to the same underlying data entity. While some noisiness and messiness is unavoidable (as we have repeatedly noted), we worked really hard to avoid that in the test set.
\nAlso - just a reminder from yesterday - we are thrilled to announce that we will be hosting the Show US the Data Conference in mid October in Washington, D.C., where we will highlight the results of this competition. As you know, the goal of the competition has been to help US federal agencies use state-of-the-art methods to quickly discover how their data are being used. Winners of this competition, as well as strong contributors, will have the opportunity to present solutions to a broader audience of federal agencies, publishers, and researchers. We welcome the Kaggle community to contribute your knowledge and expertise to these exciting efforts.
\nWe are looking to incorporate the results of the winning models in some initial tools, like data usage scorecards that summarize usage for government datasets, and Data Inventories that provide lists of agency data assets and their use. We are also looking for collaborators on industry papers to extend the research of the work done on this competition.
\nPlease fill out this form below if you want us to keep you informed on these efforts. We will only use the information here for the purposes described above.
\nThank you again for your hard work and collaboration!
Maybe you should publish 1 file from the test data containing all the labels, it would help a lot
", "votes": 1, "replies": [] }, { "id": 1336894, "author_name": "", "author_url": "", "post_date": "2021-06-05T09:13:41.297000", "content": "", "votes": 0, "replies": [] }, { "id": 1341164, "author_name": "Enrico Lohmann", "author_url": "", "post_date": "2021-06-08T13:23:44.220000", "content": "I asked for something similar in https://www.kaggle.com/c/coleridgeinitiative-show-us-the-data/discussion/235822
\nI am still waiting for an answer that was promised about a month ago. But actually this type of question has been ignored in many discussions. At this point, my assumption is that labeling is somewhat random and inconsistent which might be the cause for this avoidance.
\nThe lack of an example, that shows what perfect labeling looks like, is really frustrating and made me stop working on this competition.
", "votes": 0, "replies": [] } ] }, { "id": 1335874, "author_name": "Kramarenko Vladislav", "author_url": "", "post_date": "2021-06-04T13:42:36.080000", "content": "This is a huge problem. For example, in the first document there are two labels 'national education longitudinal study' and 'education longitudinal study' in the markup, but there is only mention of 'national education longitudinal study', so it is not clear how we should predict both cases
", "votes": 0, "replies": [ { "id": 1335941, "author_name": "GUNER", "author_url": "", "post_date": "2021-06-04T14:26:14.533000", "content": "I checked related earlier discussion(s) but it looks like those did not produce an outcome.
\nhttps://www.kaggle.com/c/coleridgeinitiative-show-us-the-data/discussion/236972
\n
The thing is, 'having one label per mention' is not a cosmetic or standardization-only issue. So deciding on one or the other label as the Ground Truth (shorter or longer version of the 'mention') is not the big issue. At the end you will have a TP in both situations as long as you predict one of those.
\nThe big thing is: having a 'non-consistent' double labeling (yes this double-labeling is not 'consistent' all along) will make the scoring simply 'false'.
\nGood thing is that 'this issue' does not require another long round of relabeling session and could be solved with a simple script running on the test set.
\nI can actually volenteer to run it on the 'test' set, no problem 😄
I agree with you. The first document mentions 'national education longitudinal study' three times. In a month of participation, I still don't know whether we should predict three 'national education longitudinal studies' or one. And why there is both 'national education longitudinal study' and 'education longitudinal study' in the markup
", "votes": 0, "replies": [] } ] }, { "id": 1334262, "author_name": "GUNER", "author_url": "", "post_date": "2021-06-03T11:18:29.980000", "content": "I guess the,
\nbottomline is, as indicated in various early discusions, if private test set includes double labeling (creating more than one ground truth label for one mention), as is the case in various publications in train and public test set (it is not hard to check that), this competition might lose [some of] its sense, because as highlighted above, the aim of an algorithm should not be trying to guess when a specific mention could be associated with two GTs and when not.
There is still time, and I don't see any other way… But since this is such a big problem, most probably, as a freshman, I am missing something,…
\nI hope we will hear from host/kaggle/Masters soon…as I know I am not the only one :-)
", "votes": 0, "replies": [ { "id": 1335823, "author_name": "GUNER", "author_url": "", "post_date": "2021-06-04T13:10:14.090000", "content": "I guess uploading attachements to kaggle in discussions is currently disabled….
\nBelow the link to the visual on the problem, if anyone wants to shed a light.
\nhttps://drive.google.com/file/d/1X807ANiWXvjxOfbC0_RIeG_O36X-rqAt/view?usp=sharing
\n