The HubMap competition just ended with what looks to be a bunch of top public LB teams blowing themselves up by using hand annotations (and pseudo-labels), and trusting the public LB over CV.
\nIt's a good reminder that this competition is going to be incredibly risky especially for top public LB teams given :-
\nwe don't have any way to perform meaningful local CV
fitting to public LB is literally an act of fitting to training data
we are likely to end the competition without being given a formal definition of what a dataset is
There's a reason why there's a conspicuous absence of GMs and other seasoned Kagglers here…
", "messages": [ { "id": 1301108, "postDate": "2021-05-11T00:28:40.767Z", "content": "The HubMap competition just ended with what looks to be a bunch of top public LB teams blowing themselves up by using hand annotations (and pseudo-labels), and trusting the public LB over CV.
\nIt's a good reminder that this competition is going to be incredibly risky especially for top public LB teams given :-
\nwe don't have any way to perform meaningful local CV
fitting to public LB is literally an act of fitting to training data
we are likely to end the competition without being given a formal definition of what a dataset is
There's a reason why there's a conspicuous absence of GMs and other seasoned Kagglers here…
", "rawMarkdown": "The HubMap competition just ended with what looks to be a bunch of top public LB teams blowing themselves up by using hand annotations (and pseudo-labels), and trusting the public LB over CV. \n\nIt's a good reminder that this competition is going to be incredibly risky especially for top public LB teams given :-\n1. we don't have any way to perform meaningful local CV\n\n2. fitting to public LB is literally an act of fitting to training data\n\n3. we are likely to end the competition without being given a formal definition of what a dataset is \n\nThere's a reason why there's a conspicuous absence of GMs and other seasoned Kagglers here...", "votes": 51 }, { "id": 1301295, "postDate": "2021-05-11T02:33:50.637Z", "content": "There will definitely be a shakeup if you look at the leaderboard alone since it doesn't reflect true performance in any way. Right now many people haven't even realized that a large chunk of public test contains training labels and happily submit the result.
\nFor people who are serious and spent a lot of time to understand the gotcha's however, I don't think the real shake up would be that big, but we will never know.
", "rawMarkdown": "There will definitely be a shakeup if you look at the leaderboard alone since it doesn't reflect true performance in any way. Right now many people haven't even realized that a large chunk of public test contains training labels and happily submit the result. \n\nFor people who are serious and spent a lot of time to understand the gotcha's however, I don't think the real shake up would be that big, but we will never know.", "votes": 12, "replies": [ { "id": 1301829, "postDate": "2021-05-11T09:05:20.397Z", "content": "Can you take me also in them team?
", "rawMarkdown": "Can you take me also in them team?" } ] }, { "id": 1303075, "postDate": "2021-05-11T22:04:21.440Z", "content": "A nagging concern I've had is whether the training data (not the weak training labels) is representative of the testing data. I understand that the testing data on which our public LB score is derived has been intentionally tainted with training data (i.e., some of the same publications), but it would be nice to know if the range of topics of the publications (and roughly their proportions) is roughly similar between training and testing data. If they are not similar, then, in my opinion, this should explicitly be stated because then the competition becomes more about just good generalization. It becomes about transfer learning / domain adaptation, which requires a modified approach.
\nFor example, hypothetically let's say that the training data we were given had no publications about COVID-19 (I know this is not the case, but just pretend for the sake of this example). I doubt that a model developed on only this training data would generalize well to capture labels in the testing data like 2019 ncov complete genome sequences, covid 19 genome sequence, or genome sequences of sars cov 2. In this case, we would need to use the testing data within our submitted notebook to adapt our model for this domain.
\nSo my concern is that the testing data could have topics from a new domain of which we are unaware. If this is the case and teams have not considered adapting to the testing data, then I think there will be a big shake up. I also think that it would be in the best interest of the host to state if there is any validity to my concern because, if this is about domain adaptation, better models will be produced if this piece of information is disclosed.
", "rawMarkdown": "A nagging concern I've had is whether the training data (not the weak training labels) is representative of the testing data. I understand that the testing data on which our public LB score is derived has been intentionally tainted with training data (i.e., some of the same publications), but it would be nice to know if the range of topics of the publications (and roughly their proportions) is roughly similar between training and testing data. If they are not similar, then, in my opinion, this should explicitly be stated because then the competition becomes more about just good generalization. It becomes about transfer learning / domain adaptation, which requires a modified approach.\n\nFor example, hypothetically let's say that the training data we were given had no publications about COVID-19 (I know this is not the case, but just pretend for the sake of this example). I doubt that a model developed on only this training data would generalize well to capture labels in the testing data like *2019 ncov complete genome sequences*, *covid 19 genome sequence*, or *genome sequences of sars cov 2*. In this case, we would need to use the testing data within our submitted notebook to adapt our model for this domain.\n\nSo my concern is that the testing data could have topics from a new domain of which we are unaware. If this is the case and teams have not considered adapting to the testing data, then I think there will be a big shake up. I also think that it would be in the best interest of the host to state if there is any validity to my concern because, if this is about domain adaptation, better models will be produced if this piece of information is disclosed.", "votes": 6, "replies": [ { "id": 1303104, "postDate": "2021-05-11T22:58:24.393Z", "content": "Yeah, Phil and Julia have been quite emphatic about the hidden test-set being disjoint and containing many more contexts and labels so it's quite possible.
\nA consolation is the lower FN penalty and the low likelihood of a team being able to create a sophisticated model that can handle OOD contexts but not excel with the training set/public test-set.
\nIn any case, I get the impression both Kaggle and Coleridge have the ability to view the private LB rankings mid-competition so I hope they're monitoring if the public-to-private LB is a random mapping. Undertaking a large re-annotation exercise mid-flight would suggest that they're being fairly proactive to reacting to issues (whatever grouses we might have).
", "rawMarkdown": "Yeah, Phil and Julia have been quite emphatic about the hidden test-set being disjoint and containing many more contexts and labels so it's quite possible. \n\nA consolation is the lower FN penalty and the low likelihood of a team being able to create a sophisticated model that can handle OOD contexts but not excel with the training set/public test-set. \n\nIn any case, I get the impression both Kaggle and Coleridge have the ability to view the private LB rankings mid-competition so I hope they're monitoring if the public-to-private LB is a random mapping. Undertaking a large re-annotation exercise mid-flight would suggest that they're being fairly proactive to reacting to issues (whatever grouses we might have). ", "votes": 3 }, { "id": 1303237, "postDate": "2021-05-12T01:22:04.507Z", "content": "I recall information about the testing data publications being disjoint and having many more labels, but I didn't remember a statement about there being more contexts.
", "rawMarkdown": "I recall information about the testing data publications being disjoint and having many more labels, but I didn't remember a statement about there being more contexts.", "votes": 1 }, { "id": 1303254, "postDate": "2021-05-12T01:30:51.867Z", "content": "It's a bit unfortunate but the discussions on this were quite fragmented. You probably should just go thru Phil's comment history but specifically :-
\nhttps://www.kaggle.com/c/coleridgeinitiative-show-us-the-data/discussion/228337#1253196
\n\n", "rawMarkdown": "It's a bit unfortunate but the discussions on this were quite fragmented. You probably should just go thru Phil's comment history but specifically :- \n\nhttps://www.kaggle.com/c/coleridgeinitiative-show-us-the-data/discussion/228337#1253196\n> \nThis is a great explanation, @julialane. The task is to identify the semantic context of a dataset's mentions. The test data contains a very large number of datasets, contexts, and publications that are entirely unseen in the training set. A model will need to generalize to the task to do well.\n@sirishks - I will be answering your follow-up questions in your thread.", "votes": 8 }, { "id": 1303349, "postDate": "2021-05-12T03:10:54.630Z", "content": "This is a great explanation, @julialane. The task is to identify the semantic context of a dataset's mentions. The test data contains a very large number of datasets, contexts, and publications that are entirely unseen in the training set. A model will need to generalize to the task to do well.
\n
\n @sirishks - I will be answering your follow-up questions in your thread.
Thank you for finding that!
", "rawMarkdown": "Thank you for finding that!", "votes": 1 }, { "id": 1303630, "postDate": "2021-05-12T06:51:33.913Z", "content": "We need to clearly distinguish two concepts: context and topic :))).
", "rawMarkdown": "We need to clearly distinguish two concepts: **context** and **topic** :))). ", "votes": 2 }, { "id": 1303909, "postDate": "2021-05-12T10:34:20.760Z", "content": "I would say to some extent it comes as a natural implication that the test set covers unseen domains, otherwise a complete disjoint of referenced datasets between train and test set is difficult to achieve (given that papers in the same domain very often cite or use the same datasets).
", "rawMarkdown": "I would say to some extent it comes as a natural implication that the test set covers unseen domains, otherwise a complete disjoint of referenced datasets between train and test set is difficult to achieve (given that papers in the same domain very often cite or use the same datasets).", "votes": 1 }, { "id": 1304869, "postDate": "2021-05-13T00:33:02.077Z", "content": "IMO, the key risk here is tuning models into making predictions \"too liberally\". FNs are penalized much less so it'd be better to withhold predictions for difficult/OOD samples. But the public LB is pushing us to make more predictions…
", "rawMarkdown": "IMO, the key risk here is tuning models into making predictions \"too liberally\". FNs are penalized much less so it'd be better to withhold predictions for difficult/OOD samples. But the public LB is pushing us to make more predictions... ", "votes": 5 } ] }, { "id": 1361592, "postDate": "2021-06-23T00:03:13.913Z", "content": "Looks like I was right ^^
", "rawMarkdown": "Looks like I was right ^^", "votes": 2, "replies": [ { "id": 1361595, "postDate": "2021-06-23T00:07:30.143Z", "content": "This is far from an armageddon, pretty normal by kaggle standard actually, and given how meaningless the public LB was, all things considered the actual shakeup would be minimal.
", "rawMarkdown": "This is far from an armageddon, pretty normal by kaggle standard actually, and given how meaningless the public LB was, all things considered the actual shakeup would be minimal." } ] }, { "id": 1332142, "postDate": "2021-06-02T00:07:40.123Z", "content": "BirdClef just ended and it's another example of a major shake-up…
", "rawMarkdown": "BirdClef just ended and it's another example of a major shake-up...", "votes": 2 }, { "id": 1336693, "postDate": "2021-06-05T06:42:49.790Z", "content": "When we talk about CV, is it like split data into N_FOLDS ( 5 ) and then ccalculate f1_beta (Jaccard) on validation data ?
", "rawMarkdown": "When we talk about CV, is it like split data into N_FOLDS ( 5 ) and then ccalculate f1_beta (Jaccard) on validation data ?" }, { "id": 1301882, "postDate": "2021-05-11T09:37:26.497Z", "content": "I am on the fence about this competition, I love the idea of looking through longer documents like this (in fact it matches closely with something I'm doing for work) but like you've pointed out there have been tons of issues so far. I think we will end up with ~10-20 teams who build out some kind of sane-ish local CV, ignore the LB and do pretty well……. at least that's my hope as I sit here on day 3 of trying to build out a sane-ish CV
", "rawMarkdown": "I am on the fence about this competition, I love the idea of looking through longer documents like this (in fact it matches closely with something I'm doing for work) but like you've pointed out there have been tons of issues so far. I think we will end up with ~10-20 teams who build out some kind of sane-ish local CV, ignore the LB and do pretty well....... at least that's my hope as I sit here on day 3 of trying to build out a sane-ish CV" }, { "id": 1301825, "postDate": "2021-05-11T09:04:26.913Z", "content": "Just now, My Two models didn't performed well, becz I was not making pipeline to go for, Can anyone help me in that? My teammate's uncle is expired so I want someone to help me, it will really help
", "rawMarkdown": "Just now, My Two models didn't performed well, becz I was not making pipeline to go for, Can anyone help me in that? My teammate's uncle is expired so I want someone to help me, it will really help " }, { "id": 1340727, "postDate": "2021-06-08T08:03:04.757Z", "rawMarkdown": "", "isDeleted": true } ], "comments": [ { "id": 1301295, "author_name": "Khoi Nguyen", "author_url": "", "post_date": "2021-05-11T02:33:50.637000", "content": "There will definitely be a shakeup if you look at the leaderboard alone since it doesn't reflect true performance in any way. Right now many people haven't even realized that a large chunk of public test contains training labels and happily submit the result.
\nFor people who are serious and spent a lot of time to understand the gotcha's however, I don't think the real shake up would be that big, but we will never know.
", "votes": 12, "replies": [ { "id": 1301829, "author_name": "Ayush Singh", "author_url": "", "post_date": "2021-05-11T09:05:20.397000", "content": "Can you take me also in them team?
", "votes": 0, "replies": [] } ] }, { "id": 1303075, "author_name": "Trent", "author_url": "", "post_date": "2021-05-11T22:04:21.440000", "content": "A nagging concern I've had is whether the training data (not the weak training labels) is representative of the testing data. I understand that the testing data on which our public LB score is derived has been intentionally tainted with training data (i.e., some of the same publications), but it would be nice to know if the range of topics of the publications (and roughly their proportions) is roughly similar between training and testing data. If they are not similar, then, in my opinion, this should explicitly be stated because then the competition becomes more about just good generalization. It becomes about transfer learning / domain adaptation, which requires a modified approach.
\nFor example, hypothetically let's say that the training data we were given had no publications about COVID-19 (I know this is not the case, but just pretend for the sake of this example). I doubt that a model developed on only this training data would generalize well to capture labels in the testing data like 2019 ncov complete genome sequences, covid 19 genome sequence, or genome sequences of sars cov 2. In this case, we would need to use the testing data within our submitted notebook to adapt our model for this domain.
\nSo my concern is that the testing data could have topics from a new domain of which we are unaware. If this is the case and teams have not considered adapting to the testing data, then I think there will be a big shake up. I also think that it would be in the best interest of the host to state if there is any validity to my concern because, if this is about domain adaptation, better models will be produced if this piece of information is disclosed.
", "votes": 6, "replies": [ { "id": 1303104, "author_name": "Chun Ming Lee", "author_url": "", "post_date": "2021-05-11T22:58:24.393000", "content": "Yeah, Phil and Julia have been quite emphatic about the hidden test-set being disjoint and containing many more contexts and labels so it's quite possible.
\nA consolation is the lower FN penalty and the low likelihood of a team being able to create a sophisticated model that can handle OOD contexts but not excel with the training set/public test-set.
\nIn any case, I get the impression both Kaggle and Coleridge have the ability to view the private LB rankings mid-competition so I hope they're monitoring if the public-to-private LB is a random mapping. Undertaking a large re-annotation exercise mid-flight would suggest that they're being fairly proactive to reacting to issues (whatever grouses we might have).
", "votes": 3, "replies": [] }, { "id": 1303237, "author_name": "Trent", "author_url": "", "post_date": "2021-05-12T01:22:04.507000", "content": "I recall information about the testing data publications being disjoint and having many more labels, but I didn't remember a statement about there being more contexts.
", "votes": 1, "replies": [] }, { "id": 1303254, "author_name": "Chun Ming Lee", "author_url": "", "post_date": "2021-05-12T01:30:51.867000", "content": "It's a bit unfortunate but the discussions on this were quite fragmented. You probably should just go thru Phil's comment history but specifically :-
\nhttps://www.kaggle.com/c/coleridgeinitiative-show-us-the-data/discussion/228337#1253196
\n\n", "votes": 8, "replies": [] }, { "id": 1303349, "author_name": "Trent", "author_url": "", "post_date": "2021-05-12T03:10:54.630000", "content": "This is a great explanation, @julialane. The task is to identify the semantic context of a dataset's mentions. The test data contains a very large number of datasets, contexts, and publications that are entirely unseen in the training set. A model will need to generalize to the task to do well.
\n
\n @sirishks - I will be answering your follow-up questions in your thread.
Thank you for finding that!
", "votes": 1, "replies": [] }, { "id": 1303630, "author_name": "Nguyen Quan Anh Minh", "author_url": "", "post_date": "2021-05-12T06:51:33.913000", "content": "We need to clearly distinguish two concepts: context and topic :))).
", "votes": 2, "replies": [] }, { "id": 1303909, "author_name": "Hannes Öhler", "author_url": "", "post_date": "2021-05-12T10:34:20.760000", "content": "I would say to some extent it comes as a natural implication that the test set covers unseen domains, otherwise a complete disjoint of referenced datasets between train and test set is difficult to achieve (given that papers in the same domain very often cite or use the same datasets).
", "votes": 1, "replies": [] }, { "id": 1304869, "author_name": "Chun Ming Lee", "author_url": "", "post_date": "2021-05-13T00:33:02.077000", "content": "IMO, the key risk here is tuning models into making predictions \"too liberally\". FNs are penalized much less so it'd be better to withhold predictions for difficult/OOD samples. But the public LB is pushing us to make more predictions…
", "votes": 5, "replies": [] } ] }, { "id": 1361592, "author_name": "Chun Ming Lee", "author_url": "", "post_date": "2021-06-23T00:03:13.913000", "content": "Looks like I was right ^^
", "votes": 2, "replies": [ { "id": 1361595, "author_name": "Khoi Nguyen", "author_url": "", "post_date": "2021-06-23T00:07:30.143000", "content": "This is far from an armageddon, pretty normal by kaggle standard actually, and given how meaningless the public LB was, all things considered the actual shakeup would be minimal.
", "votes": 0, "replies": [] } ] }, { "id": 1332142, "author_name": "Chun Ming Lee", "author_url": "", "post_date": "2021-06-02T00:07:40.123000", "content": "BirdClef just ended and it's another example of a major shake-up…
", "votes": 2, "replies": [] }, { "id": 1336693, "author_name": "aintnosunshine", "author_url": "", "post_date": "2021-06-05T06:42:49.790000", "content": "When we talk about CV, is it like split data into N_FOLDS ( 5 ) and then ccalculate f1_beta (Jaccard) on validation data ?
", "votes": 0, "replies": [] }, { "id": 1301882, "author_name": "Hamish", "author_url": "", "post_date": "2021-05-11T09:37:26.497000", "content": "I am on the fence about this competition, I love the idea of looking through longer documents like this (in fact it matches closely with something I'm doing for work) but like you've pointed out there have been tons of issues so far. I think we will end up with ~10-20 teams who build out some kind of sane-ish local CV, ignore the LB and do pretty well……. at least that's my hope as I sit here on day 3 of trying to build out a sane-ish CV
", "votes": 0, "replies": [] }, { "id": 1301825, "author_name": "Ayush Singh", "author_url": "", "post_date": "2021-05-11T09:04:26.913000", "content": "Just now, My Two models didn't performed well, becz I was not making pipeline to go for, Can anyone help me in that? My teammate's uncle is expired so I want someone to help me, it will really help
", "votes": 0, "replies": [] }, { "id": 1340727, "author_name": "", "author_url": "", "post_date": "2021-06-08T08:03:04.757000", "content": "", "votes": 0, "replies": [] } ], "raw_markdown_by_id": { "1301108": "The HubMap competition just ended with what looks to be a bunch of top public LB teams blowing themselves up by using hand annotations (and pseudo-labels), and trusting the public LB over CV. \n\nIt's a good reminder that this competition is going to be incredibly risky especially for top public LB teams given :-\n1. we don't have any way to perform meaningful local CV\n\n2. fitting to public LB is literally an act of fitting to training data\n\n3. we are likely to end the competition without being given a formal definition of what a dataset is \n\nThere's a reason why there's a conspicuous absence of GMs and other seasoned Kagglers here...", "1301295": "There will definitely be a shakeup if you look at the leaderboard alone since it doesn't reflect true performance in any way. Right now many people haven't even realized that a large chunk of public test contains training labels and happily submit the result. \n\nFor people who are serious and spent a lot of time to understand the gotcha's however, I don't think the real shake up would be that big, but we will never know.", "1303075": "A nagging concern I've had is whether the training data (not the weak training labels) is representative of the testing data. I understand that the testing data on which our public LB score is derived has been intentionally tainted with training data (i.e., some of the same publications), but it would be nice to know if the range of topics of the publications (and roughly their proportions) is roughly similar between training and testing data. If they are not similar, then, in my opinion, this should explicitly be stated because then the competition becomes more about just good generalization. It becomes about transfer learning / domain adaptation, which requires a modified approach.\n\nFor example, hypothetically let's say that the training data we were given had no publications about COVID-19 (I know this is not the case, but just pretend for the sake of this example). I doubt that a model developed on only this training data would generalize well to capture labels in the testing data like *2019 ncov complete genome sequences*, *covid 19 genome sequence*, or *genome sequences of sars cov 2*. In this case, we would need to use the testing data within our submitted notebook to adapt our model for this domain.\n\nSo my concern is that the testing data could have topics from a new domain of which we are unaware. If this is the case and teams have not considered adapting to the testing data, then I think there will be a big shake up. I also think that it would be in the best interest of the host to state if there is any validity to my concern because, if this is about domain adaptation, better models will be produced if this piece of information is disclosed.", "1361592": "Looks like I was right ^^", "1332142": "BirdClef just ended and it's another example of a major shake-up...", "1336693": "When we talk about CV, is it like split data into N_FOLDS ( 5 ) and then ccalculate f1_beta (Jaccard) on validation data ?", "1301882": "I am on the fence about this competition, I love the idea of looking through longer documents like this (in fact it matches closely with something I'm doing for work) but like you've pointed out there have been tons of issues so far. I think we will end up with ~10-20 teams who build out some kind of sane-ish local CV, ignore the LB and do pretty well....... at least that's my hope as I sit here on day 3 of trying to build out a sane-ish CV", "1301825": "Just now, My Two models didn't performed well, becz I was not making pipeline to go for, Can anyone help me in that? My teammate's uncle is expired so I want someone to help me, it will really help ", "1340727": "" } }