The last two, three days I have been trying to set up a proper CV set and came to the conclusion that it is not a trivial undertaking. Let me try to outline why:
\nI believe a proper CV strategy needs to divide the training data by dataset and publication. In other words, the train and validation set should not share any dataset and publication in order to match how the private test set is set up. Now here the troubles begin: The problem is that a significant share of publications cite more than one dataset (actually they cite even more because the training set is not comprehensively labelled, but let's ignore that for the sake of argument). So let's say, we assign a publication to the CV set, it has two datasets that means we now need to assign all other publications that cite these two datasets to the CV.. ok.. Now these publications just assigned to the CV set cite several other datasets which again have been cited by other publications and so on and so on..
\nIt actually turns out that it is possible to devide the training data into two equally sized \"disjoint\" datasets (meaning they share neither a dataset nor a publication), one of them containing 30 datasets and the other 15 (the reason why the later is about as large as the former is the dataset ADNI, which is cited a lot). This is ok but far from optimum as we disregard a high share of potential data when it comes to train our model and any kfold estimation is not really possible. (Actually there is also another possibility with a very small CV set, but this would only contain four datasets.) It can of course be that a missed a possible split (my code iterates x times over the dataset trying to make a split).
\nAnother possibility is to disregard some publications to make a kfold split possible: my best solution in this regard is a 4-fold split with 176 observations that cannot be used (observations are the publications-dataset pairs). And they 4 folds are not of equal size and cotain a somewhat different number of datasets. For sure not optimal.
\nA third option may be to say that a split by publications is not that important. In the end, we need to identify datasets mentions and identifying a dataset in a sequence may be equally difficult independently of whether another sequence has been used for training!?
\nTo sum up I think there are three options with respect to setting up a cross-validation set:
I am not sure what is the best option and I would love to hear your thoughts on this! Maybe I am also missing something and it is in the end not that difficult!?
\nSetting up a proper CV for this competition seems really important given that only a subset of the public test set (which by itself only represents 12% of the whole test set) is actually useful to assess our models.
The last two, three days I have been trying to set up a proper CV set and came to the conclusion that it is not a trivial undertaking. Let me try to outline why:
\nI believe a proper CV strategy needs to divide the training data by dataset and publication. In other words, the train and validation set should not share any dataset and publication in order to match how the private test set is set up. Now here the troubles begin: The problem is that a significant share of publications cite more than one dataset (actually they cite even more because the training set is not comprehensively labelled, but let's ignore that for the sake of argument). So let's say, we assign a publication to the CV set, it has two datasets that means we now need to assign all other publications that cite these two datasets to the CV.. ok.. Now these publications just assigned to the CV set cite several other datasets which again have been cited by other publications and so on and so on..
\nIt actually turns out that it is possible to devide the training data into two equally sized \"disjoint\" datasets (meaning they share neither a dataset nor a publication), one of them containing 30 datasets and the other 15 (the reason why the later is about as large as the former is the dataset ADNI, which is cited a lot). This is ok but far from optimum as we disregard a high share of potential data when it comes to train our model and any kfold estimation is not really possible. (Actually there is also another possibility with a very small CV set, but this would only contain four datasets.) It can of course be that a missed a possible split (my code iterates x times over the dataset trying to make a split).
\nAnother possibility is to disregard some publications to make a kfold split possible: my best solution in this regard is a 4-fold split with 176 observations that cannot be used (observations are the publications-dataset pairs). And they 4 folds are not of equal size and cotain a somewhat different number of datasets. For sure not optimal.
\nA third option may be to say that a split by publications is not that important. In the end, we need to identify datasets mentions and identifying a dataset in a sequence may be equally difficult independently of whether another sequence has been used for training!?
\nTo sum up I think there are three options with respect to setting up a cross-validation set:
I am not sure what is the best option and I would love to hear your thoughts on this! Maybe I am also missing something and it is in the end not that difficult!?
\nSetting up a proper CV for this competition seems really important given that only a subset of the public test set (which by itself only represents 12% of the whole test set) is actually useful to assess our models.
You could split a corpus like this thematically since it's neatly divided into mutually exclusive topics (e.g., aging, climate change, covid etc.) with the hold-out being a single topic, and then averaging the validation scores across topics.
\nI don't think it's productive to come up with an overelaborate CV setup for this competition. We don't have complete labels for train so your hold-out isn't clean. In circumstances like this, I'd just be mindful about not overtuning hyper-parameters or manually coming up with rules to juice your public LB score.
", "rawMarkdown": "You could split a corpus like this thematically since it's neatly divided into mutually exclusive topics (e.g., aging, climate change, covid etc.) with the hold-out being a single topic, and then averaging the validation scores across topics. \n\nI don't think it's productive to come up with an overelaborate CV setup for this competition. We don't have complete labels for train so your hold-out isn't clean. In circumstances like this, I'd just be mindful about not overtuning hyper-parameters or manually coming up with rules to juice your public LB score.", "votes": null }, { "id": "1278730", "postDate": "04/20/2021 08:25:27", "content": "Thank you for your feedback! Dividing by topic sounds like a good idea. Just not sure whether such a clear cut split exists as outlined above.
", "rawMarkdown": "Thank you for your feedback! Dividing by topic sounds like a good idea. Just not sure whether such a clear cut split exists as outlined above.", "votes": null }, { "id": "1283335", "postDate": "04/24/2021 20:36:02", "content": "I've been looking into the idea of splitting by dataset and publication as well. I was able to use networkx to create an undirected graph where the nodes are publications and datasets. I then extracted all of the disjoint subgraphs. For each subgraph I then filtered out the dataset nodes, leaving only the publication nodes. One can then combine publications from some of the small subgraphs together as desired. As long as the publications from an individual subgraph are not split across training and testing portions, then there will be no overlap in publications or datasets.
Oh thanks, very useful! I was also thinking that approaching it graphically/with a network analysis would probably be the best way to do this, but didn't investigate further. I will try out networkx, sounds like a convenient tool for this.
", "rawMarkdown": "Oh thanks, very useful! I was also thinking that approaching it graphically/with a network analysis would probably be the best way to do this, but didn't investigate further. I will try out networkx, sounds like a convenient tool for this.", "votes": null } ], "comments": [ { "id": 1278477, "author_name": "leecming", "author_url": "", "post_date": "04/20/2021 00:00:41", "content": "You could split a corpus like this thematically since it's neatly divided into mutually exclusive topics (e.g., aging, climate change, covid etc.) with the hold-out being a single topic, and then averaging the validation scores across topics.
\nI don't think it's productive to come up with an overelaborate CV setup for this competition. We don't have complete labels for train so your hold-out isn't clean. In circumstances like this, I'd just be mindful about not overtuning hyper-parameters or manually coming up with rules to juice your public LB score.
", "votes": null, "replies": [ { "id": 1278730, "author_name": "hannes82", "author_url": "", "post_date": "04/20/2021 08:25:27", "content": "Thank you for your feedback! Dividing by topic sounds like a good idea. Just not sure whether such a clear cut split exists as outlined above.
", "votes": null, "replies": [] } ] }, { "id": 1283335, "author_name": "trentb", "author_url": "", "post_date": "04/24/2021 20:36:02", "content": "I've been looking into the idea of splitting by dataset and publication as well. I was able to use networkx to create an undirected graph where the nodes are publications and datasets. I then extracted all of the disjoint subgraphs. For each subgraph I then filtered out the dataset nodes, leaving only the publication nodes. One can then combine publications from some of the small subgraphs together as desired. As long as the publications from an individual subgraph are not split across training and testing portions, then there will be no overlap in publications or datasets.
Oh thanks, very useful! I was also thinking that approaching it graphically/with a network analysis would probably be the best way to do this, but didn't investigate further. I will try out networkx, sounds like a convenient tool for this.
", "votes": null, "replies": [] } ] } ], "raw_markdown_by_id": { "1278434": "The last two, three days I have been trying to set up a proper CV set and came to the conclusion that it is not a trivial undertaking. Let me try to outline why:\nI believe a proper CV strategy needs to divide the training data by **dataset** and **publication**. In other words, the train and validation set should not share any dataset and publication in order to match how the private test set is set up. Now here the troubles begin: The problem is that a significant share of publications cite more than one dataset (actually they cite even more because the training set is not comprehensively labelled, but let's ignore that for the sake of argument). So let's say, we assign a publication to the CV set, it has two datasets that means we now need to assign all other publications that cite these two datasets to the CV.. ok.. Now these publications just assigned to the CV set cite several other datasets which again have been cited by other publications and so on and so on..\nIt actually turns out that it is possible to devide the training data into two equally sized \"disjoint\" datasets (meaning they share neither a dataset nor a publication), one of them containing 30 datasets and the other 15 (the reason why the later is about as large as the former is the dataset ADNI, which is cited a lot). This is ok but far from optimum as we disregard a high share of potential data when it comes to train our model and any kfold estimation is not really possible. (Actually there is also another possibility with a very small CV set, but this would only contain four datasets.) It can of course be that a missed a possible split (my code iterates x times over the dataset trying to make a split).\nAnother possibility is to disregard some publications to make a kfold split possible: my best solution in this regard is a 4-fold split with 176 observations that cannot be used (observations are the publications-dataset pairs). And they 4 folds are not of equal size and cotain a somewhat different number of datasets. For sure not optimal.\nA third option may be to say that a split by publications is not that important. In the end, we need to identify datasets mentions and identifying a dataset in a sequence may be equally difficult independently of whether another sequence has been used for training!?\nTo sum up I think there are three options with respect to setting up a cross-validation set:\n- Splitting the training data into equally sized training and validation sets (datasets split: 30 vs. 15)\n- 4-fold CV with losing some observations and an unequal split between folds\n- Not splitting the training data by publication which makes a groupkfold setup much more feasonable.\n\nI am not sure what is the best option and I would love to hear your thoughts on this! Maybe I am also missing something and it is in the end not that difficult!?\nSetting up a proper CV for this competition seems really important given that only a subset of the public test set (which by itself only represents 12% of the whole test set) is actually useful to assess our models.", "1278477": "You could split a corpus like this thematically since it's neatly divided into mutually exclusive topics (e.g., aging, climate change, covid etc.) with the hold-out being a single topic, and then averaging the validation scores across topics. \n\nI don't think it's productive to come up with an overelaborate CV setup for this competition. We don't have complete labels for train so your hold-out isn't clean. In circumstances like this, I'd just be mindful about not overtuning hyper-parameters or manually coming up with rules to juice your public LB score.", "1278730": "Thank you for your feedback! Dividing by topic sounds like a good idea. Just not sure whether such a clear cut split exists as outlined above.", "1283335": "I've been looking into the idea of splitting by dataset and publication as well. I was able to use `networkx` to create an undirected graph where the nodes are publications and datasets. I then extracted all of the disjoint subgraphs. For each subgraph I then filtered out the dataset nodes, leaving only the publication nodes. One can then combine publications from some of the small subgraphs together as desired. As long as the publications from an individual subgraph are not split across training and testing portions, then there will be no overlap in publications or datasets.", "1283832": "Oh thanks, very useful! I was also thinking that approaching it graphically/with a network analysis would probably be the best way to do this, but didn't investigate further. I will try out networkx, sounds like a convenient tool for this." }, "source": "meta" }