![\"CDF](\"https://i.ibb.co/zNHbcFn/distributions-submission-05-public-merge.png\")
Hi everyone ! Hope everyone is doing well.
\nI am still deeply wondering how the notebooks that ensemble a bunch of submissions based on the test public dataset can even perform well for the private test dataset ? Indeed the private test set is not the same than the public test set, so as these notebooks only take submission files as input (and not models), how can it work? Perhaps I am missing something. 😁
Hi everyone ! Hope everyone is doing well.
\nI am still deeply wondering how the notebooks that ensemble a bunch of submissions based on the test public dataset can even perform well for the private test dataset ? Indeed the private test set is not the same than the public test set, so as these notebooks only take submission files as input (and not models), how can it work? Perhaps I am missing something. 😁
Every protein in the private test set is within the visible test set that our models have already predicted. There is no 'hidden' set.
\nThis is only nominally a code competition. It is fine to prepare submission.tsv offline and then submit by attaching that file to a notebook.
", "rawMarkdown": "Every protein in the private test set is within the visible test set that our models have already predicted. There is no 'hidden' set.\n\nThis is only nominally a code competition. It is fine to prepare submission.tsv offline and then submit by attaching that file to a notebook.", "votes": null }, { "id": "2399378", "postDate": "08/20/2023 10:00:38", "content": "Top public notebook are overfitting to public LB.
\nIn particular gradually people understood that parameter \"35\" from original MT's notebook \"merge_datatasets\", should be increased - see recent public notebooks. And a couple of other changes.
PS
\nI thought it is explained in the post here
\nhttps://www.kaggle.com/competitions/cafa-5-protein-function-prediction/discussion/432907
\nbut that was misunderstanding, sorry
there is no private test dataset. There is only a public test dataset from which proteins will be selected, and the final score will be calculated based on these proteins later on.
", "rawMarkdown": "there is no private test dataset. There is only a public test dataset from which proteins will be selected, and the final score will be calculated based on these proteins later on.", "votes": null }, { "id": "2400347", "postDate": "08/21/2023 02:42:44", "content": "I show below cumulative distribution functions (CDFs) for a \"regular\" submission and for one of those public merge notebooks. Not only does public CDF predict a huge proportion of points at >0.5 probability, but you can also observe those horizontal parts where unnatural rounding or clipping was done to the data. Not saying it is impossible that these merged predictions will work on private data, but I would not put my money on that horse.
\n
Some of the models that end up within the public ensembles do produce non continuous predictions so the flat parts in the CDF is actually unsurprising! I do not completely understand why that might make these predictions less robust but it is an interesting observation.
", "rawMarkdown": "Some of the models that end up within the public ensembles do produce non continuous predictions so the flat parts in the CDF is actually unsurprising! I do not completely understand why that might make these predictions less robust but it is an interesting observation.", "votes": null }, { "id": "2403007", "postDate": "08/22/2023 13:06:02", "content": "Thanks @tilii7, that's a really interesting plot. Noting that this lies below the y=x line, and approximately differentiating in my head. I don't know if your 'regular submission' is a single model or ensemble, but it looks as I'd expect a single model to. One of the interesting things about this competition is that there are many ways to make an ensemble, which treat the 'probability' in different manners. My team have some ensembles designed to manipulate the 'probability' in ways that reflect our estimate of the precision of the scoring (i.e., above best F1 threshold) portion of each model. They will have 'weird' CDFs, I'm sure.
\nSome other thoughts on the 'probability'. I believe there's no requirement for it to scale as a probability, just for it to rank the predictions within [0.001, 1.0] as per the order of our genuine confidence (or estimate of probability) for each, and to disperse them sensibly with regard to the trial thresholds (i.e., the multiples of 0.001; namely 0.001, 0.002 … 0.998, 0.999, 1.000) such that not too many predictions get the same score to 3 dp. There may even be a small benefit in morphing the distribution to allow more trial thrsholds to appear in the region where we expect to find the scoring threshold (we didn't do this).
", "rawMarkdown": "Thanks @tilii7, that's a really interesting plot. Noting that this lies below the y=x line, and approximately differentiating in my head. I don't know if your 'regular submission' is a single model or ensemble, but it looks as I'd expect a single model to. One of the interesting things about this competition is that there are many ways to make an ensemble, which treat the 'probability' in different manners. My team have some ensembles designed to manipulate the 'probability' in ways that reflect our estimate of the precision of the scoring (i.e., above best F1 threshold) portion of each model. They will have 'weird' CDFs, I'm sure.\n\nSome other thoughts on the 'probability'. I believe there's no requirement for it to scale as a probability, just for it to rank the predictions within [0.001, 1.0] as per the order of our genuine confidence (or estimate of probability) for each, and to disperse them sensibly with regard to the trial thresholds (i.e., the multiples of 0.001; namely 0.001, 0.002 ... 0.998, 0.999, 1.000) such that not too many predictions get the same score to 3 dp. There may even be a small benefit in morphing the distribution to allow more trial thrsholds to appear in the region where we expect to find the scoring threshold (we didn't do this).", "votes": null }, { "id": "2403904", "postDate": "08/22/2023 23:54:41", "content": "\n\nI don't know if your 'regular submission' is a single model or ensemble
\n
It is an ensemble of three models (one for each ontology category) but without any funny business - a simple concatenation.
\nIt is to be seen at a later time whether manipulating probabilities by thresholding and/or weighting is a better approach. Part of me is happy that I didn't have time to dig into that and was doing the simplest ensembling possible, though I may come to regret that come Christmas.
", "rawMarkdown": "> I don't know if your 'regular submission' is a single model or ensemble\n\nIt is an ensemble of three models (one for each ontology category) but without any funny business - a simple concatenation.\n\nIt is to be seen at a later time whether manipulating probabilities by thresholding and/or weighting is a better approach. Part of me is happy that I didn't have time to dig into that and was doing the simplest ensembling possible, though I may come to regret that come Christmas.", "votes": null } ], "comments": [ { "id": 2399327, "author_name": "jbomitchell", "author_url": "", "post_date": "08/20/2023 09:13:07", "content": "Every protein in the private test set is within the visible test set that our models have already predicted. There is no 'hidden' set.
\nThis is only nominally a code competition. It is fine to prepare submission.tsv offline and then submit by attaching that file to a notebook.
", "votes": null, "replies": [] }, { "id": 2399378, "author_name": "alexandervc", "author_url": "", "post_date": "08/20/2023 10:00:38", "content": "Top public notebook are overfitting to public LB.
\nIn particular gradually people understood that parameter \"35\" from original MT's notebook \"merge_datatasets\", should be increased - see recent public notebooks. And a couple of other changes.
PS
\nI thought it is explained in the post here
\nhttps://www.kaggle.com/competitions/cafa-5-protein-function-prediction/discussion/432907
\nbut that was misunderstanding, sorry
there is no private test dataset. There is only a public test dataset from which proteins will be selected, and the final score will be calculated based on these proteins later on.
", "votes": null, "replies": [] }, { "id": 2400347, "author_name": "tilii7", "author_url": "", "post_date": "08/21/2023 02:42:44", "content": "I show below cumulative distribution functions (CDFs) for a \"regular\" submission and for one of those public merge notebooks. Not only does public CDF predict a huge proportion of points at >0.5 probability, but you can also observe those horizontal parts where unnatural rounding or clipping was done to the data. Not saying it is impossible that these merged predictions will work on private data, but I would not put my money on that horse.
\n
Some of the models that end up within the public ensembles do produce non continuous predictions so the flat parts in the CDF is actually unsurprising! I do not completely understand why that might make these predictions less robust but it is an interesting observation.
", "votes": null, "replies": [] }, { "id": 2403007, "author_name": "jbomitchell", "author_url": "", "post_date": "08/22/2023 13:06:02", "content": "Thanks @tilii7, that's a really interesting plot. Noting that this lies below the y=x line, and approximately differentiating in my head. I don't know if your 'regular submission' is a single model or ensemble, but it looks as I'd expect a single model to. One of the interesting things about this competition is that there are many ways to make an ensemble, which treat the 'probability' in different manners. My team have some ensembles designed to manipulate the 'probability' in ways that reflect our estimate of the precision of the scoring (i.e., above best F1 threshold) portion of each model. They will have 'weird' CDFs, I'm sure.
\nSome other thoughts on the 'probability'. I believe there's no requirement for it to scale as a probability, just for it to rank the predictions within [0.001, 1.0] as per the order of our genuine confidence (or estimate of probability) for each, and to disperse them sensibly with regard to the trial thresholds (i.e., the multiples of 0.001; namely 0.001, 0.002 … 0.998, 0.999, 1.000) such that not too many predictions get the same score to 3 dp. There may even be a small benefit in morphing the distribution to allow more trial thrsholds to appear in the region where we expect to find the scoring threshold (we didn't do this).
", "votes": null, "replies": [ { "id": 2403904, "author_name": "tilii7", "author_url": "", "post_date": "08/22/2023 23:54:41", "content": "\n\nI don't know if your 'regular submission' is a single model or ensemble
\n
It is an ensemble of three models (one for each ontology category) but without any funny business - a simple concatenation.
\nIt is to be seen at a later time whether manipulating probabilities by thresholding and/or weighting is a better approach. Part of me is happy that I didn't have time to dig into that and was doing the simplest ensembling possible, though I may come to regret that come Christmas.
", "votes": null, "replies": [] } ] } ] } ], "raw_markdown_by_id": { "2399303": "Hi everyone ! Hope everyone is doing well. \nI am still deeply wondering how the notebooks that ensemble a bunch of submissions based on the test public dataset can even perform well for the private test dataset ? Indeed the private test set is not the same than the public test set, so as these notebooks only take submission files as input (and not models), how can it work? Perhaps I am missing something. 😁", "2399327": "Every protein in the private test set is within the visible test set that our models have already predicted. There is no 'hidden' set.\n\nThis is only nominally a code competition. It is fine to prepare submission.tsv offline and then submit by attaching that file to a notebook.", "2399378": "Top public notebook are overfitting to public LB.\nIn particular gradually people understood that parameter \"35\" from original MT's notebook \"merge_datatasets\", should be increased - see recent public notebooks. And a couple of other changes.\n\nPS\nI thought it is explained in the post here\nhttps://www.kaggle.com/competitions/cafa-5-protein-function-prediction/discussion/432907\nbut that was misunderstanding, sorry", "2399553": "there is no private test dataset. There is only a public test dataset from which proteins will be selected, and the final score will be calculated based on these proteins later on.", "2400347": "I show below cumulative distribution functions (CDFs) for a \"regular\" submission and for one of those public merge notebooks. Not only does public CDF predict a huge proportion of points at >0.5 probability, but you can also observe those horizontal parts where unnatural rounding or clipping was done to the data. Not saying it is impossible that these merged predictions will work on private data, but I would not put my money on that horse.\n\n", "2400361": "Some of the models that end up within the public ensembles do produce non continuous predictions so the flat parts in the CDF is actually unsurprising! I do not completely understand why that might make these predictions less robust but it is an interesting observation.", "2403007": "Thanks @tilii7, that's a really interesting plot. Noting that this lies below the y=x line, and approximately differentiating in my head. I don't know if your 'regular submission' is a single model or ensemble, but it looks as I'd expect a single model to. One of the interesting things about this competition is that there are many ways to make an ensemble, which treat the 'probability' in different manners. My team have some ensembles designed to manipulate the 'probability' in ways that reflect our estimate of the precision of the scoring (i.e., above best F1 threshold) portion of each model. They will have 'weird' CDFs, I'm sure.\n\nSome other thoughts on the 'probability'. I believe there's no requirement for it to scale as a probability, just for it to rank the predictions within [0.001, 1.0] as per the order of our genuine confidence (or estimate of probability) for each, and to disperse them sensibly with regard to the trial thresholds (i.e., the multiples of 0.001; namely 0.001, 0.002 ... 0.998, 0.999, 1.000) such that not too many predictions get the same score to 3 dp. There may even be a small benefit in morphing the distribution to allow more trial thrsholds to appear in the region where we expect to find the scoring threshold (we didn't do this).", "2403904": "> I don't know if your 'regular submission' is a single model or ensemble\n\nIt is an ensemble of three models (one for each ontology category) but without any funny business - a simple concatenation.\n\nIt is to be seen at a later time whether manipulating probabilities by thresholding and/or weighting is a better approach. Part of me is happy that I didn't have time to dig into that and was doing the simplest ensembling possible, though I may come to regret that come Christmas." }, "source": "meta" }