Hi everyone,
\nAfter some experiments, I observe a large gap between local CV and public LB. From the previous forum posted by @alexandervc, we know that the public LB contains only about 100 proteins. Hence, the fluctuation in public LB score is expected. However, there exists a large gap between my local CV score and public LB, which are shown as follows:
\n| Method | \nCV | \nPublic LB | \n
|---|---|---|
| *Diamond | \n0.55110 | \n0.41578 | \n
| Blast-KNN | \n**0.58617 | \n0.41793 | \n
| MLP with LLM Embeddings V1 (5 Fold Ensemble) | \n0.55630 | \n0.51116 | \n
| MLP with LLM Embeddings V2 (Best Fold Only) | \n0.56317 | \n0.51164 | \n
| MLP with LLM Embeddings V2 (5 Fold Ensemble) | \n0.56326 | \n0.51318 | \n
| MLP with LLM Embeddings V3 (5 Fold Ensemble) | \n0.56581 | \n0.51566 | \n
\n\n*Diamond follows this great notebook shared by @geraseva. And the public LB score is directly copied from it.
\n
\n ** It seems Blast-KNN benefits too much from random KFold, which has severer leakage due to homologous proteins in both training and validation sets.
My local CV scheme is the random group kfold (k == 5) shown as follows:
\ncv = KFold(, shuffle=, random_state=)\n i, (tr_idx, val_idx) (cv.split(X=prot_ids)):\n \n prot_ids_tr, prot_ids_val = prot_ids[tr_idx], prot_ids[val_idx]\n\n \n \nwhere prot_ids contains all unique protein identifiers in training set.
Although the CV and public LB look sync here, the large gap of the scores is hard to neglect. Any thoughts on this issue? Or, maybe I do something wrong on my local CV setup?
\nThanks a lot for your help!
Hi everyone,
\nAfter some experiments, I observe a large gap between local CV and public LB. From the previous forum posted by @alexandervc, we know that the public LB contains only about 100 proteins. Hence, the fluctuation in public LB score is expected. However, there exists a large gap between my local CV score and public LB, which are shown as follows:
\n| Method | \nCV | \nPublic LB | \n
|---|---|---|
| *Diamond | \n0.55110 | \n0.41578 | \n
| Blast-KNN | \n**0.58617 | \n0.41793 | \n
| MLP with LLM Embeddings V1 (5 Fold Ensemble) | \n0.55630 | \n0.51116 | \n
| MLP with LLM Embeddings V2 (Best Fold Only) | \n0.56317 | \n0.51164 | \n
| MLP with LLM Embeddings V2 (5 Fold Ensemble) | \n0.56326 | \n0.51318 | \n
| MLP with LLM Embeddings V3 (5 Fold Ensemble) | \n0.56581 | \n0.51566 | \n
\n\n*Diamond follows this great notebook shared by @geraseva. And the public LB score is directly copied from it.
\n
\n ** It seems Blast-KNN benefits too much from random KFold, which has severer leakage due to homologous proteins in both training and validation sets.
My local CV scheme is the random group kfold (k == 5) shown as follows:
\ncv = KFold(, shuffle=, random_state=)\n i, (tr_idx, val_idx) (cv.split(X=prot_ids)):\n \n prot_ids_tr, prot_ids_val = prot_ids[tr_idx], prot_ids[val_idx]\n\n \n \nwhere prot_ids contains all unique protein identifiers in training set.
Although the CV and public LB look sync here, the large gap of the scores is hard to neglect. Any thoughts on this issue? Or, maybe I do something wrong on my local CV setup?
\nThanks a lot for your help!
Thanks for your findings. As far as I observed during past competitions, there is often a large shake up between public and private LB. Public LB is vicious because it favors overfitting a bit. As CV is a great tool to reduce overfitting and get model robust to generalization, it lowers the public LB score. But at the end it reduces the risk to get a noisy private LB score.
", "rawMarkdown": "Thanks for your findings. As far as I observed during past competitions, there is often a large shake up between public and private LB. Public LB is vicious because it favors overfitting a bit. As CV is a great tool to reduce overfitting and get model robust to generalization, it lowers the public LB score. But at the end it reduces the risk to get a noisy private LB score.", "votes": null }, { "id": "2385663", "postDate": "08/11/2023 13:40:12", "content": "Hi @henriupton,
\nThanks for the comment. My current experiments indicate that CV and public LB seem to have great correlation, but the gap is so significant that I'm afraid my CV setup might be over-optimistic. Especially, Blast-KNN (mainly based on homology) looks optimistic on local CV but worse on LB. Hence, it's hard to determine whether my CV setup is reliable or not.
", "rawMarkdown": "Hi @henriupton,\n\nThanks for the comment. My current experiments indicate that CV and public LB seem to have great correlation, but the gap is so significant that I'm afraid my CV setup might be over-optimistic. Especially, Blast-KNN (mainly based on homology) looks optimistic on local CV but worse on LB. Hence, it's hard to determine whether my CV setup is reliable or not.", "votes": null } ], "comments": [ { "id": 2385411, "author_name": "henriupton", "author_url": "", "post_date": "08/11/2023 10:19:10", "content": "Thanks for your findings. As far as I observed during past competitions, there is often a large shake up between public and private LB. Public LB is vicious because it favors overfitting a bit. As CV is a great tool to reduce overfitting and get model robust to generalization, it lowers the public LB score. But at the end it reduces the risk to get a noisy private LB score.
", "votes": null, "replies": [ { "id": 2385663, "author_name": "abaojiang", "author_url": "", "post_date": "08/11/2023 13:40:12", "content": "Hi @henriupton,
\nThanks for the comment. My current experiments indicate that CV and public LB seem to have great correlation, but the gap is so significant that I'm afraid my CV setup might be over-optimistic. Especially, Blast-KNN (mainly based on homology) looks optimistic on local CV but worse on LB. Hence, it's hard to determine whether my CV setup is reliable or not.
", "votes": null, "replies": [] } ] } ], "raw_markdown_by_id": { "2381298": "Hi everyone,\n\nAfter some experiments, I observe a large gap between local CV and public LB. From the [previous forum](https://www.kaggle.com/competitions/cafa-5-protein-function-prediction/discussion/424134) posted by @alexandervc, we know that the public LB contains only about 100 proteins. Hence, the fluctuation in public LB score is expected. However, there exists a large gap between my local CV score and public LB, which are shown as follows:\n\n| Method | CV | Public LB |\n| --- | --- | ---|\n| \\*Diamond | 0.55110 | 0.41578 |\n| Blast-KNN | \\**0.58617 | 0.41793 |\n| MLP with LLM Embeddings V1 (5 Fold Ensemble) | 0.55630 | 0.51116 |\n| MLP with LLM Embeddings V2 (Best Fold Only) | 0.56317 | 0.51164 |\n| MLP with LLM Embeddings V2 (5 Fold Ensemble) | 0.56326 | 0.51318 |\n| MLP with LLM Embeddings V3 (5 Fold Ensemble) | 0.56581 | 0.51566 |\n> \\*Diamond follows this [great notebook](https://www.kaggle.com/code/geraseva/diamond) shared by @geraseva. And the public LB score is directly copied from it.\n> \\** It seems Blast-KNN benefits too much from random KFold, which has severer leakage due to homologous proteins in both training and validation sets.\n\nMy local CV scheme is the random group kfold (k == 5) shown as follows:\n```python\ncv = KFold(5, shuffle=True, random_state=6)\nfor i, (tr_idx, val_idx) in enumerate(cv.split(X=prot_ids)):\n # Split training and validation proteins\n prot_ids_tr, prot_ids_val = prot_ids[tr_idx], prot_ids[val_idx]\n \n # Run training and evaluation processes\n # ...\n```\nwhere `prot_ids` contains **all unique protein identifiers** in training set. \n\nAlthough the CV and public LB look sync here, the **large gap** of the scores is hard to neglect. Any thoughts on this issue? Or, maybe I do something wrong on my local CV setup? \nThanks a lot for your help!", "2385411": "Thanks for your findings. As far as I observed during past competitions, there is often a large shake up between public and private LB. Public LB is vicious because it favors overfitting a bit. As CV is a great tool to reduce overfitting and get model robust to generalization, it lowers the public LB score. But at the end it reduces the risk to get a noisy private LB score.", "2385663": "Hi @henriupton,\n\nThanks for the comment. My current experiments indicate that CV and public LB seem to have great correlation, but the gap is so significant that I'm afraid my CV setup might be over-optimistic. Especially, Blast-KNN (mainly based on homology) looks optimistic on local CV but worse on LB. Hence, it's hard to determine whether my CV setup is reliable or not." }, "source": "meta" }