{ "id": 431491, "title": "3 remarkable ideas from Zoltan's notebook \"CombineEmbeddings\"", "url": "/competitions/cafa-5-protein-function-prediction/discussion/431491", "author_name": "", "post_date": "2023-08-13T19:11:14.825118100Z", "votes": 16, "comment_count": 2, "views": 0, "content": "

Zoltan's notebook \"CombineEmbeddings\"
\nhttps://www.kaggle.com/code/szabo7zoltan/combineembeddings/notebook
\ncontains not just 1 idea as may seem from the title, but actually three of them.
\nLet me comment on that , and share some elaboration.
\nIt gets LB 0.50510. Small improvement (again Ridge) will give 0.50881 and I guess more can be achieved in pretty simple way (see below). The baseline Ridge is LB 0.47717. It is remarkable that jump 0.03 can be achieved without heavy machinery, but just on ideas.
\n1) The idea from the title \"Combine embeddings\" - concatenate embeddings and train model on them - score increases - demonstrated for t5 concat esm2-size1280 for Ridge model. That is a kind of standard idea of feature engineering - combine more features , but there is always a risk to overfit. Notebook demonstrates for that case it can work. Comment: one can check that the same idea works not only for Ridge but for other models, like MLP.
\nIdeas , 2 and 3 are not that much standard:
\n2) Downsampling to 43k specially selected samples (proteins) and training model only on them. Surprisingly it gives uplift for many models ! Comment: it seems to me the rationale behind is the following: we know that there is enormous amount of missing labels for CAFA data. I.e. the experimental check of some properties have not been done yet - thus there is no label - not because protein does not actually have that property - but only just because the experiment have not yet been done yet - that is known as \"open world\". Thus it reasonable somehow to select those proteins which have more abundant labels and train only on them ! That has been done. The actual criteria proposed by Zoltan - choose only those proteins which have labels in all three ontologies. (It might be one can improve the selection of the proteins).
\n3) Improved way to select targets. In many notebooks we restrict ourselves to 500-1500 top frequent targets. But here it was proposed to do the selection more wisely: to choose top frequent targets respecting the ontologies: 1100 were selected from \"Biological processes\", 450 from \"Molecular function\", 300 \"Cellur component\". No idea how these numbers were obtained - but tried to change them - no good results (so far).
\n--- ======================================================
\nSmall improvement:
\nJust combine 5 embeddings and get LB 0.50881 with the simple Ridge model:
\nhttps://www.kaggle.com/code/alexandervc/pytorch-keras-etc-3-blend-cafa-metric-etc?scriptVersionId=139811068
\nHere are: ['t5', 'esm2S1280','esm2S320', 'esm2S480','esm2S2560' ] .
\nYou may try to add esm2-size640, protbert, and esm2 with other poolings - may be you can improve the result.
\nHere you can see that according to local scoring (random folds)
\nthe bigger embeddings are added - the better score we get:
\n\"\"
\nSee https://www.kaggle.com/code/alexandervc/pytorch-keras-etc-3-blend-cafa-metric-etc?scriptVersionId=139799984&cellId=65
\nSo it is probably not just LB artifact.
\nPS
\nAgain these ideas seems to work not only for the Ridge model, but for the other also, but seems results are more advanced for Ridge.

", "messages": [ { "id": "2389058", "postDate": "08/13/2023 19:11:14", "content": "

Zoltan's notebook \"CombineEmbeddings\"
\nhttps://www.kaggle.com/code/szabo7zoltan/combineembeddings/notebook
\ncontains not just 1 idea as may seem from the title, but actually three of them.
\nLet me comment on that , and share some elaboration.
\nIt gets LB 0.50510. Small improvement (again Ridge) will give 0.50881 and I guess more can be achieved in pretty simple way (see below). The baseline Ridge is LB 0.47717. It is remarkable that jump 0.03 can be achieved without heavy machinery, but just on ideas.
\n1) The idea from the title \"Combine embeddings\" - concatenate embeddings and train model on them - score increases - demonstrated for t5 concat esm2-size1280 for Ridge model. That is a kind of standard idea of feature engineering - combine more features , but there is always a risk to overfit. Notebook demonstrates for that case it can work. Comment: one can check that the same idea works not only for Ridge but for other models, like MLP.
\nIdeas , 2 and 3 are not that much standard:
\n2) Downsampling to 43k specially selected samples (proteins) and training model only on them. Surprisingly it gives uplift for many models ! Comment: it seems to me the rationale behind is the following: we know that there is enormous amount of missing labels for CAFA data. I.e. the experimental check of some properties have not been done yet - thus there is no label - not because protein does not actually have that property - but only just because the experiment have not yet been done yet - that is known as \"open world\". Thus it reasonable somehow to select those proteins which have more abundant labels and train only on them ! That has been done. The actual criteria proposed by Zoltan - choose only those proteins which have labels in all three ontologies. (It might be one can improve the selection of the proteins).
\n3) Improved way to select targets. In many notebooks we restrict ourselves to 500-1500 top frequent targets. But here it was proposed to do the selection more wisely: to choose top frequent targets respecting the ontologies: 1100 were selected from \"Biological processes\", 450 from \"Molecular function\", 300 \"Cellur component\". No idea how these numbers were obtained - but tried to change them - no good results (so far).
\n--- ======================================================
\nSmall improvement:
\nJust combine 5 embeddings and get LB 0.50881 with the simple Ridge model:
\nhttps://www.kaggle.com/code/alexandervc/pytorch-keras-etc-3-blend-cafa-metric-etc?scriptVersionId=139811068
\nHere are: ['t5', 'esm2S1280','esm2S320', 'esm2S480','esm2S2560' ] .
\nYou may try to add esm2-size640, protbert, and esm2 with other poolings - may be you can improve the result.
\nHere you can see that according to local scoring (random folds)
\nthe bigger embeddings are added - the better score we get:
\n\"\"
\nSee https://www.kaggle.com/code/alexandervc/pytorch-keras-etc-3-blend-cafa-metric-etc?scriptVersionId=139799984&cellId=65
\nSo it is probably not just LB artifact.
\nPS
\nAgain these ideas seems to work not only for the Ridge model, but for the other also, but seems results are more advanced for Ridge.

", "rawMarkdown": "Zoltan's notebook \"CombineEmbeddings\" \nhttps://www.kaggle.com/code/szabo7zoltan/combineembeddings/notebook\ncontains not just 1 idea as may seem from the title, but actually three of them.\n\nLet me comment on that , and share some elaboration.\n\nIt gets LB 0.50510. Small improvement (again Ridge) will give 0.50881 and I guess more can be achieved in pretty simple way (see below). The baseline Ridge is LB 0.47717. It is remarkable that jump 0.03 can be achieved without heavy machinery, but just on ideas.\n\n1) The idea from the title \"Combine embeddings\" - concatenate embeddings and train model on them - score increases - demonstrated for t5 concat esm2-size1280 for Ridge model. That is a kind of standard idea of feature engineering - combine more features , but there is always a risk to overfit. Notebook demonstrates for that case it can work. Comment: one can check that the same idea works not only for Ridge but for other models, like MLP. \n\nIdeas , 2 and 3 are not that much standard: \n\n2) Downsampling to 43k specially selected samples (proteins) and training model only on them. Surprisingly it gives uplift for many models ! Comment: it seems to me the rationale behind is the following: we know that there is enormous amount of missing labels for CAFA data. I.e. the experimental check of some properties have not been done yet - thus there is no label - not because protein does not actually have that property - but only just because the experiment have not yet been done yet - that is known as \"open world\". Thus it reasonable somehow to select those proteins which have more abundant labels and train only on them ! That has been done. The actual criteria proposed by Zoltan - choose only those proteins which have labels in all three ontologies. (It might be one can improve the selection of the proteins). \n\n3) Improved way to select targets. In many notebooks we restrict ourselves to 500-1500 top frequent targets. But here it was proposed to do the selection more wisely: to choose top frequent targets **respecting the ontologies:** 1100 were selected from \"Biological processes\", 450 from \"Molecular function\", 300 \"Cellur component\". No idea how these numbers were obtained - but tried to change them - no good results (so far). \n\n\n\n--- ======================================================\n\nSmall improvement:\n\nJust combine 5 embeddings and get LB 0.50881 with the simple Ridge model:\nhttps://www.kaggle.com/code/alexandervc/pytorch-keras-etc-3-blend-cafa-metric-etc?scriptVersionId=139811068\nHere are: ['t5', 'esm2S1280','esm2S320', 'esm2S480','esm2S2560' ] . \nYou may try to add esm2-size640, protbert, and esm2 with other poolings - may be you can improve the result. \n\nHere you can see that according to local scoring (random folds)\nthe bigger embeddings are added - the better score we get: \n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F2262596%2F7764e2d84707567f46dbe46c881f6bef%2FScreenshot%202023-08-13%20210212.png?generation=1691953521372725&alt=media)\nSee https://www.kaggle.com/code/alexandervc/pytorch-keras-etc-3-blend-cafa-metric-etc?scriptVersionId=139799984&cellId=65\nSo it is probably not just LB artifact.\n\nPS\n\nAgain these ideas seems to work not only for the Ridge model, but for the other also, but seems results are more advanced for Ridge.", "votes": null }, { "id": "2395726", "postDate": "08/17/2023 18:51:15", "content": "
\n

It is remarkable that jump 0.03 can be achieved without heavy machinery, but just on ideas.

\n
\n

In that range it can, but I doubt it holds throughout.

", "rawMarkdown": "> It is remarkable that jump 0.03 can be achieved without heavy machinery, but just on ideas.\n\nIn that range it can, but I doubt it holds throughout.", "votes": null }, { "id": "2396651", "postDate": "08/18/2023 11:43:37", "content": "

With such a small test set, I think it's just noise

", "rawMarkdown": "With such a small test set, I think it's just noise", "votes": null } ], "comments": [ { "id": 2395726, "author_name": "tilii7", "author_url": "", "post_date": "08/17/2023 18:51:15", "content": "
\n

It is remarkable that jump 0.03 can be achieved without heavy machinery, but just on ideas.

\n
\n

In that range it can, but I doubt it holds throughout.

", "votes": null, "replies": [ { "id": 2396651, "author_name": "daishu", "author_url": "", "post_date": "08/18/2023 11:43:37", "content": "

With such a small test set, I think it's just noise

", "votes": null, "replies": [] } ] } ], "raw_markdown_by_id": { "2389058": "Zoltan's notebook \"CombineEmbeddings\" \nhttps://www.kaggle.com/code/szabo7zoltan/combineembeddings/notebook\ncontains not just 1 idea as may seem from the title, but actually three of them.\n\nLet me comment on that , and share some elaboration.\n\nIt gets LB 0.50510. Small improvement (again Ridge) will give 0.50881 and I guess more can be achieved in pretty simple way (see below). The baseline Ridge is LB 0.47717. It is remarkable that jump 0.03 can be achieved without heavy machinery, but just on ideas.\n\n1) The idea from the title \"Combine embeddings\" - concatenate embeddings and train model on them - score increases - demonstrated for t5 concat esm2-size1280 for Ridge model. That is a kind of standard idea of feature engineering - combine more features , but there is always a risk to overfit. Notebook demonstrates for that case it can work. Comment: one can check that the same idea works not only for Ridge but for other models, like MLP. \n\nIdeas , 2 and 3 are not that much standard: \n\n2) Downsampling to 43k specially selected samples (proteins) and training model only on them. Surprisingly it gives uplift for many models ! Comment: it seems to me the rationale behind is the following: we know that there is enormous amount of missing labels for CAFA data. I.e. the experimental check of some properties have not been done yet - thus there is no label - not because protein does not actually have that property - but only just because the experiment have not yet been done yet - that is known as \"open world\". Thus it reasonable somehow to select those proteins which have more abundant labels and train only on them ! That has been done. The actual criteria proposed by Zoltan - choose only those proteins which have labels in all three ontologies. (It might be one can improve the selection of the proteins). \n\n3) Improved way to select targets. In many notebooks we restrict ourselves to 500-1500 top frequent targets. But here it was proposed to do the selection more wisely: to choose top frequent targets **respecting the ontologies:** 1100 were selected from \"Biological processes\", 450 from \"Molecular function\", 300 \"Cellur component\". No idea how these numbers were obtained - but tried to change them - no good results (so far). \n\n\n\n--- ======================================================\n\nSmall improvement:\n\nJust combine 5 embeddings and get LB 0.50881 with the simple Ridge model:\nhttps://www.kaggle.com/code/alexandervc/pytorch-keras-etc-3-blend-cafa-metric-etc?scriptVersionId=139811068\nHere are: ['t5', 'esm2S1280','esm2S320', 'esm2S480','esm2S2560' ] . \nYou may try to add esm2-size640, protbert, and esm2 with other poolings - may be you can improve the result. \n\nHere you can see that according to local scoring (random folds)\nthe bigger embeddings are added - the better score we get: \n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F2262596%2F7764e2d84707567f46dbe46c881f6bef%2FScreenshot%202023-08-13%20210212.png?generation=1691953521372725&alt=media)\nSee https://www.kaggle.com/code/alexandervc/pytorch-keras-etc-3-blend-cafa-metric-etc?scriptVersionId=139799984&cellId=65\nSo it is probably not just LB artifact.\n\nPS\n\nAgain these ideas seems to work not only for the Ridge model, but for the other also, but seems results are more advanced for Ridge.", "2395726": "> It is remarkable that jump 0.03 can be achieved without heavy machinery, but just on ideas.\n\nIn that range it can, but I doubt it holds throughout.", "2396651": "With such a small test set, I think it's just noise" }, "source": "meta" }