I spent some time doing exploratory analysis on the competition data, here are the top conclusions. Would love to see if you have similar or different findings and any ideas how to address them.
\nThere's a lot of potential labels (43248 nodes in the ontology graph), but we can only predict max 1500 labels per protein id. The default approach is to choose 1500 most frequent notes and use them, but we could also get more sophisticated.
\nOur predictions are automatically propagated through the ontology DAG, so that might be a way of going past 1500 labels limit. We can also use the graph structure when making predictions.
\nThe default approach is using the protein language models encodings or sequence similarity, but there's a wealth of features in UniProtKB that could be used as well, the taxonomy data etc.
\nWe can see both very short (3 residues) and very long (35375 residues sequences) and they may not be well represented by the LLMs. How to handle them?
\nWhen we look at label distributions, there are some very frequent labels and then a steep fall with a long tail of rare labels. These rare labels may also be very valuable given the competition metric (information accretion).
\nNew labels may be different from historical annotations. We can also see some differences betweent train & test, e.g. it looks like there are some species that have much more representation in test superset than train.
\nHope this is helpful. Here's the EDA notebook.
", "messages": [ { "id": 2307598, "postDate": "2023-06-18T09:54:16.933Z", "content": "I spent some time doing exploratory analysis on the competition data, here are the top conclusions. Would love to see if you have similar or different findings and any ideas how to address them.
\nThere's a lot of potential labels (43248 nodes in the ontology graph), but we can only predict max 1500 labels per protein id. The default approach is to choose 1500 most frequent notes and use them, but we could also get more sophisticated.
\nOur predictions are automatically propagated through the ontology DAG, so that might be a way of going past 1500 labels limit. We can also use the graph structure when making predictions.
\nThe default approach is using the protein language models encodings or sequence similarity, but there's a wealth of features in UniProtKB that could be used as well, the taxonomy data etc.
\nWe can see both very short (3 residues) and very long (35375 residues sequences) and they may not be well represented by the LLMs. How to handle them?
\nWhen we look at label distributions, there are some very frequent labels and then a steep fall with a long tail of rare labels. These rare labels may also be very valuable given the competition metric (information accretion).
\nNew labels may be different from historical annotations. We can also see some differences betweent train & test, e.g. it looks like there are some species that have much more representation in test superset than train.
\nHope this is helpful. Here's the EDA notebook.
", "rawMarkdown": "I spent some time doing [exploratory analysis on the competition data](https://www.kaggle.com/code/thedrcat/cafa-5-protein-function-prediction-eda), here are the top conclusions. Would love to see if you have similar or different findings and any ideas how to address them. \n\n- **How to choose labels for prediction**\n\nThere's a lot of potential labels (43248 nodes in the ontology graph), but we can only predict max 1500 labels per protein id. The default approach is to choose 1500 most frequent notes and use them, but we could also get more sophisticated. \n\n- **How to handle the ontology graph**\n\nOur predictions are automatically propagated through the ontology DAG, so that might be a way of going past 1500 labels limit. We can also use the graph structure when making predictions. \n\n- **Which features we use for predictions**\n\nThe default approach is using the protein language models encodings or sequence similarity, but there's a wealth of features in UniProtKB that could be used as well, the taxonomy data etc.\n\n- **Sequence length is important**\n\nWe can see both very short (3 residues) and very long (35375 residues sequences) and they may not be well represented by the LLMs. How to handle them? \n\n- **Handling of the long tail**\n\nWhen we look at label distributions, there are some very frequent labels and then a steep fall with a long tail of rare labels. These rare labels may also be very valuable given the competition metric (information accretion). \n\n- **Distribution shift**\n\nNew labels may be different from historical annotations. We can also see some differences betweent train & test, e.g. it looks like there are some species that have much more representation in test superset than train.\n\nHope this is helpful. Here's the [EDA notebook](https://www.kaggle.com/code/thedrcat/cafa-5-protein-function-prediction-eda). ", "votes": 28 } ], "comments": [], "raw_markdown_by_id": { "2307598": "I spent some time doing [exploratory analysis on the competition data](https://www.kaggle.com/code/thedrcat/cafa-5-protein-function-prediction-eda), here are the top conclusions. Would love to see if you have similar or different findings and any ideas how to address them. \n\n- **How to choose labels for prediction**\n\nThere's a lot of potential labels (43248 nodes in the ontology graph), but we can only predict max 1500 labels per protein id. The default approach is to choose 1500 most frequent notes and use them, but we could also get more sophisticated. \n\n- **How to handle the ontology graph**\n\nOur predictions are automatically propagated through the ontology DAG, so that might be a way of going past 1500 labels limit. We can also use the graph structure when making predictions. \n\n- **Which features we use for predictions**\n\nThe default approach is using the protein language models encodings or sequence similarity, but there's a wealth of features in UniProtKB that could be used as well, the taxonomy data etc.\n\n- **Sequence length is important**\n\nWe can see both very short (3 residues) and very long (35375 residues sequences) and they may not be well represented by the LLMs. How to handle them? \n\n- **Handling of the long tail**\n\nWhen we look at label distributions, there are some very frequent labels and then a steep fall with a long tail of rare labels. These rare labels may also be very valuable given the competition metric (information accretion). \n\n- **Distribution shift**\n\nNew labels may be different from historical annotations. We can also see some differences betweent train & test, e.g. it looks like there are some species that have much more representation in test superset than train.\n\nHope this is helpful. Here's the [EDA notebook](https://www.kaggle.com/code/thedrcat/cafa-5-protein-function-prediction-eda). " } }