{
  "id": 462605,
  "title": "9th place solution summary",
  "url": "/competitions/cafa-5-protein-function-prediction/writeups/aja-9th-place-solution-summary",
  "author_name": "",
  "post_date": "2023-12-20T19:32:39.149670200Z",
  "votes": 16,
  "comment_count": 1,
  "views": 0,
  "content": "<p>Thanks to the organizers and Kaggle for an interesting competition, along with a major thanks to the community for many helpful ideas and notebooks throughout  the submission period. Also thanks to my amazing teammates <a href=\"https://www.kaggle.com/aypyaypy\" target=\"_blank\">@aypyaypy</a> and <a href=\"https://www.kaggle.com/ahmedelfazouan\" target=\"_blank\">@ahmedelfazouan</a>.</p>\n<p>Below a short summary of our approach. The approach is an ensemble of two different ideas, that turned out to complement eachother well after merging.</p>\n<h1>Feed-forward NN</h1>\n<ol>\n<li>As input features, we went with: Prot-T5 embeddings, ESM2 embeddings, Taxo features and the (arguably more risky) QuickGO annotations.</li>\n<li>We predict the most common x_amount of annnotations as a target after propagation. In the final solutions, we ensemble models which predict different amounts of most common annotations, varying from 2000 to 3500 targets with steps of 500. This turned out to result in an important leap in terms of public score.</li>\n<li><strong>Modeling</strong>: skip layers significantly increased CV and public score. In particular, each dense layer output is forwarded as input to a last layer, which receives a concatenated set of these layers.</li>\n<li>Basic 10-fold cv scheme with seed ensembling for a slight score boost.</li>\n</ol>\n<p>Our best single model reached 0.605.</p>\n<p>Below a representation of the shift by using different targets + seed ensembling compared to single model. The mixed sub predicts pred=1 more confidently.</p>\n<p><img src=\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F11232701%2F16b9f779c022f3cf68f5d6c7f01c5cbe%2Fmixedtargets.png?generation=1703099790462286&amp;alt=media\" alt=\"\"></p>\n<h2>Propagation</h2>\n<p>The second part consists of a less ML-based  approach. </p>\n<ol>\n<li><p>Merge SprofGO and QuickGO using the method introduced by kirill’s public NB (i.e. after taking the average, select the top45 of each aspect; <a href=\"https://www.kaggle.com/code/kirilldubovik/cafa5-tuning-merge-datasets)\" target=\"_blank\">https://www.kaggle.com/code/kirilldubovik/cafa5-tuning-merge-datasets)</a>.</p></li>\n<li><p>Propagation process. If there is a gene ontology among the children whose probability is expected to be larger than its own, the probability of the parent GO is replaced by the maximum probability of the child GO. This operation is performed in such a way that it propagates from the leaf GO to the root.</p></li>\n</ol>\n<p>A visualization of the propagation process result is visualized below. As may be expected, a large amount of probabilities are set to 1. The individual public score of this approach reaches 0.597.</p>\n<p><img src=\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F11232701%2Ff0cbdd2fa1189dad1768f258b4516cc5%2Fpropagation1.png?generation=1703100755063416&amp;alt=media\" alt=\"\"></p>\n<h3>Extra</h3>\n<p>We have also tried  some other approaches, but sadly can not report on their performance due to the private score being calculated for the selected submissions only, which is understandable from a computational perspective.</p>\n<p>Some of the stuff we tried:</p>\n<ul>\n<li>Correct for the overlap in train/test set by including a nested CV with pseudo labels</li>\n<li>Replace test predictions in the rows that already exist in the training test set by using the OOF values</li>\n<li>CNNs (did not perform well at all)</li>\n<li>Any other sort of embeddings reduced model performance</li>\n</ul>",
  "messages": [
    {
      "id": "2568706",
      "postDate": "12/20/2023 19:32:39",
      "content": "<p>Thanks to the organizers and Kaggle for an interesting competition, along with a major thanks to the community for many helpful ideas and notebooks throughout  the submission period. Also thanks to my amazing teammates <a href=\"https://www.kaggle.com/aypyaypy\" target=\"_blank\">@aypyaypy</a> and <a href=\"https://www.kaggle.com/ahmedelfazouan\" target=\"_blank\">@ahmedelfazouan</a>.</p>\n<p>Below a short summary of our approach. The approach is an ensemble of two different ideas, that turned out to complement eachother well after merging.</p>\n<h1>Feed-forward NN</h1>\n<ol>\n<li>As input features, we went with: Prot-T5 embeddings, ESM2 embeddings, Taxo features and the (arguably more risky) QuickGO annotations.</li>\n<li>We predict the most common x_amount of annnotations as a target after propagation. In the final solutions, we ensemble models which predict different amounts of most common annotations, varying from 2000 to 3500 targets with steps of 500. This turned out to result in an important leap in terms of public score.</li>\n<li><strong>Modeling</strong>: skip layers significantly increased CV and public score. In particular, each dense layer output is forwarded as input to a last layer, which receives a concatenated set of these layers.</li>\n<li>Basic 10-fold cv scheme with seed ensembling for a slight score boost.</li>\n</ol>\n<p>Our best single model reached 0.605.</p>\n<p>Below a representation of the shift by using different targets + seed ensembling compared to single model. The mixed sub predicts pred=1 more confidently.</p>\n<p><img src=\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F11232701%2F16b9f779c022f3cf68f5d6c7f01c5cbe%2Fmixedtargets.png?generation=1703099790462286&amp;alt=media\" alt=\"\"></p>\n<h2>Propagation</h2>\n<p>The second part consists of a less ML-based  approach. </p>\n<ol>\n<li><p>Merge SprofGO and QuickGO using the method introduced by kirill’s public NB (i.e. after taking the average, select the top45 of each aspect; <a href=\"https://www.kaggle.com/code/kirilldubovik/cafa5-tuning-merge-datasets)\" target=\"_blank\">https://www.kaggle.com/code/kirilldubovik/cafa5-tuning-merge-datasets)</a>.</p></li>\n<li><p>Propagation process. If there is a gene ontology among the children whose probability is expected to be larger than its own, the probability of the parent GO is replaced by the maximum probability of the child GO. This operation is performed in such a way that it propagates from the leaf GO to the root.</p></li>\n</ol>\n<p>A visualization of the propagation process result is visualized below. As may be expected, a large amount of probabilities are set to 1. The individual public score of this approach reaches 0.597.</p>\n<p><img src=\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F11232701%2Ff0cbdd2fa1189dad1768f258b4516cc5%2Fpropagation1.png?generation=1703100755063416&amp;alt=media\" alt=\"\"></p>\n<h3>Extra</h3>\n<p>We have also tried  some other approaches, but sadly can not report on their performance due to the private score being calculated for the selected submissions only, which is understandable from a computational perspective.</p>\n<p>Some of the stuff we tried:</p>\n<ul>\n<li>Correct for the overlap in train/test set by including a nested CV with pseudo labels</li>\n<li>Replace test predictions in the rows that already exist in the training test set by using the OOF values</li>\n<li>CNNs (did not perform well at all)</li>\n<li>Any other sort of embeddings reduced model performance</li>\n</ul>",
      "rawMarkdown": "Thanks to the organizers and Kaggle for an interesting competition, along with a major thanks to the community for many helpful ideas and notebooks throughout  the submission period. Also thanks to my amazing teammates @aypyaypy and @ahmedelfazouan.\n\nBelow a short summary of our approach. The approach is an ensemble of two different ideas, that turned out to complement eachother well after merging.\n\n# Feed-forward NN\n\n1.  As input features, we went with: Prot-T5 embeddings, ESM2 embeddings, Taxo features and the (arguably more risky) QuickGO annotations.\n2. We predict the most common x_amount of annnotations as a target after propagation. In the final solutions, we ensemble models which predict different amounts of most common annotations, varying from 2000 to 3500 targets with steps of 500. This turned out to result in an important leap in terms of public score.\n3. **Modeling**: skip layers significantly increased CV and public score. In particular, each dense layer output is forwarded as input to a last layer, which receives a concatenated set of these layers.\n4. Basic 10-fold cv scheme with seed ensembling for a slight score boost.\n\nOur best single model reached 0.605.\n\nBelow a representation of the shift by using different targets + seed ensembling compared to single model. The mixed sub predicts pred=1 more confidently.\n\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F11232701%2F16b9f779c022f3cf68f5d6c7f01c5cbe%2Fmixedtargets.png?generation=1703099790462286&alt=media)\n\n## Propagation\n\nThe second part consists of a less ML-based  approach. \n\n1. Merge SprofGO and QuickGO using the method introduced by kirill’s public NB (i.e. after taking the average, select the top45 of each aspect; https://www.kaggle.com/code/kirilldubovik/cafa5-tuning-merge-datasets).\n\n2. Propagation process. If there is a gene ontology among the children whose probability is expected to be larger than its own, the probability of the parent GO is replaced by the maximum probability of the child GO. This operation is performed in such a way that it propagates from the leaf GO to the root.\n\nA visualization of the propagation process result is visualized below. As may be expected, a large amount of probabilities are set to 1. The individual public score of this approach reaches 0.597.\n\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F11232701%2Ff0cbdd2fa1189dad1768f258b4516cc5%2Fpropagation1.png?generation=1703100755063416&alt=media)\n\n### Extra\n\nWe have also tried  some other approaches, but sadly can not report on their performance due to the private score being calculated for the selected submissions only, which is understandable from a computational perspective.\n\nSome of the stuff we tried:\n\n- Correct for the overlap in train/test set by including a nested CV with pseudo labels\n- Replace test predictions in the rows that already exist in the training test set by using the OOF values\n- CNNs (did not perform well at all)\n- Any other sort of embeddings reduced model performance",
      "votes": null
    },
    {
      "id": "2568862",
      "postDate": "12/20/2023 23:13:34",
      "content": "<p>I was deeply impressed by the summary of your solution. Your teamwork and innovative approach were particularly remarkable. Congratulations on your achievement!</p>",
      "rawMarkdown": "I was deeply impressed by the summary of your solution. Your teamwork and innovative approach were particularly remarkable. Congratulations on your achievement!",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 2568862,
      "author_name": "yuseokjung",
      "author_url": "",
      "post_date": "12/20/2023 23:13:34",
      "content": "<p>I was deeply impressed by the summary of your solution. Your teamwork and innovative approach were particularly remarkable. Congratulations on your achievement!</p>",
      "votes": null,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "2568706": "Thanks to the organizers and Kaggle for an interesting competition, along with a major thanks to the community for many helpful ideas and notebooks throughout  the submission period. Also thanks to my amazing teammates @aypyaypy and @ahmedelfazouan.\n\nBelow a short summary of our approach. The approach is an ensemble of two different ideas, that turned out to complement eachother well after merging.\n\n# Feed-forward NN\n\n1.  As input features, we went with: Prot-T5 embeddings, ESM2 embeddings, Taxo features and the (arguably more risky) QuickGO annotations.\n2. We predict the most common x_amount of annnotations as a target after propagation. In the final solutions, we ensemble models which predict different amounts of most common annotations, varying from 2000 to 3500 targets with steps of 500. This turned out to result in an important leap in terms of public score.\n3. **Modeling**: skip layers significantly increased CV and public score. In particular, each dense layer output is forwarded as input to a last layer, which receives a concatenated set of these layers.\n4. Basic 10-fold cv scheme with seed ensembling for a slight score boost.\n\nOur best single model reached 0.605.\n\nBelow a representation of the shift by using different targets + seed ensembling compared to single model. The mixed sub predicts pred=1 more confidently.\n\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F11232701%2F16b9f779c022f3cf68f5d6c7f01c5cbe%2Fmixedtargets.png?generation=1703099790462286&alt=media)\n\n## Propagation\n\nThe second part consists of a less ML-based  approach. \n\n1. Merge SprofGO and QuickGO using the method introduced by kirill’s public NB (i.e. after taking the average, select the top45 of each aspect; https://www.kaggle.com/code/kirilldubovik/cafa5-tuning-merge-datasets).\n\n2. Propagation process. If there is a gene ontology among the children whose probability is expected to be larger than its own, the probability of the parent GO is replaced by the maximum probability of the child GO. This operation is performed in such a way that it propagates from the leaf GO to the root.\n\nA visualization of the propagation process result is visualized below. As may be expected, a large amount of probabilities are set to 1. The individual public score of this approach reaches 0.597.\n\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F11232701%2Ff0cbdd2fa1189dad1768f258b4516cc5%2Fpropagation1.png?generation=1703100755063416&alt=media)\n\n### Extra\n\nWe have also tried  some other approaches, but sadly can not report on their performance due to the private score being calculated for the selected submissions only, which is understandable from a computational perspective.\n\nSome of the stuff we tried:\n\n- Correct for the overlap in train/test set by including a nested CV with pseudo labels\n- Replace test predictions in the rows that already exist in the training test set by using the OOF values\n- CNNs (did not perform well at all)\n- Any other sort of embeddings reduced model performance",
    "2568862": "I was deeply impressed by the summary of your solution. Your teamwork and innovative approach were particularly remarkable. Congratulations on your achievement!"
  },
  "source": "meta"
}