Hi everyone,
\nGiven my late participation, I didn't spend much time exploring if this is a promising way to go. Most of my experiments are based on training level-1 models using embeddings of pLLMs. About two weeks before the competition end, I found that learning to rank (LTR) is an interesting idea to try, then I followed the research from Shanfeng's team, which is shown as follows:
\nThe basic idea is to train level-2 rankers with the probability output from level-1 models as features. I tried to combine the predictions of MLPs, Blast-KNN, and SPROF-GO using this framework. However, all of my trials failed eventually. Maybe, I did something wrong. Therefore, I'm curious about if there's anyone successfully deploying this approach here. Thanks a lot.
", "messages": [ { "id": "2404792", "postDate": "08/23/2023 14:09:26", "content": "Hi everyone,
\nGiven my late participation, I didn't spend much time exploring if this is a promising way to go. Most of my experiments are based on training level-1 models using embeddings of pLLMs. About two weeks before the competition end, I found that learning to rank (LTR) is an interesting idea to try, then I followed the research from Shanfeng's team, which is shown as follows:
\nThe basic idea is to train level-2 rankers with the probability output from level-1 models as features. I tried to combine the predictions of MLPs, Blast-KNN, and SPROF-GO using this framework. However, all of my trials failed eventually. Maybe, I did something wrong. Therefore, I'm curious about if there's anyone successfully deploying this approach here. Thanks a lot.
", "rawMarkdown": "Hi everyone,\n\nGiven my late participation, I didn't spend much time exploring if this is a promising way to go. Most of my experiments are based on training level-1 models using embeddings of pLLMs. About two weeks before the competition end, I found that learning to rank (LTR) is an interesting idea to try, then I followed the research from Shanfeng's team, which is shown as follows:\n1. [GOLabeler: improving sequence-based large-scale protein function prediction by learning to rank](https://pubmed.ncbi.nlm.nih.gov/29522145/)\n2. [NetGO: improving large-scale protein function prediction with massive network information](https://academic.oup.com/nar/article/47/W1/W379/5491749)\n3. [NetGO 2.0: improving large-scale protein function prediction with massive sequence, text, domain, family and network information](https://www.ncbi.nlm.nih.gov/pmc/articles/PMC8262706/)\n4. [NetGO 3.0: A Protein Language Model Improves Large-scale Functional Annotations](https://www.sciencedirect.com/science/article/pii/S1672022923000669)\n\nThe basic idea is to train level-2 rankers with the probability output from level-1 models as features. I tried to combine the predictions of MLPs, Blast-KNN, and SPROF-GO using this framework. However, all of my trials failed eventually. Maybe, I did something wrong. Therefore, I'm curious about if there's anyone successfully deploying this approach here. Thanks a lot.", "votes": null }, { "id": "2405026", "postDate": "08/23/2023 16:28:18", "content": "Considered it, but the usual methods are very slow with a large number of labels, as the objective function scores every pair of labels.
", "rawMarkdown": "Considered it, but the usual methods are very slow with a large number of labels, as the objective function scores every pair of labels.", "votes": null }, { "id": "2405200", "postDate": "08/23/2023 18:25:33", "content": "Thank you for the interesting post! While not LTR, I did try rank-weighted averaging. The results are inconclusive, with some increasing and some decreasing.
", "rawMarkdown": "Thank you for the interesting post! While not LTR, I did try rank-weighted averaging. The results are inconclusive, with some increasing and some decreasing.", "votes": null }, { "id": "2405706", "postDate": "08/24/2023 04:39:35", "content": "Hi @isakatsuyoshi,
\nThanks for the comment. Could you please provide some more illustration on this method,
\n\n\nI did try rank-weighted averaging.
\n
I'm curious about the implementation details, thanks a lot!
", "rawMarkdown": "Hi @isakatsuyoshi,\n\nThanks for the comment. Could you please provide some more illustration on this method,\n\n> I did try rank-weighted averaging.\n\nI'm curious about the implementation details, thanks a lot!", "votes": null }, { "id": "2405731", "postDate": "08/24/2023 04:58:39", "content": "Hi @biodav,
\nThanks for the comment.
\nWhen I implemented, I shrunk the number of terms by taking only the top-k terms for every protein, then the total number of protein-term pairs can be somewhat reduced. As for the choice of k, I run local evaluation on level-1 models with different k to see if the performance can stay the same with less output terms. I think it's similar to candidate retrieval (?), but haven't had enough time to dig deeper into this topic.
Gladly! Weighted Rank Averaging is performed by following these steps: 1. Convert to ranks, 2. Normalize, and 3. Weighted average. Here, y_pred represents the prediction values (N, K) for each model.
\nI'll provide a sample code for reference:
\n scipy.stats rankdata\n numpy np\n\n\ny_pred_path_list = [\n ,\n ,\n \n]\n\n\nweights = [, ] \n\n\ny_pred = np.zeros_like(np.load(y_pred_path_list[]))\n\n\n y_pred_path, weight (y_pred_path_list, weights):\n (y_pred_path)\n y_pred_model = np.load(y_pred_path) \n\n \n i (y_pred_model.shape[]):\n \n normalized_rank = rankdata(y_pred_model[i]) / y_pred_model[i].size\n\n \n y_pred[i] += normalized_rank * weight\nHi @isakatsuyoshi,
\nThanks a lot for the clear explanation. If I'm not mistaken, weights is adjusted based on the confidence of the corresponding method, such as local CV scores (i.e., if CV score is high, then higher weight can be assigned). Is it correct?
Yes, deciding based on local CV scores is most common. In some cases, you might also plot the correlation matrix of the predictions to weight them in a way that maintains as much diversity as possible.
", "rawMarkdown": "Yes, deciding based on local CV scores is most common. In some cases, you might also plot the correlation matrix of the predictions to weight them in a way that maintains as much diversity as possible.", "votes": null } ], "comments": [ { "id": 2405026, "author_name": "biodav", "author_url": "", "post_date": "08/23/2023 16:28:18", "content": "Considered it, but the usual methods are very slow with a large number of labels, as the objective function scores every pair of labels.
", "votes": null, "replies": [ { "id": 2405731, "author_name": "abaojiang", "author_url": "", "post_date": "08/24/2023 04:58:39", "content": "Hi @biodav,
\nThanks for the comment.
\nWhen I implemented, I shrunk the number of terms by taking only the top-k terms for every protein, then the total number of protein-term pairs can be somewhat reduced. As for the choice of k, I run local evaluation on level-1 models with different k to see if the performance can stay the same with less output terms. I think it's similar to candidate retrieval (?), but haven't had enough time to dig deeper into this topic.
Thank you for the interesting post! While not LTR, I did try rank-weighted averaging. The results are inconclusive, with some increasing and some decreasing.
", "votes": null, "replies": [ { "id": 2405706, "author_name": "abaojiang", "author_url": "", "post_date": "08/24/2023 04:39:35", "content": "Hi @isakatsuyoshi,
\nThanks for the comment. Could you please provide some more illustration on this method,
\n\n\nI did try rank-weighted averaging.
\n
I'm curious about the implementation details, thanks a lot!
", "votes": null, "replies": [ { "id": 2405782, "author_name": "isakatsuyoshi", "author_url": "", "post_date": "08/24/2023 05:37:59", "content": "Gladly! Weighted Rank Averaging is performed by following these steps: 1. Convert to ranks, 2. Normalize, and 3. Weighted average. Here, y_pred represents the prediction values (N, K) for each model.
\nI'll provide a sample code for reference:
\n scipy.stats rankdata\n numpy np\n\n\ny_pred_path_list = [\n ,\n ,\n \n]\n\n\nweights = [, ] \n\n\ny_pred = np.zeros_like(np.load(y_pred_path_list[]))\n\n\n y_pred_path, weight (y_pred_path_list, weights):\n (y_pred_path)\n y_pred_model = np.load(y_pred_path) \n\n \n i (y_pred_model.shape[]):\n \n normalized_rank = rankdata(y_pred_model[i]) / y_pred_model[i].size\n\n \n y_pred[i] += normalized_rank * weight\nHi @isakatsuyoshi,
\nThanks a lot for the clear explanation. If I'm not mistaken, weights is adjusted based on the confidence of the corresponding method, such as local CV scores (i.e., if CV score is high, then higher weight can be assigned). Is it correct?
Yes, deciding based on local CV scores is most common. In some cases, you might also plot the correlation matrix of the predictions to weight them in a way that maintains as much diversity as possible.
", "votes": null, "replies": [] } ] } ] } ] } ] } ], "raw_markdown_by_id": { "2404792": "Hi everyone,\n\nGiven my late participation, I didn't spend much time exploring if this is a promising way to go. Most of my experiments are based on training level-1 models using embeddings of pLLMs. About two weeks before the competition end, I found that learning to rank (LTR) is an interesting idea to try, then I followed the research from Shanfeng's team, which is shown as follows:\n1. [GOLabeler: improving sequence-based large-scale protein function prediction by learning to rank](https://pubmed.ncbi.nlm.nih.gov/29522145/)\n2. [NetGO: improving large-scale protein function prediction with massive network information](https://academic.oup.com/nar/article/47/W1/W379/5491749)\n3. [NetGO 2.0: improving large-scale protein function prediction with massive sequence, text, domain, family and network information](https://www.ncbi.nlm.nih.gov/pmc/articles/PMC8262706/)\n4. [NetGO 3.0: A Protein Language Model Improves Large-scale Functional Annotations](https://www.sciencedirect.com/science/article/pii/S1672022923000669)\n\nThe basic idea is to train level-2 rankers with the probability output from level-1 models as features. I tried to combine the predictions of MLPs, Blast-KNN, and SPROF-GO using this framework. However, all of my trials failed eventually. Maybe, I did something wrong. Therefore, I'm curious about if there's anyone successfully deploying this approach here. Thanks a lot.", "2405026": "Considered it, but the usual methods are very slow with a large number of labels, as the objective function scores every pair of labels.", "2405200": "Thank you for the interesting post! While not LTR, I did try rank-weighted averaging. The results are inconclusive, with some increasing and some decreasing.", "2405706": "Hi @isakatsuyoshi,\n\nThanks for the comment. Could you please provide some more illustration on this method,\n\n> I did try rank-weighted averaging.\n\nI'm curious about the implementation details, thanks a lot!", "2405731": "Hi @biodav,\n\nThanks for the comment.\nWhen I implemented, I shrunk the number of terms by taking only the top-`k` terms for every protein, then the total number of protein-term pairs can be somewhat reduced. As for the choice of `k`, I run local evaluation on level-1 models with different `k` to see if the performance can stay the same with less output terms. I think it's similar to candidate retrieval (?), but haven't had enough time to dig deeper into this topic.", "2405782": "Gladly! Weighted Rank Averaging is performed by following these steps: 1. Convert to ranks, 2. Normalize, and 3. Weighted average. Here, y_pred represents the prediction values (N, K) for each model.\n\nI'll provide a sample code for reference:\n```\nfrom scipy.stats import rankdata\nimport numpy as np\n\n# Example paths to the predictions made by different models\ny_pred_path_list = [\n 'model1_predictions.npy',\n 'model2_predictions.npy',\n # Add paths to other model predictions as needed\n]\n\n# Example weights assigned to each model, reflecting their confidence or importance\nweights = [0.4, 0.6] # These should sum to 1 if you want a weighted average\n\n# Initialize the final prediction with zeros, using the same shape as one of the models' predictions\ny_pred = np.zeros_like(np.load(y_pred_path_list[0]))\n\n# Iterate through each model's prediction and weight\nfor y_pred_path, weight in zip(y_pred_path_list, weights):\n print(y_pred_path)\n y_pred_model = np.load(y_pred_path) # Load the individual model's prediction\n \n # Perform rank transformation row-wise to prevent overflow\n for i in range(y_pred_model.shape[0]):\n # The rank is an integer, so we normalize it by dividing by the size of the predictions (K)\n normalized_rank = rankdata(y_pred_model[i]) / y_pred_model[i].size\n \n # Apply the weight to the normalized rank and accumulate it to the final prediction\n y_pred[i] += normalized_rank * weight\n\n```", "2406480": "Hi @isakatsuyoshi,\n\nThanks a lot for the clear explanation. If I'm not mistaken, `weights` is adjusted based on the confidence of the corresponding method, such as local CV scores (*i.e.,* if CV score is high, then higher weight can be assigned). Is it correct?", "2406484": "Yes, deciding based on local CV scores is most common. In some cases, you might also plot the correlation matrix of the predictions to weight them in a way that maintains as much diversity as possible." }, "source": "meta" }