About 4 years ago I created my Kaggle account, and that's when my Data Science/Machine Learning journey started. After 4 years of Data Science I felt that maybe it's about time for me to join and compete and give it my all.
\nTo be completely honest, it was my impostor syndrome that was stopping me all these years from competing.
\nSo what's the point of this post? After working day and night and juggling my real job with Kaggle, I was unable to finish my submission on time. The models have been semi-trained and are getting relatively good performance, however the only thing left is to create the inference pipeline. In hindsight, that is the first thing that I should have done⦠I am writing this post as I am writing my last function in the InferenceEngine the tensor_to_formatted_preds function.
There's just no more time, if I had one more day, I think I would have been able to finish on time. Oh well, no excuses, I should have planned ahead, I will do better next time.
\nOn the plus side I learnt a lot about graph neural networks from this competition. To summarize our approach:
\nOne huge limiting factor was the computation, and while I tried to do some last minute wrestling with AWS and GCP, I wasn't able to get them working as I needed to them to increase my quota, which usually takes time.
\nOh well, sorry for rambling on, but I really needed to get this off my chest. Forgive any spelling mistakes and grammatical errors, it's from the lack of sleep.
\nHuge shout out to the teams over at TorchDrug & PyG, Graphein, and Biopandas for the amazing tools and resources that they provide.
\nI will be sharing my github repo in the coming days, here is the link to the cafa5 pdb dataset.
\nBetter luck next time, and next time, I will be prepared π
", "messages": [ { "id": "2401970", "postDate": "08/21/2023 23:56:50", "content": "About 4 years ago I created my Kaggle account, and that's when my Data Science/Machine Learning journey started. After 4 years of Data Science I felt that maybe it's about time for me to join and compete and give it my all.
\nTo be completely honest, it was my impostor syndrome that was stopping me all these years from competing.
\nSo what's the point of this post? After working day and night and juggling my real job with Kaggle, I was unable to finish my submission on time. The models have been semi-trained and are getting relatively good performance, however the only thing left is to create the inference pipeline. In hindsight, that is the first thing that I should have done⦠I am writing this post as I am writing my last function in the InferenceEngine the tensor_to_formatted_preds function.
There's just no more time, if I had one more day, I think I would have been able to finish on time. Oh well, no excuses, I should have planned ahead, I will do better next time.
\nOn the plus side I learnt a lot about graph neural networks from this competition. To summarize our approach:
\nOne huge limiting factor was the computation, and while I tried to do some last minute wrestling with AWS and GCP, I wasn't able to get them working as I needed to them to increase my quota, which usually takes time.
\nOh well, sorry for rambling on, but I really needed to get this off my chest. Forgive any spelling mistakes and grammatical errors, it's from the lack of sleep.
\nHuge shout out to the teams over at TorchDrug & PyG, Graphein, and Biopandas for the amazing tools and resources that they provide.
\nI will be sharing my github repo in the coming days, here is the link to the cafa5 pdb dataset.
\nBetter luck next time, and next time, I will be prepared π
", "rawMarkdown": "About 4 years ago I created my Kaggle account, and that's when my Data Science/Machine Learning journey started. After 4 years of Data Science I felt that maybe it's about time for me to join and compete and give it my all. \n\nTo be completely honest, it was my impostor syndrome that was stopping me all these years from competing.\n\nSo what's the point of this post? After working day and night and juggling my real job with Kaggle, I was unable to finish my submission on time. The models have been semi-trained and are getting relatively good performance, however the only thing left is to create the inference pipeline. In hindsight, that is the first thing that I should have done... I am writing this post as I am writing my last function in the `InferenceEngine` the `tensor_to_formatted_preds` function. \n\nThere's just no more time, if I had one more day, I think I would have been able to finish on time. Oh well, no excuses, I should have planned ahead, I will do better next time.\n\nOn the plus side I learnt a lot about graph neural networks from this competition. To summarize our approach:\n* Fetch All the pdb files for the proteins\n* Convert all the pdb files to graphs and add node & edge features\n* Finetune pre-trained model (3 separate models MF, BP, CC) on the graphs using a custom weighted loss function that uses IA.txt\n* Get the predictions using the trained models\n* Format the predictions according to the submission rules -> This is where I am now \n\nOne huge limiting factor was the computation, and while I tried to do some last minute wrestling with AWS and GCP, I wasn't able to get them working as I needed to them to increase my quota, which usually takes time. \n\nOh well, sorry for rambling on, but I really needed to get this off my chest. Forgive any spelling mistakes and grammatical errors, it's from the lack of sleep.\n\nHuge shout out to the teams over at [TorchDrug](https://github.com/DeepGraphLearning/torchdrug) & [PyG](https://pytorch-geometric.readthedocs.io/en/latest/), [Graphein](https://github.com/a-r-j/graphein), and [Biopandas](https://github.com/BioPandas/biopandas/) for the amazing tools and resources that they provide.\n\nI will be sharing my github repo in the coming days, here is the link to the [cafa5 pdb dataset](https://www.kaggle.com/datasets/amerii/cafa-5-pdbs).\n\nBetter luck next time, and next time, I will be prepared π", "votes": null }, { "id": "2401978", "postDate": "08/22/2023 00:10:35", "content": "Link to GitHub repo in case anyone is interested.
", "rawMarkdown": "Link to [GitHub repo](https://github.com/AH-Merii/cafa5-competition) in case anyone is interested.", "votes": null }, { "id": "2401989", "postDate": "08/22/2023 00:20:15", "content": "I love the idea of using 3D structure data for predicting function but was always too intimidated to try that approach. Thank you for sharing!
", "rawMarkdown": "I love the idea of using 3D structure data for predicting function but was always too intimidated to try that approach. Thank you for sharing!", "votes": null }, { "id": "2401991", "postDate": "08/22/2023 00:22:41", "content": "It really was intimidating, but it was also super exciting working on it! :D
", "rawMarkdown": "It really was intimidating, but it was also super exciting working on it! :D", "votes": null }, { "id": "2401993", "postDate": "08/22/2023 00:23:12", "content": "I will be updating the README.md in the coming days.
I really like this idea, but your link to the GitHub repo seems invalid. Btw I think there might be a private dataset posted later.
", "rawMarkdown": "I really like this idea, but your link to the GitHub repo seems invalid. Btw I think there might be a private dataset posted later.", "votes": null }, { "id": "2402438", "postDate": "08/22/2023 06:41:30", "content": "Inspired by the previous NESP challenge, I tried a similiar voxel based approach using Thermonet, but couldn't get it to stop overfitting.
\nComputation was also a significant problem on my end.
", "rawMarkdown": "Inspired by the previous NESP challenge, I tried a similiar voxel based approach using Thermonet, but couldn't get it to stop overfitting.\n\nComputation was also a significant problem on my end.", "votes": null }, { "id": "2402636", "postDate": "08/22/2023 08:46:25", "content": "I also want to try it, but it very, very, very computational expensive and complicated.
\nAnd I think the Foldseek result contain the 3D information (partially), so instead of actually doing that, I gave up and ensemble the Foldseek result.
Hey Josephine,
\nThanks for pointing this out, I forgot to change the repo from private to public.
", "rawMarkdown": "Hey Josephine,\n\nThanks for pointing this out, I forgot to change the repo from private to public.", "votes": null }, { "id": "2402678", "postDate": "08/22/2023 09:21:16", "content": "The biggest hurdle was the computation.
\nI had to separate the task into different steps.
\nAs mentioned above I tried training on AWS and GCP but having them approving computational quotas last minute was very frustrating.
", "rawMarkdown": "The biggest hurdle was the computation.\n\nI had to separate the task into different steps.\n\n* Fetch all pdb data\n* Read PDB files as graphs and add features using a preprocessing data loader. (Had to do this before training as I was CPU bound)\n* Save preprocessing batches as pickle files (Total size of data was about 0.5 TB)\n* Read pickle files to training data loader and train models -> 3 different models (GPU Bound) could only use a batch size of one, as I would occasionally run out of Vram if I set it 2\n\nAs mentioned above I tried training on AWS and GCP but having them approving computational quotas last minute was very frustrating.", "votes": null }, { "id": "2402682", "postDate": "08/22/2023 09:27:20", "content": "I was surprised how well it was training, I didn't find it to be over fitting from my end, I was getting fmax score of 0.6 and above on MFO and CCO before even doing the postprocessing. And this was tested on using a 60% train 20% test and 20% val split.
\nI wasn't using all the training data as I was limited in terms of computation and time. It was trained on about 20k points as opposed to the possible 70k (per subontology) and was still performing really well despite the large train/val percentage.
\nWith some postprocessing ie making sure that each protein has no more than 1500 predictions, and by propagating the child GO terms to their parents I think the performance could improve significantly.
", "rawMarkdown": "I was surprised how well it was training, I didn't find it to be over fitting from my end, I was getting fmax score of 0.6 and above on MFO and CCO before even doing the postprocessing. And this was tested on using a 60% train 20% test and 20% val split. \n\nI wasn't using all the training data as I was limited in terms of computation and time. It was trained on about 20k points as opposed to the possible 70k (per subontology) and was still performing really well despite the large train/val percentage.\n\nWith some postprocessing ie making sure that each protein has no more than 1500 predictions, and by propagating the child GO terms to their parents I think the performance could improve significantly.", "votes": null }, { "id": "2402733", "postDate": "08/22/2023 09:58:25", "content": "You can try using gradient accumulation next time.
", "rawMarkdown": "You can try using gradient accumulation next time.", "votes": null }, { "id": "2402753", "postDate": "08/22/2023 10:20:28", "content": "I did do that! π
", "rawMarkdown": "I did do that! π", "votes": null }, { "id": "2403227", "postDate": "08/22/2023 15:09:37", "content": "Hey, what a brilliant idea!
\nCan you elaborate on how is the conversion from pdb file to graph done? Do you consider every atom in the aminoacids as a separate node or the aminoacids as whole? How is the 3d structure (mostly I mean protein coiling) handled? Are nodes related based on the aminoacid sequence or proximity in the 3D structure?
@tsunud I'm glad you liked it, but I can't take all the credit. Most of the theoretical aspect was attained by reading TorchDrug's documentation.
\nBut to shed some light on the feature engineering aspect:
\nYou take only Carbon Alpha atoms and you treat them as nodes in your graph, same for the residues (amino acids). Then you create edges between the nodes based on the distances between the atoms from each other. You can create different types of edges, for example:
\nFor more information about how the features were constructed check out my notebook which builds on the Tutorial from TorchDrug's documentation.
\nEdit: Nevermind, kaggle is not allowing me to make my notebook public :/
", "rawMarkdown": "tsunud I'm glad you liked it, but I can't take all the credit. Most of the theoretical aspect was attained by reading TorchDrug's documentation. \n\nBut to shed some light on the feature engineering aspect:\n\nYou take only Carbon Alpha atoms and you treat them as nodes in your graph, same for the residues (amino acids). Then you create edges between the nodes based on the distances between the atoms from each other. You can create different types of edges, for example:\n* KNN Atoms\n* Spherical Radius -> Create an edge between every atom and every other atom with a specified radius)\n* Sequential Edges -> Direct Edges bases on the order of the sequences, you can specify how many \"hops\", you want.\n\nFor more information about how the features were constructed check out my notebook which builds on the Tutorial from [TorchDrug's documentation](https://torchprotein.ai/tutorial_3).\n\nEdit: Nevermind, kaggle is not allowing me to make my notebook public :/", "votes": null }, { "id": "2403993", "postDate": "08/23/2023 02:14:26", "content": "Thanks for sharing! I expected that the preprocessing cost a lot of time, right?
", "rawMarkdown": "Thanks for sharing! I expected that the preprocessing cost a lot of time, right?", "votes": null }, { "id": "2403994", "postDate": "08/23/2023 02:15:15", "content": "haha, batch_size=1, definitely used gradient accumulation
", "rawMarkdown": "haha, batch_size=1, definitely used gradient accumulation", "votes": null }, { "id": "2404161", "postDate": "08/23/2023 05:15:55", "content": "Preprocessing took about 3.5 hours, and that's on a 16 core CPU.
", "rawMarkdown": "Preprocessing took about 3.5 hours, and that's on a 16 core CPU.", "votes": null } ], "comments": [ { "id": 2401978, "author_name": "amerii", "author_url": "", "post_date": "08/22/2023 00:10:35", "content": "Link to GitHub repo in case anyone is interested.
", "votes": null, "replies": [ { "id": 2401993, "author_name": "amerii", "author_url": "", "post_date": "08/22/2023 00:23:12", "content": "I will be updating the README.md in the coming days.
I love the idea of using 3D structure data for predicting function but was always too intimidated to try that approach. Thank you for sharing!
", "votes": null, "replies": [ { "id": 2401991, "author_name": "amerii", "author_url": "", "post_date": "08/22/2023 00:22:41", "content": "It really was intimidating, but it was also super exciting working on it! :D
", "votes": null, "replies": [] } ] }, { "id": 2402023, "author_name": "xiaozhuyv", "author_url": "", "post_date": "08/22/2023 00:51:25", "content": "I really like this idea, but your link to the GitHub repo seems invalid. Btw I think there might be a private dataset posted later.
", "votes": null, "replies": [ { "id": 2402670, "author_name": "amerii", "author_url": "", "post_date": "08/22/2023 09:13:54", "content": "Hey Josephine,
\nThanks for pointing this out, I forgot to change the repo from private to public.
", "votes": null, "replies": [] } ] }, { "id": 2402438, "author_name": "exjustice", "author_url": "", "post_date": "08/22/2023 06:41:30", "content": "Inspired by the previous NESP challenge, I tried a similiar voxel based approach using Thermonet, but couldn't get it to stop overfitting.
\nComputation was also a significant problem on my end.
", "votes": null, "replies": [ { "id": 2402682, "author_name": "amerii", "author_url": "", "post_date": "08/22/2023 09:27:20", "content": "I was surprised how well it was training, I didn't find it to be over fitting from my end, I was getting fmax score of 0.6 and above on MFO and CCO before even doing the postprocessing. And this was tested on using a 60% train 20% test and 20% val split.
\nI wasn't using all the training data as I was limited in terms of computation and time. It was trained on about 20k points as opposed to the possible 70k (per subontology) and was still performing really well despite the large train/val percentage.
\nWith some postprocessing ie making sure that each protein has no more than 1500 predictions, and by propagating the child GO terms to their parents I think the performance could improve significantly.
", "votes": null, "replies": [] } ] }, { "id": 2402636, "author_name": "awater1223", "author_url": "", "post_date": "08/22/2023 08:46:25", "content": "I also want to try it, but it very, very, very computational expensive and complicated.
\nAnd I think the Foldseek result contain the 3D information (partially), so instead of actually doing that, I gave up and ensemble the Foldseek result.
The biggest hurdle was the computation.
\nI had to separate the task into different steps.
\nAs mentioned above I tried training on AWS and GCP but having them approving computational quotas last minute was very frustrating.
", "votes": null, "replies": [ { "id": 2402733, "author_name": "alekseytrepetsky", "author_url": "", "post_date": "08/22/2023 09:58:25", "content": "You can try using gradient accumulation next time.
", "votes": null, "replies": [ { "id": 2402753, "author_name": "amerii", "author_url": "", "post_date": "08/22/2023 10:20:28", "content": "I did do that! π
", "votes": null, "replies": [ { "id": 2403993, "author_name": "awater1223", "author_url": "", "post_date": "08/23/2023 02:14:26", "content": "Thanks for sharing! I expected that the preprocessing cost a lot of time, right?
", "votes": null, "replies": [ { "id": 2404161, "author_name": "amerii", "author_url": "", "post_date": "08/23/2023 05:15:55", "content": "Preprocessing took about 3.5 hours, and that's on a 16 core CPU.
", "votes": null, "replies": [] } ] }, { "id": 2403994, "author_name": "awater1223", "author_url": "", "post_date": "08/23/2023 02:15:15", "content": "haha, batch_size=1, definitely used gradient accumulation
", "votes": null, "replies": [] } ] } ] } ] } ] }, { "id": 2403227, "author_name": "tsunud", "author_url": "", "post_date": "08/22/2023 15:09:37", "content": "Hey, what a brilliant idea!
\nCan you elaborate on how is the conversion from pdb file to graph done? Do you consider every atom in the aminoacids as a separate node or the aminoacids as whole? How is the 3d structure (mostly I mean protein coiling) handled? Are nodes related based on the aminoacid sequence or proximity in the 3D structure?
@tsunud I'm glad you liked it, but I can't take all the credit. Most of the theoretical aspect was attained by reading TorchDrug's documentation.
\nBut to shed some light on the feature engineering aspect:
\nYou take only Carbon Alpha atoms and you treat them as nodes in your graph, same for the residues (amino acids). Then you create edges between the nodes based on the distances between the atoms from each other. You can create different types of edges, for example:
\nFor more information about how the features were constructed check out my notebook which builds on the Tutorial from TorchDrug's documentation.
\nEdit: Nevermind, kaggle is not allowing me to make my notebook public :/
", "votes": null, "replies": [] } ] } ], "raw_markdown_by_id": { "2401970": "About 4 years ago I created my Kaggle account, and that's when my Data Science/Machine Learning journey started. After 4 years of Data Science I felt that maybe it's about time for me to join and compete and give it my all. \n\nTo be completely honest, it was my impostor syndrome that was stopping me all these years from competing.\n\nSo what's the point of this post? After working day and night and juggling my real job with Kaggle, I was unable to finish my submission on time. The models have been semi-trained and are getting relatively good performance, however the only thing left is to create the inference pipeline. In hindsight, that is the first thing that I should have done... I am writing this post as I am writing my last function in the `InferenceEngine` the `tensor_to_formatted_preds` function. \n\nThere's just no more time, if I had one more day, I think I would have been able to finish on time. Oh well, no excuses, I should have planned ahead, I will do better next time.\n\nOn the plus side I learnt a lot about graph neural networks from this competition. To summarize our approach:\n* Fetch All the pdb files for the proteins\n* Convert all the pdb files to graphs and add node & edge features\n* Finetune pre-trained model (3 separate models MF, BP, CC) on the graphs using a custom weighted loss function that uses IA.txt\n* Get the predictions using the trained models\n* Format the predictions according to the submission rules -> This is where I am now \n\nOne huge limiting factor was the computation, and while I tried to do some last minute wrestling with AWS and GCP, I wasn't able to get them working as I needed to them to increase my quota, which usually takes time. \n\nOh well, sorry for rambling on, but I really needed to get this off my chest. Forgive any spelling mistakes and grammatical errors, it's from the lack of sleep.\n\nHuge shout out to the teams over at [TorchDrug](https://github.com/DeepGraphLearning/torchdrug) & [PyG](https://pytorch-geometric.readthedocs.io/en/latest/), [Graphein](https://github.com/a-r-j/graphein), and [Biopandas](https://github.com/BioPandas/biopandas/) for the amazing tools and resources that they provide.\n\nI will be sharing my github repo in the coming days, here is the link to the [cafa5 pdb dataset](https://www.kaggle.com/datasets/amerii/cafa-5-pdbs).\n\nBetter luck next time, and next time, I will be prepared π", "2401978": "Link to [GitHub repo](https://github.com/AH-Merii/cafa5-competition) in case anyone is interested.", "2401989": "I love the idea of using 3D structure data for predicting function but was always too intimidated to try that approach. Thank you for sharing!", "2401991": "It really was intimidating, but it was also super exciting working on it! :D", "2401993": "I will be updating the `README.md` in the coming days.", "2402023": "I really like this idea, but your link to the GitHub repo seems invalid. Btw I think there might be a private dataset posted later.", "2402438": "Inspired by the previous NESP challenge, I tried a similiar voxel based approach using Thermonet, but couldn't get it to stop overfitting.\n\nComputation was also a significant problem on my end.", "2402636": "I also want to try it, but it very, very, very computational expensive and complicated.\nAnd I think the Foldseek result contain the 3D information (partially), so instead of actually doing that, I gave up and ensemble the Foldseek result.", "2402670": "Hey Josephine,\n\nThanks for pointing this out, I forgot to change the repo from private to public.", "2402678": "The biggest hurdle was the computation.\n\nI had to separate the task into different steps.\n\n* Fetch all pdb data\n* Read PDB files as graphs and add features using a preprocessing data loader. (Had to do this before training as I was CPU bound)\n* Save preprocessing batches as pickle files (Total size of data was about 0.5 TB)\n* Read pickle files to training data loader and train models -> 3 different models (GPU Bound) could only use a batch size of one, as I would occasionally run out of Vram if I set it 2\n\nAs mentioned above I tried training on AWS and GCP but having them approving computational quotas last minute was very frustrating.", "2402682": "I was surprised how well it was training, I didn't find it to be over fitting from my end, I was getting fmax score of 0.6 and above on MFO and CCO before even doing the postprocessing. And this was tested on using a 60% train 20% test and 20% val split. \n\nI wasn't using all the training data as I was limited in terms of computation and time. It was trained on about 20k points as opposed to the possible 70k (per subontology) and was still performing really well despite the large train/val percentage.\n\nWith some postprocessing ie making sure that each protein has no more than 1500 predictions, and by propagating the child GO terms to their parents I think the performance could improve significantly.", "2402733": "You can try using gradient accumulation next time.", "2402753": "I did do that! π", "2403227": "Hey, what a brilliant idea!\nCan you elaborate on how is the conversion from pdb file to graph done? Do you consider every atom in the aminoacids as a separate node or the aminoacids as whole? How is the 3d structure (mostly I mean protein coiling) handled? Are nodes related based on the aminoacid sequence or proximity in the 3D structure?", "2403410": "tsunud I'm glad you liked it, but I can't take all the credit. Most of the theoretical aspect was attained by reading TorchDrug's documentation. \n\nBut to shed some light on the feature engineering aspect:\n\nYou take only Carbon Alpha atoms and you treat them as nodes in your graph, same for the residues (amino acids). Then you create edges between the nodes based on the distances between the atoms from each other. You can create different types of edges, for example:\n* KNN Atoms\n* Spherical Radius -> Create an edge between every atom and every other atom with a specified radius)\n* Sequential Edges -> Direct Edges bases on the order of the sequences, you can specify how many \"hops\", you want.\n\nFor more information about how the features were constructed check out my notebook which builds on the Tutorial from [TorchDrug's documentation](https://torchprotein.ai/tutorial_3).\n\nEdit: Nevermind, kaggle is not allowing me to make my notebook public :/", "2403993": "Thanks for sharing! I expected that the preprocessing cost a lot of time, right?", "2403994": "haha, batch_size=1, definitely used gradient accumulation", "2404161": "Preprocessing took about 3.5 hours, and that's on a 16 core CPU." }, "source": "meta" }