![](\"https://i.ibb.co/ZLzWMvh/Sliding-Window.png\")
As the technologies have imporved, we have seen a large number of protein sequence. As the number increased , it became difficult to analyse them and their function. Thus we need the help of Deep Learning Architechtures for the function
We think of this as Classification Probelms
There are various models like TALE , DeepGO , DeepGOPlus , OntoBlast , GoFDR , PSI-BLAST , MetaGO , CATH , GOLabeler
\nAfter excessive research, it was found that Gene Ontology was the most effiecient data-source for
The model was trained on the Critical Assessment of Functional Annotation CAFA 3
Along with TEMPROT architechture, another similiar architechture TEMPROT+ was also developed which was an ensemble between TEMPROT/BLASTp
There were anomilies found in the original dataset such as
\nIdentical/Duplicate ValuesThus a new corrected version of the dataset was created
\nThe model first gets the embeddings from Fine-Tuned ProtBERT-BFD which is a transformer based architechture pre-trained for Protein Sequences on BFD
ProtBERT-BFD is a BERT/Transformer architechture and thus limits the sequence to only 512 tokens. Thus Sliding Window approach of window size of 500 without Super-Position is choosen. An additional Sliding Window is also created for the point of Ending - Starting of the sequence
Further these sliced outputs are passed through the ProtBERT-BFD. It is possible to make predictions from these embeddings at this stage
![](\"https://i.ibb.co/L5cf0Qk/Prot-BERT-Embeddings.png\")
Each slide of length 500 generated a vector of 1024 Dimensions. These values were Mean-Scaled.
The Train-Test-Split is done on the basis of time-stamp. Train-Dataset is before the September 2016 and the Test-Dataset is from September 2016 - November 2017
After Preprocessing the data was sent to a Nerual Network of 1 Hidden Layer consisting of 1000 Neurons with ReLU Activation Function , Binary-Cross Entropy and Adam Optimizer
With each Epoch, the data was also augmented with the formula
\n$$Subs(p) = L x K$$
\nAfter research, it was found that, 2 is the best value for k
\nFor TEMPROT+, an ensemble between TEMPROT and BLASTp was used with the formula
\n$$S(p , f) = αy_r + (1 - α)y_b$$
\nwhere
\n![](\"https://i.ibb.co/wpXtPLc/Model-Setup.png\")
The model was trained for 100 epochs
| \n | TEMPROT | \nTEMPROT+ | \n
|---|---|---|
| Scores | \n\n | \n |
| AuPRC | \n0.618 | \n0.584 | \n
| IAuPRC | \n0.683 | \n0.643 | \n
| Difference | \n\n | \n |
| AGTO | \n0.005 | \n0.0002 | \n
| CC | \n0.027 | \n0.01 | \n
![](\"https://media.springernature.com/lw685/springer-static/image/art%3A10.1186%2Fs12859-023-05375-0/MediaObjects/12859_2023_5375_Fig4_HTML.png\")
![](\"https://www.researchgate.net/publication/371412597/figure/fig5/AS:11431281166507842@1686276895911/Comparions-of-TEMPROT-and-TEMPROT-with-the-state-of-the-art-on-frequency-analysis_Q320.jpg\")
As the technologies have imporved, we have seen a large number of protein sequence. As the number increased , it became difficult to analyse them and their function. Thus we need the help of Deep Learning Architechtures for the function
We think of this as Classification Probelms
There are various models like TALE , DeepGO , DeepGOPlus , OntoBlast , GoFDR , PSI-BLAST , MetaGO , CATH , GOLabeler
\nAfter excessive research, it was found that Gene Ontology was the most effiecient data-source for
The model was trained on the Critical Assessment of Functional Annotation CAFA 3
Along with TEMPROT architechture, another similiar architechture TEMPROT+ was also developed which was an ensemble between TEMPROT/BLASTp
There were anomilies found in the original dataset such as
\nIdentical/Duplicate ValuesThus a new corrected version of the dataset was created
\nThe model first gets the embeddings from Fine-Tuned ProtBERT-BFD which is a transformer based architechture pre-trained for Protein Sequences on BFD
ProtBERT-BFD is a BERT/Transformer architechture and thus limits the sequence to only 512 tokens. Thus Sliding Window approach of window size of 500 without Super-Position is choosen. An additional Sliding Window is also created for the point of Ending - Starting of the sequence
Further these sliced outputs are passed through the ProtBERT-BFD. It is possible to make predictions from these embeddings at this stage
Each slide of length 500 generated a vector of 1024 Dimensions. These values were Mean-Scaled.
The Train-Test-Split is done on the basis of time-stamp. Train-Dataset is before the September 2016 and the Test-Dataset is from September 2016 - November 2017
After Preprocessing the data was sent to a Nerual Network of 1 Hidden Layer consisting of 1000 Neurons with ReLU Activation Function , Binary-Cross Entropy and Adam Optimizer
With each Epoch, the data was also augmented with the formula
\n$$Subs(p) = L x K$$
\nAfter research, it was found that, 2 is the best value for k
\nFor TEMPROT+, an ensemble between TEMPROT and BLASTp was used with the formula
\n$$S(p , f) = αy_r + (1 - α)y_b$$
\nwhere
\n
The model was trained for 100 epochs
| \n | TEMPROT | \nTEMPROT+ | \n
|---|---|---|
| Scores | \n\n | \n |
| AuPRC | \n0.618 | \n0.584 | \n
| IAuPRC | \n0.683 | \n0.643 | \n
| Difference | \n\n | \n |
| AGTO | \n0.005 | \n0.0002 | \n
| CC | \n0.027 | \n0.01 | \n
\n\nFurther these sliced outputs are passed through the `ProtBERT-BFD`. **It is possible to make predictions from these embeddings at this stage**\n\n\n\n`Each slide` of length 500 generated a `vector` of `1024 Dimensions`. These values were `Mean-Scaled`.\n\n# 4 | Model Setup ⚙️\n\nThe `Train-Test-Split` is done on the basis of `time-stamp`. `Train-Dataset` is before the `September` `2016` and the `Test-Dataset` is from `September 2016` - `November 2017`\n\nAfter Preprocessing the data was sent to a `Nerual Network` of `1 Hidden Layer` consisting of `1000 Neurons` with `ReLU Activation Function` , `Binary-Cross Entropy` and `Adam Optimizer`\n\nWith each Epoch, the data was also augmented with the formula\n\n$$Subs(p) = L x K$$\n\nAfter research, it was found that, 2 is the best value for k\n\nFor TEMPROT+, an ensemble between TEMPROT and BLASTp was used with the formula\n\n$$S(p , f) = αy_r + (1 - α)y_b$$\n\nwhere\n\n* y_r = Prediction from TEMPROT\n* y_b = Prediction from BLASTp\n\n\n\n# 5 | Training 💪\n\nThe model was trained for `100 epochs`\n\n# 6 | Evaluation 💯\n\n||TEMPROT|TEMPROT+\n|---|---|---\n|**Scores**\n|AuPRC|0.618|0.584\n|IAuPRC|0.683|0.643\n|**Difference**\n|AGTO|0.005|0.0002\n|CC|0.027|0.01\n\n\n\n# 7 | Advantages 🏆\n\n* Sequences do not have length restrictions\n* Better Results\n\n# 8 | References 📄\n\n1. Oliveira GB, Pedrini H, Dias Z. Ensemble of template-free and template-based classifers for protein secondary structure prediction. Int J Mol Sci. 2021;22(21):11449.\n2. Cao Y, Shen Y. TALE: transformer-based protein function annotation with joint sequence-label embedding. Bioinformatics. 2021;37(18):2825–33.\n3. Kagaya Y, et al. ContactPFP: protein function prediction using predicted contact information. Front Bioinform.\n2022;2(896295):1–17.\n4. Ashburner M, et al. Gene ontology: tool for the unifcation of biology. Nat Genet. 2000;25(1):25–9.\n5. Kulmanov M, et al. DeepGO: predicting protein functions from sequence and interactions using a deep ontologyaware classifer. Bioinformatics. 2018;34(4):660–8.\n6. Kulmanov M, Hoehndorf R. DeepGOPlus: improved protein function prediction from sequence. Bioinformatics.\n2019;36(2):422–9.\n7. Zhu Y-H, et al. Integrating unsupervised language model with triplet neural networks for protein gene ontology\nprediction. PLoS Comput Biol. 2022;18(12):1010793.\n8. Zehetner G. OntoBlast function: from sequence similarities directly to potential functional annotations by ontology\nterms. Nucleic Acids Res. 2003;31(13):3799–803.\n9. Gong Q, et al. GoFDR: a sequence alignment based method for predicting protein functions. Methods.\n2016;93:3–14.\n10. Altschul SF, et al. Gapped BLAST and PSI-BLAST: a new generation of protein database search programs. Nucleic\nAcids Res. 1997;25(17):3389–402.\n11. Buchfnk B, et al. Sensitive protein alignments at tree-of-life scale using DIAMOND. Nat Methods. 2021;18(4):366–8.\n12. Lai B, Xu J. Accurate protein function prediction via graph attention networks with predicted structure information.\nBrief Bioinform. 2022;23(1):502.\n13. Zhang C, et al. MetaGO: predicting gene ontology of non-homologous proteins through low-resolution protein\nstructure prediction and protein–protein network mapping. J Mol Biol. 2018;430(15):2256–65.\n14. Wang S, et al. Exploiting ontology graph for predicting sparsely annotated gene function. Bioinformatics.\n2015;31(12):357–64.\n15. Rojano E, et al. Assigning protein function from domain-function associations using DomFun. BMC Bioinform.\n2022;23(1):1–19.\n16. Das S, et al. CATH FunFHMMer web server: protein functional annotations using functional family assignments.\nNucleic Acids Res. 2015;43(W1):148–53.\n17. Fodeh S, et al. Exploiting PubMed for protein molecular function prediction via NMF based multi-label classifcation.\nIn: IEEE international conference on data mining workshops (ICDMW), 2017. p. 446–51.\n18. You R, et al. GOLabeler: improving sequence-based large-scale protein function prediction by learning to rank.\nBioinformatics. 2018;34(14):2465–73.\n19. Yao S, et al. NetGO 2.0: improving large-scale protein function prediction with massive sequence, text, domain, family and network information. Nucleic Acids Res. 2021;49(W1):469–75.\n20. Elnaggar A, et al. ProtTrans: towards cracking the language of life’s code through self-supervised deep learning and\nhigh performance computing. 2021. arXiv:2007.06225.\n21. Vaswani A, et al. Attention is all you need. In: Advances in neural information processing systems (NIPS), 2017. p.\n5998–6008.\n22. Dayhof MO. Atlas of protein sequence and structure. Washington: National Biomedical Research Foundation; 1972.\n23. Zhou N, et al. The CAFA challenge reports improved protein function prediction and new functional annotations for\nhundreds of genes through experimental screens. Genome Biol. 2019;20(1):244.\n24. Devlin J, et al. BERT: pre-training of deep bidirectional transformers for language understanding. 2018. arXiv:1810.\n04805.\n25. Jumper J, et al. Highly accurate protein structure prediction with AlphaFold. Nature. 2021;596(7873):583–9.\n26. Wolf T, et al. Huggingface’s transformers: state-of-the-art natural language processing. 2019. arXiv:1910.03771.\n27. Abadi M, et al. TensorFlow: large-scale machine learning on heterogeneous systems. 2015. https://www.tensorfow.\norg.\n28. Maiya AS. ktrain: a low-code library for augmented machine learning. 2020. arXiv:2004.10703.\n29. Kingma DP, Ba J. Adam: a method for stochastic optimization. 2014. arXiv:1412.6980.\n30. Wei J, Zou K. EDA: easy data augmentation techniques for boosting performance on text classifcation tasks. 2019.\narXiv:1901.11196.\n31. Rives A, et al. Biological structure and function emerge from scaling unsupervised learning to 250 million protein\nsequences. Proc Natl Acad Sci. 2021;118(15):2016239118.\n32. Bonetta R, Valentino G. Machine learning",
"votes": 8
}
],
"comments": [],
"raw_markdown_by_id": {
"2331547": "# 1 | Pre-Requisites 🔐\n\n* Gene Ontology\n* BERT\n* AuCPRC Metric\n\n# 2 | Starting ✨\n\nAs the `technologies have imporved`, we have seen a `large number of protein sequence`. As the `number increased` , it became `difficult to analyse` them and their `function`. Thus we need the help of `Deep Learning Architechtures` for the function\n\nWe think of this as `Classification Probelms`\n\nThere are various models like [TALE](https://academic.oup.com/bioinformatics/article/37/18/2825/6182677?login=false) , [DeepGO](https://academic.oup.com/bioinformatics/article/34/4/660/4265461) , [DeepGOPlus](https://academic.oup.com/bioinformatics/article/36/2/422/5539866) , [OntoBlast](https://pubmed.ncbi.nlm.nih.gov/12824422/) , [GoFDR](https://pubmed.ncbi.nlm.nih.gov/26277418/) , [PSI-BLAST](https://pubmed.ncbi.nlm.nih.gov/9254694/) , [MetaGO](https://pubmed.ncbi.nlm.nih.gov/29534977/) , [CATH](https://pubmed.ncbi.nlm.nih.gov/25964299/) , [GOLabeler](https://pubmed.ncbi.nlm.nih.gov/29522145/)\n\nAfter excessive research, it was found that `Gene Ontology` was the most effiecient data-source for\n* Amino-Acid Sequence Pattern Analysis\n* Homology Search\n* Training a Mathematical Model\n\nThe model was trained on the `Critical Assessment of Functional Annotation` `CAFA` `3`\n\nAlong with TEMPROT architechture, another similiar architechture `TEMPROT+` was also developed which was an ensemble between `TEMPROT`/`BLASTp`\n\n# 3 | Preprocessing 🔬\n\nThere were anomilies found in the original dataset such as\n* `Identical`/`Duplicate` Values\n\nThus a new corrected version of the dataset was created\n\n\nThe model first gets the embeddings from `Fine-Tuned` [ProtBERT-BFD](https://arxiv.org/ftp/arxiv/papers/2007/2007.06225.pdf) which is a transformer based architechture pre-trained for `Protein Sequences` on [BFD](https://bfd.mmseqs.com/)\n\n`ProtBERT-BFD` is a `BERT`/`Transformer` architechture and thus limits the sequence to only `512 tokens`. Thus `Sliding Window` approach of `window size` of `500` `without Super-Position` is choosen. An `additional Sliding Window` is also created for the point of `Ending - Starting` of the sequence\n\n\n\nFurther these sliced outputs are passed through the `ProtBERT-BFD`. **It is possible to make predictions from these embeddings at this stage**\n\n\n\n`Each slide` of length 500 generated a `vector` of `1024 Dimensions`. These values were `Mean-Scaled`.\n\n# 4 | Model Setup ⚙️\n\nThe `Train-Test-Split` is done on the basis of `time-stamp`. `Train-Dataset` is before the `September` `2016` and the `Test-Dataset` is from `September 2016` - `November 2017`\n\nAfter Preprocessing the data was sent to a `Nerual Network` of `1 Hidden Layer` consisting of `1000 Neurons` with `ReLU Activation Function` , `Binary-Cross Entropy` and `Adam Optimizer`\n\nWith each Epoch, the data was also augmented with the formula\n\n$$Subs(p) = L x K$$\n\nAfter research, it was found that, 2 is the best value for k\n\nFor TEMPROT+, an ensemble between TEMPROT and BLASTp was used with the formula\n\n$$S(p , f) = αy_r + (1 - α)y_b$$\n\nwhere\n\n* y_r = Prediction from TEMPROT\n* y_b = Prediction from BLASTp\n\n\n\n# 5 | Training 💪\n\nThe model was trained for `100 epochs`\n\n# 6 | Evaluation 💯\n\n||TEMPROT|TEMPROT+\n|---|---|---\n|**Scores**\n|AuPRC|0.618|0.584\n|IAuPRC|0.683|0.643\n|**Difference**\n|AGTO|0.005|0.0002\n|CC|0.027|0.01\n\n\n\n# 7 | Advantages 🏆\n\n* Sequences do not have length restrictions\n* Better Results\n\n# 8 | References 📄\n\n1. Oliveira GB, Pedrini H, Dias Z. Ensemble of template-free and template-based classifers for protein secondary structure prediction. Int J Mol Sci. 2021;22(21):11449.\n2. Cao Y, Shen Y. TALE: transformer-based protein function annotation with joint sequence-label embedding. Bioinformatics. 2021;37(18):2825–33.\n3. Kagaya Y, et al. ContactPFP: protein function prediction using predicted contact information. Front Bioinform.\n2022;2(896295):1–17.\n4. Ashburner M, et al. Gene ontology: tool for the unifcation of biology. Nat Genet. 2000;25(1):25–9.\n5. Kulmanov M, et al. DeepGO: predicting protein functions from sequence and interactions using a deep ontologyaware classifer. Bioinformatics. 2018;34(4):660–8.\n6. Kulmanov M, Hoehndorf R. DeepGOPlus: improved protein function prediction from sequence. Bioinformatics.\n2019;36(2):422–9.\n7. Zhu Y-H, et al. Integrating unsupervised language model with triplet neural networks for protein gene ontology\nprediction. PLoS Comput Biol. 2022;18(12):1010793.\n8. Zehetner G. OntoBlast function: from sequence similarities directly to potential functional annotations by ontology\nterms. Nucleic Acids Res. 2003;31(13):3799–803.\n9. Gong Q, et al. GoFDR: a sequence alignment based method for predicting protein functions. Methods.\n2016;93:3–14.\n10. Altschul SF, et al. Gapped BLAST and PSI-BLAST: a new generation of protein database search programs. Nucleic\nAcids Res. 1997;25(17):3389–402.\n11. Buchfnk B, et al. Sensitive protein alignments at tree-of-life scale using DIAMOND. Nat Methods. 2021;18(4):366–8.\n12. Lai B, Xu J. Accurate protein function prediction via graph attention networks with predicted structure information.\nBrief Bioinform. 2022;23(1):502.\n13. Zhang C, et al. MetaGO: predicting gene ontology of non-homologous proteins through low-resolution protein\nstructure prediction and protein–protein network mapping. J Mol Biol. 2018;430(15):2256–65.\n14. Wang S, et al. Exploiting ontology graph for predicting sparsely annotated gene function. Bioinformatics.\n2015;31(12):357–64.\n15. Rojano E, et al. Assigning protein function from domain-function associations using DomFun. BMC Bioinform.\n2022;23(1):1–19.\n16. Das S, et al. CATH FunFHMMer web server: protein functional annotations using functional family assignments.\nNucleic Acids Res. 2015;43(W1):148–53.\n17. Fodeh S, et al. Exploiting PubMed for protein molecular function prediction via NMF based multi-label classifcation.\nIn: IEEE international conference on data mining workshops (ICDMW), 2017. p. 446–51.\n18. You R, et al. GOLabeler: improving sequence-based large-scale protein function prediction by learning to rank.\nBioinformatics. 2018;34(14):2465–73.\n19. Yao S, et al. NetGO 2.0: improving large-scale protein function prediction with massive sequence, text, domain, family and network information. Nucleic Acids Res. 2021;49(W1):469–75.\n20. Elnaggar A, et al. ProtTrans: towards cracking the language of life’s code through self-supervised deep learning and\nhigh performance computing. 2021. arXiv:2007.06225.\n21. Vaswani A, et al. Attention is all you need. In: Advances in neural information processing systems (NIPS), 2017. p.\n5998–6008.\n22. Dayhof MO. Atlas of protein sequence and structure. Washington: National Biomedical Research Foundation; 1972.\n23. Zhou N, et al. The CAFA challenge reports improved protein function prediction and new functional annotations for\nhundreds of genes through experimental screens. Genome Biol. 2019;20(1):244.\n24. Devlin J, et al. BERT: pre-training of deep bidirectional transformers for language understanding. 2018. arXiv:1810.\n04805.\n25. Jumper J, et al. Highly accurate protein structure prediction with AlphaFold. Nature. 2021;596(7873):583–9.\n26. Wolf T, et al. Huggingface’s transformers: state-of-the-art natural language processing. 2019. arXiv:1910.03771.\n27. Abadi M, et al. TensorFlow: large-scale machine learning on heterogeneous systems. 2015. https://www.tensorfow.\norg.\n28. Maiya AS. ktrain: a low-code library for augmented machine learning. 2020. arXiv:2004.10703.\n29. Kingma DP, Ba J. Adam: a method for stochastic optimization. 2014. arXiv:1412.6980.\n30. Wei J, Zou K. EDA: easy data augmentation techniques for boosting performance on text classifcation tasks. 2019.\narXiv:1901.11196.\n31. Rives A, et al. Biological structure and function emerge from scaling unsupervised learning to 250 million protein\nsequences. Proc Natl Acad Sci. 2021;118(15):2016239118.\n32. Bonetta R, Valentino G. Machine learning"
}
}