{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"#
TABLE OF CONTENTS
\n1. [Introduction](#1)\n2. [Data Preparation](#2)\n3. [Model Training](#3)\n4. [Model Evaluation](#4)\n5. [Prediction and Submission](#5)","metadata":{}},{"cell_type":"markdown","source":"\n#
Introduction
\n\nGoal of the Competition:
\nThe goal of this competition is to predict the function of a set of proteins. You will develop a model trained on the amino-acid sequences of the proteins and on other data. Your work will help ​​researchers better understand the function of proteins, which is important for discovering how cells, tissues, and organs work. This may also aid in the development of new drugs and therapies for various diseases.

\nContext:
\nThe context revolves around the importance of proteins and their functions in various biological processes. Proteins are essential molecules composed of amino acids, and their sequence determines their structure and function. Assigning accurate functions to proteins is crucial for understanding biological processes and has implications in fields such as medicine and agriculture.\n\nGiven the abundance of genomic sequence data, determining the functions of proteins can be challenging due to the multiple functions proteins can have and their interactions with various partners. Researchers employ various methods, including database comparisons, literature mining, and machine learning algorithms, to infer protein functions. However, the field still faces challenges due to the complexity and ambiguity inherent in protein function prediction and the need for effective data integration.\n\nThe provided code demonstrates a machine learning approach to predict protein functions based on their amino acid sequences using the CAFA 5 protein function prediction dataset. The code utilizes TensorFlow to train a deep neural network model on protein sequence embeddings. It involves preprocessing the data, generating labels for multi-label classification, training the model, and making predictions on test data. Progress tracking using a progress bar is also incorporated.\n\nThe code aims to predict the Gene Ontology (GO) terms associated with protein sequences, which describe their molecular functions, biological processes, and cellular components. The goal is to enhance our understanding of proteins' functions and contribute to advancements in biomedical research and practical applications.","metadata":{}},{"cell_type":"markdown","source":"\n#
Data Preparation
\n\n### Protein Sequences\n\nEach protein is composed of dozens or hundreds of amino acids that are linked sequentially. Each amino acid in the sequence may be represented by a one-letter or three-letter code. Thus the sequence of a protein is often notated as a string of letters.\n\n### Data Files\n1. Gene Ontology: Contains ontology data in the file go-basic.obo, representing the GO graph structure as of the 2023-01-01 release. The file is in OBO format, and parsing libraries such as obonet (available for Python) can be used to work with it. The graph includes subontology roots for biological process (BPO), cellular component (CCO), and molecular function (MFO).\n1. Training Sequences: The file train_sequences.fasta provides protein sequences for the training dataset. The sequences are in FASTA format, a standard format for describing protein sequences. The file contains UniProt accession IDs and additional protein information. Proteins were extracted from the UniProt data set, mainly from the Swiss-Prot database, and some from the TrEMBL database.\n1. Labels: The file train_terms.tsv contains the list of annotated terms (ground truth) for the proteins in train_sequences.fasta. Each line includes a protein's UniProt accession ID, the corresponding GO term ID, and the ontology to which the term belongs.\n1. Taxonomy: The file train_taxonomy.tsv provides a list of proteins and their associated species represented by taxonomic identifier (taxon ID) numbers. Each line contains a protein's UniProt accession ID and the corresponding taxon ID.\n1. Information Accretion: The file IA.txt contains information accretion (weights) for each GO term. These weights are used to compute weighted precision and recall, as described in the evaluation section. The values in this file were computed using a specific code repository.\n1. Test Sequences: The file testsuperset.fasta contains protein sequences for which participants are asked to submit predictions. Each sequence's header includes the protein's UniProt accession ID and the taxon ID of the species to which the protein belongs. Only a subset of these sequences will accumulate functional annotations and form the test set. The file testsuperset-taxon-list.tsv provides a list of taxon IDs for the proteins in the test superset.","metadata":{}},{"cell_type":"markdown","source":"###
Importing Libraries
","metadata":{}},{"cell_type":"code","source":"import warnings\nwarnings.filterwarnings(\"ignore\")\n\nimport tensorflow as tf\nimport pandas as pd\nimport numpy as np\nimport seaborn as sns\nimport matplotlib.pyplot as plt\nfrom sklearn.model_selection import train_test_split\nfrom sklearn.preprocessing import StandardScaler","metadata":{"execution":{"iopub.status.busy":"2023-07-04T12:05:17.821523Z","iopub.execute_input":"2023-07-04T12:05:17.821843Z","iopub.status.idle":"2023-07-04T12:05:27.260338Z","shell.execute_reply.started":"2023-07-04T12:05:17.821819Z","shell.execute_reply":"2023-07-04T12:05:27.259295Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"There are a variety of protein embedding models. To make data preparation easier, we have used the precalculated protein embeddings created by Sergei Fironov using the Rost Lab's T5 protein language model in this notebook. The precalculated protein embeddings can be found here. We have added this dataset to the notebook along with the dataset made available for the competition.","metadata":{}},{"cell_type":"code","source":"%%time\n\n# Load the data\ntrain_terms = pd.read_csv(\"/kaggle/input/cafa-5-protein-function-prediction/Train/train_terms.tsv\", sep=\"\\t\")\ntrain_protein_ids = np.load('/kaggle/input/t5embeds/train_ids.npy')\ntrain_embeddings = np.load('/kaggle/input/t5embeds/train_embeds.npy')","metadata":{"execution":{"iopub.status.busy":"2023-07-04T12:05:27.261912Z","iopub.execute_input":"2023-07-04T12:05:27.262455Z","iopub.status.idle":"2023-07-04T12:05:39.880365Z","shell.execute_reply.started":"2023-07-04T12:05:27.262430Z","shell.execute_reply":"2023-07-04T12:05:39.879343Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"%%time\n\n# Convert embeddings numpy array (train_embeddings) into pandas dataframe\ntrain_df = pd.DataFrame(train_embeddings, columns=[\"Column_\" + str(i) for i in range(1, train_embeddings.shape[1] + 1)])","metadata":{"execution":{"iopub.status.busy":"2023-07-04T12:05:39.881639Z","iopub.execute_input":"2023-07-04T12:05:39.881943Z","iopub.status.idle":"2023-07-04T12:05:39.891139Z","shell.execute_reply.started":"2023-07-04T12:05:39.881918Z","shell.execute_reply":"2023-07-04T12:05:39.889940Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"%%time\n\n# Select labels\nnum_of_labels = 1500\nlabels = train_terms['term'].value_counts().index[:num_of_labels].tolist()\ntrain_terms_updated = train_terms.loc[train_terms['term'].isin(labels)]","metadata":{"execution":{"iopub.status.busy":"2023-07-04T12:05:39.893307Z","iopub.execute_input":"2023-07-04T12:05:39.893601Z","iopub.status.idle":"2023-07-04T12:05:40.795961Z","shell.execute_reply.started":"2023-07-04T12:05:39.893578Z","shell.execute_reply":"2023-07-04T12:05:40.795100Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"##
Model Training
","metadata":{}},{"cell_type":"code","source":"%%time\n\n# Create train labels dataframe\ntrain_size = train_protein_ids.shape[0]\ntrain_labels = np.zeros((train_size, num_of_labels))\nseries_train_protein_ids = pd.Series(train_protein_ids)","metadata":{"execution":{"iopub.status.busy":"2023-07-04T12:05:40.797661Z","iopub.execute_input":"2023-07-04T12:05:40.798012Z","iopub.status.idle":"2023-07-04T12:05:40.816327Z","shell.execute_reply.started":"2023-07-04T12:05:40.797973Z","shell.execute_reply":"2023-07-04T12:05:40.815016Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"%%time\n\n# Group the train_terms_updated dataframe by 'term'\ngrouped_terms = train_terms_updated.groupby('term')\n\n# Initialize the train_labels array with zeros\ntrain_labels = np.zeros((train_size, num_of_labels))\n\n# Loop through each label\nfor i, label in enumerate(labels):\n # Get the related proteins for the current label\n label_related_proteins = grouped_terms.get_group(label)['EntryID']\n \n # Use the .loc accessor to assign values to train_labels in one go\n train_labels[:, i] = series_train_protein_ids.isin(label_related_proteins).astype(float)\n\nlabels_df = pd.DataFrame(data=train_labels, columns=labels)","metadata":{"execution":{"iopub.status.busy":"2023-07-04T12:05:40.817580Z","iopub.execute_input":"2023-07-04T12:05:40.817908Z","iopub.status.idle":"2023-07-04T12:05:49.187773Z","shell.execute_reply.started":"2023-07-04T12:05:40.817852Z","shell.execute_reply":"2023-07-04T12:05:49.186312Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"%%time\n\n# Split the data into training and validation sets\ntrain_X, val_X, train_Y, val_Y = train_test_split(train_df, labels_df, test_size=0.2, random_state=42)\n\n# Standardize the input data\nscaler = StandardScaler()\ntrain_X = scaler.fit_transform(train_X)\nval_X = scaler.transform(val_X)\n\n# Reshape the input data for the GRU layer\ntrain_X = np.reshape(train_X, (train_X.shape[0], train_X.shape[1], 1))\nval_X = np.reshape(val_X, (val_X.shape[0], val_X.shape[1], 1))","metadata":{"execution":{"iopub.status.busy":"2023-07-04T12:05:49.189184Z","iopub.execute_input":"2023-07-04T12:05:49.189475Z","iopub.status.idle":"2023-07-04T12:05:50.494892Z","shell.execute_reply.started":"2023-07-04T12:05:49.189454Z","shell.execute_reply":"2023-07-04T12:05:50.494176Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"input_shape = (1024,) # Assuming input shape is (None, 1024)\n\n# Define the input layer\ninputs = tf.keras.layers.Input(shape=input_shape)\n\n# Apply batch normalization\nx = tf.keras.layers.BatchNormalization()(inputs)\n\n# Add a dense layer with ReLU activation\nx = tf.keras.layers.Dense(units=512, activation='relu')(x)\n\n# Add a dropout layer for regularization\nx = tf.keras.layers.Dropout(0.4)(x)\n\n# Add another dense layer with ReLU activation\nx = tf.keras.layers.Dense(units=256, activation='relu')(x)\n\n# Add a dense layer with ReLU activation\nx = tf.keras.layers.Dense(units=128, activation='relu')(x)\n\n# Add a dropout layer for regularization\nx = tf.keras.layers.Dropout(0.2)(x)\n\n# Add another dense layer with ReLU activation\nx = tf.keras.layers.Dense(units=64, activation='relu')(x)\n\n# Add a dense layer for the output\noutputs = tf.keras.layers.Dense(units=num_of_labels, activation='sigmoid')(x)\n\n# Create the model\nmodel = tf.keras.models.Model(inputs=inputs, outputs=outputs)","metadata":{"execution":{"iopub.status.busy":"2023-07-04T12:05:50.496002Z","iopub.execute_input":"2023-07-04T12:05:50.497308Z","iopub.status.idle":"2023-07-04T12:05:50.716524Z","shell.execute_reply.started":"2023-07-04T12:05:50.497256Z","shell.execute_reply":"2023-07-04T12:05:50.715310Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Define the optimizer and compile the model\noptimizer = tf.keras.optimizers.RMSprop(learning_rate=0.001, momentum=0.9)\nmodel.compile(optimizer=optimizer, loss='binary_crossentropy', metrics=['binary_accuracy', tf.keras.metrics.AUC()])\n\n# Define early stopping\nearly_stopping = tf.keras.callbacks.EarlyStopping(patience=3, restore_best_weights=True)","metadata":{"execution":{"iopub.status.busy":"2023-07-04T12:05:50.718123Z","iopub.execute_input":"2023-07-04T12:05:50.718648Z","iopub.status.idle":"2023-07-04T12:05:50.747194Z","shell.execute_reply.started":"2023-07-04T12:05:50.718598Z","shell.execute_reply":"2023-07-04T12:05:50.746157Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"%%time\n\n# Train the model\nhistory = model.fit(train_X, train_Y, batch_size=5120, epochs=5, validation_data=(val_X, val_Y), callbacks=[early_stopping])\n","metadata":{"execution":{"iopub.status.busy":"2023-07-04T12:05:50.750052Z","iopub.execute_input":"2023-07-04T12:05:50.750358Z","iopub.status.idle":"2023-07-04T12:06:41.475100Z","shell.execute_reply.started":"2023-07-04T12:05:50.750335Z","shell.execute_reply":"2023-07-04T12:06:41.473656Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Model Evaluation ","metadata":{}},{"cell_type":"code","source":"# Plot the training history\nhistory_df = pd.DataFrame(history.history)\nhistory_df.loc[:, ['loss', 'val_loss']].plot(title=\"Cross-entropy\")\nhistory_df.loc[:, ['binary_accuracy', 'val_binary_accuracy']].plot(title=\"Accuracy\")","metadata":{"execution":{"iopub.status.busy":"2023-07-04T12:06:41.476452Z","iopub.execute_input":"2023-07-04T12:06:41.476788Z","iopub.status.idle":"2023-07-04T12:06:42.060578Z","shell.execute_reply.started":"2023-07-04T12:06:41.476763Z","shell.execute_reply":"2023-07-04T12:06:42.059627Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Prediction and Submission ","metadata":{}},{"cell_type":"code","source":"# Make predictions on the test data\ntest_embeddings = np.load('/kaggle/input/t5embeds/test_embeds.npy')\n # Convert test_embeddings to dataframe\ncolumn_num = test_embeddings.shape[1]\ntest_df = pd.DataFrame(test_embeddings, columns = [\"Column_\" + str(i) for i in range(1, column_num+1)])","metadata":{"execution":{"iopub.status.busy":"2023-07-04T12:06:42.061779Z","iopub.execute_input":"2023-07-04T12:06:42.062068Z","iopub.status.idle":"2023-07-04T12:06:51.028618Z","shell.execute_reply.started":"2023-07-04T12:06:42.062045Z","shell.execute_reply":"2023-07-04T12:06:51.027942Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Create submission file\npredictions = model.predict(test_df)\nsubmission = pd.DataFrame(columns = ['Protein Id', 'GO Term Id','Prediction'])\ntest_protein_ids = np.load('/kaggle/input/t5embeds/test_ids.npy')\nl = []\nfor k in list(test_protein_ids):\n l += [ k] * predictions.shape[1] \n","metadata":{"execution":{"iopub.status.busy":"2023-07-04T12:06:51.029674Z","iopub.execute_input":"2023-07-04T12:06:51.030222Z","iopub.status.idle":"2023-07-04T12:07:04.404012Z","shell.execute_reply.started":"2023-07-04T12:06:51.030189Z","shell.execute_reply":"2023-07-04T12:07:04.402908Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"submission['Protein Id'] = l\nsubmission['GO Term Id'] = labels * predictions.shape[0]\nsubmission['Prediction'] = predictions.ravel()\n","metadata":{"execution":{"iopub.status.busy":"2023-07-04T12:07:04.405658Z","iopub.execute_input":"2023-07-04T12:07:04.406061Z","iopub.status.idle":"2023-07-04T12:07:30.797933Z","shell.execute_reply.started":"2023-07-04T12:07:04.406034Z","shell.execute_reply":"2023-07-04T12:07:30.795778Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"print(submission)","metadata":{"execution":{"iopub.status.busy":"2023-07-04T12:07:30.800424Z","iopub.execute_input":"2023-07-04T12:07:30.800778Z","iopub.status.idle":"2023-07-04T12:07:30.812493Z","shell.execute_reply.started":"2023-07-04T12:07:30.800750Z","shell.execute_reply":"2023-07-04T12:07:30.811381Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"submission.to_csv(\"submission.tsv\", header=False, index=False, sep=\"\\t\")","metadata":{"execution":{"iopub.status.busy":"2023-07-04T12:07:30.813504Z","iopub.execute_input":"2023-07-04T12:07:30.813800Z","iopub.status.idle":"2023-07-04T12:09:40.733452Z","shell.execute_reply.started":"2023-07-04T12:07:30.813778Z","shell.execute_reply":"2023-07-04T12:09:40.731802Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"This code was inspired from https://www.kaggle.com/code/gusthema/cafa-5-protein-function-with-tensorflow/notebook and in case you find this version useful, please upvote the original version.","metadata":{}}]}