{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"code","source":"import numpy as np\nimport pandas as pd\nimport os\nimport random\nfrom sklearn.utils import shuffle\n\n!pip install spektral","metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### 1. Basic Graph Theory\nLet’s start from the beginning — basic graph theory. \n\nNowadays, a lot of information are represented in graphs. \n\nFor example Google’s Knowledge Graph that helps with the Search Engine Optimization (SEO), chemical molecular structure, document citation networks (document A has cited document B), and social media networks (who is connected to who?). \n\nA graph consists of 2 main elements, nodes (vertices or points) and edges (links or lines) where the nodes are connected by edges.\n\nNow comes the next question, which part of the data are nodes, and which one are edges? \n\nThere is no strict answer to this as we should define nodes and edges ourselves. \n\nFor example, in a chemical molecule that consists multiple atoms, the atoms can be defined as nodes and the bond between atoms can be defined as edges.\n\nIllustrations of checmial molecules\n<img src = \"https://miro.medium.com/max/500/1*_IhgiTj5GC92HVUfQDy-6g.png\">","metadata":{}},{"cell_type":"markdown","source":"Another example is document citation networks from the [Cora dataset](https://relational.fit.cvut.cz/dataset/CORA). \n\nThe nodes represent individual document and each edge represents whether that document is cited by the other.\n\nHow the edges link the nodes allows us to distinguish between a directed vs an undirected graph. \n\nSimply put, in a directed graph, direction matters, and edges cannot be used in the other direction. \nUndirected graphs behave in the opposite manner, the edges follow no direction and can be used interchangeably.\n\n<img src = \"https://miro.medium.com/max/875/1*zXvtHcbKASzD0vPSh3EoCA.png\">\n\n<img src = \"https://miro.medium.com/max/293/1*v8jNKaprhHJj8RNjDYYPNw.png\">","metadata":{}},{"cell_type":"markdown","source":"Besides that, there is also a special type of graph where each node is connected to all other nodes, this is called a complete graph.\n\n<img src = \"https://miro.medium.com/max/321/1*JA9xZHR9PEk2wFdLbYpqkw.png\">","metadata":{}},{"cell_type":"markdown","source":"### 2. Translating Graph into Features for Neural Networks","metadata":{}},{"cell_type":"markdown","source":"#### 2.1. Adjacency matrix (A)\nAn adjacency matrix is a N x N matrix filled with either 0 or 1, where N is the total number of nodes. \n\nAdjacency matrices are able to represent the existence of edges the connect the node pairs through the value in the matrices. \n\nFor example, if we have 5 nodes in our graph, then the shape of the matrix is [5, 5]. \n\nMatrix element Aᵢⱼ is 1 if an edge exists between node i and j. \n\nFrom the adjacency matrix below, we can see that the connection between node 2 and 3 (A₂₃) is colored yellow to represent 1 as they’re connected, while A₂₁ is dark purple as node 2 and 1 are not connected to each other.\n\n<img src = \"https://miro.medium.com/max/875/1*lvWOW6EyxXi3nSn7CqdQQw.png\">","metadata":{}},{"cell_type":"markdown","source":"#### 2.2 Node attributes matrix (X)\nUnlike adjacency matrices that models the relationship between nodes, this matrix represents the features or attributes of each node. \n\nIf there are N nodes and the size of node attributes is F, then the shape of this matrix is N x F.\n\nIn the example of the CORA dataset, we will have a corpus that contains words from all the documents. \n\nThe node attributes would be bag-of-words that indicate the presence of a word in the document, while each document is represented by a node. \n\nIn this case, F will represent the size of the corpus (the total number of unique words) while N is the total number of documents available.\n\n<img src = \"https://miro.medium.com/max/595/1*9obE8LS2gPSFjbxR8yml4Q.png\">","metadata":{}},{"cell_type":"markdown","source":"#### 2.3. Edge attributes matrix (E)\nSometimes, edges can have its own attributes too, just like nodes. \n\nIf the size of edge attributes is S and the number of edges available is n_edges, the shape of this matrix is n_edges x S.","metadata":{}},{"cell_type":"markdown","source":"### 3. Graph Neural Networks Data Modes\nIf we look back at the examples of the chemical molecular structures and document citation networks mentioned earlier, we will realize that both of them have different graph representation settings.\n\nFor example, if we want to classify chemical molecules, we will consider each molecule as 1 different graph; \nso in this setting, the number of the graphs we have will be as many as the number of the molecules. We call this Batch Mode. \n\nOn the other hand, if we want to classify documents within a document citation network, we will only have 1 big graph consisting of all the documents as the nodes. In this setting, we will call it Single Mode.","metadata":{}},{"cell_type":"markdown","source":"### 4. Graph Convolutional Networks\n\nIn GCN, we will take into account the Adjacency Matrix (A) in the forward propagation equation in addition to the node features (or so-called input features). \n\n`A` is a matrix that represents the edges or connection between the nodes in the forward propagation equation. \n\nThe insertion of A in the forward pass equation enables the model to learn the feature representations based on nodes connectivity. \n\nFor the sake of simplicity, the bias b is omitted. The resulting GCN can be seen as the first-order approximation of Spectral Graph Convolution in the form of a message passing network where the information is propagated along the neighboring nodes within the graph.\n\nBy adding the adjacency matrix as an additional element, the forward pass equation will then be:\n<img src = \"https://miro.medium.com/max/600/1*2cT063K_PIvJVRqFn8c5gg.png\">\n\n`A*` is the normalized version of A. To get better understanding on why we need to normalize A and what happens during forward pass in GCNs, let’s do an experiment.","metadata":{}},{"cell_type":"markdown","source":"### 5. Building Graph Convolutional Networks\n\n#### 5.1. Initializing the Graph G\nLet’s start by building a simple undirected graph (G) using NetworkX. \n\nThe graph G will consist of 6 nodes and the feature of each node will correspond to that particular node number. \n\nFor example, node 1 will have a node feature of 1, node 2 will have a node feature of 2, and so on. To simplify, we are not going to assign edge features in this experiment.","metadata":{}},{"cell_type":"code","source":"import networkx as nx\nimport numpy as np\nimport matplotlib.pyplot as plt\nfrom scipy.linalg import fractional_matrix_power\n\nimport warnings\nwarnings.filterwarnings(\"ignore\", category=UserWarning)\n\n\n#Initialize the graph\nG = nx.Graph(name='G')\n\n#Create nodes\n#In this example, the graph will consist of 6 nodes.\n#Each node is assigned node feature which corresponds to the node name\nfor i in range(6):\n    G.add_node(i, name=i)\n\n\n#Define the edges and the edges to the graph\nedges = [(0,1),(0,2),(1,2),(0,3),(3,4),(3,5),(4,5)]\nG.add_edges_from(edges)\n\n#See graph info\nprint('Graph Info:\\n', nx.info(G))\n\n#Inspect the node features\nprint('\\nGraph Nodes: ', G.nodes.data())\n\n#Plot the graph\nnx.draw(G, with_labels=True, font_weight='bold')\nplt.show()","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Since we only have 1 graph, this data configuration is an example of a Single Mode representation. \n\nWe will build a GCN that will learn the nodes features representation.","metadata":{}},{"cell_type":"markdown","source":"#### 5.2 Inserting Adjacency Matrix (A) to Forward Pass Equation\n##### 5.2.1 The next step is to obtain the Adjacency Matrix (A) and Node Features Matrix (X) from graph G.","metadata":{}},{"cell_type":"code","source":"#Get the Adjacency Matrix (A) and Node Features Matrix (X) as numpy array\nA = np.array(nx.attr_matrix(G, node_attr='name')[0])\nX = np.array(nx.attr_matrix(G, node_attr='name')[1])\nX = np.expand_dims(X,axis=1)\n\nprint('Shape of A: ', A.shape)\nprint('\\nShape of X: ', X.shape)\nprint('\\nAdjacency Matrix (A):\\n', A)\nprint('\\nNode Features Matrix (X):\\n', X)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Now, let’s investigate how by inserting A into the forward pass equation adds to richer feature representation of the model. We are going to perform dot product of A and X. Let’s call the result of this dot product operation as AX in this notebook.","metadata":{}},{"cell_type":"code","source":"#Dot product Adjacency Matrix (A) and Node Features (X)\nAX = np.dot(A,X)\nprint(\"Dot product of A and X (AX):\\n\", AX)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"From the results, it is apparent that AX represents the sum of neighboring nodes features. \n\nFor example, the first row of AX corresponds to the sum of nodes features connected to node 0, which is node 1, 2, and 3. \n\nThis gives us an idea how the propagation mechanism is happening in GCNs and how the node connectivity impacts the hidden features representation seen by GCNs.\n\n`There is one problem here, while AX sums up the adjacent node features, it does not take into account the features of the node itself.`","metadata":{}},{"cell_type":"markdown","source":"##### 5.2.2 Inserting Self-Loops and Normalizing A\nTo address this problem, we now add self-loops to each node of A. \n\nAdding self-loops is basically a mechanism to connect a node to itself. \n\nThat being said, all the diagonal elements of Adjacency Matrix A will now become 1 because each node is connected to itself. \n\nLet’s call A with self-loops added as A_hat and recalculate AX, which is now the dot product of A_hat and X.","metadata":{}},{"cell_type":"code","source":"#Add Self Loops\nG_self_loops = G.copy()\n\nself_loops = []\nfor i in range(G.number_of_nodes()):\n    self_loops.append((i,i))\n\nG_self_loops.add_edges_from(self_loops)\n\n#Check the edges of G_self_loops after adding the self loops\nprint('Edges of G with self-loops:\\n', G_self_loops.edges)\n\n#Get the Adjacency Matrix (A) and Node Features Matrix (X) of added self-lopps graph\nA_hat = np.array(nx.attr_matrix(G_self_loops, node_attr='name')[0])\nprint('Adjacency Matrix of added self-loops G (A_hat):\\n', A_hat)\n\n#Calculate the dot product of A_hat and X (AX)\nAX = np.dot(A_hat, X)\nprint('AX:\\n', AX)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Now we can see AX has now considered features of the nodes itself.\n\nBut theres is still another problem. \n\n`The elements of AX are not normalized.` \n\nSimilar to data pre-processing for any Neural Networks operation, we need to normalize the features to prevent numerical instabilities and vanishing/exploding gradients in order for the model to converge. \n\nIn GCNs, we normalize our data by calculating the Degree Matrix (D) and performing dot product operation of the inverse of D with AX.\n\nIn graph terminology, the term “degree” refers to the number of edges a node is connected to.\n\n`normalized features = (Inverse)D A X` \n\nWe will call normalized features`DAX` in this notebook. ","metadata":{}},{"cell_type":"code","source":"#Get the Degree Matrix of the added self-loops graph\nDeg_Mat = G_self_loops.degree()\nprint('Degree Matrix of added self-loops G (D): ', Deg_Mat)\n\n#Convert the Degree Matrix to a N x N matrix where N is the number of nodes\nD = np.diag([deg for (n,deg) in list(Deg_Mat)])\nprint('Degree Matrix of added self-loops G as numpy array (D):\\n', D)\n\n#Find the inverse of Degree Matrix (D)\nD_inv = np.linalg.inv(D)\nprint('Inverse of D:\\n', D_inv)\n\n#Dot product of D and AX for normalization\nDAX = np.dot(D_inv,AX)\nprint('DAX:\\n', DAX)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"If we compare DAX with AX, we will notice that:\n<img src = \"https://miro.medium.com/max/353/1*ltJAgs2TYVIW3j16H3oWzg.png\">\n\nWe can see the impact normalization has on DAX, where the element that corresponds to node 3 has lower values compared to node 4 and 5. `But why would node 3 have different values after normalization if it has the same initial value as node 4 and 5?`\n\nLet’s take a look back at our graph. \n\nNode 3 has 3 incident edges, while nodes 4 and 5 only have 2 incident edges. \n\nThe fact that node 3 has a higher degree than node 4 and 5 leads to a lower weighting of node 3’s features in DAX. \n\n`In other words, the lower the degree of a node, the stronger that a node belongs to a certain group or cluster.`\n\nIn the original GCN paper, authors `Kipf and Welling` states that doing symmetric normalization will make dynamics more interesting, hence, the normalization equation is modified from:\n\n<img src = \"https://miro.medium.com/max/530/1*yd8uL8Ewj_C4ES5faZVUxg.png\">\n\nLet’s calculate the normalized values using the new symmetric normalization equation!","metadata":{}},{"cell_type":"code","source":"#Symmetrically-normalization\nD_half_norm = fractional_matrix_power(D, -0.5)\nDADX = D_half_norm.dot(A_hat).dot(D_half_norm).dot(X)\nprint('DADX:\\n', DADX)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Looking back at \"Forward pass equation\" that we saw earlier, we will realize that we now have the answers to what is `A*`! In the paper, `A*` is referred to as renormalization trick","metadata":{}},{"cell_type":"markdown","source":"### 6.  Implementation of Graph Convolution Neural Networks (GCNN) using Spektral API\n\nSpektral API, is a Python library for graph deep learning based on Tensorflow 2. \n\nWe are going to perform Semi-Supervised Node Classification using CORA dataset, similar to the work presented in the original GCN paper by Thomas Kipf and Max Welling (2017).\n\n#### 6.1 Dataset Overview\n\n`CORA citation network dataset consists of 2708 nodes, where each node represents a document or a technical paper.` \n\nThe node features are bag-of-words representation that indicates the presence of a word in the document. \n\n`The vocabulary — hence, also the node features — contains 1433 words.`\n\nWe will treat the dataset as an undirected graph where the edge represents whether one document cites the other or vice versa. \n\n`There is no edge feature in this dataset.` \n\nThe goal of this task is to classify the nodes (or the documents) into 7 different classes which correspond to the papers’ research areas. \n\nThis is a single-label multi-class classification problem with `Single Mode` data representation setting.\n\nThis implementation is also an example of `Transductive Learning`, where the neural network sees all data, including the test dataset, during the training. \n\nThis is contrast to `Inductive Learning` — which is the typical Supervised Learning — where the test data is kept separate during the training.","metadata":{}},{"cell_type":"markdown","source":"#### 6.2 Text Classification Problem\n\nSince we are going to classify documents based on their textual features, a common machine learning way to look at this problem is by seeing it as a supervised text classification problem. \n\nUsing this approach, the machine learning model will learn each document’s hidden representation only based on its own features.\n\nBelow image is an illustration of the text classification apporach on a document classification problem:\n<img src = \"https://miro.medium.com/max/700/1*JC9WB-6BbH9hxsU4PlYgcg.png\">\n\nThis approach might work well if there are enough labeled examples for each class. \n\nUnfortunately, in real world cases, labeling data might be expensive.\n\nLet's look at another approach:\n\nBesides its own text content, normally, a technical paper also cites other related papers. \n\nIntuitively, the cited papers are likely to belong to similar research area.\n\nIn this citation network dataset, we want to leverage the citation information from each paper in addition to its own textual content. Hence, the dataset has now turned into a network of papers.\n<img src = \"https://miro.medium.com/max/700/1*gT9NBC2Ybl7w7V_RGEAILQ.png\">\n\n\nUsing this configuration, we can utilize Graph Neural Networks, such as Graph Convolutional Networks (GCNs), to build a model that learns the documents interconnection in addition to their own textual features. \n\nThe GCN model will learn the nodes (or documents) hidden representation not only based on its own features, but also its neighboring nodes’ features. \n\nHence, we can reduce the number of necessary labeled examples and implement semi-supervised learning utilizing the Adjacency Matrix (A) or the nodes connectivity within a graph.\n\nAnother case where Graph Neural Networks might be useful is when each example does not have distinct features on its own, but the relations between the examples can enrich the feature representations.","metadata":{}},{"cell_type":"markdown","source":"#### 6.3 Implementation of Graph Convolutional Networks\n\n##### 6.3.1 Loading and Parsing the Dataset\n\nIn this experiment, we are going to build and train a GCN model using Spektral API that is built on Tensorflow 2. \n\nAlthough Spektral provides built-in functions to load and preprocess CORA dataset, but we are going to use the raw dataset in order to gain deeper understanding on the data preprocessing and configuration. \n\nWe will use `cora.content` and `cora.cites` files and we will randomly shuffle the data.\n\nIn `cora.content` file, each line consists of several elements:\n1. first element indicates the document (or node) ID,\n2. 2nd until the last second elements indicate the node features,\n3. last element indicates the label of that particular node.\n\nIn `cora.cites` file, each line contains a tuple of documents (or nodes) IDs. \n1. The first element of the tuple indicates the ID of the paper being cited\n2. while the second element indicates the paper containing the citation. \n\nAlthough this configuration represents a directed graph, in this approach we treat the dataset as an undirected graph.","metadata":{}},{"cell_type":"code","source":"#loading the data\n\nall_data = []\nall_edges = []\n\nfor root,dirs,files in os.walk('../input/coradataset'):\n    #print(root)\n    for file in files:\n        #print(file)\n        if '.content' in file:\n            with open(os.path.join(root,file),'r') as f:                \n                all_data.extend(f.read().splitlines())\n        elif 'cites' in file:\n            with open(os.path.join(root,file),'r') as f:\n                all_edges.extend(f.read().splitlines())\n\n                \n#Shuffle the data because the raw data is ordered based on the label\nrandom_state = 77\nall_data = shuffle(all_data,random_state=random_state)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"After loading the data, we build `Node Features Matrix (X)` and `a list containing tuples of adjacent nodes`. \n\nThis edges list will be used to build a graph from where we can obtain the Adjacency Matrix (A).","metadata":{}},{"cell_type":"code","source":"#parse the data\nlabels = []\nnodes = []\nX = []\n\nfor i,data in enumerate(all_data):\n    elements = data.split('\\t')\n    labels.append(elements[-1])\n    X.append(elements[1:-1])\n    nodes.append(elements[0])\n\nX = np.array(X,dtype=int)\nN = X.shape[0] #the number of nodes\nF = X.shape[1] #the size of node features\nprint('X shape: ', X.shape)\n\n\n#parse the edge\nedge_list=[]\nfor edge in all_edges:\n    e = edge.split('\\t')\n    edge_list.append((e[0],e[1]))\n\nprint('\\nNumber of nodes (N): ', N)\nprint('\\nNumber of features (F) of each node: ', F)\nprint('\\nCategories: ', set(labels))\n\nnum_classes = len(set(labels))\nprint('\\nNumber of classes: ', num_classes)\n","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"##### 6.3.2 Setting the Train, Validation, and Test Mask\nWe will feed in the Node Features Matrix (X) and Adjacency Matrix (A) to the neural networks. \n\nWe are also going to set Boolean masks with a length of N for each training, validation, and testing dataset.\n\nThe elements of those masks are True when they belong to corresponding training, validation, or test dataset. \n\nFor example, the elements of train mask are True for those which belong to training data.\n\n<img src=\"https://miro.medium.com/max/875/1*nuanGfgWcumBM-jb6cOtiA.png\">","metadata":{}},{"cell_type":"markdown","source":"In the original paper, they picked 20 labeled examples for each class. \n\nHence, with 7 classes, we will have a total of 140 labeled training examples. \n\nWe will also use 500 labeled validation examples and 1000 labeled testing examples.","metadata":{}},{"cell_type":"code","source":"def limit_data(labels,limit=20,val_num=500,test_num=1000):\n    '''\n    Get the index of train, validation, and test data\n    '''\n    label_counter = dict((l, 0) for l in labels)\n    train_idx = []\n\n    for i in range(len(labels)):\n        label = labels[i]\n        if label_counter[label]<limit:\n            #add the example to the training data\n            train_idx.append(i)\n            label_counter[label]+=1\n        \n        #exit the loop once we found 20 examples for each class\n        if all(count == limit for count in label_counter.values()):\n            break\n    \n    #get the indices that do not go to traning data\n    rest_idx = [x for x in range(len(labels)) if x not in train_idx]\n    val_idx = rest_idx[:val_num]\n    test_idx = rest_idx[val_num:(val_num+test_num)]\n    return train_idx, val_idx,test_idx\n\ntrain_idx,val_idx,test_idx = limit_data(labels)\n\n#set the mask\ntrain_mask = np.zeros((N,),dtype=bool)\ntrain_mask[train_idx] = True\n\nval_mask = np.zeros((N,),dtype=bool)\nval_mask[val_idx] = True\n\ntest_mask = np.zeros((N,),dtype=bool)\ntest_mask[test_idx] = True","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"##### 6.3.3 Obtaining the Adjacency Matrix\nThe next step is to obtain the Adjacency Matrix (A) of the graph. \n\nWe use NetworkX to help us do this. \n\nWe will initialize a graph and then add the nodes and edges lists to the graph.","metadata":{}},{"cell_type":"code","source":"#build the graph\nG = nx.Graph()\nG.add_nodes_from(nodes)\nG.add_edges_from(edge_list)\n\n#obtain the adjacency matrix (A)\nA = nx.adjacency_matrix(G)\nprint('Graph info: ', nx.info(G))","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"##### 6.3.4 Converting the label to one-hot encoding\nThe last step before building our GCN is, just like any other machine learning model, encoding the labels and then converting them to one-hot encoding.","metadata":{}},{"cell_type":"code","source":"from sklearn import preprocessing\nfrom keras.utils import to_categorical\ndef encode_label(labels):\n    label_encoder = preprocessing.LabelEncoder()\n    labels = label_encoder.fit_transform(labels)\n    labels = to_categorical(labels)\n    return labels, label_encoder.classes_\n\nlabels_encoded, classes = encode_label(labels)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"We are now done with data preprocessing and ready to build our GCN!","metadata":{}},{"cell_type":"markdown","source":"##### 6.3.5 Build the Graph Convolutional Networks\nThe GCN model architectures and hyperparameters follow the design from GCN original paper. \n\nThe GCN model will take 2 inputs:\n1. the Node Features Matrix (X) \n2. Adjacency Matrix (A). \n\nWe are going to implement 2-layer GCN with Dropout layers and L2 regularization. \n\nWe are also going to set the maximum training epochs to be 200 and implement Early Stopping with patience of 10. \n\nIt means that the training will be stopped once the validation loss does not decrease for 10 consecutive epochs.\n\nTo monitor the training and validation accuracy and loss, we are also going to call TensorBoard in the callbacks.\n\nBefore feeding in the Adjacency Matrix (A) to the GCN, we need to do extra preprocessing by performing renormalization trick according to the original paper. \n\nThe code to train GCN below was originally obtained from [Spektral GitHub page](https://github.com/danielegrattarola/spektral/blob/master/examples/node_prediction/citation_gcn.py).","metadata":{}},{"cell_type":"code","source":"from spektral.layers import GCNConv\nfrom keras.layers import Input,Dropout\nfrom tensorflow.keras import regularizers\nfrom keras import Model\nfrom keras.optimizers import Adam\nimport tensorflow as tf\n\nchannels = 16           # Number of channels in the first layer\ndropout = 0.5           # Dropout rate for the features\nl2_reg = 5e-4           # L2 regularization rate\nlearning_rate = 1e-2    # Learning rate\nepochs = 200            # Number of training epochs\nes_patience = 10        # Patience for early stopping\n\n# Preprocessing operations\nA = GCNConv.preprocess(A).astype('f4')\n\n# Model definition\nX_in = Input(shape=(F, ))\nfltr_in = Input((N, ), sparse=True)\n\ndropout_1 = Dropout(dropout)(X_in)\ngraph_conv_1 = GCNConv(channels,\n                         activation='relu',\n                         kernel_regularizer=regularizers.l2(l2_reg),\n                         use_bias=False)([dropout_1, fltr_in])\n\ndropout_2 = Dropout(dropout)(graph_conv_1)\ngraph_conv_2 = GCNConv(num_classes,\n                         activation='softmax',\n                         use_bias=False)([dropout_2, fltr_in])\n\n# Build model\nmodel = Model(inputs=[X_in, fltr_in], outputs=graph_conv_2)\noptimizer = Adam(lr=learning_rate)\nmodel.compile(optimizer=optimizer,\n              loss='categorical_crossentropy',\n              weighted_metrics=['acc'])\nmodel.summary()\n\ntbCallBack_GCN = tf.keras.callbacks.TensorBoard(\n    log_dir='./Tensorboard_GCN_cora',\n)\ncallback_GCN = [tbCallBack_GCN]","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"##### Train the Graph Convolutional Networks\nWe are implementing Transductive Learning, which means we will feed the whole graph to both training and testing. \n\nWe separate the training, validation, and testing data using the Boolean masks we have constructed before. \n\nThese masks will be passed to sample_weight argument. We set the batch_size to be the whole graph size, otherwise the graph will be shuffled.\n\nTo better evaluate the model performance for each class, we use F1-score instead of accuracy and loss metrics.\n","metadata":{}},{"cell_type":"code","source":"from keras.callbacks import EarlyStopping\n# Train model\nvalidation_data = ([X, A], labels_encoded, val_mask)\nmodel.fit([X, A],\n          labels_encoded,\n          sample_weight=train_mask,\n          epochs=epochs,\n          batch_size=N,\n          validation_data=validation_data,\n          shuffle=False,\n          callbacks=[\n              EarlyStopping(patience=es_patience,  restore_best_weights=True),\n              tbCallBack_GCN\n          ])","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"y = labels_encoded","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"from sklearn.metrics import classification_report\ny_pred = model.predict([X, A],batch_size=N)\nreport = classification_report(np.argmax(y,axis=1), np.argmax(y_pred,axis=1), target_names=classes)\nprint('GCN Classification Report: \\n {}'.format(report))","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Hope you liked this notebook!","metadata":{}},{"cell_type":"code","source":"","metadata":{},"execution_count":null,"outputs":[]}]}