{"cells":[{"metadata":{},"cell_type":"markdown","source":"\n# Binary Neural Networks in TensorFlow - A Friendly Introduction\nFrom someone who has mainly worked with PyTorch, this notebook is an introduction to the keras library. Furthermore, it is also an introduction to Binary Neural Networks(BNNs) using the larq framework.\n\nBNNs attempt to tackle one of the problems with NNs and CNNs: expensive computation and memory usage. The concept is to turn most of the network weights into binary values (-1 or 1), heavily reducing the memory footprint of the model. Moreover, with adequated hardware, the model also runs much faster.\n\nI will be tuning this model to try and achieve the best accuracy I can on the MNIST dataset while also providing code explanations to the best of my ability.\n\nTensorFlow: https://www.tensorflow.org/\n\nlarq docs: https://docs.larq.dev/larq/\n"},{"metadata":{},"cell_type":"markdown","source":"# Importing necessary libraries"},{"metadata":{"trusted":false},"cell_type":"code","source":"import numpy as np \nimport matplotlib.pyplot as plt\nimport pandas as pd\n\nimport tensorflow as tf\nimport larq as lq \nfrom sklearn.model_selection import train_test_split\n%matplotlib inline","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Loading, processing and checking csv data"},{"metadata":{"trusted":false},"cell_type":"code","source":"# Loading data from csv files using pandas\ntrain_data = pd.read_csv(\"train.csv\")\ntest_data = pd.read_csv(\"test.csv\")","execution_count":null,"outputs":[]},{"metadata":{"trusted":false},"cell_type":"code","source":"# Checking data\nprint(train_data.shape) # we have 42000\ntrain_data.head()\n\nprint(test_data.shape)\ntest_data.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":false},"cell_type":"code","source":"# Assembling data sets\n\n# Train data\nX = train_data.iloc[:, 1:].values.astype(\"float32\") \ny = train_data.iloc[:, 0].values.astype(\"int32\")\n\n# Test data\nX_test = test_data.values.astype(\"float32\")","execution_count":null,"outputs":[]},{"metadata":{"trusted":false},"cell_type":"code","source":"# Converting to image 28x28 shapes instead of 1-d pixel arrays\n# Adding one dimension for gray scale color\nX = X.reshape(-1, 28, 28, 1)\nX_test = X_test.reshape(-1, 28, 28, 1)\n\n# Plotting a couple of images to see the digits\nfor i in range(10, 13) :\n    plt.subplot(330 + (i + 1))\n    plt.imshow(X[i], cmap=plt.get_cmap(\"gray\"))\n    plt.title(y[i])","execution_count":null,"outputs":[]},{"metadata":{"trusted":false},"cell_type":"code","source":"# Normalizing pixel values to lie between -1 and 1\nX = X/(255/2 - 1)\n\n# Validation data\n# Take some data from the training set to evaluate our model before submitting to Kaggle\n# use whole X when training final submission\nX_train, X_val, y_train, y_val = train_test_split(X, y, test_size=0.2, random_state=42)\n\nX_test = X_test/(255/2 - 1)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Defining the model using tensorflow and larq\n\nlarq is a framework used on top of Keras to construct and train binary weights and inputs neural networks. BNNs are of interest due to how efficient they can be, using much less memory than their full-precision counterparts. Of course, the major drawback of such NNs is the possible reduced accuracy. However, depending on the application at hand, the trade-off might be well worth it, as certain systems require light-weight models.\n\nFollowing are the two main concepts from larq used in this model.\n\n**ste_sign (quantizer)**: defines how a full-precision weight or input is to be converted into a binary one. In particular, this quantizer converts all negative weights to -1 and all positive or zero weights to 1.\n\n**weight_clip (constraint)**: contrains the pre-binarized weights to be within a range of values [-1, 1]."},{"metadata":{"trusted":false},"cell_type":"code","source":"# All binary layers apart from the first will use these settings\nsettings = dict(\n    input_quantizer=\"ste_sign\",\n    kernel_quantizer=\"ste_sign\",\n    kernel_constraint=\"weight_clip\"\n    )\n\nmodel = tf.keras.models.Sequential()\n\n# First later: quantize only weights, not inputs\nmodel.add(\n    lq.layers.QuantConv2D(64, (3, 3),\n    kernel_quantizer= \"ste_sign\",\n    kernel_constraint=\"weight_clip\",\n    use_bias = False,\n    input_shape = (28, 28, 1))\n)\nmodel.add(tf.keras.layers.MaxPooling2D((2, 2)))\nmodel.add(tf.keras.layers.BatchNormalization(scale=False))\n\nmodel.add(lq.layers.QuantConv2D(64, (3, 3), use_bias=False, **settings))\nmodel.add(tf.keras.layers.MaxPooling2D((2, 2)))\nmodel.add(tf.keras.layers.BatchNormalization(scale=False))\n\nmodel.add(lq.layers.QuantConv2D(64, (3, 3), use_bias=False, **settings))\nmodel.add(tf.keras.layers.BatchNormalization(scale=False))\nmodel.add(tf.keras.layers.Flatten())\n\nmodel.add(lq.layers.QuantDense(120, use_bias=False, **settings))\nmodel.add(tf.keras.layers.BatchNormalization(scale=False))\nmodel.add(lq.layers.QuantDense(84, use_bias=False, **settings))\nmodel.add(tf.keras.layers.BatchNormalization(scale=False))\nmodel.add(lq.layers.QuantDense(10, use_bias=False, **settings))\nmodel.add(tf.keras.layers.BatchNormalization(scale=False))\nmodel.add(tf.keras.layers.Activation(\"softmax\"))","execution_count":null,"outputs":[]},{"metadata":{"trusted":false},"cell_type":"code","source":"# Visualizing our model\nlq.models.summary(model)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Training the binary net"},{"metadata":{"trusted":false},"cell_type":"code","source":"model.compile(\n    optimizer=\"adam\",\n    loss=\"sparse_categorical_crossentropy\",\n    metrics=[\"accuracy\"]\n)\n\n#model.fit(X_train, y_train, batch_size=64, epochs=15) # Use during dev\nmodel.fit(X, y, batch_size=64, epochs=10) # Use for final deployment","execution_count":null,"outputs":[]},{"metadata":{"trusted":false},"cell_type":"code","source":"# Evaluating our model on our validation set\nval_loss, val_acc = model.evaluate(X_val, y_val)","execution_count":null,"outputs":[]},{"metadata":{"tags":[],"trusted":false},"cell_type":"code","source":"# Making predictions on test data \npredictions = np.argmax(model.predict(X_test) > 0.5, axis = -1)","execution_count":null,"outputs":[]},{"metadata":{"trusted":false},"cell_type":"code","source":"# Saving predictions for submission\nsubmissions=pd.DataFrame({\"ImageId\": list(range(1,len(predictions)+1)),\n                         \"Label\": predictions})\nsubmissions.to_csv(\"sub.csv\", index=False, header=True)","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}