{"cells":[{"metadata":{"_uuid":"f6c69a6a212dd89c545c01df9bd02a4b8461ff6d"},"cell_type":"markdown","source":"# Multilayer Perceptron\nThis notebook is quick MLP (multilayer perceptron) predicting digit for [kaggle competition](https://www.kaggle.com/c/digit-recognizer). \n\nPlease consider follow before reading this notebook.\n- We don't verify data in this notebook.\n- MLP isn't the suitable model for image classification, but digit recognition problem is simply enough to use, however for more complex problem consider CNN (Convolution Neural Network)\n- I would like to make this notebook simple enough for anyone who new to MLP can follow the code easily."},{"metadata":{"_uuid":"509e7775a97af2d80be0dec92b38a29f0d067ae2"},"cell_type":"markdown","source":"## 1. Load data"},{"metadata":{"_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","collapsed":true,"trusted":true},"cell_type":"code","source":"import pandas as pd\nimport tensorflow as tf\n\ntrain_dataset = pd.read_csv('../input/train.csv')\ntest_dataset = pd.read_csv('../input/test.csv')","execution_count":12,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"3e69457ecde17a276f719e013a70db96e10e2797"},"cell_type":"code","source":"print(\"Train Dataset\")\nprint(train_dataset[:5])\nprint(\"Test Dataset\")\nprint(test_dataset[:5])\nprint(\"Train Size : {}\".format(train_dataset.shape[0]))\nprint(\"Test Size : {}\".format(test_dataset.shape[0]))","execution_count":13,"outputs":[]},{"metadata":{"_uuid":"c1c3f75fc205ee6a92f8ac5a56a2516f246263fc"},"cell_type":"markdown","source":"## 2. Prepare Data\nBefore training we need to prepare data a little bit. here is the detail\n- We need to scale each pixel to the range of 0 - 1\n- Since Kaggle don't have validate data for us,so we have to split some from training data.\n- We need to spearate label (train_y) out of feature (train_x) so we can feed it to our network as label and feature. "},{"metadata":{"trusted":true,"_uuid":"0e6c3670b5cc2afa225d39ae5a7dba61401d6ffe"},"cell_type":"code","source":"from sklearn.preprocessing import MinMaxScaler\nfrom sklearn.preprocessing import LabelBinarizer\n\nvalidation_ratio = 0.1\n\npixel_scaler = MinMaxScaler()\none_hot = LabelBinarizer()\none_hot.fit(range(10))\ntrain_y = one_hot.transform(train_dataset['label'])\ntrain_x = train_dataset.drop('label', axis=1)\ntrain_x = pixel_scaler.fit_transform(train_x)\ntest_x = pixel_scaler.transform(test_dataset)\n\nsplit = int(train_dataset.shape[0] * (1 - validation_ratio))\n\nvalid_x = train_x[split:train_x.shape[0]]\nvalid_y = train_y[split:train_x.shape[0]]\ntrain_x = train_x[0:split]\ntrain_y = train_y[0:split]\nprint(\"Train Size : {}\".format(train_x.shape[0]))\nprint(\"Validation Size : {}\".format(valid_x.shape[0]))","execution_count":14,"outputs":[]},{"metadata":{"_uuid":"96813ec6602dc492988f8e4e5798ea8afa93be2a"},"cell_type":"markdown","source":"## 3. Create Model\nBelow is the model code.\n- Initialize DigitalRecognizerMLP at first.\n- Call make_nn() to set up tensorflow graph.\n- Call train() for train network after training the network has been stored and ready for prediction.\n- Call predict() for make a prediction with the latest saved network."},{"metadata":{"collapsed":true,"trusted":true,"_uuid":"93199e00fd38eee61995430038bee99b1d7d624d"},"cell_type":"code","source":"class DigitRecognizerMLP(object):\n    def __init__(self, feature_size=784, label_size=10):\n        self.feature_size = feature_size\n        self.label_size = label_size\n    \n    def build_input(self):\n        model_x = tf.placeholder(tf.float32, [None, self.feature_size])\n        model_y = tf.placeholder(tf.int32, [None, self.label_size])\n        model_lr = tf.placeholder(tf.float32)\n        return model_x, model_y, model_lr\n    \n    def build_mlp(self, model_x, model_y):\n        hidden_1 = tf.layers.dense(model_x, 1024, activation=tf.nn.relu)\n        hidden_2 = tf.layers.dense(hidden_1, 512, activation=tf.nn.relu)\n        hidden_3 = tf.layers.dense(hidden_2, 256, activation=tf.nn.relu)\n        hidden_4 = tf.layers.dense(hidden_2, 128, activation=tf.nn.relu)\n        logits = tf.layers.dense(hidden_3, self.label_size, activation=None)\n        return logits\n    \n    def build_output(self, model_y, logits):\n        output = tf.argmax(logits, axis=1)\n        loss = tf.losses.softmax_cross_entropy(model_y, logits)\n        accuracy = tf.reduce_mean( tf.cast( tf.equal( tf.argmax(model_y, axis=1), output ), tf.float32) )\n        return output, loss, accuracy\n    \n    def model_opt(self, model_lr, loss):\n        opt = tf.train.AdamOptimizer(model_lr).minimize(loss)\n        return opt\n    \n    def make_nn(self):\n        tf.reset_default_graph()\n        self.model_x, self.model_y, self.model_lr = self.build_input()\n        self.logits = self.build_mlp(self.model_x, self.model_y)\n        self.output, self.loss, self.accuracy = self.build_output(self.model_y, self.logits)\n        self.opt = self.model_opt(self.model_lr, self.loss)\n        \n    def train(self, epoch, learning_rate, train_x, train_y, batch_size, valid_x, valid_y, get_batch_func, print_every):\n        t_loss_list = []\n        v_loss_list = []\n        accuracy_list = []\n        saver = tf.train.Saver()\n        with tf.Session() as sess :\n            sess.run(tf.global_variables_initializer())\n            counter = 0\n            for e in range(epoch) :\n                for x,y in get_batch_func(train_x, train_y, batch_size):\n                    feed_dict = {self.model_x:x, self.model_y:y, self.model_lr:learning_rate}\n                    loss, _ = sess.run([self.loss, self.opt], feed_dict=feed_dict)\n                    if counter % print_every == 0 :\n                        feed_dict = {self.model_x:valid_x, self.model_y:valid_y, self.model_lr:learning_rate}\n                        v_loss, accuracy = sess.run([self.loss, self.accuracy], feed_dict=feed_dict)\n                        print(\"Epoch: {}/{}, Step: {}, T Loss: {:.4f}, V Loss: {:.4f}, Accuracy: {:.4f}\".format(e+1, \n                                    epoch, counter, loss, v_loss, accuracy))\n                        t_loss_list.append( loss )\n                        v_loss_list.append( v_loss )\n                        accuracy_list.append( accuracy )\n                        save_path = saver.save(sess, \"tensor/model.ckpt\")\n                    counter += 1\n        return t_loss_list, v_loss_list, accuracy_list\n    \n    def predict(self, predict_x):\n        saver = tf.train.Saver()\n        with tf.Session() as sess:\n            sess.run(tf.global_variables_initializer())\n            saver.restore(sess, \"tensor/model.ckpt\")\n            feed_dict = {self.model_x:predict_x}\n            output = sess.run(self.output, feed_dict=feed_dict)\n            return output","execution_count":15,"outputs":[]},{"metadata":{"collapsed":true,"trusted":true,"_uuid":"11d576c81dc664a3b9e20700370fecb0c3c7a8cb"},"cell_type":"code","source":"# batch create function\ndef get_batch(feature, label, batch_size):\n    num_batch = feature.shape[0] // batch_size\n    for i in range(num_batch):\n        start = i * batch_size\n        end = (i + 1) * batch_size\n        x = feature[start:end]\n        y = label[start:end]\n        yield x, y","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"43d8d87c2a57d7d62afade1aaee094936b205026"},"cell_type":"markdown","source":"## 4. Train"},{"metadata":{"trusted":true,"_uuid":"a43cdf7e21e57e1b3a378c063f81df2fce3869a3"},"cell_type":"code","source":"learning_rate = 0.0001\nbatch_size = 64\nepoch = 50\nprint_every = 1000\n\nmlp = DigitRecognizerMLP()\nmlp.make_nn()\nt_loss, v_loss, acc = mlp.train(epoch, learning_rate, train_x, train_y, batch_size, valid_x, valid_y, get_batch, print_every)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"68ed419ef0998e1abd2d070b3cabcdf81b9c9265"},"cell_type":"markdown","source":"## 5. Predict"},{"metadata":{"collapsed":true,"trusted":true,"_uuid":"77b5465530c1131ed184f3520183e90c2e9f29e8"},"cell_type":"code","source":"mlp = DigitRecognizerMLP()\nmlp.make_nn()\npredict = mlp.predict(test_x)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"761a89c52ca8f690a13b32cc4010b7032a34385b"},"cell_type":"markdown","source":"## 6. Make CSV\nThis is just making csv file as kaggle required for submittion."},{"metadata":{"collapsed":true,"trusted":true,"_uuid":"db094f7d9ea6c291a6f70aaddc2c3ce1013b6b11"},"cell_type":"code","source":"import csv\n\nwith open('predict.csv', 'w') as csv_file :\n    writer = csv.writer(csv_file)\n    writer.writerow(['ImageId', 'Label'])\n    for i in range(len(predict)) :\n        writer.writerow([str(i+1), str(predict[i])])\n    csv_file.close()","execution_count":null,"outputs":[]},{"metadata":{"collapsed":true,"trusted":true,"_uuid":"ba3db14c73a1eb992bfb24715d70b13657868b9e"},"cell_type":"code","source":"","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"name":"python","version":"3.6.5","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"}},"nbformat":4,"nbformat_minor":1}