{"nbformat": 4, "metadata": {"language_info": {"mimetype": "text/x-python", "name": "python", "version": "3.6.3", "file_extension": ".py", "nbconvert_exporter": "python", "codemirror_mode": {"name": "ipython", "version": 3}, "pygments_lexer": "ipython3"}, "kernelspec": {"name": "python3", "display_name": "Python 3", "language": "python"}}, "nbformat_minor": 1, "cells": [{"cell_type": "markdown", "metadata": {"_cell_guid": "1e2f8788-b0c3-479e-8d3b-870361afd74a", "_uuid": "74625ed45323430ae4f8d9ec7fa9ef49695a14bc"}, "source": ["Andrew Dodd\n", "11/8/17\n", "\n", "The following notebook is a somewhat Frankstein monster version of a few TF tutorials I saw online. For the most part, it is based on this tutorial:\n", "https://www.tensorflow.org/get_started/mnist/pros#train-and-evaluate-the-model"]}, {"execution_count": null, "cell_type": "code", "metadata": {"_cell_guid": "6feac2d4-0495-472c-8b99-31c35b59a4b6", "collapsed": true, "_uuid": "3888e34062f8a31d70307765f65cbb5af5a79eee"}, "outputs": [], "source": ["# This Python 3 environment comes with many helpful analytics libraries installed\n", "# It is defined by the kaggle/python docker image: https://github.com/kaggle/docker-python\n", "# For example, here's several helpful packages to load in \n", "\n", "import numpy as np # linear algebra\n", "import pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\n", "\n", "# Input data files are available in the \"../input/\" directory.\n", "# For example, running this (by clicking run or pressing Shift+Enter) will list the files in the input directory\n", "\n", "from subprocess import check_output\n", "print(check_output([\"ls\", \"../input\"]).decode(\"utf8\"))\n", "\n", "# Any results you write to the current directory are saved as output."]}, {"cell_type": "markdown", "metadata": {"_cell_guid": "570bc78f-5fdc-4eee-a3c4-b4a4f03aae71", "_uuid": "4565e54d87ba726ac8fd62a7605360c6eaba16e2"}, "source": ["(1) Load Datasets"]}, {"execution_count": null, "cell_type": "code", "metadata": {"_cell_guid": "62160947-842a-4542-b686-c28d950b6c86", "collapsed": true, "_uuid": "56f9c330a0fbd8e5e6d7a4f9a4a9c8a881f55bae"}, "outputs": [], "source": ["train_data = pd.read_csv('../input/train.csv')\n", "test_data = pd.read_csv('../input/test.csv')\n", "#sample_sub = pd.read_csv('../input/sample_submission.csv')"]}, {"cell_type": "markdown", "metadata": {"_cell_guid": "2aeb8d10-57a0-4aac-826c-b00d8fd6cfc8", "_uuid": "77ced48d4e6d7d0c28dcb2a478bc5a05d51a75a9"}, "source": ["(2) Imports"]}, {"execution_count": null, "cell_type": "code", "metadata": {"_cell_guid": "2ae864a0-0957-49a4-b9e8-2eb23e03c589", "collapsed": true, "_uuid": "23fad40e024a980102b33a452229c9f6bfd86658"}, "outputs": [], "source": ["import tensorflow as tf\n", "from sklearn.model_selection import train_test_split\n", "import random\n", "sess = tf.InteractiveSession()"]}, {"cell_type": "markdown", "metadata": {"_cell_guid": "b5977f9c-7864-4c82-a08f-8557a14a1dc4", "_uuid": "56a2dc8ca6c2e54e30d761a37b5c9bc7cd0f18ef"}, "source": ["(3) Preprocess data, label data 0-9, etc."]}, {"execution_count": null, "cell_type": "code", "metadata": {"_cell_guid": "be20bf65-92ba-42ab-b6c6-be995b2a32c0", "collapsed": true, "_uuid": "dacb0143030748b19cbb7d2fddb87a8e24f72899"}, "outputs": [], "source": ["# Defining all of the x data values, y data values, and x test data values.\n", "# We may also want to split the current x and y data into validation data and training data.\n", "\n", "x_data = train_data.values   # 42000 x 785, 42000 samples, 784 pixels per sample, need to remove label\n", "y_data = x_data[:,0]     # 42000 labels, later converted to one hot vectors\n", "x_data = np.delete(x_data, [0], axis=1)\n", "x_data = x_data.astype(np.float)\n", "\n", "x_test_data = test_data.values\n", "x_test_data = x_test_data.astype(np.float)\n", "\n", "#One hot vectors for y labels.\n", "y_data_OneHot = np.zeros((42000, 10))\n", "y_data_OneHot[np.arange(42000), y_data] = 1\n", "\n", "#Tensorflow Variable set up\n", "x = tf.placeholder(tf.float32, shape=[None, 784])\n", "y_ = tf.placeholder(tf.float32, shape=[None, 10])\n", "\n", "#Splitting data here, x_data, y_data\n", "X_train, X_val, y_train, y_val = train_test_split(x_data, y_data_OneHot, test_size=0.2)\n"]}, {"cell_type": "markdown", "metadata": {"_cell_guid": "2e2d53ca-9c62-46f2-a79d-e23d53a10f8d", "_uuid": "2fc9c58fee9f73a29c17dd642b77b5fd49fe4371"}, "source": ["(4) Here I am implementing a two layer model, softmax linear regression. This will not do great. Then I will try a convolutional model.\n", "\n", "Note: the tutorial I am using called: https://www.tensorflow.org/get_started/mnist/pros, this uses MNIST's class calls. I don't have those so I need my own methods of generating batches for testing. This should be as easy as generating random numbers and getting 100 or so random samples with their y values.\n", "\n", "We also need to split data into train/test split."]}, {"execution_count": null, "cell_type": "code", "metadata": {"_cell_guid": "123b950a-0ab8-44c7-ab55-ef36c090449f", "collapsed": true, "_uuid": "0067998d549135759490f742247e6681409ff9e5"}, "outputs": [], "source": ["W = tf.Variable(tf.zeros([784,10]))\n", "b = tf.Variable(tf.zeros([10]))\n", "\n", "sess.run(tf.global_variables_initializer())\n", "\n", "y = tf.matmul(x,W) + b\n", "\n", "cross_entropy = tf.reduce_mean(\n", "    tf.nn.softmax_cross_entropy_with_logits(labels = y_, logits = y))\n", "\n", "train_step = tf.train.GradientDescentOptimizer(.5).minimize(cross_entropy)\n", "\n", "epochs = 30\n", "\n", "for __ in range (epochs):\n", "    if __%20 == 0:\n", "        print (\"epoch\", __)\n", "    if __ == 99:\n", "        print (\"Done\")\n", "    for _ in range (1000):\n", "        temp = np.random.randint(X_train.shape[0], size=100)\n", "        X_train_batch = X_train[temp, :]\n", "        y_train_batch = y_train[temp, :]\n", "        train_step.run(feed_dict={x: X_train_batch, y_: y_train_batch})"]}, {"cell_type": "markdown", "metadata": {"_cell_guid": "06c8ee6a-8109-46ed-8e2f-710ca51bab50", "_uuid": "51a38388b997ca2644b543fcaa554bdfc3e7dcc1"}, "source": ["(4.1) Evaluation: Now that this model is trained, we will test accuracy..."]}, {"execution_count": null, "cell_type": "code", "metadata": {"_cell_guid": "2f143a7c-49cb-44cf-a3d3-773e1a1146d7", "collapsed": true, "_uuid": "2f07ae7e9b2bd26a3a1155336e9d5a03ec6a4e0c"}, "outputs": [], "source": ["correct_prediction = tf.equal(tf.argmax(y,1), tf.argmax(y_,1))\n", "\n", "accuracy = tf.reduce_mean(tf.cast(correct_prediction, tf.float32))\n", "\n", "print(accuracy.eval(feed_dict={x: X_val, y_: y_val}))"]}, {"cell_type": "markdown", "metadata": {"_cell_guid": "acf77951-6fb3-441f-927f-b32dc142bfe0", "_uuid": "e245464cc6ec09d2622222bce0aa9ff3ce41e649"}, "source": ["(5) Multilayer NN: We should see an accuracy in the range of 88-90% for the linear regression model with a softmax output layer. What about with a multilayer NN?\n", "\n", "Here we create a few useful functions for later..."]}, {"execution_count": null, "cell_type": "code", "metadata": {"_cell_guid": "e646d0de-7896-4242-9d3c-95cdbf95e742", "collapsed": true, "_uuid": "8c351dbf8199cb07cd30c7cf6c59207c6577be3a"}, "outputs": [], "source": ["def weight_variable(shape):\n", "    initial = tf.truncated_normal(shape, stddev = .1)\n", "    return tf.Variable(initial)\n", "\n", "def bias_variable(shape):\n", "    initial = tf.constant(.1, shape = shape)\n", "    return tf.Variable(initial)\n", "\n", "def conv2d(x, W):\n", "    return tf.nn.conv2d(x,W, strides=[1,1,1,1], padding = 'SAME')\n", "\n", "def max_pool_2x2(x):\n", "    return tf.nn.max_pool(x, ksize=[1,2,2,1], \n", "                          strides = [1,2,2,1], padding = 'SAME')"]}, {"cell_type": "markdown", "metadata": {"_cell_guid": "58fa1c4b-ba76-43c2-b57e-ec8c82754e8f", "_uuid": "0babba80f33497fed00a1122ef712add544bf86c"}, "source": ["(5.1) First convolutional layer"]}, {"execution_count": null, "cell_type": "code", "metadata": {"_cell_guid": "33b7e06c-84d3-45ce-a15f-3855b5540e71", "collapsed": true, "_uuid": "7966f12a2f5b5a5472871ae832eeeff5f3361ca8"}, "outputs": [], "source": ["W_conv1 = weight_variable([5,5,1,32])\n", "b_conv1 = bias_variable([32])\n", "\n", "x_image = tf.reshape(x, [-1,28,28,1])\n", "\n", "h_conv1 = tf.nn.relu(conv2d(x_image, W_conv1) + b_conv1)\n", "h_pool1 = max_pool_2x2(h_conv1)"]}, {"cell_type": "markdown", "metadata": {"_cell_guid": "1604ea0c-0757-4640-877d-9d68c4239e33", "_uuid": "cf81933b39e9a38a23ef8fbd2970ed4ca2f0685f"}, "source": ["(5.2) Second convolutional layer\n"]}, {"execution_count": null, "cell_type": "code", "metadata": {"_cell_guid": "8d94a889-e3a3-4f38-95b1-c09e25b87213", "collapsed": true, "_uuid": "c239086e998755b790c4887ab7340e8bfc314247"}, "outputs": [], "source": ["W_conv2 = weight_variable([5,5,32,64])\n", "b_conv2 = bias_variable([64])\n", "\n", "h_conv2 = tf.nn.relu(conv2d(h_pool1, W_conv2) + b_conv2)\n", "h_pool2 = max_pool_2x2(h_conv2)"]}, {"cell_type": "markdown", "metadata": {"_cell_guid": "89e432b9-6c26-4bf7-af36-808dd0fd9219", "_uuid": "7296ecdd04e376a4c2dfaed33e7fef2c2210c6c9"}, "source": ["(5.3) Densely Connected Layer, Dropout"]}, {"execution_count": null, "cell_type": "code", "metadata": {"_cell_guid": "07df3594-01d4-429a-8945-7a09a9824c00", "collapsed": true, "_uuid": "da5744e100a37fed4fbe37476f2496d8d3191734"}, "outputs": [], "source": ["W_fc1 = weight_variable([7 * 7 * 64, 1024])\n", "b_fc1 = bias_variable([1024])\n", "\n", "h_pool2_flat = tf.reshape(h_pool2, [-1, 7*7*64])\n", "h_fc1 = tf.nn.relu(tf.matmul(h_pool2_flat, W_fc1) + b_fc1)\n", "\n", "#Dropout\n", "keep_prob = tf.placeholder(tf.float32)\n", "h_fc1_drop = tf.nn.dropout(h_fc1, keep_prob)"]}, {"cell_type": "markdown", "metadata": {"_cell_guid": "f97c1070-628e-4b7f-9c18-c39bfa56549f", "_uuid": "6ea2fefce59586ac3d92eb3e4fb75c5bc0088c83"}, "source": ["(5.4) Readout Layer"]}, {"execution_count": null, "cell_type": "code", "metadata": {"_cell_guid": "1c98177e-5d32-4bde-a6a8-5b39459e4db9", "collapsed": true, "_uuid": "2a311e1a86446235d21bb25444ee758a37a815bb"}, "outputs": [], "source": ["W_fc2 = weight_variable([1024, 10])\n", "b_fc2 = bias_variable([10])\n", "\n", "y_conv = tf.matmul(h_fc1_drop, W_fc2) + b_fc2"]}, {"cell_type": "markdown", "metadata": {"_cell_guid": "651fd131-f4d4-4f21-b716-7fc2ee8f2561", "_uuid": "2ce33f864354198d3225b82f6f123400e34e1665"}, "source": ["(5.5) Train and Evaluate Model"]}, {"execution_count": null, "cell_type": "code", "metadata": {"scrolled": false, "_cell_guid": "950a0cc9-dceb-4951-b261-4ea4f7c878c7", "collapsed": true, "_uuid": "07603ade6e7119818e9166f9a0bced62c07ee5a7"}, "outputs": [], "source": ["cross_entropy = tf.reduce_mean(\n", "    tf.nn.softmax_cross_entropy_with_logits(labels=y_, logits=y_conv))\n", "train_step = tf.train.AdamOptimizer(1e-4).minimize(cross_entropy)\n", "correct_prediction = tf.equal(tf.argmax(y_conv, 1), tf.argmax(y_, 1))\n", "accuracy = tf.reduce_mean(tf.cast(correct_prediction, tf.float32))\n", "\n", "BATCH_SIZE = 100\n", "\n", "init = tf.global_variables_initializer()#\n", "sess = tf.InteractiveSession()#\n", "\n", "sess.run(init)#\n", "\n", "#with tf.Session() as sess:\n", "#    sess.run(tf.global_variables_initializer())\n", "for i in range (8000):\n", "    #Make a batch\n", "    temp = np.random.randint(X_train.shape[0], size=BATCH_SIZE)\n", "    X_train_batch = X_train[temp, :]\n", "    y_train_batch = y_train[temp, :]\n", "        \n", "    #Print the training accuracy\n", "    if i%100 == 0:\n", "        train_accuracy = accuracy.eval(feed_dict={\n", "            x: X_train_batch, y_: y_train_batch, keep_prob: 1.0})\n", "        print('Step %d, training accuracy %g' % (i, train_accuracy))\n", "        \n", "    #Train\n", "    train_step.run(feed_dict ={x: X_train_batch, y_: y_train_batch, keep_prob: .5})\n", "    \n", "print ('validation data accuracy %g' % accuracy.eval(feed_dict = {\n", "x: X_val, y_: y_val, keep_prob: 1.0}))\n", "    \n"]}, {"execution_count": null, "cell_type": "code", "metadata": {"_cell_guid": "b24a1c9d-d613-491c-9c87-80a8e11b6460", "collapsed": true, "_uuid": "edf982e14e45c4a99d1ac17f949782e2714230ca"}, "outputs": [], "source": ["    # prediction function\n", "    #[0.1, 0.9, 0.2, 0.1, 0.1 0.3, 0.5, 0.1, 0.2, 0.3] => 1\n", "    predict = tf.argmax(y_conv,1)\n", "    \n", "    #print (x_test_data[0])\n", "    \n", "    #predicted_labels = predict.eval(feed_dict={x: x_test_data, keep_prob: 1.0})\n", "    \n", "    predicted_labels = np.zeros(x_test_data.shape[0])\n", "    for i in range(0,x_test_data.shape[0]//BATCH_SIZE):\n", "        if i%100 == 0:\n", "            print(i)\n", "            print (predict.eval(feed_dict\n", "                        ={x: x_test_data[i*BATCH_SIZE : (i+1)*BATCH_SIZE], keep_prob: 1.0}))\n", "            \n", "        predicted_labels[i*BATCH_SIZE : (i+1)*BATCH_SIZE] = predict.eval(feed_dict\n", "                                ={x: x_test_data[i*BATCH_SIZE : (i+1)*BATCH_SIZE], keep_prob: 1.0})\n", "        #First batch computes 0-100, next batch is 101-200, etc\n", "        \n", "    np.savetxt('submission_dodd.csv', \n", "           np.c_[range(1,len(x_test_data)+1),predicted_labels],\n", "           delimiter=',', \n", "           header = 'ImageId,Label', \n", "           comments = '', \n", "           fmt='%d')                 \n", "    \n", "    print (predicted_labels)\n", "    print (predicted_labels.shape)\n", "    print (np.mean(predicted_labels))\n", "                                  "]}, {"execution_count": null, "cell_type": "code", "metadata": {"_cell_guid": "71bf6c87-e795-4c9c-93e4-01fac1a03191", "collapsed": true, "_uuid": "603ddccd459e7a80d9714a1959b014d379e16e57"}, "outputs": [], "source": ["print (np.mean(x_test_data))\n", "sess.close()"]}]}