{"cells":[{"cell_type":"markdown","metadata":{"_cell_guid":"f6b04f34-d978-b8f4-9c94-8fbcdf452b1b"},"source":"The goal for this Kernel is simple:\nAdapt the \"original\" [Tensorflow tutorial example][1] to produce a Kaggle submission.\nWe will also try to add some explanation where this seems plausible. However, most of the explanation will be imported from the original tutorial. It kind of works now ;)\n\n  [1]: https://www.tensorflow.org/get_started/mnist/beginners\n\nThis is Version 1.1 of the notebook. It adds evaluation against the test data and renames some variables."},{"cell_type":"code","execution_count":null,"metadata":{"_cell_guid":"b681764e-88e3-ca94-5e03-716e980beb60"},"outputs":[],"source":"# Start with reading and \"preprocessing\" some data\n\n# This Python 3 environment comes with many helpful analytics libraries installed\n# It is defined by the kaggle/python docker image: https://github.com/kaggle/docker-python\n# For example, here's several helpful packages to load in \n\nimport numpy as np # linear algebra\nimport pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\nimport matplotlib.pyplot as plt\nimport tensorflow as tf\n\n# Input data files are available in the \"../input/\" directory.\n# For example, running this (by clicking run or pressing Shift+Enter) will list the files in the input directory\n\nfrom subprocess import check_output\nprint(check_output([\"ls\", \"../input\"]).decode(\"utf8\"))\n\n# Read in training data from train.csv\ntrain = pd.read_csv('../input/train.csv')\ntrainfv = train.drop(['label'], axis=1).values.astype(dtype=np.float32)\n# trainFeatureVectorsConvoFormat = trainFeatureVectors.reshape(42000, 28, 28, 1)\n\n# Naming might be improveable... ;)\ntrainLabels = train['label'].tolist()\nohtrainLabels = tf.one_hot(trainLabels, depth=10)\nohtrainLabelsNdarray = tf.Session().run(ohtrainLabels).astype(dtype=np.float64)\n\n# Read in testing data from test.csv\ntest = pd.read_csv('../input/test.csv')\ntestfv = test.drop(['label'], axis=1).values.astype(dtype=np.float32)\n# testFeatureVectorsConvoFormat = testFeatureVectors.reshape(28000, 28, 28, 1)\ntestLabels = test['label'].tolist()\nohtestLabels = tf.one_hot(testLabels, depth=10)\nohtestLabelsNdarray = tf.Session().run(ohtestLabels).astype(dtype=np.float64)\n\nprint(trainfv[0],ohtrainLabelsNdarray[0])\n\n# Show some training data and there labels\nfor i in range(5):\n    pixels = trainfv[i].reshape((28, 28))\n    plt.imshow(pixels, cmap='gray')\n    plt.show()\n    print(\"Label: \",trainLabels[i],\" - OneHot: \",ohtrainLabelsNdarray[i])\n\n# Convert data from 0..255 to 0..1\n# If you forget this (as I did), the model will not converge (well, it will converge, but to \n# something like 0.0921 accuracy - which is really, really BAD ;)\ntrainfv = np.multiply(trainfv, 1.0 / 255.0)    \n\nprint(trainfv[0],ohtrainLabelsNdarray[0])"},{"cell_type":"code","execution_count":null,"metadata":{"_cell_guid":"7a164ac6-6b50-5598-1daf-2de41b3dd2f2"},"outputs":[],"source":"# Now build the simple model: all 784 inputs are connected to 10 \"decision nodes\", which use \n# a thresholded and normalized softmax to compute a \"probability\" for each of the ten digits\nx = tf.placeholder(tf.float32, [None, 784])\nW = tf.Variable(tf.zeros([784, 10]))\nb = tf.Variable(tf.zeros([10]))\n\n# Define the way we evaluate the network\ny = tf.nn.softmax(tf.matmul(x, W) + b)\n\n# To implement cross-entropy we need to first add a new placeholder to input the correct answers:\ny_ = tf.placeholder(tf.float32, [None, 10])\n\n# numerically unstable implementation of cross entroy\ncross_entropy = tf.reduce_mean(-tf.reduce_sum(y_ * tf.log(y), reduction_indices=[1]))\n\n# Explaination of the cross entropy formula above:\n# First, tf.log computes the logarithm of each element of y. \n# Next, we multiply each element of y_ with the corresponding element of tf.log(y). \n# Then tf.reduce_sum adds the elements in the second dimension of y, due to the \n# reduction_indices=[1] parameter. Finally, tf.reduce_mean computes the mean over \n# all the examples in the batch.\n\n# No, define a training step \ntrain_step = tf.train.GradientDescentOptimizer(0.5).minimize(cross_entropy)"},{"cell_type":"markdown","metadata":{"_cell_guid":"386bdb46-0d39-4ba7-6d64-91007c203161"},"source":"![The weights learned by the model][1]\n\n\n  [1]: https://www.tensorflow.org/images/softmax-weights.png\n\nThis is a visualisation, embedded from the Tensorflow tutorial, of the weights learned by the model."},{"cell_type":"code","execution_count":null,"metadata":{"_cell_guid":"11d8c333-0bdf-2f1d-1cfa-721013f1a12a"},"outputs":[],"source":"import math\n\n# A mildly random version....simply pick one of the (consecutive) slices of size \"size\"\nfrom random import randint\n\n# This is used below, for simplicity and for faster convergence ;)\ndef random_batch(data,labels,size):\n    value = math.floor(len(data) / size)    \n    intervall = randint(0,value-1)\n    return data[intervall*size:intervall*(size+1)],labels[intervall*size:intervall*(size+1)]\n\n# print(random_batch(trainfv,ohtrainLabelsNdarray,3))\n\nfrom random import sample\n\n# of course, this could be done in a myriad of (better) ways...\n# This \"random\" picking (not subarray-wise, as above) leads to far slower\n# learning - which justifies the assumption that \"similar\" digits (from\n# the same writer perhaps) which we want too learn to differentiate, \n# often are close together in the original data, making it useful to learn\n# them together in batches\ndef really_random_batch(data,labels,size):\n    res_data = np.empty((size,784))\n    res_label = np.empty((size,10))\n    \n    l = len(data)\n    s = sample(range(l), size)\n    \n    for i in range(size):\n        res_data[i] = data[i]\n        res_label[i] = labels[i]\n        \n    return np.array(res_data),np.array(res_label)\n\n# A sample sample... ;)\n# print(really_random_batch(trainfv,ohtrainLabelsNdarray,3))"},{"cell_type":"code","execution_count":null,"metadata":{"_cell_guid":"19d06e8d-d6d3-a396-dae1-fdb61943d025"},"outputs":[],"source":"sess = tf.InteractiveSession()\ntf.global_variables_initializer().run()\n\n# Training (go higher for better results, e.g. 1500)\nfor i in range(500):\n  batch_xs, batch_ys = random_batch(trainfv,ohtrainLabelsNdarray,150)\n  sess.run(train_step, feed_dict={x: batch_xs, y_: batch_ys})\n  if not(i % 50): \n    print(i)\n    \n# Test our prediction \ncorrect_prediction = tf.equal(tf.argmax(y, 1), tf.argmax(y_, 1))\naccuracy = tf.reduce_mean(tf.cast(correct_prediction, tf.float32))\nprint(\"\\nAccuracy of the current model (against training samples): \",sess.run(accuracy, feed_dict={x: trainfv[0:10000],\n                                    y_: ohtrainLabelsNdarray[0:10000]}))\n\nprint(\"\\nAccuracy of the current model (against first 10000 test samples): \",sess.run(accuracy, feed_dict={x: testfv[0:10000],\n                                    y_: ohtestLabelsNdarray[0:10000]}))\n\n\n# With 1500 batches of 150 elements, you should reach an accuracy of about 0.92 (92% correct\n# decisions...still a lot of errors, but, heh, it's just a start!)\n\n# Re-running this code cell will produce a different accuracy, if you want to keep the result\n# \"fixed\", you could set a random seed prior to calling random_batch"},{"cell_type":"markdown","metadata":{"_cell_guid":"55d98e68-73ea-75d0-7275-94a78a188474"},"source":"Missing: Build a submission file from the model\n"}],"metadata":{"_change_revision":0,"_is_fork":false,"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"codemirror_mode":{"name":"ipython","version":3},"file_extension":".py","mimetype":"text/x-python","name":"python","nbconvert_exporter":"python","pygments_lexer":"ipython3","version":"3.6.0"}},"nbformat":4,"nbformat_minor":0}