{"cells":[{"metadata":{"_uuid":"7e12af316f963a341c16939dfb2babb4db96c0d6","_cell_guid":"48e92c46-676f-2117-5b5f-f0c9962c2c9e"},"cell_type":"markdown","source":"### Building a Convolutional Network on only 2 features:\nAuthor - Alexandru Papiu\n\nConvolutional Nets can be thought of as feature extractors - they take in an image that is very high dimensional and return a learned representation of this image that consists of higher level features that help with the learning task at hand. Unfortunately these representations are themselves high dimensional so it's hard to see exactly what is going on. To try to understand things better I'll build a CNN on the MNIST dataset that has a 2-unit layer right before the classification layer. This will force the model to capture as much information as possible in those two features. Let's see how well the 2-D model does!"},{"metadata":{"_uuid":"8fc865e4f152ed94d574b51ce643e69272e12776","_cell_guid":"39c53576-448d-343f-e881-5e9f1965b3f1"},"cell_type":"markdown","source":"###Loading modules and data:"},{"metadata":{"_uuid":"12e93789a8273f88e51c061783c0061868b528b8","_cell_guid":"22a49f86-5711-7df7-6d57-fd20952173a7","collapsed":true,"trusted":false},"cell_type":"code","source":"%matplotlib inline\nimport numpy as np\nimport pandas as pd\nimport matplotlib.pyplot as plt\nimport matplotlib\nfrom sklearn import metrics\nfrom sklearn.neighbors import NearestNeighbors\n\nfrom keras.models import Sequential, Model\nfrom keras.layers import Dense, Dropout, Convolution2D, MaxPooling2D, Flatten, Input\nfrom keras.optimizers import adam\nfrom keras.utils.np_utils import to_categorical\n\nimport matplotlib.pyplot as plt\nplt.style.use('seaborn')\n\nimport seaborn as sns\n\n%config InlineBackend.figure_format = 'retina'","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"5aa88f829665f0262835c11c6c18d126caf01c2f","_cell_guid":"934c60ad-43fd-d3f1-3127-3d8a333ecc6b","collapsed":true,"trusted":false},"cell_type":"code","source":"train = pd.read_csv(\"../input/train.csv\")","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"bd01ca7ac5026774aef35bf7e92a659285c5b135","_cell_guid":"c6ccf89d-9bc3-2ddb-ed86-7b7f40beeb90","collapsed":true,"trusted":false},"cell_type":"code","source":"X_train = train.iloc[:,1:].values\nX_train = X_train.reshape(X_train.shape[0], 28, 28, 1) #reshape to rectangular\nX_train = X_train/255 #pixel values are 0 - 255 - this makes puts them in the range 0 - 1\n\ny_train = train[\"label\"].values","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"943e027d5bb73748eb7235b4b3ae5d221b9d2b1e","_cell_guid":"d7384fd8-4b3d-8253-4f98-51ddf53e3b9e","collapsed":true,"trusted":false},"cell_type":"code","source":"y_ohe = to_categorical(y_train)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"3681f5214defd93a9d3c0de7a15af50866fdf702","_cell_guid":"be121053-4ac8-69a9-eb53-292c4fb83aa1"},"cell_type":"markdown","source":"### Building the Model:"},{"metadata":{"_uuid":"4afaa1488967addba0f32e6cc7ee945467e1e746","_cell_guid":"61eb3765-fdf5-485f-a532-3d1171116ff2"},"cell_type":"markdown","source":"Let's build the CNN. The architecture is pretty \"classic\" - a bunch of convolutional layers with relu activations followed by some fully connected layers.  The only thing that's different is the we make the output gradually smaller so that we can see how much signal the model can capture in only 2 dimensions."},{"metadata":{"_uuid":"2103ec267476a281ad79f2d6d4088cd0970f41fd","_cell_guid":"12c7213e-cfee-c98f-33b7-6c71dd82ce07","collapsed":true,"trusted":false},"cell_type":"code","source":"model = Sequential()\nmodel.add(Convolution2D(32, 3, 3, input_shape = (28, 28, 1), activation=\"relu\"))\nmodel.add(Convolution2D(32, 3, 3, activation=\"relu\"))\nmodel.add(MaxPooling2D(pool_size=(2,2)))\n\nmodel.add(Convolution2D(32, 3, 3, activation=\"relu\"))\n#model.add(Convolution2D(32, 3, 3, activation=\"relu\"))\nmodel.add(MaxPooling2D(pool_size=(2, 2)))\n\nmodel.add(Flatten())\n\nmodel.add(Dense(128, activation = \"relu\"))\nmodel.add(Dense(16, activation = \"relu\"))\nmodel.add(Dense(2))\nmodel.add(Dense(10, activation=\"softmax\"))","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"37ac1759e3c4aa53ca1833546ad2382682e003a4","_cell_guid":"20a4e417-d2b9-4da8-c497-df9da32cb14a","collapsed":true,"trusted":false},"cell_type":"code","source":"model.compile(loss='categorical_crossentropy', \n              optimizer = adam(lr=0.001), metrics = [\"accuracy\"])","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"e8cbfbc1ae855ea540992cb4dd6a9b0d1943c42e","_cell_guid":"b510da1b-d2f1-574c-6fe4-8a6836ac863e","collapsed":true,"trusted":false},"cell_type":"code","source":"hist = model.fit(X_train, y_ohe,\n          validation_split = 0.05, batch_size = 128, nb_epoch = 7)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"be74683d112b99294504289ee873292dc00a1f76","_cell_guid":"734aa251-95d8-2324-cced-b84e440ad856"},"cell_type":"markdown","source":"97% accuracy! That's not bad at all for only 2 dimensions.  Let's all try to visualize projections of the digits onto the 2-d feature space:"},{"metadata":{"_uuid":"87db9e0c9f335bc53ed045de78d3072cb6a23d79","_cell_guid":"4b2d9295-22bd-ba15-fa10-e0ab0443f534","collapsed":true,"trusted":false},"cell_type":"code","source":"#getting the 2D output:\noutput = model.get_layer(\"dense_3\").output\nextr = Model(model.input, output)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"adc423dcd414acda2a8a7a622d76d9b148455932","_cell_guid":"8c1d2d12-6e93-ba82-f047-0a30384ea484"},"cell_type":"markdown","source":"### The Learned 2-D representation:"},{"metadata":{"_uuid":"44049d271ff7b343eed51c7842607fcc7df33b6a","_cell_guid":"1a076c99-d0ba-23d6-6068-3b414dc1b7b0","collapsed":true,"trusted":false},"cell_type":"code","source":"\nX_proj = extr.predict(X_train[:10000])\nX_proj.shape\n\nproj = pd.DataFrame(X_proj[:,:2])\nproj.columns = [\"comp_1\", \"comp_2\"]\nproj[\"labels\"] = y_train[:10000]\n\n\nmatplotlib.rcParams['figure.figsize'] = (10.0, 10.0)\nsns.lmplot(\"comp_1\", \"comp_2\",hue = \"labels\", data = proj, fit_reg=False, size = 8)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"33f1b677a76aa0d01ff713b18420ca8bff4e7fe3","_cell_guid":"5c9f4c09-1d61-dfcf-12ed-6626f58a2986"},"cell_type":"markdown","source":"That is really impressive. The CNN has managed to encode these high dimensional images into only 2 dimensions in such a way the classes are separated in a clean symmetric pattern. It would be interesting to dig a little deeper to see exactly what the two axes represent here - at the end of the day they're just mathematical functions from a 784 dimensional space to a 2 dimensional one. However that might not be too illuminating, the transformations are just messy compositions of convolutions, linear combinations and relu's.  What's really striking I think, is how the model starts with some raw inputs and by simply minimizing a loss function builds this beautiful symmetric structure. I guess there's beauty in math after all :)"}],"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"_is_fork":false,"language_info":{"codemirror_mode":{"version":3,"name":"ipython"},"nbconvert_exporter":"python","mimetype":"text/x-python","name":"python","pygments_lexer":"ipython3","version":"3.6.4","file_extension":".py"},"_change_revision":0},"nbformat":4,"nbformat_minor":1}