{"cells":[{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"code","source":"# This Python 3 environment comes with many helpful analytics libraries installed\n# It is defined by the kaggle/python docker image: https://github.com/kaggle/docker-python\n# For example, here's several helpful packages to load in \n\nimport numpy as np # linear algebra\nimport pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\n\n# Input data files are available in the \"../input/\" directory.\n# For example, running this (by clicking run or pressing Shift+Enter) will list all files under the input directory\n\nimport os\nfor dirname, _, filenames in os.walk('/kaggle/input'):\n    for filename in filenames:\n        print(os.path.join(dirname, filename))\n\n# Any results you write to the current directory are saved as output.","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"'''Importing Data Manipulattion Modules'''\nimport numpy as np\nimport pandas as pd\n\n'''Seaborn and Matplotlib Visualization'''\nimport matplotlib.pyplot as plt\nimport seaborn as sns\n%matplotlib inline\n\n'''Importing preprocessing libraries'''\nfrom sklearn.model_selection import train_test_split\nfrom sklearn.metrics import confusion_matrix\n\nfrom sklearn import preprocessing\n\n'''Importing keras libraries'''\nfrom keras.utils.np_utils import to_categorical\nfrom keras.models import Sequential\nfrom keras.layers import Dense,Conv2D,Flatten,MaxPool2D,Dropout,BatchNormalization\nfrom keras.optimizers import RMSprop,Adam\nfrom keras.callbacks import ReduceLROnPlateau\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Lets load the data"},{"metadata":{"trusted":true},"cell_type":"code","source":"train=pd.read_csv('../input/Kannada-MNIST/train.csv')\ntest1=pd.read_csv('../input/Kannada-MNIST/test.csv')\nsample_sub=pd.read_csv('../input/Kannada-MNIST/sample_submission.csv')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Lets understand the data\n"},{"metadata":{"trusted":true},"cell_type":"code","source":"train.info()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train.head()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Our training data set has 60000 rows amd 785 columns. First column is label and rest of the 784 colummns are pixel0 to pixel 783.The first column, called label, is the digit that was drawn by the user.The rest of the columns contain the pixel-values of the associated image."},{"metadata":{"trusted":true},"cell_type":"code","source":"test1.info()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"test1.head()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Our testing data set has 5000 rows and 785 columns.It does not have label column , but has first column as id column and rest of the columns are pixel0 to pixel783\n"},{"metadata":{},"cell_type":"markdown","source":"As per data description , each image is 28 pixels in height and 28 pixels in width, for a total of 784 pixels in total (so each row represent one image in data set). Each pixel has a single pixel-value associated with it, indicating the lightness or darkness of that pixel, with higher numbers meaning darker. This pixel-value is an integer between 0 and 255, inclusive.\n\n  \n\nEach pixel column in the training set has a name like pixel{x}, where x is an integer between 0 and 783, inclusive. To locate this pixel on the image, suppose that we have decomposed x as x = i * 28 + j, where i and j are integers between 0 and 27, inclusive. Then pixel{x} is located on row i and column j of a 28 x 28 matrix, (indexing by zero).\n\nFor example, pixel31 indicates the pixel that is in the fourth column from the left, and the second row from the top, as in the ascii-diagram below.\n\nVisually, if we omit the \"pixel\" prefix, the pixels make up the image like this:\n\n000 001 002 003 ----------------------------------------------------------------------------------------------------------------------------------------028\n029 030 031 032-----------------------------------------------------------------------------------------------------------------------------------------056\n"},{"metadata":{},"cell_type":"markdown","source":"Lets visualize distribution of training set"},{"metadata":{"trusted":true},"cell_type":"code","source":"\ny=train.label.value_counts()\nsns.barplot(y.index,y)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"We have equal counts for all the labels in training set\n"},{"metadata":{},"cell_type":"markdown","source":"Lets do data preperation ."},{"metadata":{"trusted":true},"cell_type":"markdown","source":"Lets seperate features and labels of our training set.And divide our data in taining and vaildation set"},{"metadata":{"trusted":true},"cell_type":"code","source":"X_train, X_valid, y_train, y_valid =\\\n    train_test_split(train.drop('label', axis=1).values, train.iloc[:,0].values, \n                     test_size=0.2, random_state=1, stratify=train.iloc[:,0])\n\ntest=test1.drop('id',axis=1)\nX_test = test.values\n\nprint('y_train Shape:', y_train.shape)\nprint('X_train Shape:', X_train.shape)\nprint('y_valid Shape:', y_valid.shape)\nprint('X_valid Shape:', X_valid.shape)\nprint('X_test Shape: ', X_test.shape)\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Next, we need to reshape our dataset inputs ( Xs_train and Xs_test) to the shape that our model expects when we train the model. The first number is the number of images (-1). Then comes the shape of each image (28x28). The last number is 1, which signifies that the images are greyscale.if 3 implies RGB mode"},{"metadata":{"trusted":true},"cell_type":"code","source":"X_train = X_train.reshape(-1, 28, 28, 1)\nX_valid = X_valid.reshape(-1, 28, 28, 1)\nX_test = X_test.reshape(-1, 28, 28, 1)\n\nprint('X_train Shape:', X_train.shape)\nprint('X_valid Shape:', X_valid.shape)\nprint('X_test Shape: ', X_test.shape)\n\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Plot the first image in the data set\n"},{"metadata":{"trusted":true},"cell_type":"code","source":"plt.imshow(X_train[0][:,:,0])","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Neural networks process inputs using small weight values, and inputs with large integer values can disrupt or slow down the learning process. we need to scale pixel values between 0 and 1"},{"metadata":{"trusted":true},"cell_type":"code","source":"Xs_train = X_train / 255\nXs_valid = X_valid / 255","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"lets encode our target value.The categorical value assignment can be done using sklearn’s LabelEncoder.Problem with label encoding is that it assumes higher the categorical value, better the category, WHICH WE DONT WANT!lets use Keras inbuild library to_categorical() for one-hot encoding"},{"metadata":{"trusted":true},"cell_type":"code","source":"y_train=to_categorical(y_train)\ny_valid=to_categorical(y_valid)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"[](http://keras.dhpit.com)\n"},{"metadata":{},"cell_type":"markdown","source":"I found these  easy to understand article \n\nhttps://www.edureka.co/blog/convolutional-neural-network/\n\nhttps://adeshpande3.github.io/adeshpande3.github.io/A-Beginner's-Guide-To-Understanding-Convolutional-Neural-Networks/\n\n\nhttp://keras.dhpit.com\n\n\nhttps://www.dlology.com/blog/one-simple-trick-to-train-keras-model-faster-with-batch-normalization/\n\nhttps://missinglink.ai/guides/keras/using-keras-flatten-operation-cnn-models-code-examples/\n\nhttps://keras.io/getting-started/sequential-model-guide/\n\nhttps://towardsdatascience.com/covolutional-neural-network-cb0883dd6529\n\nhttp://cs231n.github.io/convolutional-networks/"},{"metadata":{},"cell_type":"markdown","source":"#define model\n"},{"metadata":{"trusted":true},"cell_type":"markdown","source":"#The Sequential model is a linear stack of layers.\nmodel_kannada = Sequential()"},{"metadata":{"trusted":true},"cell_type":"code","source":"np.random.seed(1)\n\n#The Sequential model is a linear stack of layers.I used the Keras Sequential API, \n#where we add one layer at a time, starting from the input.\nfrom keras.models import Sequential\nKannad_cnn = Sequential()\n\n# adding layers to model\nKannad_cnn.add(Conv2D(32, (3,3), padding='same', input_shape=(28,28,1), activation = 'relu') )\nKannad_cnn.add(MaxPool2D(2,2))\nKannad_cnn.add(BatchNormalization())\n#32 is number of neurons(number of convolutional filters to use) in hidden layer\n#(3,3)kernel_size\n\nKannad_cnn.add(Conv2D(64, (3,3), padding='same', activation = 'relu') )\nKannad_cnn.add(MaxPool2D(2,2))\nKannad_cnn.add(BatchNormalization())\n\nKannad_cnn.add(Flatten())\nKannad_cnn.add(Dense(64, activation='relu'))\nKannad_cnn.add(BatchNormalization())\n#In between the convolutional layer and the fully connected layer, there is a ‘Flatten’ layer. Flattening transforms \n#a two-dimensional matrix of features into a vector that can be fed into a fully connected neural network classifier.\n\nKannad_cnn.add(Dense(10, activation='softmax'))\n\nKannad_cnn.summary()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"#compile Model"},{"metadata":{"trusted":true},"cell_type":"code","source":"optimizer=Adam(learning_rate=0.001,beta_1=0.9,beta_2=0.999,epsilon=1e-08)\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Optimizers update the weight parameters to minimize the loss function. Loss function acts as guides to the terrain telling optimizer if it is moving in the right direction to reach the bottom of the valley, the global minimum.There are many optimizers for example adam, SGD, GradientDescent, Adagrad, Adadelta , Adamax,rmsprops optimizer.  \n\n\nThe Adam paper suggests:\n\nGood default settings for the tested machine learning problems are alpha=0.001, beta1=0.9, beta2=0.999 and epsilon=10−8\n\nThe TensorFlow documentation suggests some tuning of epsilon:\n\nThe default value of 1e-8 for epsilon might not be a good default in general. For example, when training an Inception network on ImageNet a current good choice is 1.0 or 0.1.\n\nWe can see that the popular deep learning libraries generally use the default parameters recommended by the paper.\n\nTensorFlow: learning_rate=0.001, beta1=0.9, beta2=0.999, epsilon=1e-08.\nKeras: lr=0.001, beta_1=0.9, beta_2=0.999, epsilon=1e-08, decay=0.0."},{"metadata":{"trusted":true},"cell_type":"code","source":"Kannad_cnn.compile(optimizer=optimizer,loss=['categorical_crossentropy'],metrics=['accuracy'])","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"A loss function is used to optimize a machine learning algorithm. An accuracy metric is used to measure the algorithm’s performance (accuracy) in an interpretable way. \nLoss is defined as the difference between the predicted value by model and the true value. The most common loss function used in deep neural networks is cross-entropy. \nAccuracy is one of the metrics to measure the performance of  model It’s defined as:\nAcccuracy = No of correct predictions/Total no of predictions"},{"metadata":{"trusted":true},"cell_type":"code","source":"h1 = Kannad_cnn.fit(Xs_train, y_train, batch_size=64, epochs=20, verbose=1, \n                   validation_data = [Xs_valid, y_valid]) ","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":""},{"metadata":{},"cell_type":"markdown","source":"A batch size is a small part of the dataset.\nIterations is the number of batches of data the algorithm has seen (or simply the number of passes the algorithm has done on the dataset). \nEpochs is the number of times a learning algorithm sees the complete dataset.\n\nBy setting verbose 0, 1 or 2 you just say how do you want to 'see' the training progress for each epoch.\nverbose=0 will show you nothing\nverbose=1 will show you an animated progress bar\nverbose=2 will just mention the number of epoch like this\n"},{"metadata":{},"cell_type":"markdown","source":"What Batch size to choose?\n\nToo large a mini-batch size usually leads to a lower accuracy!\n\nThere are two notions of speed:\n\nComputational speed\nSpeed of convergence of an algorithm\nComputational speed is simply the speed of performing numerical calculations in hardware. As you said, it is usually higher with a larger mini-batch size. \n\nthe other notion of speed  tells us how quickly our algorithm converges (loss and accuracy we want)\n\nAnother \"catch\" - When using a smaller batch size, calculation of the error has more noise than when we use a larger batch size. however thats not bad as that noise can help the algorithm jump out of a bad local minimum and have more chance of finding either a better local minimum, or hopefully the global minimum.In general, batch size of 32 is a good starting point, and we should also try with 64, 128, and 256. Other values (lower or higher) may be fine for some data sets, but the given range is generally the best to start experimenting with. Though, under 32, it might get too slow because of significantly lower computational speed, because of not exploiting vectorization to the full extent. If you get an \"out of memory\" error, you should try reducing the mini-batch size anyway.\n\nTo conclude,a smaller mini-batch size (not too small) usually leads not only to a smaller number of iterations of a training algorithm, than a large batch size, but also to a higher accuracy overall\n"},{"metadata":{"trusted":true},"cell_type":"code","source":"plt.figure(figsize=[12,6])\nplt.subplot(1,2,1)\nplt.plot(range(1,21), h1.history['accuracy'], label='Training Accuracy')\nplt.plot(range(1,21), h1.history['val_accuracy'], label='Validation Accuracy')\nplt.xlabel('Epoch')\nplt.legend()\n\nplt.subplot(1,2,2)\nplt.plot(range(1,21), h1.history['loss'], label='Training Loss')\nplt.plot(range(1,21), h1.history['val_loss'], label='Validation Loss')\nplt.xlabel('Epoch')\nplt.legend()\n\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"Y_test = Kannad_cnn.predict_classes(X_test)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"my_submission = pd.DataFrame({'id' :test1. iloc[:,0], 'label':Y_test })\n\nprint(my_submission.iloc[:3])\n\nmy_submission.to_csv('submission.csv', index=False)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":""}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}