{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# List of Content\n\n1.[Data Import](#dataImport)\n\n2.[Data Transformation](#dataTransformation)\n\n3.[Model Definition](#modelDefinition)\n\n4.[Model Training](#modelTraining)\n\n5.[Training Results](#results)","metadata":{}},{"cell_type":"markdown","source":"Importing necesairy libraries ","metadata":{"execution":{"iopub.status.busy":"2022-10-27T13:29:08.879601Z","iopub.execute_input":"2022-10-27T13:29:08.880400Z","iopub.status.idle":"2022-10-27T13:29:08.886726Z","shell.execute_reply.started":"2022-10-27T13:29:08.880364Z","shell.execute_reply":"2022-10-27T13:29:08.885075Z"}}},{"cell_type":"code","source":"#library import\nimport pandas as pd\nimport numpy as np\nimport matplotlib.pyplot as plt","metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","execution":{"iopub.status.busy":"2022-10-27T13:29:08.845397Z","iopub.execute_input":"2022-10-27T13:29:08.846424Z","iopub.status.idle":"2022-10-27T13:29:08.877244Z","shell.execute_reply.started":"2022-10-27T13:29:08.846311Z","shell.execute_reply":"2022-10-27T13:29:08.875983Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"<a id=\"dataImport\"></a>\n# Data Import\n\n\nFirst step is data import. Pictures from dataset are saved in .csv format. I used pd.read_csv i consider it the most straight forward way to handle this task. Index is saved in column one and label in last column. There are also 784 pixel values stored  in between so data set consist 786 columns in total. Train data set consist 42000 entities on the other hand test data set have 28000 entities. ","metadata":{}},{"cell_type":"code","source":"#training data set import\ntrain = pd.read_csv(r'../input/digit-recognizer/train.csv')\n#test data set import\ntest = pd.read_csv(r'../input/digit-recognizer/test.csv')","metadata":{"execution":{"iopub.status.busy":"2022-10-25T11:49:01.763113Z","iopub.execute_input":"2022-10-25T11:49:01.763539Z","iopub.status.idle":"2022-10-25T11:49:04.453300Z","shell.execute_reply.started":"2022-10-25T11:49:01.763485Z","shell.execute_reply":"2022-10-25T11:49:04.452217Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"<a id=\"dataTransformation\"></a>\n# Data Transformation\n\nAfter importing dataset I transformed DataFrame to Numpy arrays. I also checked the data cohesion to make sure that import went all right.","metadata":{}},{"cell_type":"code","source":"#transforming data from pandas data frame to numpy and splitting them to y and x\n#labels\ny = train['label'].to_numpy()\n#training independet variables\nxTrain = train.drop('label',axis=1).to_numpy()\n#test independent variable\nxTest = test.to_numpy()\n\n#data cohesion checking\n#y lenght check\nprint(\"\\nNumber of labels :\" + str(len(y)))\n#x col numbers\nprint(\"\\nNumber of individuals in train data set :\" + str(len(xTrain[:,1])))\n#x rows numbers\nprint(\"\\nNumber of individual features in train data set  :\" + str(len(xTrain[1,:])))\n#test columns number\nprint(\"\\nNumber of individuals in test data set :\" + str(len(xTest[:,1])))\n#test rows number\nprint(\"\\nNumber of individual features in train data set :\" + str(len(xTest[1,:]))+'\\n')","metadata":{"execution":{"iopub.status.busy":"2022-10-25T11:49:04.455492Z","iopub.execute_input":"2022-10-25T11:49:04.456261Z","iopub.status.idle":"2022-10-25T11:49:04.504756Z","shell.execute_reply.started":"2022-10-25T11:49:04.456210Z","shell.execute_reply":"2022-10-25T11:49:04.503367Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Printing out some pictures.","metadata":{}},{"cell_type":"code","source":"#ploting images no. 0,1,3\nplt.imshow(xTrain[2,:].reshape(28,28),cmap=\"gray\")\nplt.show()\nplt.imshow(xTrain[1,:].reshape(28,28),cmap=\"gray\")\nplt.show()\nplt.imshow(xTrain[3,:].reshape(28,28),cmap=\"gray\")\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2022-10-25T11:49:04.506353Z","iopub.execute_input":"2022-10-25T11:49:04.506698Z","iopub.status.idle":"2022-10-25T11:49:04.856709Z","shell.execute_reply.started":"2022-10-25T11:49:04.506673Z","shell.execute_reply":"2022-10-25T11:49:04.855642Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Saving data to external files is a next step. I did this because originally i was working in PyCharm and i was using two files - one for import and one for prediction.","metadata":{}},{"cell_type":"code","source":"#saving numpy arrays to external files\nnp.save(\"xTrain.npy\", xTrain)\nnp.save(\"xTest.npy\", xTest)\nnp.save(\"labels.npy\", y)","metadata":{"execution":{"iopub.status.busy":"2022-10-25T11:49:04.859101Z","iopub.execute_input":"2022-10-25T11:49:04.860056Z","iopub.status.idle":"2022-10-25T11:49:05.179587Z","shell.execute_reply.started":"2022-10-25T11:49:04.860025Z","shell.execute_reply":"2022-10-25T11:49:05.178431Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Moving on to next part which consist prediction I feel obliged to explain why part of import and data transformation was done in import file but second part was performed in prediction file. This code is part of my enginnering thesis where I compared CNN with classic neural net in picture classification tasks so imported data set was used for CNN and classic neural net which have diffrent input size. Due to this reason I made common part of data transformation in one file and parts that are specific for CNN or classic neural net in prediction file.","metadata":{}},{"cell_type":"code","source":"import numpy as np\nimport pandas as pd\nimport torch\nimport matplotlib.pyplot as plt\nfrom torch import nn, optim\nfrom torch.utils.data import TensorDataset,DataLoader\nfrom sklearn.model_selection import train_test_split\nimport time\nimport torch.nn.functional as F","metadata":{"execution":{"iopub.status.busy":"2022-10-25T11:49:05.180718Z","iopub.execute_input":"2022-10-25T11:49:05.180958Z","iopub.status.idle":"2022-10-25T11:49:05.185474Z","shell.execute_reply.started":"2022-10-25T11:49:05.180935Z","shell.execute_reply":"2022-10-25T11:49:05.184840Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Checking if CUDA is available and setting device.","metadata":{}},{"cell_type":"code","source":"if torch.cuda.is_available():\n    print('Cuda')\n    dev = \"cuda\"\nelse:\n    print('CPU')\n    dev = \"cpu\"\ndevice = torch.device(dev)","metadata":{"execution":{"iopub.status.busy":"2022-10-25T11:49:05.186671Z","iopub.execute_input":"2022-10-25T11:49:05.186890Z","iopub.status.idle":"2022-10-25T11:49:05.202093Z","shell.execute_reply.started":"2022-10-25T11:49:05.186868Z","shell.execute_reply":"2022-10-25T11:49:05.200988Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Importing Numpy arrays from files and further data transformation.","metadata":{}},{"cell_type":"code","source":"#import data\nxTrain = np.load('xTrain.npy')\ny = np.load('labels.npy')\nxTestFinal = np.load('xTest.npy')\n\n#train test split\nxTrain,xTest,yTrain,yTest = train_test_split(xTrain,y,test_size=0.2)\n\n#converting to tensor\nxTestFinal = torch.from_numpy(xTestFinal.reshape(xTestFinal.shape[0],1,28,28))\nxTrain = torch.from_numpy(xTrain.reshape(xTrain.shape[0],1,28,28))\nyTrain = torch.from_numpy(yTrain)\nxTest = torch.from_numpy(xTest.reshape(xTest.shape[0],1,28,28))\nyTest = torch.from_numpy(yTest)\n\n#changeing type due to errors\nxTestFinal = xTestFinal.type(torch.float32)\nxTrain = xTrain.type(torch.float32)\nxTest = xTest.type(torch.float32)\n\n#sending tensors to device\nxTestFinal = xTestFinal.to(device)\nxTrain = xTrain.to(device)\nyTrain = yTrain.to(device)\nxTest = xTest.to(device)\nyTest = yTest.to(device)\n\n#creating data set\ntrainDataSet = TensorDataset(xTrain,yTrain)\n#definig batch size\nbatchSize = 5000\n#creating data loader\ndataLoad = DataLoader(trainDataSet, batchSize, shuffle='True')","metadata":{"execution":{"iopub.status.busy":"2022-10-25T11:49:05.203431Z","iopub.execute_input":"2022-10-25T11:49:05.203678Z","iopub.status.idle":"2022-10-25T11:49:05.599454Z","shell.execute_reply.started":"2022-10-25T11:49:05.203649Z","shell.execute_reply":"2022-10-25T11:49:05.598161Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"<a id=\"modelDefinition\"></a>\n# Model Definition\n\nDefining neural net parameters. Purpose of offset and numOfLayers parameters will be explained later. Here numOfEpoch = 4 due to omiting Kaggle Notebook computation. ","metadata":{}},{"cell_type":"code","source":"#neural net parameters\n#offset\nz=0\n#number of layers\nnumOfLayers = 4\n#defining input, hidden layers and output size, number of epochs\ninputSize = 147\nhiddenLayersSizes = [512,256,128,64,36,16]\noutputSize = 10\nnumOfEpochs = 4\n#learning rate\nlearningRate = 0.001\n#loss function\nlossFunction = nn.CrossEntropyLoss","metadata":{"execution":{"iopub.status.busy":"2022-10-25T11:49:05.600567Z","iopub.execute_input":"2022-10-25T11:49:05.600796Z","iopub.status.idle":"2022-10-25T11:49:05.606815Z","shell.execute_reply.started":"2022-10-25T11:49:05.600773Z","shell.execute_reply":"2022-10-25T11:49:05.605741Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Model definition. Here you cane see why i defined offset and numOfLayers parameters before. It allowed me to quickly change neural net hiperparameters which will normally take a lot of time while testing more neural net architecture concepts. You can define layer size list and then slide through it changing just one parameter. In term of hidden layers witch same size it's not that usefull but when u have diffrent layer sizes which are dependent it's reallu useful and time saving feature.","metadata":{}},{"cell_type":"code","source":"#defining class model\nclass Model(nn.Module):\n    def __init__(self,inputSize, hiddenLayersSizes, outputSize):\n        super().__init__()\n        self.conv1 = nn.Conv2d(in_channels=1,out_channels=2,kernel_size=(3,3),stride=(1,1),padding=(1,1))\n        self.conv2 = nn.Conv2d(in_channels=2, out_channels=3, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))\n        self.pool1 = nn.MaxPool2d(kernel_size=(2,2),stride=(2,2),padding=(0,0))\n        self.layer1 = nn.Linear(inputSize,hiddenLayersSizes[0+z])\n        self.layer2 = nn.Linear(hiddenLayersSizes[0+z], hiddenLayersSizes[1+z])\n        self.layer3 = nn.Linear(hiddenLayersSizes[1+z], hiddenLayersSizes[2+z])\n        self.layer4 = nn.Linear(hiddenLayersSizes[2+z], hiddenLayersSizes[3+z])\n        self.layer5 = nn.Linear(hiddenLayersSizes[(numOfLayers-1)+z], outputSize)\n    def forward(self, x):\n        x = F.relu(self.conv1(x))\n        x = self.pool1(x)\n        x = F.relu(self.conv2(x))\n        x = self.pool1(x)\n        x = torch.flatten(x,1)\n        x = self.layer1(x)\n        x = nn.LeakyReLU()(x)\n        x = self.layer2(x)\n        x = nn.Sigmoid()(x)\n        x = self.layer3(x)\n        x = nn.Tanh()(x)\n        x = self.layer4(x)\n        x = nn.Sigmoid()(x)\n        x = self.layer5(x)\n        x = nn.LogSoftmax(dim=1)(x)\n        return x\n","metadata":{"execution":{"iopub.status.busy":"2022-10-25T11:49:05.607985Z","iopub.execute_input":"2022-10-25T11:49:05.608322Z","iopub.status.idle":"2022-10-25T11:49:05.620926Z","shell.execute_reply.started":"2022-10-25T11:49:05.608290Z","shell.execute_reply":"2022-10-25T11:49:05.619701Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"<a id=\"modelTraining\"></a>\n# Model Training\n\nWhen model is defined it can be train but it's worth to remember that learning process is nondeterministic so neural net may stuck in local minimum sometimes and significantly lower it's performance. While comparing diffrent models this characteristic may affect results so training process repetition is recomended. Number of repetitions may vary depend on time we have, computing power of our PC and size of neural net. I made ten repetitions for each neural net and calculate mean from results. Here repetition is set to 2 due to omiting Kaggle Notebook computation. ","metadata":{}},{"cell_type":"code","source":"#assigning model to variable\nmodel = Model(inputSize,hiddenLayersSizes,outputSize)\nmodel = model.to(device)\n#optimizer\noptimizer = torch.optim.Adam(model.parameters(), lr = learningRate)\n#num of repetition\nrepetition = 2\n#repetition results saving\nrepTable = [0] * repetition\nrepEpochTable = [0] * repetition\n#repetition loop\nfor x in range(repetition):\n    #reseting model parameters (weights)\n    for layer in model.children():\n        if hasattr(layer, 'reset_parameters'):\n            layer.reset_parameters()\n    #reseting repetition best and bestEpoch beforenew repetition\n    best = 0\n    bestEpoch = 0\n    #tables for over/under fitting check\n    lossHistory = [0] * numOfEpochs\n    accTrainHistory = [0] * numOfEpochs\n    accTestHistory = [0] * numOfEpochs\n    timeHist = [0] * numOfEpochs\n    #learning loop\n    for epoch in range(numOfEpochs):\n        # time measure start\n        start = time.time()\n        #batch training loop\n        for xBatch, yBatch in dataLoad:\n            #preiction\n            pred = model(xBatch)\n            #loss calculation\n            loss = lossFunction()(pred, yBatch)\n            #calculating dloss/dx\n            loss.backward()\n            #optimization step\n            optimizer.step()\n            optimizer.zero_grad()\n            #lossHistory saving\n            lossHistory[epoch] += loss.item()*yBatch.size(0)\n        # time measure stop\n        stop = time.time()\n        #time Hist save\n        timeHist[epoch] += (stop-start) * (10**3)\n        #prediction on train data set\n        predTrain = model(xTrain)\n        accTrain = (torch.argmax(predTrain, dim=1)==yTrain).float().mean()\n        lossHistory[epoch] /= len(dataLoad.dataset)\n        #prediction on test data set\n        predTest = model(xTest)\n        accTest = (torch.argmax(predTest, dim=1)==yTest).float().mean()\n        accTrainHistory[epoch] = accTrain.cpu()\n        accTestHistory[epoch] = accTest.cpu()\n        #checking if it's best result in this repetition\n        if accTest > best:\n            best = accTest\n            bestEpoch = epoch\n        #printing out results\n        print(\"Learning Epoch: \" + str(epoch))\n        print(\"Repetition: \" + str(x))\n        print(\"Train data set accuracy: \" + str(accTrain))\n        print(\"Test data set accuracy: \" + str(accTest))\n        print(\"Time of epoch:    \" + str((stop-start) * (10**3))+\" ms\\n\")\n    #saving results to repetition table\n    repTable[x] = float(best.cpu())\n    repEpochTable[x] = bestEpoch","metadata":{"execution":{"iopub.status.busy":"2022-10-25T11:49:05.623684Z","iopub.execute_input":"2022-10-25T11:49:05.623979Z","iopub.status.idle":"2022-10-25T11:49:42.619147Z","shell.execute_reply.started":"2022-10-25T11:49:05.623955Z","shell.execute_reply":"2022-10-25T11:49:42.618292Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"<a id=\"results\"></a>\n# Training Results\n\nLast step is printing out the final results.","metadata":{}},{"cell_type":"code","source":"print(repTable)\nprint(repEpochTable)\n\nrepMean = sum(repTable)/len(repTable)\nprint(\"Acc mean : \" + str(repMean))\n\n#wyświtlanie rezultatu\nplt.plot(accTrainHistory,lw=3)\nplt.plot(accTestHistory,lw=3)\nplt.title('Train Test ACC comparison')\nplt.xlabel('Epoch')\nplt.ylabel('Accuracy')\nplt.legend(['Train DS','Test DS'])\nplt.show()\n\n#accuracy na zbiorze testowym\npred = model(xTest)\ncorr = (torch.argmax(pred,dim=1)==yTest).float()\nacc = corr.mean()\nprint(\"Accuracy on test data set is equal to :  \" + str(acc))\nprint(\"Training time is equal to :   \" + str(sum(timeHist)) + \" ms\\n\")\n","metadata":{"execution":{"iopub.status.busy":"2022-10-25T11:49:42.620318Z","iopub.execute_input":"2022-10-25T11:49:42.621131Z","iopub.status.idle":"2022-10-25T11:49:43.048607Z","shell.execute_reply.started":"2022-10-25T11:49:42.621097Z","shell.execute_reply":"2022-10-25T11:49:43.046711Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Finally I achived accuracy equal to 0.98189 (neural net configuration: conv(1,3,(3,3),(1,1)(1,1)) - conv(3,5,(3,3),(1,1)(1,1)) - 512 * TanH - 256 * Sigmoid - 128 * LeakyReLu - 64 * TanH) so not bad but not good either. In futere I will try to improve this result by using One versus All instead of Multiclass Classification. ","metadata":{}}]}