{"cells":[{"metadata":{},"cell_type":"markdown","source":" \n"},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"code","source":"import numpy as np # linear algebra\nimport pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\nfrom matplotlib import pyplot as plt\nimport torch ","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Tensors\nTorch use tensors to represent data in a very efficent way, you can perfom several mathematical operations efficiently and fast"},{"metadata":{"trusted":true},"cell_type":"code","source":"#we create some random input data\ninputs=torch.randn(1,5)\nprint(inputs,\"\\n\",inputs.shape)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"We can build neural networks perfoming the corresponding mathematical operations between tensors for example let's say we want to create a single layer network with 3 hidden units (neurons) and 4 inputs so\nin this case our inputs X should be multiplied by the weight matrix W of the layer \n$$\n\\begin{equation*}\n{X} \\times {W}= \\begin{bmatrix}\n\\mathbf{x}_1 & \\mathbf{x}_2 & \\mathbf{x}_3 & \\mathbf{x}_4\\\\\n\\end{bmatrix}\n\\times\n\\begin{vmatrix}\n\\mathbf{h}_1 & \\mathbf{h}_2 & \\mathbf{h}_3 \\\\\n{w}_{11}&{w}_{21}&{w}_{31}\\\\\n{w}_{12}&{w}_{22}&{w}_{32}\\\\\n{w}_{13}&{w}_{23}&{w}_{33}\\\\\n{w}_{14}&{w}_{24}&{w}_{34}\n\\end{vmatrix}\n\\end{equation*}\n$$\nWhere each column of W represent the 4 weights of a hidden unit $$h_{1-3}$$ corresponding to each input $$x_{1-4}$$ \n\nW will have as many columns as neurons and as many rows as inputs,we are performing the dot product of each column and the inputs to get a score for each neuron,those scores will be stored in a hidden vector h\n\nthen we will sum the bias to the hidden state vector h = X * W \n$$\n\\begin{bmatrix}\n\\mathbf{h}_1 & \\mathbf{h}_2 & \\mathbf{h}_3 \\\\\n\\end{bmatrix}\n+\n\\begin{bmatrix}\n\\mathbf{b}_1 & \\mathbf{b}_2 & \\mathbf{b}_3 \\\\\n\\end{bmatrix}\n$$\nat the end our network will be \n$$\n\\mathbf{Y}_{prediction}=W\\times X + b\n$$\n\n"},{"metadata":{},"cell_type":"markdown","source":"Implementing the basic dense layer"},{"metadata":{"trusted":true},"cell_type":"code","source":"def no_activation(x):\n    return x","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def sigmoid(x):\n    return 1/(1+torch.exp(x))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"\nclass dense():\n    def __init__(self,input_d,units):\n        self.weights=torch.randn((input_d,units))\n        self.bias=torch.randn((1,units))\n    def __call__(self,inputs,activation=no_activation):\n        return activation(torch.mm(inputs,self.weights)+self.bias)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Creating a simple model using 1 hidden layer with two neurons and and output layer with one neuron "},{"metadata":{"trusted":true},"cell_type":"code","source":"class model():\n    def __init__(self,input_d):\n        self.layer1=dense(input_d,2)\n        self.layer2=dense(self.layer1.weights.shape[1],1)\n    def __call__(self,inputs):\n        h1= self.layer1(inputs,activation=sigmoid)\n        h2= self.layer2(h1,activation=sigmoid)\n        return h2","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"\ntorch.manual_seed(7)\nfeatures = torch.randn((1, 3))\nfirstnet=model(features.shape[1])\nfirstnet(features)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"we can create tensors from numpy arrays "},{"metadata":{"trusted":true},"cell_type":"code","source":"a=np.random.rand(4,3)\na","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"in this case the new tensor will share mamoery whit the numpy array so wherever change we made in any of them will affect each other"},{"metadata":{"trusted":true},"cell_type":"code","source":"b=torch.from_numpy(a)\nb","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"b.mul_(2)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"a","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"We can also create new tensor from numpy arrays without sharing memory between them, so this new tensor will have its own memory, and the changes on it will not affect the numpy array"},{"metadata":{"trusted":true},"cell_type":"code","source":"b_nosharedmemory = torch.Tensor(a)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"b_nosharedmemory","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"b_nosharedmemory.mul_(2)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"a","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"b_nosharedmemory","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Lets create a neural net two solve the mnist from of predict the number appearing on an image"},{"metadata":{},"cell_type":"markdown","source":"We defined a transform to normalize the data , we are going to tranform each image to a vectorand also normalize it in a range of [-1,1] (standart normal distribution) image = (image - mean) / std the function normalize recieves two parameters mean and std in our case mean=0.5 and std=0.5 this normalization provides a better workfield to gradient descent since in the end we want to capture the distribution of the data its easier to capture a normal distribution 0 mean and equal variance"},{"metadata":{"trusted":true},"cell_type":"code","source":"from torchvision import datasets,transforms\n\n\n\ntransform = transforms.Compose([transforms.ToTensor(),\n                              transforms.Normalize((0.5,), (0.5,)),\n                              ])\n# Download and load the data\ndata= datasets.MNIST('~/.pytorch/MNIST_data/', download=True, train=True, transform=transform)\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"#this dataloader will split the data in batches of 64 samples and will shuffle them \ndataloader = torch.utils.data.DataLoader(data, batch_size=64, shuffle=True)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"\n\nAs we can see the data loader iterates giving us a batch of 64  images  each time"},{"metadata":{"trusted":true},"cell_type":"code","source":"dataiter = iter(dataloader)\nimages, labels = dataiter.next()\nprint(type(images))\nprint(images.shape)\nprint(labels.shape)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"plt.imshow(images[1].numpy().squeeze(), cmap='Greys_r');","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"[](http://)"},{"metadata":{},"cell_type":"markdown","source":"we will apply exp to all the elements in our data, then we know each row of our data is a prediction, so we should sum by rows (in dim=1) and also in order to divide 10 values by 10 values we should transpose that summaroty of the rows two divide the the row by a value "},{"metadata":{"trusted":true},"cell_type":"code","source":"def softmax(x):\n    return torch.exp(x)/torch.sum(torch.exp(x), dim=1).view(-1, 1)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"class model(object):\n    def __init__(self,input_d):\n        self.layer1=dense(input_d,256)\n        self.outputlayer=dense(self.layer1.weights.shape[1],10)\n    def __call__(self,inputs):\n        h1=self.layer1(inputs,activation=sigmoid)\n        out=self.outputlayer(h1,activation=softmax)\n        return out","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"inputs = images.view(images.shape[0], -1)\ninputs.shape","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"mnist_net=model(inputs.shape[1])\noutputs=mnist_net(inputs)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"print(outputs.shape)\nprint(torch.sum(outputs,dim=1))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Using the nn module to create models"},{"metadata":{"trusted":true},"cell_type":"code","source":"import torch.nn as nn\nclass model(torch.nn.Module):\n    def __init__(self,input_d):\n        super().__init__()\n        self.layer1=nn.Linear(784,256)\n        self.out_Layer=nn.Linear(256,10)\n    def __call__(self,inputs):\n        h1=torch.sigmoid(self.layer1(inputs))\n        out=torch.softmax(self.out_Layer(h1),dim=1) #we will calculate the softamx by row \n        return out\n    ","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"torchmodel=model(inputs.shape[1])\ntorchmodel","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"outputs=torchmodel(inputs)\noutputs.shape","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"torch.sum(outputs,dim=1)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Lets build a more complex architecture also using other activation functions such as ReLU"},{"metadata":{"trusted":true},"cell_type":"code","source":"import torch.nn.functional as F\nclass Model(torch.nn.Module):\n    def __init__(self,input_d):\n        super().__init__()\n        self.layer1=nn.Linear(input_d,128)\n        self.layer2=nn.Linear(self.layer1.out_features,64)\n        self.out_layer=nn.Linear(self.layer2.out_features,10)\n    def __call__(self,inputs):\n        h1=F.relu(self.layer1(inputs))\n        h2=F.relu(self.layer2(h1))\n        out=torch.softmax(self.out_layer(h2),dim=1)\n        return out\n        \n        \n    ","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"relumodel=Model(inputs.shape[1])\nrelumodel","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"torch.sum(relumodel(inputs),dim=1)\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Let's build a model and calculate the loss noticed that we calculate the loss with the output scores and not whit the probabilities coming from the activation function (softmax)"},{"metadata":{"trusted":true},"cell_type":"code","source":"\n# Build a feed-forward network\nmodel = nn.Sequential(nn.Linear(784, 128),\n                      nn.ReLU(),\n                      nn.Linear(128, 64),\n                      nn.ReLU(),\n                      nn.Linear(64, 10))\n\n# Define the loss\ncriterion = nn.CrossEntropyLoss()\n\n# Get our data\nimages, labels = next(iter(dataloader))\n# Flatten images\nimages = images.view(images.shape[0], -1)\n\n# Forward pass, get our logits\nlogits = model(images)\n# Calculate the loss with the logits and the labels\nloss = criterion(logits, labels)\n\nprint(loss)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":" it's more convenient to build the model with a log-softmax output ,then you can get the actual probabilities by taking the exponential torch.exp(output). With a log-softmax output, we should use  to use the negative log likelihood loss\n \n \nThe softmax function returns probabilities between [0, 1].\nThe log of these probabilities returns values between [-inf, 0], since log(0) = -inf and log(1) = 0.\nThat is why the order won’t change.\n\nHowever, you should use the NLLLoss with a log_softmax output\nor CrossEntropyLoss with logits if you prefer not to add an extra log_softmax layer into your model."},{"metadata":{"trusted":true},"cell_type":"code","source":"\n# TODO: Build a feed-forward network\nmodel = nn.Sequential(nn.Linear(images.shape[1],250),\n                      nn.ReLU(),\n                      nn.Linear(250,120),\n                      nn.ReLU(),\n                      nn.Linear(120,10),\n                      nn.LogSoftmax(dim=1) \n                      #because we want to softmax by row (each row contains the score of a input), \n                      #we have 64 rows because our batch is of 64 samples\n                      )\n\n# TODO: Define the loss\ncriterion = nn.NLLLoss()\n\n\n# Forward pass, get our logits\nlogits = model(images)\nprint(logits.shape)\nprint(torch.sum(torch.exp(logits),dim=1))\n# Calculate the loss with the logits and the labels\nloss = criterion(logits, labels)\n\nprint(loss)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"![](http://)"},{"metadata":{"trusted":true},"cell_type":"code","source":"images.shape","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Autograd\n\nAutograd allow us to track the operations done on a tensor and when you tell it to do a backward pass , it will go backwards  trought  each of these operations and calculate the gradients whit resect to the input parameters "},{"metadata":{},"cell_type":"markdown","source":"In torch you need to indicate that you require to calculate the gradients of a tensor"},{"metadata":{"trusted":true},"cell_type":"code","source":"tensor=torch.randn(2,2,requires_grad=True)\ntensor","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"You also can turn the grads of and turn in on again on a tensor"},{"metadata":{"trusted":true},"cell_type":"code","source":"tensor.requires_grad=False\nprint(\"grads off:\\n\",tensor)\ntensor.requires_grad=True\nprint(\"grads on:\\n\",tensor)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"you can also use contexts to control the grads tracking"},{"metadata":{"trusted":true},"cell_type":"code","source":"with torch.no_grad():\n    y=tensor**2\ny.requires_grad","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"print(tensor)\nwith torch.no_grad():\n    tensor=tensor**2\nprint(tensor)\ntensor.requires_grad=True","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"in this case we see the torch wil track the operations in the tensor since we stablish requieres_grad as True\nwe can also globally enable or disable grads globally using \n```python\ntorch.set_grad_enabled(True|false)\n```"},{"metadata":{"trusted":true},"cell_type":"code","source":"tensor=torch.sqrt(tensor)\ntensor","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"then let's say we have a function \n$$\ny=x^2\n$$\n\nand also\n$$\nz = \\left[\\frac{1}{n}\\sum_i^n y_i\\right]\n$$\n\nso then we want to calculate the gradients of z with respect x\n\n$$\n\\frac{\\partial z}{\\partial x} = \\frac{\\partial}{\\partial x}\\left[\\frac{1}{n}\\sum_i^n x_i^2\\right] = \\frac{x}{2}\n$$\n\nwe can easily perfom this with autograd in pytorch"},{"metadata":{"trusted":true},"cell_type":"code","source":"x=torch.randn(2,2,requires_grad=True)\nprint(f\"x: \\n{x}\")\ny=x**2\nprint(f\"y: \\n{y}\")\nz=torch.mean(y)\nprint(f\"z: \\n{z}\")\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"So then to perfom the gradients of z with respect x we must perfom z.backward() and check the result in x.grad, we know that the derivative with respect of x of z are x/2 so lets check if are the same"},{"metadata":{"trusted":true},"cell_type":"code","source":"z.backward()\nprint(f\"gradients of z with respect x: \\n{x.grad}\")\nprint(f\"x/2: \\n{x/2}\")\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Optimizers\n\nThen  we know that in this way we can calculate the gradients of the loss and then update our parameters.\nPytorch provides a set of optimizers that made this process automatically, such as stochastic gradient descent or adam."},{"metadata":{"trusted":true},"cell_type":"code","source":"from torch import optim\n\nOptimizer = optim.SGD(model.parameters(),lr=0.01)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Then using this optimizer we can actually train our network:\nfirst we need two perfom a feedforward pass, calculate the loss with those values, then backpropagate to calculate our gradients (backward pass) and finally update our parameters(weights and biases)\n"},{"metadata":{"trusted":true},"cell_type":"code","source":"# Get our data\nimages, labels = next(iter(dataloader))\n# Flatten images\nimages = images.view(images.shape[0], -1)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"Optimizer.zero_grad()\noutput=model(images)\nloss=criterion(output,labels) #this is not reciprocable take care\nloss.backward()#we calculate the gradients of the loss with respect the parameters\nprint(f\"Weights before optimize(layer1): {model[0].weight}\")\nprint(f\"Example gradients(layer1): {model[0].weight.grad}\")","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Then after calculate the loss and it's gradients we use our optimizer to update te parameters \nit's important to clean the gradients of the Optimizer cause they maybe acumulated (clean it before calulate the gradients of the loss) if you clean it here you will delete the new gradients"},{"metadata":{"trusted":true},"cell_type":"code","source":"Optimizer.step()\nprint(f\"Updated Weights after optimize(layer1): {model[0].weight}\")","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Ok now lets put all togheter below, first lets import our data,define our model, define our loss, create and optimizer and code the training loop"},{"metadata":{"trusted":true},"cell_type":"code","source":"from torchvision import datasets,transforms\n\n\n\ntransform = transforms.Compose([transforms.ToTensor(),\n                              transforms.Normalize((0.5,), (0.5,)),\n                              ])\n# Download and load the data\ndata= datasets.MNIST('~/.pytorch/MNIST_data/', download=True, train=True, transform=transform)\ndataloader = torch.utils.data.DataLoader(data, batch_size=64, shuffle=True)\ndataloader.dataset.train_data.shape","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"\n# TODO: Build a feed-forward network\nmodel = nn.Sequential(nn.Linear(784,250),\n                      nn.ReLU(),\n                      nn.Linear(250,120),\n                      nn.ReLU(),\n                      nn.Linear(120,10),\n                      nn.LogSoftmax(dim=1) \n                      #because we want to softmax by row (each row contains the score of a input), \n                      #we have 64 rows because our batch is of 64 samples\n                      )","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"criterion = nn.NLLLoss()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def train(model,n_epochs,criterion):\n    Optimizer = optim.SGD(model.parameters(),lr=0.003)\n    for epoch in range(n_epochs):#epochs\n        epoch_loss=0\n        for images,labels in dataloader: #batches\n            Optimizer.zero_grad()\n            images=images.view(images.shape[0], -1)\n            predictions=model(images)\n            loss=criterion(predictions,labels)\n            loss.backward()\n            Optimizer.step()\n            epoch_loss+=loss.item()#we want just the number inside the tensor\n        print(f\"Epoch: {epoch} Training loss: {epoch_loss/len(dataloader)}\")        ","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train(model,5,criterion)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def view_classify(img, ps, version=\"MNIST\"):\n    ''' Function for viewing an image and it's predicted classes.\n    '''\n    ps = ps.data.numpy().squeeze()\n\n    fig, (ax1, ax2) = plt.subplots(figsize=(6,9), ncols=2)\n    ax1.imshow(img.resize_(1, 28, 28).numpy().squeeze())\n    ax1.axis('off')\n    ax2.barh(np.arange(10), ps)\n    ax2.set_aspect(0.1)\n    ax2.set_yticks(np.arange(10))\n    if version == \"MNIST\":\n        ax2.set_yticklabels(np.arange(10))\n    elif version == \"Fashion\":\n        ax2.set_yticklabels(['T-shirt/top',\n                            'Trouser',\n                            'Pullover',\n                            'Dress',\n                            'Coat',\n                            'Sandal',\n                            'Shirt',\n                            'Sneaker',\n                            'Bag',\n                            'Ankle Boot'], size='small');\n    ax2.set_title('Class Probability')\n    ax2.set_xlim(0, 1.1)\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"%matplotlib inline\nimport helper\n\nimages, labels = next(iter(dataloader))\n\nimg = images[0].view(1, 784)\n# Turn off gradients to speed up this part\nwith torch.no_grad():\n    logps = model(img)\n\n# Output of the network are log-probabilities, need to take exponential for probabilities\nps = torch.exp(logps)\nview_classify(img.view(1, 28, 28), ps)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"So then let's move into a complex model, so we are trying to predict over 10 different classes of clothes where also there is more vaiance in the data"},{"metadata":{"trusted":true},"cell_type":"code","source":"import torch\nfrom torchvision import datasets, transforms\nimport helper\n\n# Define a transform to normalize the data\ntransform = transforms.Compose([transforms.ToTensor(),\n                                transforms.Normalize((0.5,), (0.5,))])\n# Download and load the training data\ntrainset = datasets.FashionMNIST('~/.pytorch/F_MNIST_data/', download=True, train=True, transform=transform)\ntrainloader = torch.utils.data.DataLoader(trainset, batch_size=64, shuffle=True)\n\n# Download and load the test data\ntestset = datasets.FashionMNIST('~/.pytorch/F_MNIST_data/', download=True, train=False, transform=transform)\ntestloader = torch.utils.data.DataLoader(testset, batch_size=64, shuffle=True)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"import torch.nn.functional as F\nfrom torch import optim\n\nclass fashionModel(torch.nn.Module):\n    def __init__(self,input_d):\n        super().__init__()\n        self.layer1=torch.nn.Linear(input_d,256)\n        self.nivel1Layer1 =torch.nn.Linear(256,80)\n        self.nivel1Layer2 =torch.nn.Linear(256,80)\n        self.Layer3=torch.nn.Linear(80+80,50)#we will concatenate the output of the 2 previous layers 200,200=400\n        self.Layer4=torch.nn.Linear(256+50,34)\n        self.outLayer=torch.nn.Linear(34,10)\n\n    \n    def forward(self,inputs):\n        h1=F.relu(self.layer1(inputs))\n        h2_1=F.relu(self.nivel1Layer1(h1))\n        h2_2=F.relu(self.nivel1Layer2(h1))\n        h3=F.relu(self.Layer3(torch.cat((h2_1,h2_2),dim=1)))\n        h4=F.relu(self.Layer4(torch.cat((h3,h1),dim=1)))\n        outputs=F.log_softmax(self.outLayer(h4),dim=1)\n        return outputs\n\n\n    def fit(self,batch_generator,num_epochs,criterion,Optimizer=None):\n        if not Optimizer:\n            Optimizer=optim.Adam(self.parameters(),lr=0.003)\n        train_samples=testloader.dataset.data.shape[0]\n        print(f\"{train_samples} training samples\")\n        for epoch in range(num_epochs):\n            epoch_loss=0\n            for images,labels in batch_generator:\n                Optimizer.zero_grad()#clean the gradients of the optimizer\n                outputs=self.forward(images.view(images.shape[0], -1))\n                loss=criterion(outputs,labels)#calculates the loss\n                loss.backward()#calculate the gradients of the loss with respect the model parameters\n                Optimizer.step()#we update the model parameters\n                epoch_loss+=loss.item()\n            print(f\"EPOCH:{epoch} loss:{epoch_loss}\")\n                ","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"x = torch.zeros(20,784, requires_grad=False)\nprint(\"Test tensor: \",x.shape)\nmodel=fashionModel(784)\nout=model(x)\nprint(\"Test output:\",out.shape)\nprint(\"Check if output its ok (sum1):\\n\",torch.sum(torch.exp(out),dim=1))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"criterion = torch.nn.NLLLoss()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"model.fit(trainloader,7,criterion)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"%matplotlib inline\n%config InlineBackend.figure_format = 'retina'\n\nimport helper\n\n# Test out your network!\n\ndataiter = iter(testloader)\nimages, labels = dataiter.next()\nimg = images[0]\n# Convert 2D image to 1D vector\nimg = img.resize_(1, 784)\n\n# TODO: Calculate the class probabilities (softmax) for img\nps=torch.exp(model(img))\nprint(ps)\n# Plot the image and probabilities\nview_classify(img.resize_(1, 28, 28), ps, version='Fashion')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"pip install hiddenlayer","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"import hiddenlayer as hl\nhl.build_graph(model, torch.zeros(1, 784))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Validation\nWe also have to validate our models using some matrics in order to know if is generalizing well or  if we have to implement some techniques to improve generalization such as dropout or L regularizations"},{"metadata":{},"cell_type":"markdown","source":"### Accuracy\nwe'll calculate the accuracy on our test set for our previous model"},{"metadata":{"trusted":true},"cell_type":"code","source":"dataiter = iter(testloader)\nimages, labels = dataiter.next()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"with torch.no_grad(): #we don't need grads in the predictions so let's speed up this part\n    predictions=model(images.view(images.shape[0],-1))\n    predictions=torch.exp(predictions)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"in order to get the predicted class since our predictions give us a probability for each class we can pickup the highest probability using the method .topk(n,dim) of our prediction tensors, this method returns a tuple (value,index) so for us will be (probability,class) "},{"metadata":{"trusted":true},"cell_type":"code","source":"probabilities,p_classes=predictions.topk(1,dim=1) #to pickup the top 1 of the row (since each row is a probability vector of the image)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"print(p_classes.shape,\"\\n\",p_classes[:10])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"print(labels.shape,\"\\n\",labels[:10])","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"In torch NLLoss allows to pass our prediction as a vector of probabilities [y1,y2,...yn] where n is the number of classes and our actual labels as simple numbers instead of having to one hot encoding it"},{"metadata":{},"cell_type":"markdown","source":""},{"metadata":{},"cell_type":"markdown","source":"then we want two know how many samples have been correctly classified so to perfom == operations p_classes and labels should have the same shapes so we can squeeze p_classes or unsqueeze labels (also view(*p_classes.shape)"},{"metadata":{"trusted":true},"cell_type":"code","source":"equals = p_classes.squeeze(1) == labels","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"the we just calculate the mean of corectly classified samples\nnote*: we cast the equals vector to a float vector in order to get a float division and not a integer division"},{"metadata":{"trusted":true},"cell_type":"code","source":"print(f\"Accuracy: {torch.mean(equals.type(torch.FloatTensor)).item()*100}\")","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"let's implement the validation step in the trainig pass"},{"metadata":{"trusted":true},"cell_type":"code","source":"import torch.nn.functional as F\nfrom torch import optim\n\nclass fashionModel(torch.nn.Module):\n    def __init__(self,input_d):\n        super().__init__()\n        self.layer1=torch.nn.Linear(input_d,256)\n        self.nivel1Layer1 =torch.nn.Linear(256,80)\n        self.nivel1Layer2 =torch.nn.Linear(256,80)\n        self.Layer3=torch.nn.Linear(80+80,50)#we will concatenate the output of the 2 previous layers 200,200=400\n        self.Layer4=torch.nn.Linear(256+50,34)\n        self.outLayer=torch.nn.Linear(34,10)\n\n    \n    def forward(self,inputs):\n        h1=F.relu(self.layer1(inputs))\n        h2_1=F.relu(self.nivel1Layer1(h1))\n        h2_2=F.relu(self.nivel1Layer2(h1))\n        h3=F.relu(self.Layer3(torch.cat((h2_1,h2_2),dim=1)))\n        h4=F.relu(self.Layer4(torch.cat((h3,h1),dim=1)))\n        outputs=F.log_softmax(self.outLayer(h4),dim=1)\n        return outputs\n    \n    def accuracy(self,predictions,labels):\n        predictions=torch.exp(predictions)#as we have log softmax at the end we need to calculate te actual outputs\n        probabilities,p_classes=predictions.topk(1,dim=1) #to pickup the top 1 of the row (since each row is a probability vector of the image)\n        equals = p_classes.squeeze(1) == labels\n        return torch.mean(equals.type(torch.FloatTensor)).item()\n    #todo\n    def metrics():\n        pass\n        \n\n    def fit(self,batch_generator,criterion,num_epochs=20,Optimizer=None,validation_generator=None):\n        if not Optimizer:\n            Optimizer=optim.Adam(self.parameters(),lr=0.003)\n        train_batch_size=len(batch_generator)\n        train_metrics=[] #(loss,accuracy) size:number of batches\n        if validation_generator:\n            val_batch_size=len(validation_generator)\n            val_metrics=[] #(loss,accuracy) size:number of batches\n        for epoch in range(num_epochs):\n            epoch_train_loss=0\n            epoch_train_accuracy=0\n            for images,labels in batch_generator:\n                Optimizer.zero_grad()#clean the gradients of the optimizer\n                outputs=self.forward(images.view(images.shape[0], -1))\n                loss=criterion(outputs,labels)#calculates the loss\n                loss.backward()#calculate the gradients of the loss with respect the model parameters\n                Optimizer.step()#we update the model parameters\n                accuracy=self.accuracy(outputs,labels)\n                epoch_train_loss+=loss.item()\n                epoch_train_accuracy+=accuracy\n            train_metrics.append((epoch_train_loss/train_batch_size,epoch_train_accuracy/train_batch_size))\n            print(f\"EPOCH:{epoch}\\nTrain loss:{train_metrics[-1][0]:.3f}.. Train accuracy:{train_metrics[-1][1]:.3f}..\")\n            if validation_generator:\n                epoch_val_loss=0\n                epoch_val_accuracy=0\n                with torch.no_grad():\n                    for images_val,labels_val in validation_generator:\n                        outputs=self.forward(images_val.view(images_val.shape[0],-1))\n                        val_loss=criterion(outputs,labels_val)\n                        val_accuracy=self.accuracy(outputs,labels_val)\n                        epoch_val_loss+=val_loss.item()\n                        epoch_val_accuracy+=val_accuracy\n                val_metrics.append((epoch_val_loss/val_batch_size,epoch_val_accuracy/val_batch_size))\n                print(f\"Validation loss:{val_metrics[-1][0]:.3f}.. Validation accuracy:{val_metrics[-1][1]:.3f}..\")\n        if validation_generator:\n            return train_metrics,val_metrics\n        else:\n            return train_metrics","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"lets train the model remeber we have a batch generator or training called  trainloader and another for validation called testloader"},{"metadata":{"trusted":true},"cell_type":"code","source":"model=fashionModel(784)\ncriterion = torch.nn.NLLLoss()\ntrain_metrics,val_metrics=model.fit(trainloader,criterion,validation_generator=testloader)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"import matplotlib.pyplot as plt\ndef plot_train_history(train_metrics,val_metrics):\n    train_loss,train_accuracy = zip(*train_metrics)\n    val_loss,val_accuracy = zip(*val_metrics)\n    plt.plot(train_loss, label='Training loss')\n    plt.plot(val_loss, label='Validation loss')\n    plt.legend(frameon=False)\n    plt.show()\n    plt.plot(train_accuracy, label='Training accuracy')\n    plt.plot(val_accuracy, label='Validation accuracy')\n    plt.legend(frameon=False)\n    plt.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"plot_train_history(train_metrics,val_metrics)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"As we can see there is a serious overfitting in the model let's try to use some generalization techniques to improve the perfomance of the model"},{"metadata":{},"cell_type":"markdown","source":"we use nn.dropout to turn off some percentage of the model during training and force the netork to share information between layer and not only train some of them\n\nwe can unable dropout to enter in evaluation mode using model.eval() and enable again for training using model.train()"},{"metadata":{"trusted":true},"cell_type":"code","source":"import torch.nn.functional as F\nfrom torch import optim\n\nclass fashionModel(torch.nn.Module):\n    def __init__(self,input_d):\n        super().__init__()\n        self.layer1=torch.nn.Linear(input_d,256)\n        self.nivel1Layer1 =torch.nn.Linear(256,120)\n        self.nivel1Layer2 =torch.nn.Linear(256,120)\n        self.Layer3=torch.nn.Linear(self.nivel1Layer1.out_features+\n                                    self.nivel1Layer2.out_features,\n                                    50)#we will concatenate the output of the 2 previous layers 200,200=400\n        self.Layer4=torch.nn.Linear(self.layer1.out_features\n                                    +self.Layer3.out_features,\n                                    34)\n        self.outLayer=torch.nn.Linear(34,10)\n        \n        self.dropout=torch.nn.Dropout(p=0.6)\n\n    \n    def forward(self,inputs):\n        h1= F.relu(self.layer1(inputs))\n        h2_1= self.dropout(F.relu(self.nivel1Layer1(h1)))\n        h2_2= self.dropout(F.relu(self.nivel1Layer2(h1)))\n        h3= self.dropout(F.relu(self.Layer3(torch.cat((h2_1,h2_2),dim=1))))\n        h4= self.dropout(F.relu(self.Layer4(torch.cat((h3,h1),dim=1))))\n        outputs=F.log_softmax(self.outLayer(h4),dim=1)\n        return outputs\n    \n    def accuracy(self,predictions,labels):\n        predictions=torch.exp(predictions)#as we have log softmax at the end we need to calculate te actual outputs\n        probabilities,p_classes=predictions.topk(1,dim=1) #to pickup the top 1 of the row (since each row is a probability vector of the image)\n        equals = p_classes.squeeze(1) == labels\n        return torch.mean(equals.type(torch.FloatTensor)).item()\n        #todo\n    def metrics():\n        pass\n        \n\n    def fit(self,batch_generator,criterion,num_epochs=20,Optimizer=None,validation_generator=None):\n        if not Optimizer:\n            Optimizer=optim.Adam(self.parameters(),lr=0.003)\n        train_batch_size=len(batch_generator)\n        train_metrics=[] #(loss,accuracy) size:number of batches\n        if validation_generator:\n            val_batch_size=len(validation_generator)\n            val_metrics=[] #(loss,accuracy) size:number of batches\n        for epoch in range(num_epochs):\n            epoch_train_loss=0\n            epoch_train_accuracy=0\n            self.train()#we put our model in train mode (we enable dropout)\n            for images,labels in batch_generator:\n                Optimizer.zero_grad()#clean the gradients of the optimizer\n                outputs=self.forward(images.view(images.shape[0], -1))\n                loss=criterion(outputs,labels)#calculates the loss\n                loss.backward()#calculate the gradients of the loss with respect the model parameters\n                Optimizer.step()#we update the model parameters\n                accuracy=self.accuracy(outputs,labels)\n                epoch_train_loss+=loss.item()\n                epoch_train_accuracy+=accuracy\n            train_metrics.append((epoch_train_loss/train_batch_size,epoch_train_accuracy/train_batch_size))\n            print(f\"EPOCH:{epoch}\\nTrain loss:{train_metrics[-1][0]:.3f}.. Train accuracy:{train_metrics[-1][1]:.3f}..\")\n            if validation_generator:\n                epoch_val_loss=0\n                epoch_val_accuracy=0\n                with torch.no_grad():\n                    self.eval()#we enter to evaluation mode so dropout isn't enabled\n                    for images_val,labels_val in validation_generator:\n                        outputs=self.forward(images_val.view(images_val.shape[0],-1))\n                        val_loss=criterion(outputs,labels_val)\n                        val_accuracy=self.accuracy(outputs,labels_val)\n                        epoch_val_loss+=val_loss.item()\n                        epoch_val_accuracy+=val_accuracy\n                val_metrics.append((epoch_val_loss/val_batch_size,epoch_val_accuracy/val_batch_size))\n                print(f\"Validation loss:{val_metrics[-1][0]:.3f}.. Validation accuracy:{val_metrics[-1][1]:.3f}..\")\n        if validation_generator:\n            return train_metrics,val_metrics\n        else:\n            return train_metrics","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"model=fashionModel(784)\ncriterion = torch.nn.NLLLoss()\ntrain_metrics,val_metrics=model.fit(trainloader,criterion,validation_generator=testloader)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"plot_train_history(train_metrics,val_metrics)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"applying dropout to our models could improve generalization significantly cause we force the neurons to learn important relationships and share information throught layers.\n\nin the case of our model we have a lot of parameters because of the paralel layers concatenation and the residual conection, so in this case the model could find hard to find  important relationships in such a varianced dataset algo we are maybe training a lot of not relevant relationships so dropout especially in the concatenation layers could help to find important relationship in all those neurons"},{"metadata":{},"cell_type":"markdown","source":"# Saving and loading models\nwe can save a model in order to use it for predictions in the future or even continue training or editing some parts the parameters(weights,bias matrices and more ..) of a torch.nn.Module can be obtained with ```state_dict() ```"},{"metadata":{"trusted":true},"cell_type":"code","source":"from pprint import pprint\nprint(f\"Model description {model}\")\nprint(\"\\nModel Parameters:\")\nfor p,v in model.state_dict().items():\n    print(f\"{p: <20}--->  {v.type(),v.shape}\")\n    \n    ","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"the simplest way to do it is by simply save the ```state_dict()``` with ```torch.save(\"path.pth\")``` where the .pth extension is the common extension for torch checkpoints "},{"metadata":{"trusted":true},"cell_type":"code","source":"torch.save(model.state_dict(),'model_checkpoint.pth')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"then we can load the parameters to the network again by open the file, charging the checkpoint to a dict and then loading it using model.load_state_dict"},{"metadata":{"trusted":true},"cell_type":"code","source":"model=fashionModel(784)\nprint(\"New model:\\n\",list(model.state_dict().items())[-1])\ncheckpoint=torch.load('model_checkpoint.pth')\nmodel.load_state_dict(checkpoint)\nprint(\"Restored model:\\n\",list(model.state_dict().items())[-1])\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Unfortunally this method only stored the parameters matrices of the model so we need to rewrite the model in the same way we trained it,we can preserve other information about the architecture creating a dictionary that contains the kind of layers, the input/output shape,etc and then define how the network should be constructed."},{"metadata":{},"cell_type":"markdown","source":"# Loading Data\n\nAs we have seen before pytorch includes a dataset tool to load and preprocess data\n"},{"metadata":{"trusted":true},"cell_type":"code","source":"!ls -l ../input/dataset/dataset/","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Preprocessing\nPytorch vision provides use several tranformations to apply to our data we just need to define a preprocessing pipeline"},{"metadata":{"trusted":true},"cell_type":"code","source":"from torchvision import datasets,transforms\n\ntransform = transforms.Compose([transforms.Resize(255),#resizes the image\n                                transforms.CenterCrop(224),#crops the image from the center\n                                transforms.ToTensor(),#converts to tensors\n                                transforms.Normalize((0.5,), (0.5,)),#normalize(scale) the tensors\n                              ])\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Then we will charge our data also applying the transform pipeline we defined before\nwe use te ```datasets.ImageFolder``` from torchvision which takes the data from a ```\"path\"``` folder then it searches for subfolders and labels the data as the name of the folder\n\n```/path/dogs/hjkhkj.jpg -> dogs\n   /path/cats/hjkhkj.jpg -> cats```\n\nThen we will use a dataloader to generate batches of size 64 and shuffle our data"},{"metadata":{"trusted":true},"cell_type":"code","source":"train_data = datasets.ImageFolder(\"../input/dataset/dataset/training_set\", transform=transform)\ntrain_generator = torch.utils.data.DataLoader(train_data, batch_size=64, shuffle=True)\nprint(f\"Training Data: {train_generator.dataset}\\nClasses:{train_generator.dataset.classes}\")","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"val_data = datasets.ImageFolder(\"../input/dataset/dataset/test_set\", transform=transform)\nval_generator = torch.utils.data.DataLoader(val_data, batch_size=64, shuffle=True)\nprint(f\"Validation Data: {val_generator.dataset}\\nClasses:{val_generator.dataset.classes}\")","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Data Augmentation\nWe can also use the torch transforms to perfom Data Augmentation which is a technique that allow us to introduce randomness in our data in order not only to increase our training data but also to teach our model how to perfom in very difficult enviroments low light, high contrast, different perspectives and rotations, etc."},{"metadata":{"trusted":true},"cell_type":"code","source":"## we will change only apply data augmentation in our train set\ntrain_transforms = transforms.Compose([transforms.RandomRotation(30),#rotates our images randmly in 0 to 30 degrees\n                                       transforms.RandomResizedCrop(224), #will resize and crop our images randmly in pictures of size 224\n                                       transforms.RandomHorizontalFlip(),#will flip random images horizontally\n                                       transforms.ToTensor()]) #transforms our images to tensors\ntrain_data = datasets.ImageFolder(\"../input/dataset/dataset/training_set\", transform=train_transforms)\ntrain_generator = torch.utils.data.DataLoader(train_data, batch_size=64, shuffle=True)\nprint(f\"Training Data: {train_generator.dataset}\\nClasses:{train_generator.dataset.classes}\")","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def imshow(image, ax=None, title=None, normalize=True):\n    \"\"\"Imshow for Tensor.\"\"\"\n    if ax is None:\n        fig, ax = plt.subplots()\n    image = image.numpy().transpose((1, 2, 0))\n\n    if normalize:\n        mean = np.array([0.485, 0.456, 0.406])\n        std = np.array([0.229, 0.224, 0.225])\n        image = std * image + mean\n        image = np.clip(image, 0, 1)\n    ax.imshow(image)\n    ax.spines['top'].set_visible(False)\n    ax.spines['right'].set_visible(False)\n    ax.spines['left'].set_visible(False)\n    ax.spines['bottom'].set_visible(False)\n    ax.tick_params(axis='both', length=0)\n    ax.set_xticklabels('')\n    ax.set_yticklabels('')\n\n    return ax","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"#this is how our images locks like"},{"metadata":{"trusted":true},"cell_type":"code","source":"# change this to the trainloader or testloader \ndata_iter = iter(train_generator)\ndata_iter_val = iter(val_generator)\n\nimages, labels = next(data_iter)\nfig, axes = plt.subplots(figsize=(10,4), ncols=4)\nfig.suptitle('Training Images')\nfor ii in range(4):\n    ax = axes[ii]\n    imshow(images[ii], ax=ax, normalize=False)\n    \nimages, labels = next(data_iter_val)\nfig, axes = plt.subplots(figsize=(10,4), ncols=4)\nfig.suptitle('Validation Images')\nfor ii in range(4):\n    ax = axes[ii]\n    imshow(images[ii], ax=ax, normalize=False)\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"from torch import nn\nfrom torch import optim\nfrom torch import functional as F\nfrom torchvision import models","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Transfer learning\n\nwe want to work whit this data set so torch already provides us a lot of models that perfoms very well on image classification tasks in the torch.vision package ,since this models has been trained in recognize 1000 different types of objects they provide a strong and robust set of kernels to recognize patters and shapes on images, then so, we just need to re train the classifier part of the network in order to recognize our classes.\n\nIn this case I pickup the mobilenet_v2 since it provides a lightweight and accurate network architecture"},{"metadata":{"trusted":true},"cell_type":"code","source":"model = models.resnet50(pretrained=True)\nmodel","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"we are going to replace the classifier on our network with a custom architecture capable of recognize two classes  "},{"metadata":{"trusted":true},"cell_type":"code","source":"from collections import OrderedDict\nnewClassifier=nn.Sequential(OrderedDict([\n                          ('fc1', nn.Linear(2048, 512)),\n                          ('relu', nn.ReLU()),\n                          ('Dropout2',nn.Dropout(0.2\n                                                )),\n                          ('fc2', nn.Linear(512, 2)),\n                          ('output', nn.LogSoftmax(dim=1)) #compute logsoftmax over rows\n                          ]))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"then we need to \"freeze\" all the weights of the mobilenet and replace the classifier with out custom model, in this way we will not track the operations and calculate gradients to those layers and also tell the optimizer not to update the weights,so then we can use those ways to extract features and then classifying the images using our custom classifier."},{"metadata":{"trusted":true},"cell_type":"code","source":"for p in model.parameters():\n    p.requires_grad=False\ni=iter(model.parameters())\nnext(model.parameters()).requires_grad","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"finally we replace the mobilenet classifier with ours in order to train it for our task"},{"metadata":{"trusted":true},"cell_type":"code","source":"model.fc=newClassifier\nmodel.fc","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# GPU Training \ntorch provides us a cuda mode to train our models over gpu's that parallelize the linear algebra computations over 1000x times \n\n```cuda()``` moves our model parameters and tensors to the gpu memory, we can also move it back to the cpu using ```cpu()``` or alternativily model|tensor.todevice(device)"},{"metadata":{"trusted":true},"cell_type":"code","source":"device = torch.device(\"cuda:0\" if torch.cuda.is_available() else \"cpu\")\ndevice","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"#model.cuda() #move the paameters of the model to the gpu memory \n#images.cuda()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"import time\n\nfor device  in [\"cuda\",\"cpu\"]:\n    criterion = nn.NLLLoss()#negative log-likelihood\n    optimizer = optim.Adam(model.fc.parameters(),lr=0.001) # we only want to optimize(train) the classifier parameters\n    #lets training for 3 samples and compare the times\n    model.to(device)\n    for i ,(images,labels) in enumerate(train_generator):\n        images,labels = images.to(device),labels.to(device)\n        start=time.time()\n        outputs = model(images)\n        loss=criterion(outputs,labels)\n        loss.backward()# we compute the gradients with respect the lost and store it in the model parameters.grad\n        optimizer.step()# we update the parameters using our gradients\n        if i==4:\n            break\n    print(f\"Device = {device}; Time per batch: {(time.time() - start)/4:.3f} seconds\")\n        ","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"There is a serious improvement using GPU instead of cpu for parallelisation\nso then we are actually perfoming the training of the model an see how it performs"},{"metadata":{},"cell_type":"markdown","source":"The we are ready to retrain our network so as we doo before we will only train the classifier module that we have replace in order to use the model for our task"},{"metadata":{"trusted":true},"cell_type":"code","source":"from tqdm import tqdm_notebook as tqdm\ndef f_accuracy(predictions,labels):\n    top_p, top_class = predictions.topk(1, dim=1)\n    equals = top_class == labels.view(*top_class.shape)\n    return torch.mean(equals.type(torch.FloatTensor)).item()\n    \ndef fit(model,train_generator,val_generator,optimizer,epochs=1,verbose=5,device='cuda',f_accuracy=f_accuracy):\n    model.to(device)\n    steps = 0\n    running_loss = 0\n    running_accuracy = 0\n    for epoch in range(epochs):\n        for inputs, labels in tqdm(train_generator,desc=\"Train Step\"):\n            steps += 1\n            # Move input and label tensors to the default device\n            inputs, labels = inputs.to(device), labels.to(device)\n            optimizer.zero_grad()#clean the optimizer gradients\n            logps = model.forward(inputs)\n            loss = criterion(logps, labels)\n            loss.backward()\n            optimizer.step()\n            running_loss += loss.item()\n            if steps % verbose == 0:\n                test_loss = 0\n                test_accuracy = 0\n                model.eval()\n                with torch.no_grad():\n                    for inputs, labels in tqdm(val_generator,desc=\"validation step\"):\n                        inputs, labels = inputs.to(device), labels.to(device)\n                        logps = model.forward(inputs)\n                        batch_loss = criterion(logps, labels)\n                        test_loss += batch_loss.item()\n                        # Calculate accuracy\n                        accuracy=f_accuracy(torch.exp(logps),labels)\n                        test_accuracy += accuracy\n                print(f\"Epoch: {epoch+1}/Step: {steps}.. \"\n                      f\"Train loss: {running_loss/verbose:.3f}.. \"\n                      f\"Val loss: {test_loss/len(val_generator):.3f}.. \"\n                      f\"Val accuracy: {test_accuracy/len(val_generator):.3f}\")\n            if steps>25:\n                break\n            running_loss = 0\n            model.train()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"model = models.resnet50(pretrained=True)\nfor p in model.parameters():\n    p.requires_grad=False\nnewClassifier=nn.Sequential(nn.Linear(2048, 512),\n                                         nn.ReLU(),\n                                         nn.Dropout(0.2),\n                                         nn.Linear(512,2),\n                                         nn.LogSoftmax(dim=1) #compute logsoftmax over rows\n                                        )\nmodel.fc=newClassifier\noptimizer = optim.Adam(model.fc.parameters(), lr=0.003)\nval_metrics =fit(model,train_generator,val_generator,optimizer)","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"name":"python","version":"3.6.4","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"}},"nbformat":4,"nbformat_minor":1}