{ "id": 187831, "title": "How NLLOSS work?", "url": "/competitions/digit-recognizer/discussion/187831", "author_name": "JeongHwan Choi", "post_date": "2020-09-30T13:13:30.899000", "votes": 1, "comment_count": 2, "views": null, "content": "

I try to understand the whole algorithm of deep learning so that I can use it to other case.

\n

However I can not understand how NLLOSS work.

\n

In this digit recognizer case, we make 10 output layer and through softmax function, we can see that output is probability like [0.1341,0.2313,0.2234,…]

\n

But how can error can be decreased because our target is one number like [3]

\n

I check NLLOSS function and it looks like this.

\n

def NLLLoss(logs, targets):
\n out = torch.zeros_like(targets, dtype=torch.float)
\n for i in range(len(targets)):
\n out[i] = logs[i][targets[i]]
\n return -out.sum()/len(out)

\n

In this case in logs, tensor that include probability is input and targets is just tensor that has number like [3]

", "messages": [ { "id": 1051326, "postDate": "2020-10-16T12:02:45.017Z", "content": "

@cauthur , you are right that softmax function returns an array of probability for every input that you have. Here is a simplifying diagram:

\n

\"Softmax

\n

Here is the NLLLoss function as you described:

\n
def NLLLoss(logs, targets):\n    out = torch.zeros_like(targets, dtype=torch.float)\n    for i in range(len(targets)):\n        out[i] = logs[i][targets[i]]\nreturn -out.sum()/len(out)\n
\n

Let's take an example, we want to predict a 3 in our dataset, our softmax output is like this:
\n[0.1, 0.1, 0.2, 0.05, 0.05, 0.1, 0.1, 0.1, 0.1, 0.1] (it adds to 1)

\n

Using the function above, NLLLoss = sum([-0.2])/1 = -0.2

\n

We want to minimize this loss (ideal array returned from softmax should be [0, 0, 1, 0, 0, 0, 0, 0, 0, 0], ideal NLLLoss = sum([-1])/1 = -1.

\n

Hence, we can see that, the boundaries between our class predictions are still not so close (our prediction for this data being 3 is only 0.2 probability which is quite close to other prediction (for eg, prediction for 1 is 0.1 probability). In order to minimize this loss, we backpropagate it (take gradients/derivative of loss with respect to our weights at every iteration and update our weights). Hope this helps!

\n

Some additional resources which I find helpful:

\n
    \n

  1. https://ljvmiranda921.github.io/notebook/2017/08/13/softmax-and-the-negative-log-likelihood/

    2. \n

  2. https://www.pluralsight.com/guides/building-your-first-pytorch-solution

    3. \n

  3. https://medium.com/data-science-bootcamp/softmax-function-beyond-the-basics-51f09ce11154

    4. \n
\n

Image from:
\nhttps://ljvmiranda921.github.io/notebook/2017/08/13/softmax-and-the-negative-log-likelihood/

", "rawMarkdown": "@cauthur , you are right that softmax function returns an array of probability for every input that you have. Here is a simplifying diagram:\n\n![Softmax Diagram](https://miro.medium.com/max/700/1*jb1i7A5R-9thuSGZL3vtpw.png)\n\nHere is the NLLLoss function as you described:\n```\ndef NLLLoss(logs, targets):\n out = torch.zeros_like(targets, dtype=torch.float)\n for i in range(len(targets)):\n out[i] = logs[i][targets[i]]\nreturn -out.sum()/len(out)\n```\n\nLet's take an example, we want to predict a 3 in our dataset, our softmax output is like this: \n```[0.1, 0.1, 0.2, 0.05, 0.05, 0.1, 0.1, 0.1, 0.1, 0.1]``` (it adds to 1)\n\nUsing the function above, NLLLoss = sum([-0.2])/1 = -0.2\n\nWe want to minimize this loss (ideal array returned from softmax should be ```[0, 0, 1, 0, 0, 0, 0, 0, 0, 0]```, ideal NLLLoss = sum([-1])/1 = -1.\n\nHence, we can see that, the boundaries between our class predictions are still not so close (our prediction for this data being 3 is only 0.2 probability which is quite close to other prediction (for eg, prediction for 1 is 0.1 probability). In order to minimize this loss, we backpropagate it (take gradients/derivative of loss with respect to our weights at every iteration and update our weights). Hope this helps!\n\nSome additional resources which I find helpful:\n\n1. https://ljvmiranda921.github.io/notebook/2017/08/13/softmax-and-the-negative-log-likelihood/\n\n2. https://www.pluralsight.com/guides/building-your-first-pytorch-solution\n\n3. https://medium.com/data-science-bootcamp/softmax-function-beyond-the-basics-51f09ce11154\n\nImage from: \nhttps://ljvmiranda921.github.io/notebook/2017/08/13/softmax-and-the-negative-log-likelihood/", "votes": 1, "replies": [ { "id": 1060254, "postDate": "2020-10-26T01:24:53.340Z", "content": "

Wow thanks! I got it easily. I was busy so I couldn't reply quickly. Thanks a lot!!!! Have a nice day.

", "rawMarkdown": "Wow thanks! I got it easily. I was busy so I couldn't reply quickly. Thanks a lot!!!! Have a nice day." } ] }, { "id": 1032818, "postDate": "2020-09-30T13:13:30.900Z", "content": "

I try to understand the whole algorithm of deep learning so that I can use it to other case.

\n

However I can not understand how NLLOSS work.

\n

In this digit recognizer case, we make 10 output layer and through softmax function, we can see that output is probability like [0.1341,0.2313,0.2234,…]

\n

But how can error can be decreased because our target is one number like [3]

\n

I check NLLOSS function and it looks like this.

\n

def NLLLoss(logs, targets):
\n out = torch.zeros_like(targets, dtype=torch.float)
\n for i in range(len(targets)):
\n out[i] = logs[i][targets[i]]
\n return -out.sum()/len(out)

\n

In this case in logs, tensor that include probability is input and targets is just tensor that has number like [3]

", "rawMarkdown": "I try to understand the whole algorithm of deep learning so that I can use it to other case.\n\nHowever I can not understand how NLLOSS work.\n\nIn this digit recognizer case, we make 10 output layer and through softmax function, we can see that output is probability like [0.1341,0.2313,0.2234,...] \n\nBut how can error can be decreased because our target is one number like [3]\n\nI check NLLOSS function and it looks like this.\n\n`def NLLLoss(logs, targets):`\n `out = torch.zeros_like(targets, dtype=torch.float)`\n `for i in range(len(targets)):`\n `out[i] = logs[i][targets[i]]`\n `return -out.sum()/len(out)`\n\nIn this case in logs, tensor that include probability is input and targets is just tensor that has number like [3]\n", "votes": 1 } ], "comments": [ { "id": 1051326, "author_name": "Alan Choon", "author_url": "", "post_date": "2020-10-16T12:02:45.017000", "content": "

@cauthur , you are right that softmax function returns an array of probability for every input that you have. Here is a simplifying diagram:

\n

\"Softmax

\n

Here is the NLLLoss function as you described:

\n
def NLLLoss(logs, targets):\n    out = torch.zeros_like(targets, dtype=torch.float)\n    for i in range(len(targets)):\n        out[i] = logs[i][targets[i]]\nreturn -out.sum()/len(out)\n
\n

Let's take an example, we want to predict a 3 in our dataset, our softmax output is like this:
\n[0.1, 0.1, 0.2, 0.05, 0.05, 0.1, 0.1, 0.1, 0.1, 0.1] (it adds to 1)

\n

Using the function above, NLLLoss = sum([-0.2])/1 = -0.2

\n

We want to minimize this loss (ideal array returned from softmax should be [0, 0, 1, 0, 0, 0, 0, 0, 0, 0], ideal NLLLoss = sum([-1])/1 = -1.

\n

Hence, we can see that, the boundaries between our class predictions are still not so close (our prediction for this data being 3 is only 0.2 probability which is quite close to other prediction (for eg, prediction for 1 is 0.1 probability). In order to minimize this loss, we backpropagate it (take gradients/derivative of loss with respect to our weights at every iteration and update our weights). Hope this helps!

\n

Some additional resources which I find helpful:

\n
    \n

  1. https://ljvmiranda921.github.io/notebook/2017/08/13/softmax-and-the-negative-log-likelihood/

    2. \n

  2. https://www.pluralsight.com/guides/building-your-first-pytorch-solution

    3. \n

  3. https://medium.com/data-science-bootcamp/softmax-function-beyond-the-basics-51f09ce11154

    4. \n
\n

Image from:
\nhttps://ljvmiranda921.github.io/notebook/2017/08/13/softmax-and-the-negative-log-likelihood/

", "votes": 1, "replies": [ { "id": 1060254, "author_name": "JeongHwan Choi", "author_url": "", "post_date": "2020-10-26T01:24:53.340000", "content": "

Wow thanks! I got it easily. I was busy so I couldn't reply quickly. Thanks a lot!!!! Have a nice day.

", "votes": 0, "replies": [] } ] } ], "raw_markdown_by_id": { "1051326": "@cauthur , you are right that softmax function returns an array of probability for every input that you have. Here is a simplifying diagram:\n\n![Softmax Diagram](https://miro.medium.com/max/700/1*jb1i7A5R-9thuSGZL3vtpw.png)\n\nHere is the NLLLoss function as you described:\n```\ndef NLLLoss(logs, targets):\n out = torch.zeros_like(targets, dtype=torch.float)\n for i in range(len(targets)):\n out[i] = logs[i][targets[i]]\nreturn -out.sum()/len(out)\n```\n\nLet's take an example, we want to predict a 3 in our dataset, our softmax output is like this: \n```[0.1, 0.1, 0.2, 0.05, 0.05, 0.1, 0.1, 0.1, 0.1, 0.1]``` (it adds to 1)\n\nUsing the function above, NLLLoss = sum([-0.2])/1 = -0.2\n\nWe want to minimize this loss (ideal array returned from softmax should be ```[0, 0, 1, 0, 0, 0, 0, 0, 0, 0]```, ideal NLLLoss = sum([-1])/1 = -1.\n\nHence, we can see that, the boundaries between our class predictions are still not so close (our prediction for this data being 3 is only 0.2 probability which is quite close to other prediction (for eg, prediction for 1 is 0.1 probability). In order to minimize this loss, we backpropagate it (take gradients/derivative of loss with respect to our weights at every iteration and update our weights). Hope this helps!\n\nSome additional resources which I find helpful:\n\n1. https://ljvmiranda921.github.io/notebook/2017/08/13/softmax-and-the-negative-log-likelihood/\n\n2. https://www.pluralsight.com/guides/building-your-first-pytorch-solution\n\n3. https://medium.com/data-science-bootcamp/softmax-function-beyond-the-basics-51f09ce11154\n\nImage from: \nhttps://ljvmiranda921.github.io/notebook/2017/08/13/softmax-and-the-negative-log-likelihood/", "1032818": "I try to understand the whole algorithm of deep learning so that I can use it to other case.\n\nHowever I can not understand how NLLOSS work.\n\nIn this digit recognizer case, we make 10 output layer and through softmax function, we can see that output is probability like [0.1341,0.2313,0.2234,...] \n\nBut how can error can be decreased because our target is one number like [3]\n\nI check NLLOSS function and it looks like this.\n\n`def NLLLoss(logs, targets):`\n `out = torch.zeros_like(targets, dtype=torch.float)`\n `for i in range(len(targets)):`\n `out[i] = logs[i][targets[i]]`\n `return -out.sum()/len(out)`\n\nIn this case in logs, tensor that include probability is input and targets is just tensor that has number like [3]\n" } }