{ "id": 11450, "title": "When is it good to use Softmax?", "url": "/competitions/datasciencebowl/discussion/11450", "author_name": "", "post_date": "2015-01-07T22:43:19.047Z", "votes": null, "comment_count": 4, "views": 12928, "content": "

I've been doing a little playing around with the idea of using softmax algorithm on my answers.  From my playing around, it looks like if you are doing really well predicting the correct class of plankton, you will benefit from softmax, but if your predictions aren't all that accurate, softmax will hurt your score. (And can hurt it big time!)

\n

So my question is, how good do you need to be before softmax is worth your while?  Anyone with good math chops got an answer to this? Maybe a good intuitive answer that helps those of us without a PhD in math to understand?

\n

Thanks,

\n

Chip

", "messages": [ { "id": "62426", "postDate": "01/07/2015 22:43:19", "content": "

I've been doing a little playing around with the idea of using softmax algorithm on my answers.  From my playing around, it looks like if you are doing really well predicting the correct class of plankton, you will benefit from softmax, but if your predictions aren't all that accurate, softmax will hurt your score. (And can hurt it big time!)

\n

So my question is, how good do you need to be before softmax is worth your while?  Anyone with good math chops got an answer to this? Maybe a good intuitive answer that helps those of us without a PhD in math to understand?

\n

Thanks,

\n

Chip

", "rawMarkdown": "", "votes": null }, { "id": "62471", "postDate": "01/08/2015 23:32:24", "content": "

hey Chip...

\n

I'm interested to learn more.  Can you describe the context of your use of softmax?  i.e. what are you doing before you apply it?  

\n

My understanding is that a softmax is required for a multiclass classifier or else the sum of your output probabilities may not add up to 1, which be nonsense and not be true probabilities.  So if implementing logistic regression, your output would be:  softmax((X dot W) + b).

\n

I think it's built in under the hood for any canned ML algorithms you're using, but it sounds like maybe you're doing something from scratch...  or more interesting.  If you care to share, I'd love to hear.

\n\n

Kevin

", "rawMarkdown": "", "votes": null }, { "id": "62475", "postDate": "01/09/2015 00:26:27", "content": "

Kevin,

\n

Depending on what software you're using it might implement softmax for you. I'm currently using some neural net code that I wrote. The output from the neural net is 121 values. Each value is approximately the probability of the input being that class.  Many other ML algos work the same way, giving you a set of probabilities (or relative probabilities) that the input is that class.

\n

Using softmax, you search through the outputs from your ML algo, find the one with the highest probability and set that value to 1, and all the other values to 0.

\n

I've looked at two of the starter code examples provided for this competition. One of them takes the output from a CNN and then softmax's the output. The other uses a random forest which outputs probabilities and does not softmax the output.

\n

I think the answer of whether to softmax is tied to the evaluation algorithm and how well you have predicted the classes.  If you are predicting really well, softmax helps, but if you are not predicting well, softmax will actually hurt you. (Again, this is due to the evaluation algo)

\n

Unfortunately my math isn't strong enough to figure out what the break point is.  I've thought about running various test cases to see if I could figure this out largely by trial and error, but I'm still hoping someone will step up to the plate and explain the math of when to use softmax and when not to.

\n

Chip

\n\n

might also be worth mentioning, that maybe I've misunderstood what a softmax algo does. I've always thought that softmax picks the most likely and sets that to 1 and sets the others to 0. If that's not softmax, then my question should be edited to when should you use whatever the name of the function that would do that is. (hope that's not too confusing)

", "rawMarkdown": "", "votes": null }, { "id": "62478", "postDate": "01/09/2015 01:50:49", "content": "

@Chip

\n

Softmax in its standard form doesn't set max probability to 1 and rest to 0. No decent algorithm should ever produce 1 or 0 probability because in case of a mistake the log error goes to infinity. 

\n

Intuitively when normalizing the multiclass probabilities you have 2 extreme options. To simplify things let's assume that our hypothesis is (X dot W) + b:

\n

1. all probabilities are equal (your model is biased; or in other words your b coefficient is very large.

\n

2. the maximum probability is 1 and the rest are 0 (as in your case) (your model has high variance because it has some very large w and low b coefficient)

\n

I guess the softmax function lies somewhere in between to balance those 2 options.

\n

Refer to wikipedia:

\n

http://en.wikipedia.org/wiki/Softmax_function

", "rawMarkdown": "", "votes": null }, { "id": "62517", "postDate": "01/09/2015 18:00:00", "content": "

Hi Chip,

\n

Just to add to other excellent answers, softmax regression can be seen as a multiclass generalization of logistic regression. I wrote about it some time ago:

\n

http://cjauvin.blogspot.ca/2013/10/softmax-regression-101.html

", "rawMarkdown": "", "votes": null } ], "comments": [ { "id": 62471, "author_name": "kevinhinson", "author_url": "", "post_date": "01/08/2015 23:32:24", "content": "

hey Chip...

\n

I'm interested to learn more.  Can you describe the context of your use of softmax?  i.e. what are you doing before you apply it?  

\n

My understanding is that a softmax is required for a multiclass classifier or else the sum of your output probabilities may not add up to 1, which be nonsense and not be true probabilities.  So if implementing logistic regression, your output would be:  softmax((X dot W) + b).

\n

I think it's built in under the hood for any canned ML algorithms you're using, but it sounds like maybe you're doing something from scratch...  or more interesting.  If you care to share, I'd love to hear.

\n\n

Kevin

", "votes": null, "replies": [] }, { "id": 62475, "author_name": "chipburwell", "author_url": "", "post_date": "01/09/2015 00:26:27", "content": "

Kevin,

\n

Depending on what software you're using it might implement softmax for you. I'm currently using some neural net code that I wrote. The output from the neural net is 121 values. Each value is approximately the probability of the input being that class.  Many other ML algos work the same way, giving you a set of probabilities (or relative probabilities) that the input is that class.

\n

Using softmax, you search through the outputs from your ML algo, find the one with the highest probability and set that value to 1, and all the other values to 0.

\n

I've looked at two of the starter code examples provided for this competition. One of them takes the output from a CNN and then softmax's the output. The other uses a random forest which outputs probabilities and does not softmax the output.

\n

I think the answer of whether to softmax is tied to the evaluation algorithm and how well you have predicted the classes.  If you are predicting really well, softmax helps, but if you are not predicting well, softmax will actually hurt you. (Again, this is due to the evaluation algo)

\n

Unfortunately my math isn't strong enough to figure out what the break point is.  I've thought about running various test cases to see if I could figure this out largely by trial and error, but I'm still hoping someone will step up to the plate and explain the math of when to use softmax and when not to.

\n

Chip

\n\n

might also be worth mentioning, that maybe I've misunderstood what a softmax algo does. I've always thought that softmax picks the most likely and sets that to 1 and sets the others to 0. If that's not softmax, then my question should be edited to when should you use whatever the name of the function that would do that is. (hope that's not too confusing)

", "votes": null, "replies": [] }, { "id": 62478, "author_name": "paweljankiewicz", "author_url": "", "post_date": "01/09/2015 01:50:49", "content": "

@Chip

\n

Softmax in its standard form doesn't set max probability to 1 and rest to 0. No decent algorithm should ever produce 1 or 0 probability because in case of a mistake the log error goes to infinity. 

\n

Intuitively when normalizing the multiclass probabilities you have 2 extreme options. To simplify things let's assume that our hypothesis is (X dot W) + b:

\n

1. all probabilities are equal (your model is biased; or in other words your b coefficient is very large.

\n

2. the maximum probability is 1 and the rest are 0 (as in your case) (your model has high variance because it has some very large w and low b coefficient)

\n

I guess the softmax function lies somewhere in between to balance those 2 options.

\n

Refer to wikipedia:

\n

http://en.wikipedia.org/wiki/Softmax_function

", "votes": null, "replies": [] }, { "id": 62517, "author_name": "cjauvin", "author_url": "", "post_date": "01/09/2015 18:00:00", "content": "

Hi Chip,

\n

Just to add to other excellent answers, softmax regression can be seen as a multiclass generalization of logistic regression. I wrote about it some time ago:

\n

http://cjauvin.blogspot.ca/2013/10/softmax-regression-101.html

", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "62426": "", "62471": "", "62475": "", "62478": "", "62517": "" }, "source": "meta" }