{ "id": 11290, "title": "Classification error", "url": "/competitions/datasciencebowl/discussion/11290", "author_name": "", "post_date": "2014-12-18T09:05:20.357Z", "votes": null, "comment_count": 12, "views": 3541, "content": "

hi,

\n

the error measure seems not very natural for me.

\n

The formula on the evaluation page says that it counts only the columns where target is 1.

\n

Anyway, since this is a classification problem I also evaluated the accuracy.

\n

I got 55% accuracy on a 20% random validation set.

\n

Submitting this leads me to 1.9 lb or so.

\n

The model is a drednet (neural net with ReLu's) on a 50x50 scaled version of the data.

\n

Have anyone evaluated accuracy too ?

", "messages": [ { "id": "61456", "postDate": "12/18/2014 09:05:20", "content": "

hi,

\n

the error measure seems not very natural for me.

\n

The formula on the evaluation page says that it counts only the columns where target is 1.

\n

Anyway, since this is a classification problem I also evaluated the accuracy.

\n

I got 55% accuracy on a 20% random validation set.

\n

Submitting this leads me to 1.9 lb or so.

\n

The model is a drednet (neural net with ReLu's) on a 50x50 scaled version of the data.

\n

Have anyone evaluated accuracy too ?

", "rawMarkdown": "", "votes": null }, { "id": "61458", "postDate": "12/18/2014 09:29:44", "content": "

According to the evaluation page:

\n

The submitted probabilities for a given image are not required to sum to one because they are rescaled prior to being scored (each row is divided by the row sum).

\n

With that pre-processing, then the log loss function does take account of all output predictions, and not just the target column. If you are using a softmax output layer, you won't notice the difference though, as your model is already scaling the outputs in the same way

", "rawMarkdown": "", "votes": null }, { "id": "61459", "postDate": "12/18/2014 09:35:34", "content": "

yep, but compare the formulas here on evaluation page and this one:

\n

https://www.kaggle.com/wiki/LogarithmicLoss

\n

There is a term missing "(1-yi)*log(1-yi^)" which counts the error for "0" targets.

", "rawMarkdown": "", "votes": null }, { "id": "61460", "postDate": "12/18/2014 09:46:05", "content": "

You should note that yi is either 0 or 1. So, yi*log(yi^) + (1-yi)*log(1-yi^) is essentially one term. The two formulas are the same.

\n

As for the accuracy, I got similar result.

", "rawMarkdown": "", "votes": null }, { "id": "61462", "postDate": "12/18/2014 09:50:13", "content": "

[quote=Michael Jahrer;61459]

\n

yep, but compare the formulas here on evaluation page and this one:

\n

https://www.kaggle.com/wiki/LogarithmicLoss

\n

There is a term missing "(1-yi)*log(1-yi^)" which counts the error for "0" targets.

\n

[/quote]

\n

It doesn't really matter, the 0's are being accounted for by the normalisation, although you are right it is not the same numerically.

\n

Adding the (1-yi) terms would increase the error value for classification errors which were "really confident it is this specific wrong thing" as opposed to "not really sure what it is".

", "rawMarkdown": "", "votes": null }, { "id": "61463", "postDate": "12/18/2014 09:57:22", "content": "

[quote=yr;61460]

\n

You should note that yi is either 0 or 1. So, yi*log(yi^) + (1-yi)*log(1-yi^) is essentially one term. The two formulas are the same.

\n

[/quote]

\n

I don't think the last sentence here is correct. The formulae are different, and would result in different metrics for the competition.

", "rawMarkdown": "", "votes": null }, { "id": "61464", "postDate": "12/18/2014 09:57:23", "content": "

e.g. 4 classes, M=4
y=[0 0 1 0] <-- targets
yp=[0.1 0.1 0.5 0.3] <-- predictions

\n

sum(j=1,M,yj*log(ypj))
= 0*log(0.1) + 0*log(0.1) + 1*log(0.5) + 0*log(0.3) = -0.6931

\n

sum(j=1,M,yj*log(ypj)+(1-yj)*log(1-ypj))
= 0*log(0.1)+(1-0)*log(1-0.1) + 0*log(0.1)+(1-0)*log(1-0.1) + 1*log(0.5)+(1-1)*log(0.5) + 0*log(0.3)+(1-0)*log(1-0.3) = -1.2605

\n

with "y*log(yp)" -> -0.6931
with "y*log(yp)+(1-y)*log(1-yp)" -> -1.2605

\n

so there is a difference, or am I miss something ?

", "rawMarkdown": "", "votes": null }, { "id": "61465", "postDate": "12/18/2014 10:14:26", "content": "

[quote=Michael Jahrer;61464]

\n

so there is a difference, or am I miss something ?

\n

[/quote]

\n

I was just about to post a similar numerical example.

\n

You are right, but as I said, it is not such a big difference in practical terms. The competition is focussing on confidence in the correct class, and does not care about your distribution of incorrect classes. However, with the output scaling, your confidence in the correct class does depend on all the outputs, not just the correct one.

\n

There may be good design reasons for this choice, such as not wanting to penalise poor categorisation to similar-looking items (the same kind of error a human might make).

", "rawMarkdown": "", "votes": null }, { "id": "61469", "postDate": "12/18/2014 11:13:46", "content": "

LogLoss is usually used in binary classification, y \\in {0, 1}. Therefore, "y*log(yp)+(1-y)*log(1-yp)" reduces to one term that is associated with the true target, i.e.,

\n

y = 1 : log(yp)

\n

y= 0 : log(1-yp) (Note: 1-yp here is the probability that the example is "0")

\n

This is just the (negative) log likelihood of the data.

\n

Softmax loss/cross entry as in "sum(j=1,M,yj*log(ypj))" is a generalization of logloss to multi-class problem. It is the log likelihood of the data too. See: http://deeplearning.stanford.edu/wiki/index.php/Softmax_Regression

\n

[quote=Michael Jahrer;61464]

\n

e.g. 4 classes, M=4
y=[0 0 1 0] <-- targets
yp=[0.1 0.1 0.5 0.3] <-- predictions

\n

sum(j=1,M,yj*log(ypj))
= 0*log(0.1) + 0*log(0.1) + 1*log(0.5) + 0*log(0.3) = -0.6931

\n

sum(j=1,M,yj*log(ypj)+(1-yj)*log(1-ypj))
= 0*log(0.1)+(1-0)*log(1-0.1) + 0*log(0.1)+(1-0)*log(1-0.1) + 1*log(0.5)+(1-1)*log(0.5) + 0*log(0.3)+(1-0)*log(1-0.3) = -1.2605

\n

with "y*log(yp)" -> -0.6931
with "y*log(yp)+(1-y)*log(1-yp)" -> -1.2605

\n

so there is a difference, or am I miss something ?

\n

[/quote]

\n

In your above example, if you are using the second method to compute the error, then you are treating each target as a binary classification problem (multi-label classification in terminology). It is not the same as multi-class classification, which should use the first method if you are trying to get the multi-class logloss / softmax loss.

", "rawMarkdown": "", "votes": null }, { "id": "61485", "postDate": "12/18/2014 14:31:32", "content": "

thanks yr, this makes sense

", "rawMarkdown": "", "votes": null }, { "id": "61505", "postDate": "12/18/2014 22:00:11", "content": "

The model currently in first has an accuracy of around 71-72% on a 20% random validation set.

Still got plenty of ground to cover to compete with humans =P

", "rawMarkdown": "", "votes": null }, { "id": "61508", "postDate": "12/18/2014 23:18:50", "content": "

The attachment file is a validation log of my model. includes NLL/Accuracy score.

\n

My model is CNN using Torch7.

", "rawMarkdown": "", "votes": null }, { "id": "65900", "postDate": "03/10/2015 18:17:29", "content": "

Looking into parallel multinomial options using R. Seems like there are more use cases not using multinomial outputs in this area, where each of the 121 classes predicted sum over a more even range of probabilities--multi-class classification each row sums to 1.

\n

Does the competition allow multi-label classification submissions (e.g., [.15, .15, .4, .15, .15]) where each element in a given row will equal either 0 or 1 (e.g., [0, 0, 1, 0, 0]) where each row sums 1?

", "rawMarkdown": "", "votes": null } ], "comments": [ { "id": 61458, "author_name": "slobo777", "author_url": "", "post_date": "12/18/2014 09:29:44", "content": "

According to the evaluation page:

\n

The submitted probabilities for a given image are not required to sum to one because they are rescaled prior to being scored (each row is divided by the row sum).

\n

With that pre-processing, then the log loss function does take account of all output predictions, and not just the target column. If you are using a softmax output layer, you won't notice the difference though, as your model is already scaling the outputs in the same way

", "votes": null, "replies": [] }, { "id": 61459, "author_name": "mjahrer", "author_url": "", "post_date": "12/18/2014 09:35:34", "content": "

yep, but compare the formulas here on evaluation page and this one:

\n

https://www.kaggle.com/wiki/LogarithmicLoss

\n

There is a term missing "(1-yi)*log(1-yi^)" which counts the error for "0" targets.

", "votes": null, "replies": [] }, { "id": 61460, "author_name": "chenglongchen", "author_url": "", "post_date": "12/18/2014 09:46:05", "content": "

You should note that yi is either 0 or 1. So, yi*log(yi^) + (1-yi)*log(1-yi^) is essentially one term. The two formulas are the same.

\n

As for the accuracy, I got similar result.

", "votes": null, "replies": [] }, { "id": 61462, "author_name": "slobo777", "author_url": "", "post_date": "12/18/2014 09:50:13", "content": "

[quote=Michael Jahrer;61459]

\n

yep, but compare the formulas here on evaluation page and this one:

\n

https://www.kaggle.com/wiki/LogarithmicLoss

\n

There is a term missing "(1-yi)*log(1-yi^)" which counts the error for "0" targets.

\n

[/quote]

\n

It doesn't really matter, the 0's are being accounted for by the normalisation, although you are right it is not the same numerically.

\n

Adding the (1-yi) terms would increase the error value for classification errors which were "really confident it is this specific wrong thing" as opposed to "not really sure what it is".

", "votes": null, "replies": [] }, { "id": 61463, "author_name": "slobo777", "author_url": "", "post_date": "12/18/2014 09:57:22", "content": "

[quote=yr;61460]

\n

You should note that yi is either 0 or 1. So, yi*log(yi^) + (1-yi)*log(1-yi^) is essentially one term. The two formulas are the same.

\n

[/quote]

\n

I don't think the last sentence here is correct. The formulae are different, and would result in different metrics for the competition.

", "votes": null, "replies": [] }, { "id": 61464, "author_name": "mjahrer", "author_url": "", "post_date": "12/18/2014 09:57:23", "content": "

e.g. 4 classes, M=4
y=[0 0 1 0] <-- targets
yp=[0.1 0.1 0.5 0.3] <-- predictions

\n

sum(j=1,M,yj*log(ypj))
= 0*log(0.1) + 0*log(0.1) + 1*log(0.5) + 0*log(0.3) = -0.6931

\n

sum(j=1,M,yj*log(ypj)+(1-yj)*log(1-ypj))
= 0*log(0.1)+(1-0)*log(1-0.1) + 0*log(0.1)+(1-0)*log(1-0.1) + 1*log(0.5)+(1-1)*log(0.5) + 0*log(0.3)+(1-0)*log(1-0.3) = -1.2605

\n

with "y*log(yp)" -> -0.6931
with "y*log(yp)+(1-y)*log(1-yp)" -> -1.2605

\n

so there is a difference, or am I miss something ?

", "votes": null, "replies": [] }, { "id": 61465, "author_name": "slobo777", "author_url": "", "post_date": "12/18/2014 10:14:26", "content": "

[quote=Michael Jahrer;61464]

\n

so there is a difference, or am I miss something ?

\n

[/quote]

\n

I was just about to post a similar numerical example.

\n

You are right, but as I said, it is not such a big difference in practical terms. The competition is focussing on confidence in the correct class, and does not care about your distribution of incorrect classes. However, with the output scaling, your confidence in the correct class does depend on all the outputs, not just the correct one.

\n

There may be good design reasons for this choice, such as not wanting to penalise poor categorisation to similar-looking items (the same kind of error a human might make).

", "votes": null, "replies": [] }, { "id": 61469, "author_name": "chenglongchen", "author_url": "", "post_date": "12/18/2014 11:13:46", "content": "

LogLoss is usually used in binary classification, y \\in {0, 1}. Therefore, "y*log(yp)+(1-y)*log(1-yp)" reduces to one term that is associated with the true target, i.e.,

\n

y = 1 : log(yp)

\n

y= 0 : log(1-yp) (Note: 1-yp here is the probability that the example is "0")

\n

This is just the (negative) log likelihood of the data.

\n

Softmax loss/cross entry as in "sum(j=1,M,yj*log(ypj))" is a generalization of logloss to multi-class problem. It is the log likelihood of the data too. See: http://deeplearning.stanford.edu/wiki/index.php/Softmax_Regression

\n

[quote=Michael Jahrer;61464]

\n

e.g. 4 classes, M=4
y=[0 0 1 0] <-- targets
yp=[0.1 0.1 0.5 0.3] <-- predictions

\n

sum(j=1,M,yj*log(ypj))
= 0*log(0.1) + 0*log(0.1) + 1*log(0.5) + 0*log(0.3) = -0.6931

\n

sum(j=1,M,yj*log(ypj)+(1-yj)*log(1-ypj))
= 0*log(0.1)+(1-0)*log(1-0.1) + 0*log(0.1)+(1-0)*log(1-0.1) + 1*log(0.5)+(1-1)*log(0.5) + 0*log(0.3)+(1-0)*log(1-0.3) = -1.2605

\n

with "y*log(yp)" -> -0.6931
with "y*log(yp)+(1-y)*log(1-yp)" -> -1.2605

\n

so there is a difference, or am I miss something ?

\n

[/quote]

\n

In your above example, if you are using the second method to compute the error, then you are treating each target as a binary classification problem (multi-label classification in terminology). It is not the same as multi-class classification, which should use the first method if you are trying to get the multi-class logloss / softmax loss.

", "votes": null, "replies": [] }, { "id": 61485, "author_name": "mjahrer", "author_url": "", "post_date": "12/18/2014 14:31:32", "content": "

thanks yr, this makes sense

", "votes": null, "replies": [] }, { "id": 61505, "author_name": "alecradford", "author_url": "", "post_date": "12/18/2014 22:00:11", "content": "

The model currently in first has an accuracy of around 71-72% on a 20% random validation set.

Still got plenty of ground to cover to compete with humans =P

", "votes": null, "replies": [] }, { "id": 61508, "author_name": "nagadomi", "author_url": "", "post_date": "12/18/2014 23:18:50", "content": "

The attachment file is a validation log of my model. includes NLL/Accuracy score.

\n

My model is CNN using Torch7.

", "votes": null, "replies": [] }, { "id": 65900, "author_name": "zeccalehn", "author_url": "", "post_date": "03/10/2015 18:17:29", "content": "

Looking into parallel multinomial options using R. Seems like there are more use cases not using multinomial outputs in this area, where each of the 121 classes predicted sum over a more even range of probabilities--multi-class classification each row sums to 1.

\n

Does the competition allow multi-label classification submissions (e.g., [.15, .15, .4, .15, .15]) where each element in a given row will equal either 0 or 1 (e.g., [0, 0, 1, 0, 0]) where each row sums 1?

", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "61456": "", "61458": "", "61459": "", "61460": "", "61462": "", "61463": "", "61464": "", "61465": "", "61469": "", "61485": "", "61505": "", "61508": "", "65900": "" }, "source": "meta" }