{ "id": 11419, "title": "logloss ->0 if all p(i,j)->1", "url": "/competitions/datasciencebowl/discussion/11419", "author_name": "", "post_date": "2015-01-03T21:45:19.510Z", "votes": null, "comment_count": 15, "views": 4152, "content": "

I am new to this board. Per multiclass logarithmic loss function=> 0 as p(i,j)=>1
. Am I missing something here?

", "messages": [ { "id": "62247", "postDate": "01/03/2015 21:45:19", "content": "

I am new to this board. Per multiclass logarithmic loss function=> 0 as p(i,j)=>1
. Am I missing something here?

", "rawMarkdown": "", "votes": null }, { "id": "62251", "postDate": "01/04/2015 01:14:58", "content": "

[quote=Parkinson Zabelin;62247]

\n

I am new to this board. Per multiclass logarithmic loss function=> 0 as p(i,j)=>1
. Am I missing something here?

\n

[/quote]

\n

Yes, you are missing that:

\n

$$\\sum_{j=1}^{M}p_{i,j}=1$$

\n

That is, for each example(i), the probabilities of the classes(the j's) must sum to 1.   

\n

Also, from the Evaluation page:   

\n

The submitted probabilities for a given image are not required to sum to one because they are rescaled prior to being scored (each row is divided by the row sum).   

\n

In other words, they force your probabilities to sum to one if the ones you provide don't. So if you submit all ones, your prediction will be rescaled to a uniform probability of 1/M for each class. That won't have a zero loss.

", "rawMarkdown": "", "votes": null }, { "id": "62255", "postDate": "01/04/2015 14:55:32", "content": "

[quote=David Thaler;62251]

\n

In other words, they force your probabilities to sum to one if the ones you provide don't. So if you submit all ones, your prediction will be rescaled to a uniform probability of 1/M for each class. That won't have a zero loss.

\n

[/quote]

\n

To add to this, due to the scaling this is the same as the "uniform probability" benchmark, and will score 4.795791 on the public leaderboard

", "rawMarkdown": "", "votes": null }, { "id": "62608", "postDate": "01/11/2015 18:30:54", "content": "

Neil, and others,

\n

What is the (most likely) best score possible?

\n

Plankton experts are rated in the high 95-98% accuracy and this data base was hand labeled as well (according to the FAQ and other statements), so if a theoretical team of plankton experts were to take the time (eons) what would their score be?

", "rawMarkdown": "", "votes": null }, { "id": "62651", "postDate": "01/12/2015 15:17:04", "content": "

There isn't a simple 1:1 mapping between the classification error and the softmax loss, but to get a feeling you can create an 'artificial classifier' that is correct p% of the time and calculate its loss.

a simple example that gives you different sofmax loss values for 0% classification error can be seen if you assign to the correct class the value "Pmax" and randomly spread the remaining (1-Pmax) along all the other classes. even though for Pmax > 0.5 you will get no classification error at all, the softmax loss will greatly depend on the value of Pmax itself.

", "rawMarkdown": "", "votes": null }, { "id": "62708", "postDate": "01/14/2015 01:52:57", "content": "

Thanks SG, and others

\n

But does any one want to make a semi-math-based guess on the best score possible?

\n

(plus or minus some good guesstimate)?

\n

I am just trying to get a handle on how low the scores could possibly go.....

\n

thanks

\n

(full disclosure: I am log loss challenged)

", "rawMarkdown": "", "votes": null }, { "id": "62718", "postDate": "01/14/2015 14:41:04", "content": "

We have that the log-loss for multiclass is defined here.  Noting the convention that 0 log(0) = 0, if the probabilities were not bounded below/above by 10**(-15), 1-10**(-15), the minimum log loss is 0, which corresponds to the correct class being selected with probability 1, and the other classes having probability 0.

\n

Since our log-losses are bounded, if this labeling is given, the log loss is:

\n

-(1 * 1 *  log(1-120*(10**(-15))) + 0 * 120 * log(10**(-15))) * 130400/130400 = 1.200151e-13,

\n

approximately.

\n

The 1 * 1 * log(1-120*(10**(-15)))  is 1 for y_{i, j}  times 1 for the correct classification times log(1-120*(10**(-15))) for the log-probability (assuming that's how they compute the probability when all are 1's and 0's), and 0 is for y_{i, j} times 120 for the other 120 classes times the log-probability.  The 1 = 130400/130400 is the number of classifications (N) divided by N. 

\n

So, the best result is very, very small, and corresponds to perfect classification.  What's a reasonable "best" result?  Not a clue.

", "rawMarkdown": "", "votes": null }, { "id": "62719", "postDate": "01/14/2015 14:49:51", "content": "

I don't think it is possible to determine a "best" result, without a strong model of how errors will appear in the predictions.

\n

Human classification errors are most likely to be confusion between small groups of similar-looking objects, but defining this, and putting a numerical value in based on just starting with the accuracy value is going to lead to wide variations.

\n

In fact, determining a lower bound on the score given this data is likely harder than making a winning entry into the competition. You are better off making a naive interpolation of the leaderboard scores. In which case, I am going to completely guess at around 0.5 

", "rawMarkdown": "", "votes": null }, { "id": "62733", "postDate": "01/15/2015 02:20:00", "content": "

Karle, I had the same question ("What is the best score possible?").  Instead I answered a similar one; "What score would a perfect model get".

\n

What I did was assume that I could build a model that predicted the class exactly.  Such a model would give p(i,j) = 1 if image i belonged to class j, and zero otherwise.  I then evaluated the model after having randomly shuffled the class of 2% of the images (to simulate the 98% human accuracy they quoted).  The resulting score is about 0.68.

\n

I suppose this is somewhat of an upper bound on the best score (as evidenced by the leaderboard!).  You could improve upon it (as Neil suggested) using a model that knows how a human is likely to mis-classify images.

", "rawMarkdown": "", "votes": null }, { "id": "62734", "postDate": "01/15/2015 02:50:50", "content": "

Another way of calculating a potential lower bound (given that humans have 98% agreement) is to assume that on 98% of the examples the model assigns p~1 and for the other 2% the model is unsure and assigns everything a probability of 1/121.

\n

This gives nll~.1, which contrasts Alex's calculation where in the 2%, the model assigns high probability to the wrong class (and very low probability to the correct class.)

", "rawMarkdown": "", "votes": null }, { "id": "62735", "postDate": "01/15/2015 03:26:46", "content": "

Too build on that a bit further I took the 'perfect' model and added a very simple human error model, so that instead of giving p(i,j) = 1, I set p(i,j) = 0.98 if image i belongs to class j and 0.02 / 120 otherwise.  Then did the same shuffling of 2% described above.  This gives a score of 0.19.

\n

Like Jesse mentioned the difference in score is due to the 'perfect' model assigning very low probability to 2% of the samples.

\n

Interesting that if someone managed to build a perfect model (and if the test set contains 2% errors) they'd get a 0.68 but by intentionally making the model 'worse' they could get a much better score!

", "rawMarkdown": "", "votes": null }, { "id": "62744", "postDate": "01/15/2015 09:32:37", "content": "

I agree with that.

\n

The problem is not to find a "true" classifier but to reproduce the experts work which is not perfect.

", "rawMarkdown": "", "votes": null }, { "id": "62756", "postDate": "01/15/2015 17:04:35", "content": "

thanks for the thought provoking replies, especially this one from Ben: The problem is not to find a "true" classifier but to reproduce the experts work which is not perfect.

\n

wow!

", "rawMarkdown": "", "votes": null }, { "id": "62758", "postDate": "01/15/2015 18:45:26", "content": "

No provocation here. I just wanted to point out that doing better than experts should lead to worse evalutation on learderboard.

\n

Here is a semi-math-based guess of the best possible score:

\n

Let's say there is a very large group of experts. Given a sample s , we note pi the proportion of experts who think the sample s is in the class i.

\n

Let's say a classifier attributes the probability pi' at the class i for this sample s.

\n

For an expert who thinks that the sample s is of class i, the evaluation score of the classifier will be : -log(pi').

\n

So the expected value of the evaluation over all the experts is : -p1×log(p1') - p2×log(p2') - p3×log(p3') + ...

\n

To minimize this expression, the classifier should attribute pi' = pi .

\n

Now let's try to guess the best possible score. In fact, we just assume that experts are right 95% to 98% of the time. So there is a lot of different configurations that lead to different best possible scores. I imagine two cases:

\n

1) worst case : 95% of the samples are well classed by all experts and the other 5% are classed uniformly over all the 121 classes.

\n

Then the best evaluation score will be : - 5% × ln(1/121) = 0.24

\n

2) best case : 96% of the samples are well classed by all experts and the other 4% correspond to two classes (2% each) that are totally similar.

\n

Then the best evaluation score will be : 2 × (- 2% × ln(1/2)) = 0.028

\n

I'm not entirely sure this two cases are the worst and the best ones but it lead me to think that 0.24 is a reachable score.

\n

Sorry if my english is bad, I'm french ;)

\n\n

Edit : Karle, I just understand what you mean by "thought provoking replies", I had read "provoking replies" without the "thought". As I said, my english ... :D

", "rawMarkdown": "", "votes": null }, { "id": "62771", "postDate": "01/16/2015 00:44:14", "content": "

Ben , votre anglais est tellement mieux que mon français . Merci pour l'explication de LOG .

\n

(Ben, well, your English is so much better than my French. Thank you for the LOG explanation)

\n

And to all the others too..Merci!.                       ( I needed Google translate to fix my message....)

", "rawMarkdown": "", "votes": null }, { "id": "65233", "postDate": "03/02/2015 13:09:39", "content": "

In all this discussion I'm missing one thought: there might be a pattern on how the experts fail to classify certain images. If so, our model might well be detecting that pattern and reproducing it. In which case, they would improve the 2% of the experts down to what is purely noise, wherever that might be.

", "rawMarkdown": "", "votes": null } ], "comments": [ { "id": 62251, "author_name": "davidthaler", "author_url": "", "post_date": "01/04/2015 01:14:58", "content": "

[quote=Parkinson Zabelin;62247]

\n

I am new to this board. Per multiclass logarithmic loss function=> 0 as p(i,j)=>1
. Am I missing something here?

\n

[/quote]

\n

Yes, you are missing that:

\n

$$\\sum_{j=1}^{M}p_{i,j}=1$$

\n

That is, for each example(i), the probabilities of the classes(the j's) must sum to 1.   

\n

Also, from the Evaluation page:   

\n

The submitted probabilities for a given image are not required to sum to one because they are rescaled prior to being scored (each row is divided by the row sum).   

\n

In other words, they force your probabilities to sum to one if the ones you provide don't. So if you submit all ones, your prediction will be rescaled to a uniform probability of 1/M for each class. That won't have a zero loss.

", "votes": null, "replies": [] }, { "id": 62255, "author_name": "slobo777", "author_url": "", "post_date": "01/04/2015 14:55:32", "content": "

[quote=David Thaler;62251]

\n

In other words, they force your probabilities to sum to one if the ones you provide don't. So if you submit all ones, your prediction will be rescaled to a uniform probability of 1/M for each class. That won't have a zero loss.

\n

[/quote]

\n

To add to this, due to the scaling this is the same as the "uniform probability" benchmark, and will score 4.795791 on the public leaderboard

", "votes": null, "replies": [] }, { "id": 62608, "author_name": "karle1", "author_url": "", "post_date": "01/11/2015 18:30:54", "content": "

Neil, and others,

\n

What is the (most likely) best score possible?

\n

Plankton experts are rated in the high 95-98% accuracy and this data base was hand labeled as well (according to the FAQ and other statements), so if a theoretical team of plankton experts were to take the time (eons) what would their score be?

", "votes": null, "replies": [] }, { "id": 62651, "author_name": "selfishgene", "author_url": "", "post_date": "01/12/2015 15:17:04", "content": "

There isn't a simple 1:1 mapping between the classification error and the softmax loss, but to get a feeling you can create an 'artificial classifier' that is correct p% of the time and calculate its loss.

a simple example that gives you different sofmax loss values for 0% classification error can be seen if you assign to the correct class the value "Pmax" and randomly spread the remaining (1-Pmax) along all the other classes. even though for Pmax > 0.5 you will get no classification error at all, the softmax loss will greatly depend on the value of Pmax itself.

", "votes": null, "replies": [] }, { "id": 62708, "author_name": "karle1", "author_url": "", "post_date": "01/14/2015 01:52:57", "content": "

Thanks SG, and others

\n

But does any one want to make a semi-math-based guess on the best score possible?

\n

(plus or minus some good guesstimate)?

\n

I am just trying to get a handle on how low the scores could possibly go.....

\n

thanks

\n

(full disclosure: I am log loss challenged)

", "votes": null, "replies": [] }, { "id": 62718, "author_name": "jqveenstra", "author_url": "", "post_date": "01/14/2015 14:41:04", "content": "

We have that the log-loss for multiclass is defined here.  Noting the convention that 0 log(0) = 0, if the probabilities were not bounded below/above by 10**(-15), 1-10**(-15), the minimum log loss is 0, which corresponds to the correct class being selected with probability 1, and the other classes having probability 0.

\n

Since our log-losses are bounded, if this labeling is given, the log loss is:

\n

-(1 * 1 *  log(1-120*(10**(-15))) + 0 * 120 * log(10**(-15))) * 130400/130400 = 1.200151e-13,

\n

approximately.

\n

The 1 * 1 * log(1-120*(10**(-15)))  is 1 for y_{i, j}  times 1 for the correct classification times log(1-120*(10**(-15))) for the log-probability (assuming that's how they compute the probability when all are 1's and 0's), and 0 is for y_{i, j} times 120 for the other 120 classes times the log-probability.  The 1 = 130400/130400 is the number of classifications (N) divided by N. 

\n

So, the best result is very, very small, and corresponds to perfect classification.  What's a reasonable "best" result?  Not a clue.

", "votes": null, "replies": [] }, { "id": 62719, "author_name": "slobo777", "author_url": "", "post_date": "01/14/2015 14:49:51", "content": "

I don't think it is possible to determine a "best" result, without a strong model of how errors will appear in the predictions.

\n

Human classification errors are most likely to be confusion between small groups of similar-looking objects, but defining this, and putting a numerical value in based on just starting with the accuracy value is going to lead to wide variations.

\n

In fact, determining a lower bound on the score given this data is likely harder than making a winning entry into the competition. You are better off making a naive interpolation of the leaderboard scores. In which case, I am going to completely guess at around 0.5 

", "votes": null, "replies": [] }, { "id": 62733, "author_name": "alexkleeman2", "author_url": "", "post_date": "01/15/2015 02:20:00", "content": "

Karle, I had the same question ("What is the best score possible?").  Instead I answered a similar one; "What score would a perfect model get".

\n

What I did was assume that I could build a model that predicted the class exactly.  Such a model would give p(i,j) = 1 if image i belonged to class j, and zero otherwise.  I then evaluated the model after having randomly shuffled the class of 2% of the images (to simulate the 98% human accuracy they quoted).  The resulting score is about 0.68.

\n

I suppose this is somewhat of an upper bound on the best score (as evidenced by the leaderboard!).  You could improve upon it (as Neil suggested) using a model that knows how a human is likely to mis-classify images.

", "votes": null, "replies": [] }, { "id": 62734, "author_name": "jesselivezey", "author_url": "", "post_date": "01/15/2015 02:50:50", "content": "

Another way of calculating a potential lower bound (given that humans have 98% agreement) is to assume that on 98% of the examples the model assigns p~1 and for the other 2% the model is unsure and assigns everything a probability of 1/121.

\n

This gives nll~.1, which contrasts Alex's calculation where in the 2%, the model assigns high probability to the wrong class (and very low probability to the correct class.)

", "votes": null, "replies": [] }, { "id": 62735, "author_name": "alexkleeman2", "author_url": "", "post_date": "01/15/2015 03:26:46", "content": "

Too build on that a bit further I took the 'perfect' model and added a very simple human error model, so that instead of giving p(i,j) = 1, I set p(i,j) = 0.98 if image i belongs to class j and 0.02 / 120 otherwise.  Then did the same shuffling of 2% described above.  This gives a score of 0.19.

\n

Like Jesse mentioned the difference in score is due to the 'perfect' model assigning very low probability to 2% of the samples.

\n

Interesting that if someone managed to build a perfect model (and if the test set contains 2% errors) they'd get a 0.68 but by intentionally making the model 'worse' they could get a much better score!

", "votes": null, "replies": [] }, { "id": 62744, "author_name": "irmo322", "author_url": "", "post_date": "01/15/2015 09:32:37", "content": "

I agree with that.

\n

The problem is not to find a "true" classifier but to reproduce the experts work which is not perfect.

", "votes": null, "replies": [] }, { "id": 62756, "author_name": "karle1", "author_url": "", "post_date": "01/15/2015 17:04:35", "content": "

thanks for the thought provoking replies, especially this one from Ben: The problem is not to find a "true" classifier but to reproduce the experts work which is not perfect.

\n

wow!

", "votes": null, "replies": [] }, { "id": 62758, "author_name": "irmo322", "author_url": "", "post_date": "01/15/2015 18:45:26", "content": "

No provocation here. I just wanted to point out that doing better than experts should lead to worse evalutation on learderboard.

\n

Here is a semi-math-based guess of the best possible score:

\n

Let's say there is a very large group of experts. Given a sample s , we note pi the proportion of experts who think the sample s is in the class i.

\n

Let's say a classifier attributes the probability pi' at the class i for this sample s.

\n

For an expert who thinks that the sample s is of class i, the evaluation score of the classifier will be : -log(pi').

\n

So the expected value of the evaluation over all the experts is : -p1×log(p1') - p2×log(p2') - p3×log(p3') + ...

\n

To minimize this expression, the classifier should attribute pi' = pi .

\n

Now let's try to guess the best possible score. In fact, we just assume that experts are right 95% to 98% of the time. So there is a lot of different configurations that lead to different best possible scores. I imagine two cases:

\n

1) worst case : 95% of the samples are well classed by all experts and the other 5% are classed uniformly over all the 121 classes.

\n

Then the best evaluation score will be : - 5% × ln(1/121) = 0.24

\n

2) best case : 96% of the samples are well classed by all experts and the other 4% correspond to two classes (2% each) that are totally similar.

\n

Then the best evaluation score will be : 2 × (- 2% × ln(1/2)) = 0.028

\n

I'm not entirely sure this two cases are the worst and the best ones but it lead me to think that 0.24 is a reachable score.

\n

Sorry if my english is bad, I'm french ;)

\n\n

Edit : Karle, I just understand what you mean by "thought provoking replies", I had read "provoking replies" without the "thought". As I said, my english ... :D

", "votes": null, "replies": [] }, { "id": 62771, "author_name": "karle1", "author_url": "", "post_date": "01/16/2015 00:44:14", "content": "

Ben , votre anglais est tellement mieux que mon français . Merci pour l'explication de LOG .

\n

(Ben, well, your English is so much better than my French. Thank you for the LOG explanation)

\n

And to all the others too..Merci!.                       ( I needed Google translate to fix my message....)

", "votes": null, "replies": [] }, { "id": 65233, "author_name": "atreo1", "author_url": "", "post_date": "03/02/2015 13:09:39", "content": "

In all this discussion I'm missing one thought: there might be a pattern on how the experts fail to classify certain images. If so, our model might well be detecting that pattern and reproducing it. In which case, they would improve the 2% of the experts down to what is purely noise, wherever that might be.

", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "62247": "", "62251": "", "62255": "", "62608": "", "62651": "", "62708": "", "62718": "", "62719": "", "62733": "", "62734": "", "62735": "", "62744": "", "62756": "", "62758": "", "62771": "", "65233": "" }, "source": "meta" }