{ "id": 12465, "title": "Can you guys share percentage accuracy on validation data?", "url": "/competitions/datasciencebowl/discussion/12465", "author_name": "", "post_date": "2015-02-10T09:35:46.050Z", "votes": 1, "comment_count": 36, "views": 9582, "content": "

Hi,

\n

Can you guys share percentage accuracy versus log loss score on your training data?

\n

I have a score of 1.06 with 67.5% accuracy on validation data.

", "messages": [ { "id": "63903", "postDate": "02/10/2015 09:35:46", "content": "

Hi,

\n

Can you guys share percentage accuracy versus log loss score on your training data?

\n

I have a score of 1.06 with 67.5% accuracy on validation data.

", "rawMarkdown": "", "votes": null }, { "id": "63904", "postDate": "02/10/2015 09:41:21", "content": "

Hi, very similar to my results.

\n

valid (random 20%): Accuracy=0.683 LogLossMulti=1.1244

\n

leaderboard: 1.067027

", "rawMarkdown": "", "votes": null }, { "id": "63910", "postDate": "02/10/2015 10:53:22", "content": "

Thanks Michael.

\n

I wanted to know the maximum percentage accuracy people have got. It looks like the log-loss function needs to be optimized separately, I have found that even with 35% accuracy you can beat the top score if you are lucky with random submission. So basically log-loss is bit fishy.

", "rawMarkdown": "", "votes": null }, { "id": "63912", "postDate": "02/10/2015 11:19:14", "content": "

[quote=Sash;63910]

\n

Thanks Michael.

\n

I wanted to know the maximum percentage accuracy people have got. It looks like the log-loss function needs to be optimized separately, I have found that even with 35% accuracy you can beat the top score if you are lucky with random submission. So basically log-loss is bit fishy.

\n

[/quote]

\n

Yep, logloss sounds also not natural for me given the problem here.

\n

By changing my train method I can achieve 71% accuracy, but logloss dropped a lot.

", "rawMarkdown": "", "votes": null }, { "id": "63933", "postDate": "02/10/2015 16:40:22", "content": "

[quote=Sash;63910]

\n

Thanks Michael.

\n

I have found that even with 35% accuracy you can beat the top score if you are lucky with random submission.

\n

[/quote]

\n

Can you explain this a bit more? 

", "rawMarkdown": "", "votes": null }, { "id": "63940", "postDate": "02/10/2015 17:13:46", "content": "

My cv accuracy is as high as 74% but, as other folks have reported, that doesn't always go hand in hand with lower logloss.

", "rawMarkdown": "", "votes": null }, { "id": "63942", "postDate": "02/10/2015 17:20:30", "content": "

[quote=Michael Jahrer;63904]

\n

Hi, very similar to my results.

\n

valid (random 20%): Accuracy=0.683 LogLossMulti=1.1244

\n

leaderboard: 1.067027

\n

[/quote]

\n

Is it correct that you don't use CNNs? This would be an accomplishment.

\n

btw 0.87 logloss at ~74% acc

", "rawMarkdown": "", "votes": null }, { "id": "63959", "postDate": "02/10/2015 20:00:27", "content": "

[quote=Rafael;63942]

\n

[quote=Michael Jahrer;63904]

\n

Hi, very similar to my results.

\n

valid (random 20%): Accuracy=0.683 LogLossMulti=1.1244

\n

leaderboard: 1.067027

\n

[/quote]

\n

Is it correct that you don't use CNNs? This would be an accomplishment.

\n

btw 0.87 logloss at ~74% acc

\n

[/quote]

\n\n

yep, no CNN, but NN with geometric data augmentation :) - I cracked the 1.0 on the leaderboard. Thats it for now

", "rawMarkdown": "", "votes": null }, { "id": "63994", "postDate": "02/11/2015 02:39:38", "content": "

[quote=emolson;63933]

\n

[quote=Sash;63910]

\n

Thanks Michael.

\n

I have found that even with 35% accuracy you can beat the top score if you are lucky with random submission.

\n

[/quote]

\n

Can you explain this a bit more? 

\n

[/quote]

\n

I just meant it is possible mathematically.. but looks impossible probabilistically.

\n

Say percentage accuracy is 35. so if we have probability of 1 for those images, contribution to CV score from 35% of data is 0. that is 0.35*ln(1) = 0.

\n

For the rest 65% which is erroneous - let us say we have probability of 0.49 for those locations where y(i,j) in log loss function so it sums up to -0.65*ln(0.49) = 0.4636.. so the total score is 0.4636...

", "rawMarkdown": "", "votes": null }, { "id": "64066", "postDate": "02/11/2015 21:11:42", "content": "

Mayby its goes as follows :

\n

for one observation LogLoss takes the following form : LL = -y*log(p), but we don't know `y`, if we could estimate probability of correct guess as 'q', then we obtain expected LogLoss in the form : E(LL) = -q*log(p) -(1-q)*log((1-p)/(n-1)), where n is number of classes, and probability assigned, to each of them is (1-p)/(n-1), now take derivative over p, and find p in terms of q and n, for n=1 case  solution is simply p = q :-) for other values of n please calculate yourself, you can further optimize with use of a priori classes probabilities, deriving analytical solution should not be as hard as it seems

", "rawMarkdown": "", "votes": null }, { "id": "64087", "postDate": "02/12/2015 03:35:57", "content": "

[quote=xvnv;64066]

\n

Mayby its goes as follows :

\n

for one observation LogLoss takes the following form : LL = -y*log(p), but we don't know `y`, if we could estimate probability of correct guess as 'q', then we obtain expected LogLoss in the form : E(LL) = -q*log(p) -(1-q)*log((1-p)/(n-1)), where n is number of classes, and probability assigned, to each of them is (1-p)/(n-1), now take derivative over p, and find p in terms of q and n, for n=1 case  solution is simply p = q :-) for other values of n please calculate yourself, you can further optimize with use of a priori classes probabilities, deriving analytical solution should not be as hard as it seems

\n

[/quote]

\n\n

This is the first thing I tried, but with this we cannot beat the best score unless we have an accuracy of 90+ percentage...may be 93% can't recall exactly..

", "rawMarkdown": "", "votes": null }, { "id": "65232", "postDate": "03/02/2015 12:54:58", "content": "

Logloss is there to penalize wildly wrong guesses. They are telling you it's better to have two options with 60%-40% accuracy (for example) than saying "this one has 60%, so I'll go with this as the right option". My problem in this is that it creates some sort of moral dilema, since there is a lot of tricks you could use to reduce your logloss artificially, without improving your accuracy or even being realistic. I want to really give the probabilities as my model predicts them, without having to look at the logloss score.

", "rawMarkdown": "", "votes": null }, { "id": "65374", "postDate": "03/04/2015 02:13:39", "content": "

Just to jump in here and offer my 2 cents. In our field (plankton imaging), one of the early papers put out ~83% overall accuracy as a high benchmark, and it has become a sort of "gold standard." While I will - and have - argued that this benchmark is arbitrary and somewhat questionably derived, it remains that our field has yet to see this threshold broken with a real dataset (not an artificially curated learning set). 

\n

I'm seeing on this forum and this thread accuracy rates of 78% - and possibly higher? - so I'm wondering whether this competition can break through this arbitrary limit of 83% accuracy. It would be very exciting to see it happen. 

", "rawMarkdown": "", "votes": null }, { "id": "65377", "postDate": "03/04/2015 03:53:41", "content": "

@Jessica, I believe top ten players already went beyond 83%:)

", "rawMarkdown": "", "votes": null }, { "id": "65385", "postDate": "03/04/2015 06:27:43", "content": "

@Jessica

\n

Your are the competition admin, do you not know if it has broken 83% accuracy limit?

", "rawMarkdown": "", "votes": null }, { "id": "65386", "postDate": "03/04/2015 06:47:19", "content": "

My statistic on 10% validation set is: logloss~0.64 and accuracy~80%. So I support @Guocong Song's guess.

\n

If the evaluation is the accuracy itself, a higher accuracy can be reached using other loss functions, e.g., hinge loss.

", "rawMarkdown": "", "votes": null }, { "id": "65387", "postDate": "03/04/2015 07:02:48", "content": "

[quote=yr;65386]

\n

My statistic on 10% validation set is: logloss~0.64 and accuracy~80%. So I support @Guocong Song's guess.

\n

If the evaluation is the accuracy itself, a higher accuracy can be reached using other loss functions, e.g., hinge loss.

\n

[/quote]

\n

Can you explain how we can achieve higher accuracy with other loss functions? Is it the objective function of neural network that we need to change?

", "rawMarkdown": "", "votes": null }, { "id": "65388", "postDate": "03/04/2015 07:05:15", "content": "

I don`t support, may be top-2 only :) Around 0.6 I have noticed that my more accurate models scores worse in logloss and visa-versa.

", "rawMarkdown": "", "votes": null }, { "id": "65389", "postDate": "03/04/2015 07:13:36", "content": "

for people who would want to get percentage accuracy on training set, all you have to do is submit an entry containing only ones and zeros. That is in each row replace the max probability with 1 and rest with zeros.

\n

Then if you get a score X, the percentage accuracy would be

\n

1+ (x/log(10^-15))

\n

log is natural logarithm.

\n

So with this you can share your actual test data percentage accuracy if you wish.

", "rawMarkdown": "", "votes": null }, { "id": "65390", "postDate": "03/04/2015 07:13:45", "content": "

10% validation set is not stable. The same model has ~+-0.04 difference on logloss.

", "rawMarkdown": "", "votes": null }, { "id": "65394", "postDate": "03/04/2015 10:08:03", "content": "

[quote=Sachin;65387]

\n

[quote=yr;65386]

\n

My statistic on 10% validation set is: logloss~0.64 and accuracy~80%. So I support @Guocong Song's guess.

\n

If the evaluation is the accuracy itself, a higher accuracy can be reached using other loss functions, e.g., hinge loss.

\n

[/quote]

\n

Can you explain how we can achieve higher accuracy with other loss functions? Is it the objective function of neural network that we need to change?

\n

[/quote]

\n\n

If you would like to get higher accuracy using other loss function, you should change loss function. In Caffe you should change "SOFTMAX_LOOS" to "HINGE_LOSS". This will change from LinerarClassifier to SVM. But for your information, SVM does not care about loss, only accuracy.

\n

I think that about is Sachin was thinking about 

", "rawMarkdown": "", "votes": null }, { "id": "65431", "postDate": "03/04/2015 19:24:05", "content": "

[quote=Jessica Luo;65374]

\n

Just to jump in here and offer my 2 cents. In our field (plankton imaging), one of the early papers put out ~83% overall accuracy as a high benchmark, and it has become a sort of "gold standard." While I will - and have - argued that this benchmark is arbitrary and somewhat questionably derived, it remains that our field has yet to see this threshold broken with a real dataset (not an artificially curated learning set). 

\n

I'm seeing on this forum and this thread accuracy rates of 78% - and possibly higher? - so I'm wondering whether this competition can break through this arbitrary limit of 83% accuracy. It would be very exciting to see it happen. 

\n

[/quote]

\n

I am curious about this paper.  Do you have a link?  I searched and found quite a few references to RAPID and an accuracy of 70-80% on 10-20 classes.  There was also mention that the classification was often subjective among scientists, which is why we get class overlap like protist_other and protist_noctiluca.  

\n

At any rate, it seems that this dataset is the most extensive to date.  This should help get great results.

", "rawMarkdown": "", "votes": null }, { "id": "65437", "postDate": "03/04/2015 20:43:22", "content": "

[quote=Ryan Munion;65431]

\n

[quote=Jessica Luo;65374]

\n

Just to jump in here and offer my 2 cents. In our field (plankton imaging), one of the early papers put out ~83% overall accuracy as a high benchmark, and it has become a sort of "gold standard." While I will - and have - argued that this benchmark is arbitrary and somewhat questionably derived, it remains that our field has yet to see this threshold broken with a real dataset (not an artificially curated learning set). 

\n

I'm seeing on this forum and this thread accuracy rates of 78% - and possibly higher? - so I'm wondering whether this competition can break through this arbitrary limit of 83% accuracy. It would be very exciting to see it happen. 

\n

[/quote]

\n

I am curious about this paper.  Do you have a link?  I searched and found quite a few references to RAPID and an accuracy of 70-80% on 10-20 classes.  There was also mention that the classification was often subjective among scientists, which is why we get class overlap like protist_other and protist_noctiluca.  

\n

At any rate, it seems that this dataset is the most extensive to date.  This should help get great results.

\n

[/quote]

\n

Hi Ryan,

\n

The RAPID article (Benfield et al. 2007) that you mention is a great one for an intro to field and techniques used, and it has a more complete overview of the history of these imaging systems than what I am going to provide here. But nonetheless - to answer your question, the Video Plankton Recorder ("VPR"; Davis et al. 1992) was one of the first - and still widely used - plankton imaging systems developed. The VPR team also developed one of the early automatic plankton classification techniques using SVMs and Neural Nets (Hu and Davis 2005, Hu and Davis 2006). That approach, along with Gabriel Gorsky's team's Random Forest based approach (Gorsky et al. 2010) remains among the more widely used / accepted approaches within plankton imaging today. 

\n

Hu and Davis (VPR; 2005) were among the first to put out a benchmark for "acceptable accuracy" in classification: 67-83%, which is based on the work of Culverhouse et al. (2003). For better or for worse, that number has more or less stuck (though some research groups functionally ignore accuracy and recall and optimize their workflow for analysis/processing time) -- and has become that high benchmark in our field. Of course, you can curate any kind of ideal learning set from your data and obtain a near perfect classification accuracy on it, but to then take your model and apply to your real data -- particularly in an unbounded problem, with Unknowns -- I have yet to see this value exceeded. So -- while it's really exciting to see us in an unbounded, 120+ class problem that represents real world data(*) get so close to breaking this benchmark, the real test would be when we take these algorithms to the whole dataset to see how they perform. That's when the magic will happen!

\n

Cheers,

\n

Jessica

\n

* - Of course, this competition dataset is not completely representative of our actual dataset. If so, you would get about 1 fish to 1,000 detritus and that would be no fun at all. Plus, everyone would optimize their models to perfectly predict the detritus and get great log-loss scores whereas what we're really interested in is classifying the fish and other organisms.

", "rawMarkdown": "", "votes": null }, { "id": "65462", "postDate": "03/04/2015 22:49:50", "content": "

Thank you.

\n

I will be interested in seeing how the results fair in practice.

", "rawMarkdown": "", "votes": null }, { "id": "66018", "postDate": "03/12/2015 08:16:06", "content": "

My accuracy on test data is 73.5% with a log-loss of 0.862932 on the leader board.

", "rawMarkdown": "", "votes": null }, { "id": "66026", "postDate": "03/12/2015 09:12:28", "content": "

We are seeing accuracy of 80% with log-loss around 1.0. Any advice on how to optimize log-loss?

", "rawMarkdown": "", "votes": null }, { "id": "66028", "postDate": "03/12/2015 09:42:15", "content": "

[quote=Alex Zam;66026]

\n

We are seeing accuracy of 80% with log-loss around 1.0. Any advice on how to optimize log-loss?

\n

[/quote]

\n

Is it 80% accurate on test data - that is 130400 images?

", "rawMarkdown": "", "votes": null }, { "id": "66033", "postDate": "03/12/2015 12:09:33", "content": "

Guys, my best single model performs 71.22% accuracy on validation data and it is 0.93 on LB.

", "rawMarkdown": "", "votes": null }, { "id": "66049", "postDate": "03/12/2015 16:19:42", "content": "

They couldn't possibly know if it were  80% accurate on the test data without cheating and hand labeling, Sachin.

\n\n

Anyway, I'm at ~73-74% accuracy.

", "rawMarkdown": "", "votes": null }, { "id": "66051", "postDate": "03/12/2015 17:17:37", "content": "

[quote=Torgos;66049]

\n

They couldn't possibly know if it were  80% accurate on the test data without cheating and hand labeling, Sachin.

\n

Anyway, I'm at ~73-74% accuracy.

\n

[/quote]

\n

Hi Torgos, you need not cheat and hand label to get percentage accuracy on test data.

\n

On this thread I have given a method to deduce percentage accuracy from log-loss, you can try that.

", "rawMarkdown": "", "votes": null }, { "id": "66170", "postDate": "03/14/2015 09:32:13", "content": "

@Sachin, would like to explain your aforementioned equation in details?

", "rawMarkdown": "", "votes": null }, { "id": "66173", "postDate": "03/14/2015 10:45:01", "content": "

[quote=PengPai;66170]

\n

@Sachin, would like to explain your aforementioned equation in details?

\n

[/quote]

\n

Sure PengPai I will try my best.

\n\n

Note: All logarithms are to the base "e"

\n

Lets say there are 2 classes- class 0 and class 1. And there are 5 images. Let the probabilities predicated for each class be as follows

\n

class0   class1   actual_class

\n

0.1          0.9            class1

\n

0.2          0.8            class1

\n

0.3          0.7            class1

\n

0.6          0.4            class1

\n

0.25        0.75          class0

\n\n

so the percentage accuracy would be 3*100/5 = 60%  which is unknown at the moment for the test data. But the log-loss calculated based on the evaluation metric is

\n

-(log(0.9)+log(0.8)+log(0.7)+log(0.4)+log(0.25))/5 = 0.59755282

\n

But in order to get your percentage accuracy you need to make another submission making the class with highest probability in each row to be 1 and the rest to be 0.

\n

that is

\n

class0       class1      actual_class

\n

0                   1                  class1

\n

0                   1                   class1

\n

0                    1                  class1

\n

1                   0                    class1

\n

0                  1                     class0

\n\n

so your CV score would be  -(log(1) + log(1) + log(1)+log(10^-15)+log(10^-15))/5 = 13.81551056

\n

since log(0) is -infinity the smallest number they consider is 10^-15.

\n\n

In order to get the percentage accuracy from this do

\n

1+ (13.81551056/log(10^-15)) = 0.6 = 60%

\n

If it is still not clear you can ask me again.

\n

Thanks

", "rawMarkdown": "", "votes": null }, { "id": "66174", "postDate": "03/14/2015 10:48:51", "content": "

[quote=Sachin;66051]

\n

[quote=Torgos;66049]

\n

They couldn't possibly know if it were  80% accurate on the test data without cheating and hand labeling, Sachin.

\n

Anyway, I'm at ~73-74% accuracy.

\n

[/quote]

\n

Hi Torgos, you need not cheat and hand label to get percentage accuracy on test data.

\n

On this thread I have given a method to deduce percentage accuracy from log-loss, you can try that.

\n

[/quote]

\n\n

Hi Torgos,

\n

I don't really understand why you down voted my comment, but please have a look at the solution I have provided to get the percentage accuracy and then you can decide.

\n

Thanks

\n

-Sachin

", "rawMarkdown": "", "votes": null }, { "id": "66232", "postDate": "03/15/2015 12:36:23", "content": "

@Jessica Luo: After the competition could you please post some of the leading accuracy scores as well?  Just for curiosity :-).  Thanks!

\n\n

[quote=Jessica Luo;65437]

\n

Hu and Davis (VPR; 2005) were among the first to put out a benchmark for "acceptable accuracy" in classification: 67-83%, which is based on the work of Culverhouse et al. (2003).

\n

[/quote]

", "rawMarkdown": "", "votes": null }, { "id": "66281", "postDate": "03/15/2015 20:56:00", "content": "

Instead of just looking at the overall accuracy, I did a breakdown of the per class percentage accuracy.

The charts are for: (i) total counts (ii) miss counts (iii) miss percentages – all as a function of class id (121 plankton classes, alphabetically sorted). For each image, a “hit” is attributed to a class only if the highest probability class prediction matches the actual class in the training set (last layer is CNN softmax). These stats are based on 80% of the ~30k training set from which the model predictions were created - this is usually a no no, but I also ran this procedure on the validation set and got very similar results (the problem with the validation set is that some classes are very small).

\n

In order to get a good overall score, a model needs to somehow lower those high bars in the middle misses chart, especially those with high miss rates (bottom chart).

The usual suspects for poorly predicted classes are the aptly named “others” and “unknowns” e.g. class ids: 11,85,118,119,120

", "rawMarkdown": "", "votes": null }, { "id": "66282", "postDate": "03/15/2015 20:57:41", "content": "

... inline image didn't work, please see attachment.

", "rawMarkdown": "", "votes": null }, { "id": "66430", "postDate": "03/16/2015 18:15:54", "content": "

[quote=Moy;66232]

\n

@Jessica Luo: After the competition could you please post some of the leading accuracy scores as well?  Just for curiosity :-).  Thanks!

\n

[/quote]

\n

Yup - sure! I'll pull a few from the top of the leaderboard after the competition ends. Good luck to all in the final push!

", "rawMarkdown": "", "votes": null } ], "comments": [ { "id": 63904, "author_name": "mjahrer", "author_url": "", "post_date": "02/10/2015 09:41:21", "content": "

Hi, very similar to my results.

\n

valid (random 20%): Accuracy=0.683 LogLossMulti=1.1244

\n

leaderboard: 1.067027

", "votes": null, "replies": [] }, { "id": 63910, "author_name": "sachinvernekar", "author_url": "", "post_date": "02/10/2015 10:53:22", "content": "

Thanks Michael.

\n

I wanted to know the maximum percentage accuracy people have got. It looks like the log-loss function needs to be optimized separately, I have found that even with 35% accuracy you can beat the top score if you are lucky with random submission. So basically log-loss is bit fishy.

", "votes": null, "replies": [] }, { "id": 63912, "author_name": "mjahrer", "author_url": "", "post_date": "02/10/2015 11:19:14", "content": "

[quote=Sash;63910]

\n

Thanks Michael.

\n

I wanted to know the maximum percentage accuracy people have got. It looks like the log-loss function needs to be optimized separately, I have found that even with 35% accuracy you can beat the top score if you are lucky with random submission. So basically log-loss is bit fishy.

\n

[/quote]

\n

Yep, logloss sounds also not natural for me given the problem here.

\n

By changing my train method I can achieve 71% accuracy, but logloss dropped a lot.

", "votes": null, "replies": [] }, { "id": 63933, "author_name": "emolson", "author_url": "", "post_date": "02/10/2015 16:40:22", "content": "

[quote=Sash;63910]

\n

Thanks Michael.

\n

I have found that even with 35% accuracy you can beat the top score if you are lucky with random submission.

\n

[/quote]

\n

Can you explain this a bit more? 

", "votes": null, "replies": [] }, { "id": 63940, "author_name": "adjgiulio", "author_url": "", "post_date": "02/10/2015 17:13:46", "content": "

My cv accuracy is as high as 74% but, as other folks have reported, that doesn't always go hand in hand with lower logloss.

", "votes": null, "replies": [] }, { "id": 63942, "author_name": "potamitis", "author_url": "", "post_date": "02/10/2015 17:20:30", "content": "

[quote=Michael Jahrer;63904]

\n

Hi, very similar to my results.

\n

valid (random 20%): Accuracy=0.683 LogLossMulti=1.1244

\n

leaderboard: 1.067027

\n

[/quote]

\n

Is it correct that you don't use CNNs? This would be an accomplishment.

\n

btw 0.87 logloss at ~74% acc

", "votes": null, "replies": [] }, { "id": 63959, "author_name": "mjahrer", "author_url": "", "post_date": "02/10/2015 20:00:27", "content": "

[quote=Rafael;63942]

\n

[quote=Michael Jahrer;63904]

\n

Hi, very similar to my results.

\n

valid (random 20%): Accuracy=0.683 LogLossMulti=1.1244

\n

leaderboard: 1.067027

\n

[/quote]

\n

Is it correct that you don't use CNNs? This would be an accomplishment.

\n

btw 0.87 logloss at ~74% acc

\n

[/quote]

\n\n

yep, no CNN, but NN with geometric data augmentation :) - I cracked the 1.0 on the leaderboard. Thats it for now

", "votes": null, "replies": [] }, { "id": 63994, "author_name": "sachinvernekar", "author_url": "", "post_date": "02/11/2015 02:39:38", "content": "

[quote=emolson;63933]

\n

[quote=Sash;63910]

\n

Thanks Michael.

\n

I have found that even with 35% accuracy you can beat the top score if you are lucky with random submission.

\n

[/quote]

\n

Can you explain this a bit more? 

\n

[/quote]

\n

I just meant it is possible mathematically.. but looks impossible probabilistically.

\n

Say percentage accuracy is 35. so if we have probability of 1 for those images, contribution to CV score from 35% of data is 0. that is 0.35*ln(1) = 0.

\n

For the rest 65% which is erroneous - let us say we have probability of 0.49 for those locations where y(i,j) in log loss function so it sums up to -0.65*ln(0.49) = 0.4636.. so the total score is 0.4636...

", "votes": null, "replies": [] }, { "id": 64066, "author_name": "oneoff", "author_url": "", "post_date": "02/11/2015 21:11:42", "content": "

Mayby its goes as follows :

\n

for one observation LogLoss takes the following form : LL = -y*log(p), but we don't know `y`, if we could estimate probability of correct guess as 'q', then we obtain expected LogLoss in the form : E(LL) = -q*log(p) -(1-q)*log((1-p)/(n-1)), where n is number of classes, and probability assigned, to each of them is (1-p)/(n-1), now take derivative over p, and find p in terms of q and n, for n=1 case  solution is simply p = q :-) for other values of n please calculate yourself, you can further optimize with use of a priori classes probabilities, deriving analytical solution should not be as hard as it seems

", "votes": null, "replies": [] }, { "id": 64087, "author_name": "sachinvernekar", "author_url": "", "post_date": "02/12/2015 03:35:57", "content": "

[quote=xvnv;64066]

\n

Mayby its goes as follows :

\n

for one observation LogLoss takes the following form : LL = -y*log(p), but we don't know `y`, if we could estimate probability of correct guess as 'q', then we obtain expected LogLoss in the form : E(LL) = -q*log(p) -(1-q)*log((1-p)/(n-1)), where n is number of classes, and probability assigned, to each of them is (1-p)/(n-1), now take derivative over p, and find p in terms of q and n, for n=1 case  solution is simply p = q :-) for other values of n please calculate yourself, you can further optimize with use of a priori classes probabilities, deriving analytical solution should not be as hard as it seems

\n

[/quote]

\n\n

This is the first thing I tried, but with this we cannot beat the best score unless we have an accuracy of 90+ percentage...may be 93% can't recall exactly..

", "votes": null, "replies": [] }, { "id": 65232, "author_name": "atreo1", "author_url": "", "post_date": "03/02/2015 12:54:58", "content": "

Logloss is there to penalize wildly wrong guesses. They are telling you it's better to have two options with 60%-40% accuracy (for example) than saying "this one has 60%, so I'll go with this as the right option". My problem in this is that it creates some sort of moral dilema, since there is a lot of tricks you could use to reduce your logloss artificially, without improving your accuracy or even being realistic. I want to really give the probabilities as my model predicts them, without having to look at the logloss score.

", "votes": null, "replies": [] }, { "id": 65374, "author_name": "jessicaluo", "author_url": "", "post_date": "03/04/2015 02:13:39", "content": "

Just to jump in here and offer my 2 cents. In our field (plankton imaging), one of the early papers put out ~83% overall accuracy as a high benchmark, and it has become a sort of "gold standard." While I will - and have - argued that this benchmark is arbitrary and somewhat questionably derived, it remains that our field has yet to see this threshold broken with a real dataset (not an artificially curated learning set). 

\n

I'm seeing on this forum and this thread accuracy rates of 78% - and possibly higher? - so I'm wondering whether this competition can break through this arbitrary limit of 83% accuracy. It would be very exciting to see it happen. 

", "votes": null, "replies": [] }, { "id": 65377, "author_name": "songgc", "author_url": "", "post_date": "03/04/2015 03:53:41", "content": "

@Jessica, I believe top ten players already went beyond 83%:)

", "votes": null, "replies": [] }, { "id": 65385, "author_name": "sachinvernekar", "author_url": "", "post_date": "03/04/2015 06:27:43", "content": "

@Jessica

\n

Your are the competition admin, do you not know if it has broken 83% accuracy limit?

", "votes": null, "replies": [] }, { "id": 65386, "author_name": "chenglongchen", "author_url": "", "post_date": "03/04/2015 06:47:19", "content": "

My statistic on 10% validation set is: logloss~0.64 and accuracy~80%. So I support @Guocong Song's guess.

\n

If the evaluation is the accuracy itself, a higher accuracy can be reached using other loss functions, e.g., hinge loss.

", "votes": null, "replies": [] }, { "id": 65387, "author_name": "sachinvernekar", "author_url": "", "post_date": "03/04/2015 07:02:48", "content": "

[quote=yr;65386]

\n

My statistic on 10% validation set is: logloss~0.64 and accuracy~80%. So I support @Guocong Song's guess.

\n

If the evaluation is the accuracy itself, a higher accuracy can be reached using other loss functions, e.g., hinge loss.

\n

[/quote]

\n

Can you explain how we can achieve higher accuracy with other loss functions? Is it the objective function of neural network that we need to change?

", "votes": null, "replies": [] }, { "id": 65388, "author_name": "oldufo", "author_url": "", "post_date": "03/04/2015 07:05:15", "content": "

I don`t support, may be top-2 only :) Around 0.6 I have noticed that my more accurate models scores worse in logloss and visa-versa.

", "votes": null, "replies": [] }, { "id": 65389, "author_name": "sachinvernekar", "author_url": "", "post_date": "03/04/2015 07:13:36", "content": "

for people who would want to get percentage accuracy on training set, all you have to do is submit an entry containing only ones and zeros. That is in each row replace the max probability with 1 and rest with zeros.

\n

Then if you get a score X, the percentage accuracy would be

\n

1+ (x/log(10^-15))

\n

log is natural logarithm.

\n

So with this you can share your actual test data percentage accuracy if you wish.

", "votes": null, "replies": [] }, { "id": 65390, "author_name": "liangpang", "author_url": "", "post_date": "03/04/2015 07:13:45", "content": "

10% validation set is not stable. The same model has ~+-0.04 difference on logloss.

", "votes": null, "replies": [] }, { "id": 65394, "author_name": "melgor", "author_url": "", "post_date": "03/04/2015 10:08:03", "content": "

[quote=Sachin;65387]

\n

[quote=yr;65386]

\n

My statistic on 10% validation set is: logloss~0.64 and accuracy~80%. So I support @Guocong Song's guess.

\n

If the evaluation is the accuracy itself, a higher accuracy can be reached using other loss functions, e.g., hinge loss.

\n

[/quote]

\n

Can you explain how we can achieve higher accuracy with other loss functions? Is it the objective function of neural network that we need to change?

\n

[/quote]

\n\n

If you would like to get higher accuracy using other loss function, you should change loss function. In Caffe you should change "SOFTMAX_LOOS" to "HINGE_LOSS". This will change from LinerarClassifier to SVM. But for your information, SVM does not care about loss, only accuracy.

\n

I think that about is Sachin was thinking about 

", "votes": null, "replies": [] }, { "id": 65431, "author_name": "iamthep", "author_url": "", "post_date": "03/04/2015 19:24:05", "content": "

[quote=Jessica Luo;65374]

\n

Just to jump in here and offer my 2 cents. In our field (plankton imaging), one of the early papers put out ~83% overall accuracy as a high benchmark, and it has become a sort of "gold standard." While I will - and have - argued that this benchmark is arbitrary and somewhat questionably derived, it remains that our field has yet to see this threshold broken with a real dataset (not an artificially curated learning set). 

\n

I'm seeing on this forum and this thread accuracy rates of 78% - and possibly higher? - so I'm wondering whether this competition can break through this arbitrary limit of 83% accuracy. It would be very exciting to see it happen. 

\n

[/quote]

\n

I am curious about this paper.  Do you have a link?  I searched and found quite a few references to RAPID and an accuracy of 70-80% on 10-20 classes.  There was also mention that the classification was often subjective among scientists, which is why we get class overlap like protist_other and protist_noctiluca.  

\n

At any rate, it seems that this dataset is the most extensive to date.  This should help get great results.

", "votes": null, "replies": [] }, { "id": 65437, "author_name": "jessicaluo", "author_url": "", "post_date": "03/04/2015 20:43:22", "content": "

[quote=Ryan Munion;65431]

\n

[quote=Jessica Luo;65374]

\n

Just to jump in here and offer my 2 cents. In our field (plankton imaging), one of the early papers put out ~83% overall accuracy as a high benchmark, and it has become a sort of "gold standard." While I will - and have - argued that this benchmark is arbitrary and somewhat questionably derived, it remains that our field has yet to see this threshold broken with a real dataset (not an artificially curated learning set). 

\n

I'm seeing on this forum and this thread accuracy rates of 78% - and possibly higher? - so I'm wondering whether this competition can break through this arbitrary limit of 83% accuracy. It would be very exciting to see it happen. 

\n

[/quote]

\n

I am curious about this paper.  Do you have a link?  I searched and found quite a few references to RAPID and an accuracy of 70-80% on 10-20 classes.  There was also mention that the classification was often subjective among scientists, which is why we get class overlap like protist_other and protist_noctiluca.  

\n

At any rate, it seems that this dataset is the most extensive to date.  This should help get great results.

\n

[/quote]

\n

Hi Ryan,

\n

The RAPID article (Benfield et al. 2007) that you mention is a great one for an intro to field and techniques used, and it has a more complete overview of the history of these imaging systems than what I am going to provide here. But nonetheless - to answer your question, the Video Plankton Recorder ("VPR"; Davis et al. 1992) was one of the first - and still widely used - plankton imaging systems developed. The VPR team also developed one of the early automatic plankton classification techniques using SVMs and Neural Nets (Hu and Davis 2005, Hu and Davis 2006). That approach, along with Gabriel Gorsky's team's Random Forest based approach (Gorsky et al. 2010) remains among the more widely used / accepted approaches within plankton imaging today. 

\n

Hu and Davis (VPR; 2005) were among the first to put out a benchmark for "acceptable accuracy" in classification: 67-83%, which is based on the work of Culverhouse et al. (2003). For better or for worse, that number has more or less stuck (though some research groups functionally ignore accuracy and recall and optimize their workflow for analysis/processing time) -- and has become that high benchmark in our field. Of course, you can curate any kind of ideal learning set from your data and obtain a near perfect classification accuracy on it, but to then take your model and apply to your real data -- particularly in an unbounded problem, with Unknowns -- I have yet to see this value exceeded. So -- while it's really exciting to see us in an unbounded, 120+ class problem that represents real world data(*) get so close to breaking this benchmark, the real test would be when we take these algorithms to the whole dataset to see how they perform. That's when the magic will happen!

\n

Cheers,

\n

Jessica

\n

* - Of course, this competition dataset is not completely representative of our actual dataset. If so, you would get about 1 fish to 1,000 detritus and that would be no fun at all. Plus, everyone would optimize their models to perfectly predict the detritus and get great log-loss scores whereas what we're really interested in is classifying the fish and other organisms.

", "votes": null, "replies": [] }, { "id": 65462, "author_name": "iamthep", "author_url": "", "post_date": "03/04/2015 22:49:50", "content": "

Thank you.

\n

I will be interested in seeing how the results fair in practice.

", "votes": null, "replies": [] }, { "id": 66018, "author_name": "sachinvernekar", "author_url": "", "post_date": "03/12/2015 08:16:06", "content": "

My accuracy on test data is 73.5% with a log-loss of 0.862932 on the leader board.

", "votes": null, "replies": [] }, { "id": 66026, "author_name": "alexzam", "author_url": "", "post_date": "03/12/2015 09:12:28", "content": "

We are seeing accuracy of 80% with log-loss around 1.0. Any advice on how to optimize log-loss?

", "votes": null, "replies": [] }, { "id": 66028, "author_name": "sachinvernekar", "author_url": "", "post_date": "03/12/2015 09:42:15", "content": "

[quote=Alex Zam;66026]

\n

We are seeing accuracy of 80% with log-loss around 1.0. Any advice on how to optimize log-loss?

\n

[/quote]

\n

Is it 80% accurate on test data - that is 130400 images?

", "votes": null, "replies": [] }, { "id": 66033, "author_name": "pengpai", "author_url": "", "post_date": "03/12/2015 12:09:33", "content": "

Guys, my best single model performs 71.22% accuracy on validation data and it is 0.93 on LB.

", "votes": null, "replies": [] }, { "id": 66049, "author_name": "telser", "author_url": "", "post_date": "03/12/2015 16:19:42", "content": "

They couldn't possibly know if it were  80% accurate on the test data without cheating and hand labeling, Sachin.

\n\n

Anyway, I'm at ~73-74% accuracy.

", "votes": null, "replies": [] }, { "id": 66051, "author_name": "sachinvernekar", "author_url": "", "post_date": "03/12/2015 17:17:37", "content": "

[quote=Torgos;66049]

\n

They couldn't possibly know if it were  80% accurate on the test data without cheating and hand labeling, Sachin.

\n

Anyway, I'm at ~73-74% accuracy.

\n

[/quote]

\n

Hi Torgos, you need not cheat and hand label to get percentage accuracy on test data.

\n

On this thread I have given a method to deduce percentage accuracy from log-loss, you can try that.

", "votes": null, "replies": [] }, { "id": 66170, "author_name": "pengpai", "author_url": "", "post_date": "03/14/2015 09:32:13", "content": "

@Sachin, would like to explain your aforementioned equation in details?

", "votes": null, "replies": [] }, { "id": 66173, "author_name": "sachinvernekar", "author_url": "", "post_date": "03/14/2015 10:45:01", "content": "

[quote=PengPai;66170]

\n

@Sachin, would like to explain your aforementioned equation in details?

\n

[/quote]

\n

Sure PengPai I will try my best.

\n\n

Note: All logarithms are to the base "e"

\n

Lets say there are 2 classes- class 0 and class 1. And there are 5 images. Let the probabilities predicated for each class be as follows

\n

class0   class1   actual_class

\n

0.1          0.9            class1

\n

0.2          0.8            class1

\n

0.3          0.7            class1

\n

0.6          0.4            class1

\n

0.25        0.75          class0

\n\n

so the percentage accuracy would be 3*100/5 = 60%  which is unknown at the moment for the test data. But the log-loss calculated based on the evaluation metric is

\n

-(log(0.9)+log(0.8)+log(0.7)+log(0.4)+log(0.25))/5 = 0.59755282

\n

But in order to get your percentage accuracy you need to make another submission making the class with highest probability in each row to be 1 and the rest to be 0.

\n

that is

\n

class0       class1      actual_class

\n

0                   1                  class1

\n

0                   1                   class1

\n

0                    1                  class1

\n

1                   0                    class1

\n

0                  1                     class0

\n\n

so your CV score would be  -(log(1) + log(1) + log(1)+log(10^-15)+log(10^-15))/5 = 13.81551056

\n

since log(0) is -infinity the smallest number they consider is 10^-15.

\n\n

In order to get the percentage accuracy from this do

\n

1+ (13.81551056/log(10^-15)) = 0.6 = 60%

\n

If it is still not clear you can ask me again.

\n

Thanks

", "votes": null, "replies": [] }, { "id": 66174, "author_name": "sachinvernekar", "author_url": "", "post_date": "03/14/2015 10:48:51", "content": "

[quote=Sachin;66051]

\n

[quote=Torgos;66049]

\n

They couldn't possibly know if it were  80% accurate on the test data without cheating and hand labeling, Sachin.

\n

Anyway, I'm at ~73-74% accuracy.

\n

[/quote]

\n

Hi Torgos, you need not cheat and hand label to get percentage accuracy on test data.

\n

On this thread I have given a method to deduce percentage accuracy from log-loss, you can try that.

\n

[/quote]

\n\n

Hi Torgos,

\n

I don't really understand why you down voted my comment, but please have a look at the solution I have provided to get the percentage accuracy and then you can decide.

\n

Thanks

\n

-Sachin

", "votes": null, "replies": [] }, { "id": 66232, "author_name": "moyeaswaran", "author_url": "", "post_date": "03/15/2015 12:36:23", "content": "

@Jessica Luo: After the competition could you please post some of the leading accuracy scores as well?  Just for curiosity :-).  Thanks!

\n\n

[quote=Jessica Luo;65437]

\n

Hu and Davis (VPR; 2005) were among the first to put out a benchmark for "acceptable accuracy" in classification: 67-83%, which is based on the work of Culverhouse et al. (2003).

\n

[/quote]

", "votes": null, "replies": [] }, { "id": 66281, "author_name": "gala1276881", "author_url": "", "post_date": "03/15/2015 20:56:00", "content": "

Instead of just looking at the overall accuracy, I did a breakdown of the per class percentage accuracy.

The charts are for: (i) total counts (ii) miss counts (iii) miss percentages – all as a function of class id (121 plankton classes, alphabetically sorted). For each image, a “hit” is attributed to a class only if the highest probability class prediction matches the actual class in the training set (last layer is CNN softmax). These stats are based on 80% of the ~30k training set from which the model predictions were created - this is usually a no no, but I also ran this procedure on the validation set and got very similar results (the problem with the validation set is that some classes are very small).

\n

In order to get a good overall score, a model needs to somehow lower those high bars in the middle misses chart, especially those with high miss rates (bottom chart).

The usual suspects for poorly predicted classes are the aptly named “others” and “unknowns” e.g. class ids: 11,85,118,119,120

", "votes": null, "replies": [] }, { "id": 66282, "author_name": "gala1276881", "author_url": "", "post_date": "03/15/2015 20:57:41", "content": "

... inline image didn't work, please see attachment.

", "votes": null, "replies": [] }, { "id": 66430, "author_name": "jessicaluo", "author_url": "", "post_date": "03/16/2015 18:15:54", "content": "

[quote=Moy;66232]

\n

@Jessica Luo: After the competition could you please post some of the leading accuracy scores as well?  Just for curiosity :-).  Thanks!

\n

[/quote]

\n

Yup - sure! I'll pull a few from the top of the leaderboard after the competition ends. Good luck to all in the final push!

", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "63903": "", "63904": "", "63910": "", "63912": "", "63933": "", "63940": "", "63942": "", "63959": "", "63994": "", "64066": "", "64087": "", "65232": "", "65374": "", "65377": "", "65385": "", "65386": "", "65387": "", "65388": "", "65389": "", "65390": "", "65394": "", "65431": "", "65437": "", "65462": "", "66018": "", "66026": "", "66028": "", "66033": "", "66049": "", "66051": "", "66170": "", "66173": "", "66174": "", "66232": "", "66281": "", "66282": "", "66430": "" }, "source": "meta" }