{ "id": 9584, "title": "Average Precision at k (AP@k) ", "url": "/competitions/avito-prohibited-content/discussion/9584", "author_name": "", "post_date": "2014-06-27T04:37:35.103Z", "votes": null, "comment_count": 25, "views": 15131, "content": "

Can anyone elaborate on what this metric is?

\n

Thanks

", "messages": [ { "id": "49671", "postDate": "06/27/2014 04:37:35", "content": "

Can anyone elaborate on what this metric is?

\n

Thanks

", "rawMarkdown": "", "votes": null }, { "id": "49680", "postDate": "06/27/2014 06:51:40", "content": "

There is a good article here: https://sanchom.wordpress.com/tag/average-precision/

", "rawMarkdown": "", "votes": null }, { "id": "49723", "postDate": "06/27/2014 19:54:07", "content": "

I'm not clear how this metric is calculated for the public LB. I can see two different ways the scores for the LB could be generated:

\n\n

1. The top 32500 ads of a submission are taken for the metric, the rest are discarded. The score for the public leaderboard is the precision for the ads that appear in the public LB set. The ads that appear in the private LB set are ignored.

\n\n

2. The ads that are part of the public LB set are taken from the submission for the metric, the rest are discarded. The top 32500 public LB ads are used to score the precision.

\n\n

Could someone please clarify how the metric is computed for the LB?

", "rawMarkdown": "", "votes": null }, { "id": "49728", "postDate": "06/27/2014 20:40:24", "content": "

The metric is computed for the LB exactly as you described in 2.

", "rawMarkdown": "", "votes": null }, { "id": "49729", "postDate": "06/27/2014 20:51:17", "content": "

[quote=Jack Shih Tzu;49723]

\n

he top 32500 public LB ads 

\n

[/quote]

\n\n

I'm sorry, I feel like an idiot. Where if the 32500 number coming from?

", "rawMarkdown": "", "votes": null }, { "id": "49731", "postDate": "06/27/2014 20:58:05", "content": "

32500 is the value we are using for k in the AP@k metric. It is shown when you hover the "Score?" title in the leaderboard page. We chose that number because it approximately represents 5% of the test data.

", "rawMarkdown": "", "votes": null }, { "id": "49733", "postDate": "06/27/2014 21:10:04", "content": "

Evaluating Ranked Search Engine Results

\n

.

\n

What makes for a good search engine?

\n

.

\n

Mainly:

\n\n

Optionally:

\n\n

.

\n

Measuring Quality

\n

How de we assess the quality of a search engine result page (SERP)?

\n\n

.

\n

Feedback & Labels

\n

Quality feedback can be binary (0 for not relevant, 1 for relevant) or non-binary (0-3 scale).

\n

.

\n

Example binary feedback for 15 results:

\n

.

\n

[(1,1),(2,1),(3,1),(4,0),(5,1),(6,1),(7,1),(8,1),(9,1),(10,1),(11,0),(12,1),(13,0),(14,1),(15,0)]
(Rank 1, 2, 3 are relevant, rank 4 is irrelevant, rank 5 to 10 are relevant etc.)

\n

.

\n

rank_eval.py

\n

Functions for:

\n\n

Sample output:

\n

.

\n

INPUT: [(1,1),(2,1),(3,1),(4,0),(5,1),(6,1),(7,1),(8,1),(9,1),(10,1),(11,0),(12,1),(13,0),(14,1),(15,0)]

\n

P: [1.0, 1.0, 1.0, 0.75, 0.8, 0.83, 0.85, 0.87, 0.88, 0.9, 0.81, 0.83, 0.76, 0.78, 0.73]
R: [0.09, 0.18, 0.27, 0.27, 0.36, 0.45, 0.54, 0.63, 0.72, 0.81, 0.81, 0.90, 0.90, 1.0, 1.0]
IP: [1.0, 1.0, 1.0, 0.9, 0.9, 0.9, 0.9, 0.9, 0.9, 0.9, 0.83, 0.83, 0.78, 0.78, 0.73]
AP: [1.0, 1.0, 1.0, 0.93, 0.90, 0.89, 0.89, 0.88, 0.88, 0.89, 0.88, 0.87, 0.87, 0.86, 0.85]
PaK: 0.8 // precision at 5
CG: 11
DCG: 6.69277
IDCG: 6.95740
NDCG: 0.96196

\n\n

RANK RELEVANT? PRECISION RECALL INTERPOLATED PRECISION
1 1 1.00000 0.09091 1.00000
2 1 1.00000 0.18182 1.00000
3 1 1.00000 0.27273 1.00000
4 0 0.75000 0.27273 0.90000
5 1 0.80000 0.36364 0.90000
6 1 0.83333 0.45455 0.90000
7 1 0.85714 0.54545 0.90000
8 1 0.87500 0.63636 0.90000
9 1 0.88889 0.72727 0.90000
10 1 0.90000 0.81818 0.90000
11 0 0.81818 0.81818 0.83333
12 1 0.83333 0.90909 0.83333
13 0 0.76923 0.90909 0.78571
14 1 0.78571 1.00000 0.78571
15 0 0.73333 1.00000 0.73333
.

\n

Further reading:

\n

intro-to-text-retrieval.pdf

", "rawMarkdown": "", "votes": null }, { "id": "49734", "postDate": "06/27/2014 21:14:59", "content": "

[quote=Luis Tandalla;49731]

\n

32500 is the value we are using for k in the AP@k metric. It is shown when you hover the "Score?" title in the leaderboard page. We chose that number because it approximately represents 5% of the test data.

\n

[/quote]

\n\n

Thanks Luis. This is helpful. You might want to have that added to the evaluation page as well.

", "rawMarkdown": "", "votes": null }, { "id": "49882", "postDate": "07/01/2014 00:16:07", "content": "

Hi, can anyone provide the AP@k evaluation code in R, C#, C, or C++ ?

", "rawMarkdown": "", "votes": null }, { "id": "50103", "postDate": "07/07/2014 15:51:38", "content": "

Hi B Yang,

\n

A python code that evaluates AP@k has been posted in the Data page.

", "rawMarkdown": "", "votes": null }, { "id": "50347", "postDate": "07/11/2014 12:39:59", "content": "

In case anybody will find this usefull:

\n

 same code posted above, except that it does not read solution and predictions from the files, but keeps solution in memory, so fast evaluation is possible.

", "rawMarkdown": "", "votes": null }, { "id": "50361", "postDate": "07/11/2014 16:14:41", "content": "

Why post sample APK code to the data page when the wiki already has an implementation?  https://www.kaggle.com/wiki/MeanAveragePrecision

", "rawMarkdown": "", "votes": null }, { "id": "50368", "postDate": "07/11/2014 18:53:17", "content": "

I'm having some troubles when implementing cross-validation for this competitions. Currently, I'm planning to do the following.

\n\n

Calculate the prediction probability on each cross-validation fold and sort those values in descending order. Next, take the top 5 percent of it and compares with top 5 percent of the actual labels. But, I don't have an idea of how to select top 5 percent of the cross-validation according the their actual probability values. Since, class label is either zero or one ( as given in the training set) once sorted the relative position of each training example in the cross-validation fold is not unique.

\n\n

You helps are highly appreciated.

", "rawMarkdown": "", "votes": null }, { "id": "50510", "postDate": "07/15/2014 00:46:08", "content": "

[quote=Synthient;50361]

\n

Why post sample APK code to the data page when the wiki already has an implementation?  https://www.kaggle.com/wiki/MeanAveragePrecision

\n

[/quote]

\n

I haven't looked at the code yet, but the wiki describes "mean average precision" while the term used for this contest is "average precision". Are they the same thing or is there extra meaning to the word "mean" ?

", "rawMarkdown": "", "votes": null }, { "id": "50919", "postDate": "07/20/2014 10:51:06", "content": "

It seems that something that is calculated by the APatK.py is not quite Avarage Precision at K as discribed in the article mentioned above and in wiki page here

\n

For instance let us take an example from the article about planes and birds.

\n

The author got the answer 0.782

\n

I've created two files solution.csv and predictions.csv which represent the same problem, set K = 10 and got answer 0.669920634921

\n

Probably the reason for this result is that Avarage Precision at K is an integral of precision-recall curve and the given code, in fact, calculates integral of "precision-shareofpositives curve".

\n

Were am I wrong? And if i'm right, which metric is actually used in the competition?

", "rawMarkdown": "", "votes": null }, { "id": "50926", "postDate": "07/20/2014 14:28:26", "content": "

Георгий, I'm totally agree with you. Seems actual metric is NOT average precison at K. I am not python user and tryied to cross-validate using self-programmed AvPatK metric in R. It gives ~ 0.75 on my train set. But on public LB set I recieve ~0.9. After your post I checked python code from data page and saw that it actually calculate something like "precision-shareofpositives curve" as you pointed.

\n

Organization of this competition is very very poor. Why should I browse forums to find actual metrics? Administrators, why you can't write and DOUBLE CHECK CORRECT evaluation rules, formulas and code at Evaluation page?

", "rawMarkdown": "", "votes": null }, { "id": "50935", "postDate": "07/20/2014 19:26:28", "content": "

As far as I can see from the given APatK.py code, and the discussion above in this thread, and my observation on the leaderboard, the metric goes like this:

\n

For the public leaderboard score:

\n

take the submission file and eliminate all ids that are not in the 50% public

\n

take the top 32500 submitted ids from what remains

\n

count the ids that are actually spam

\n

score = (number of correctly guessed spam) / 32500

\n

And for the private LB, just adjust the first step.

\n

Not sure if it fits the announced name academically, but this is how it's calculated as far as I can tell.

\n

By the way, I too don't think this is the best metric, that's why I double checked. But apparently this is what the sponsor is paying for. IMHO, good old AUC would be a better choice.

\n

Hope this helps.

", "rawMarkdown": "", "votes": null }, { "id": "50936", "postDate": "07/20/2014 19:38:12", "content": "

Good old auc is like 0.99 already from the sample benchmark.  

", "rawMarkdown": "", "votes": null }, { "id": "50938", "postDate": "07/20/2014 20:08:40", "content": "

[quote=Abhishek;50936]

\n

Good old auc is like 0.99 already from the sample benchmark.  

\n

[/quote]

\n

And with the current metric, the top score is 0.98 with 42 days to go and only 114 competitors so far. I expect to see 0.99999 close to the end.

\n

AUC takes into consideration the whole data set, that's why I suggested it would have been better. The current metric does not. I've only played with 10% of the data, and I have 0.96 on the LB.

", "rawMarkdown": "", "votes": null }, { "id": "50976", "postDate": "07/21/2014 00:33:37", "content": "

To avoid any further confusion, the metric is calculated as:

\n

\\[AP@K = \\frac{\\sum_{i=1}^K P@i}{K} \\]

\n

where

\n

\\[ P@i = \\frac{ \\text{#relevant - forbidden ads retrieved in the first ith places} }{i} \\]

\n

As Abhishek mentioned, AUC metric would already give scores of 0.99+, so the differences in the top leaderboard would be in the last decimal places, and chance would play a big role at the end. The chosen metric gives a little more space to improve and to differentiate between scores.

\n

[quote=Георгий Калашнов;50919]

\n

The author got the answer 0.782 [but] got answer 0.669920634921

\n

[/quote]

\n

The definition in wikipedia,

\n

\\[AP@K = \\frac{\\sum P@i * rel(i)}{\\text{# relevant at K}} \\]

\n

would also give higher scores as AUC would, so the top leaderboard would already be populated with 0.99+ scores. 

\n

[quote=barisumog;50938]

\n

AUC takes into consideration the whole data set.

\n

[/quote]

\n

Sometimes one may not be interested in the predictions of the whole data set but only in the most confident (top) predictions. 

", "rawMarkdown": "", "votes": null }, { "id": "51079", "postDate": "07/23/2014 04:42:34", "content": "

How close is your CV score to LB? And do you find it directionally correct? I just started doing some CV and I find the score off by 0.008-0.016 and not always directionally correct. I'm working with k based on 5% of my test set instead of a static 32k. So far I've only used the benchmark features.

", "rawMarkdown": "", "votes": null }, { "id": "51085", "postDate": "07/23/2014 09:27:52", "content": "

[quote=Giulio;51079]

\n

I find the score off by 0.008-0.016 and not always directionally correct

\n

[/quote]

\n

More or less the same here.

", "rawMarkdown": "", "votes": null }, { "id": "51148", "postDate": "07/24/2014 17:29:08", "content": "

[quote=Triskelion;49733]

\n

Evaluating Ranked Search Engine Results

\n

.

\n

What makes for a good search engine?

\n

.

\n

.

\n

.

\n

[/quote]

\n

Please forgive the newb question, but I take it that in the context of this competition and the information you provided on search engine performance that correctly predicting prohibited content is equivalent to relevance in search engine performance.

\n

Thank you

", "rawMarkdown": "", "votes": null }, { "id": "51167", "postDate": "07/25/2014 06:13:19", "content": "

[quote=HeatfanJohn;51148]

\n

Please forgive the newb question, but I take it that in the context of this competition and the information you provided on search engine performance that correctly predicting prohibited content is equivalent to relevance in search engine performance.

\n

Thank you

\n

[/quote]

\n

It is much the same yes, average precision at K is a metric used in evaluating search engine relevance, and the metric in use for this competition. I suspect that Avito wants a ranked result page with illicit ads, the worst offenders on top, so moderators can act on it.

", "rawMarkdown": "", "votes": null }, { "id": "51842", "postDate": "08/10/2014 19:49:27", "content": "

[quote=Peter Pišljar;50347]

\n

In case anybody will find this usefull:

\n

 same code posted above, except that it does not read solution and predictions from the files, but keeps solution in memory, so fast evaluation is possible.

\n

[/quote]

\n

A faster implementation using cPickle and pandas :)

", "rawMarkdown": "", "votes": null }, { "id": "52196", "postDate": "08/19/2014 18:22:29", "content": "

This might be a very stupid question, but I cannot figure out how to use the APatK.py file to compute a score for my solution.

\n

I understand that the predictions.csv file should be the submission file. This is the one that contains item_ids from the avito_test.tsv file, arranged in order from most illicit to least illicit content. 

\n

However, I don't understand how to generate the solutions.tsv file. 

\n\n

The post above says:

\n

"take the submission file and eliminate all ids that are not in the 50% public

\n

take the top 32500 submitted ids from what remains

\n

count the ids that are actually spam"

\n\n

Now, since the avito_test.tsv file does not mention the is_blocked parameter, how do we count the ids that are actually spam?

\n\n

Equivalently, if I look at Peter's post above, where he has attached a version of the APatK.py file,

\n

I don't understand how to get the data_test.target variable in line 6?

", "rawMarkdown": "", "votes": null } ], "comments": [ { "id": 49680, "author_name": "sashikanthdareddy", "author_url": "", "post_date": "06/27/2014 06:51:40", "content": "

There is a good article here: https://sanchom.wordpress.com/tag/average-precision/

", "votes": null, "replies": [] }, { "id": 49723, "author_name": "tbrichie", "author_url": "", "post_date": "06/27/2014 19:54:07", "content": "

I'm not clear how this metric is calculated for the public LB. I can see two different ways the scores for the LB could be generated:

\n\n

1. The top 32500 ads of a submission are taken for the metric, the rest are discarded. The score for the public leaderboard is the precision for the ads that appear in the public LB set. The ads that appear in the private LB set are ignored.

\n\n

2. The ads that are part of the public LB set are taken from the submission for the metric, the rest are discarded. The top 32500 public LB ads are used to score the precision.

\n\n

Could someone please clarify how the metric is computed for the LB?

", "votes": null, "replies": [] }, { "id": 49728, "author_name": "luistandalla", "author_url": "", "post_date": "06/27/2014 20:40:24", "content": "

The metric is computed for the LB exactly as you described in 2.

", "votes": null, "replies": [] }, { "id": 49729, "author_name": "adjgiulio", "author_url": "", "post_date": "06/27/2014 20:51:17", "content": "

[quote=Jack Shih Tzu;49723]

\n

he top 32500 public LB ads 

\n

[/quote]

\n\n

I'm sorry, I feel like an idiot. Where if the 32500 number coming from?

", "votes": null, "replies": [] }, { "id": 49731, "author_name": "luistandalla", "author_url": "", "post_date": "06/27/2014 20:58:05", "content": "

32500 is the value we are using for k in the AP@k metric. It is shown when you hover the "Score?" title in the leaderboard page. We chose that number because it approximately represents 5% of the test data.

", "votes": null, "replies": [] }, { "id": 49733, "author_name": "triskelion", "author_url": "", "post_date": "06/27/2014 21:10:04", "content": "

Evaluating Ranked Search Engine Results

\n

.

\n

What makes for a good search engine?

\n

.

\n

Mainly:

\n\n

Optionally:

\n\n

.

\n

Measuring Quality

\n

How de we assess the quality of a search engine result page (SERP)?

\n\n

.

\n

Feedback & Labels

\n

Quality feedback can be binary (0 for not relevant, 1 for relevant) or non-binary (0-3 scale).

\n

.

\n

Example binary feedback for 15 results:

\n

.

\n

[(1,1),(2,1),(3,1),(4,0),(5,1),(6,1),(7,1),(8,1),(9,1),(10,1),(11,0),(12,1),(13,0),(14,1),(15,0)]
(Rank 1, 2, 3 are relevant, rank 4 is irrelevant, rank 5 to 10 are relevant etc.)

\n

.

\n

rank_eval.py

\n

Functions for:

\n\n

Sample output:

\n

.

\n

INPUT: [(1,1),(2,1),(3,1),(4,0),(5,1),(6,1),(7,1),(8,1),(9,1),(10,1),(11,0),(12,1),(13,0),(14,1),(15,0)]

\n

P: [1.0, 1.0, 1.0, 0.75, 0.8, 0.83, 0.85, 0.87, 0.88, 0.9, 0.81, 0.83, 0.76, 0.78, 0.73]
R: [0.09, 0.18, 0.27, 0.27, 0.36, 0.45, 0.54, 0.63, 0.72, 0.81, 0.81, 0.90, 0.90, 1.0, 1.0]
IP: [1.0, 1.0, 1.0, 0.9, 0.9, 0.9, 0.9, 0.9, 0.9, 0.9, 0.83, 0.83, 0.78, 0.78, 0.73]
AP: [1.0, 1.0, 1.0, 0.93, 0.90, 0.89, 0.89, 0.88, 0.88, 0.89, 0.88, 0.87, 0.87, 0.86, 0.85]
PaK: 0.8 // precision at 5
CG: 11
DCG: 6.69277
IDCG: 6.95740
NDCG: 0.96196

\n\n

RANK RELEVANT? PRECISION RECALL INTERPOLATED PRECISION
1 1 1.00000 0.09091 1.00000
2 1 1.00000 0.18182 1.00000
3 1 1.00000 0.27273 1.00000
4 0 0.75000 0.27273 0.90000
5 1 0.80000 0.36364 0.90000
6 1 0.83333 0.45455 0.90000
7 1 0.85714 0.54545 0.90000
8 1 0.87500 0.63636 0.90000
9 1 0.88889 0.72727 0.90000
10 1 0.90000 0.81818 0.90000
11 0 0.81818 0.81818 0.83333
12 1 0.83333 0.90909 0.83333
13 0 0.76923 0.90909 0.78571
14 1 0.78571 1.00000 0.78571
15 0 0.73333 1.00000 0.73333
.

\n

Further reading:

\n

intro-to-text-retrieval.pdf

", "votes": null, "replies": [] }, { "id": 49734, "author_name": "adjgiulio", "author_url": "", "post_date": "06/27/2014 21:14:59", "content": "

[quote=Luis Tandalla;49731]

\n

32500 is the value we are using for k in the AP@k metric. It is shown when you hover the "Score?" title in the leaderboard page. We chose that number because it approximately represents 5% of the test data.

\n

[/quote]

\n\n

Thanks Luis. This is helpful. You might want to have that added to the evaluation page as well.

", "votes": null, "replies": [] }, { "id": 49882, "author_name": "byang1", "author_url": "", "post_date": "07/01/2014 00:16:07", "content": "

Hi, can anyone provide the AP@k evaluation code in R, C#, C, or C++ ?

", "votes": null, "replies": [] }, { "id": 50103, "author_name": "luistandalla", "author_url": "", "post_date": "07/07/2014 15:51:38", "content": "

Hi B Yang,

\n

A python code that evaluates AP@k has been posted in the Data page.

", "votes": null, "replies": [] }, { "id": 50347, "author_name": "peterpisljar", "author_url": "", "post_date": "07/11/2014 12:39:59", "content": "

In case anybody will find this usefull:

\n

 same code posted above, except that it does not read solution and predictions from the files, but keeps solution in memory, so fast evaluation is possible.

", "votes": null, "replies": [] }, { "id": 50361, "author_name": "brundage", "author_url": "", "post_date": "07/11/2014 16:14:41", "content": "

Why post sample APK code to the data page when the wiki already has an implementation?  https://www.kaggle.com/wiki/MeanAveragePrecision

", "votes": null, "replies": [] }, { "id": 50368, "author_name": "upulbandara", "author_url": "", "post_date": "07/11/2014 18:53:17", "content": "

I'm having some troubles when implementing cross-validation for this competitions. Currently, I'm planning to do the following.

\n\n

Calculate the prediction probability on each cross-validation fold and sort those values in descending order. Next, take the top 5 percent of it and compares with top 5 percent of the actual labels. But, I don't have an idea of how to select top 5 percent of the cross-validation according the their actual probability values. Since, class label is either zero or one ( as given in the training set) once sorted the relative position of each training example in the cross-validation fold is not unique.

\n\n

You helps are highly appreciated.

", "votes": null, "replies": [] }, { "id": 50510, "author_name": "byang1", "author_url": "", "post_date": "07/15/2014 00:46:08", "content": "

[quote=Synthient;50361]

\n

Why post sample APK code to the data page when the wiki already has an implementation?  https://www.kaggle.com/wiki/MeanAveragePrecision

\n

[/quote]

\n

I haven't looked at the code yet, but the wiki describes "mean average precision" while the term used for this contest is "average precision". Are they the same thing or is there extra meaning to the word "mean" ?

", "votes": null, "replies": [] }, { "id": 50919, "author_name": "user173369", "author_url": "", "post_date": "07/20/2014 10:51:06", "content": "

It seems that something that is calculated by the APatK.py is not quite Avarage Precision at K as discribed in the article mentioned above and in wiki page here

\n

For instance let us take an example from the article about planes and birds.

\n

The author got the answer 0.782

\n

I've created two files solution.csv and predictions.csv which represent the same problem, set K = 10 and got answer 0.669920634921

\n

Probably the reason for this result is that Avarage Precision at K is an integral of precision-recall curve and the given code, in fact, calculates integral of "precision-shareofpositives curve".

\n

Were am I wrong? And if i'm right, which metric is actually used in the competition?

", "votes": null, "replies": [] }, { "id": 50926, "author_name": "dselivanov", "author_url": "", "post_date": "07/20/2014 14:28:26", "content": "

Георгий, I'm totally agree with you. Seems actual metric is NOT average precison at K. I am not python user and tryied to cross-validate using self-programmed AvPatK metric in R. It gives ~ 0.75 on my train set. But on public LB set I recieve ~0.9. After your post I checked python code from data page and saw that it actually calculate something like "precision-shareofpositives curve" as you pointed.

\n

Organization of this competition is very very poor. Why should I browse forums to find actual metrics? Administrators, why you can't write and DOUBLE CHECK CORRECT evaluation rules, formulas and code at Evaluation page?

", "votes": null, "replies": [] }, { "id": 50935, "author_name": "barisumog", "author_url": "", "post_date": "07/20/2014 19:26:28", "content": "

As far as I can see from the given APatK.py code, and the discussion above in this thread, and my observation on the leaderboard, the metric goes like this:

\n

For the public leaderboard score:

\n

take the submission file and eliminate all ids that are not in the 50% public

\n

take the top 32500 submitted ids from what remains

\n

count the ids that are actually spam

\n

score = (number of correctly guessed spam) / 32500

\n

And for the private LB, just adjust the first step.

\n

Not sure if it fits the announced name academically, but this is how it's calculated as far as I can tell.

\n

By the way, I too don't think this is the best metric, that's why I double checked. But apparently this is what the sponsor is paying for. IMHO, good old AUC would be a better choice.

\n

Hope this helps.

", "votes": null, "replies": [] }, { "id": 50936, "author_name": "abhishek", "author_url": "", "post_date": "07/20/2014 19:38:12", "content": "

Good old auc is like 0.99 already from the sample benchmark.  

", "votes": null, "replies": [] }, { "id": 50938, "author_name": "barisumog", "author_url": "", "post_date": "07/20/2014 20:08:40", "content": "

[quote=Abhishek;50936]

\n

Good old auc is like 0.99 already from the sample benchmark.  

\n

[/quote]

\n

And with the current metric, the top score is 0.98 with 42 days to go and only 114 competitors so far. I expect to see 0.99999 close to the end.

\n

AUC takes into consideration the whole data set, that's why I suggested it would have been better. The current metric does not. I've only played with 10% of the data, and I have 0.96 on the LB.

", "votes": null, "replies": [] }, { "id": 50976, "author_name": "luistandalla", "author_url": "", "post_date": "07/21/2014 00:33:37", "content": "

To avoid any further confusion, the metric is calculated as:

\n

\\[AP@K = \\frac{\\sum_{i=1}^K P@i}{K} \\]

\n

where

\n

\\[ P@i = \\frac{ \\text{#relevant - forbidden ads retrieved in the first ith places} }{i} \\]

\n

As Abhishek mentioned, AUC metric would already give scores of 0.99+, so the differences in the top leaderboard would be in the last decimal places, and chance would play a big role at the end. The chosen metric gives a little more space to improve and to differentiate between scores.

\n

[quote=Георгий Калашнов;50919]

\n

The author got the answer 0.782 [but] got answer 0.669920634921

\n

[/quote]

\n

The definition in wikipedia,

\n

\\[AP@K = \\frac{\\sum P@i * rel(i)}{\\text{# relevant at K}} \\]

\n

would also give higher scores as AUC would, so the top leaderboard would already be populated with 0.99+ scores. 

\n

[quote=barisumog;50938]

\n

AUC takes into consideration the whole data set.

\n

[/quote]

\n

Sometimes one may not be interested in the predictions of the whole data set but only in the most confident (top) predictions. 

", "votes": null, "replies": [] }, { "id": 51079, "author_name": "adjgiulio", "author_url": "", "post_date": "07/23/2014 04:42:34", "content": "

How close is your CV score to LB? And do you find it directionally correct? I just started doing some CV and I find the score off by 0.008-0.016 and not always directionally correct. I'm working with k based on 5% of my test set instead of a static 32k. So far I've only used the benchmark features.

", "votes": null, "replies": [] }, { "id": 51085, "author_name": "barisumog", "author_url": "", "post_date": "07/23/2014 09:27:52", "content": "

[quote=Giulio;51079]

\n

I find the score off by 0.008-0.016 and not always directionally correct

\n

[/quote]

\n

More or less the same here.

", "votes": null, "replies": [] }, { "id": 51148, "author_name": "heatfanjohn", "author_url": "", "post_date": "07/24/2014 17:29:08", "content": "

[quote=Triskelion;49733]

\n

Evaluating Ranked Search Engine Results

\n

.

\n

What makes for a good search engine?

\n

.

\n

.

\n

.

\n

[/quote]

\n

Please forgive the newb question, but I take it that in the context of this competition and the information you provided on search engine performance that correctly predicting prohibited content is equivalent to relevance in search engine performance.

\n

Thank you

", "votes": null, "replies": [] }, { "id": 51167, "author_name": "triskelion", "author_url": "", "post_date": "07/25/2014 06:13:19", "content": "

[quote=HeatfanJohn;51148]

\n

Please forgive the newb question, but I take it that in the context of this competition and the information you provided on search engine performance that correctly predicting prohibited content is equivalent to relevance in search engine performance.

\n

Thank you

\n

[/quote]

\n

It is much the same yes, average precision at K is a metric used in evaluating search engine relevance, and the metric in use for this competition. I suspect that Avito wants a ranked result page with illicit ads, the worst offenders on top, so moderators can act on it.

", "votes": null, "replies": [] }, { "id": 51842, "author_name": "chenglongchen", "author_url": "", "post_date": "08/10/2014 19:49:27", "content": "

[quote=Peter Pišljar;50347]

\n

In case anybody will find this usefull:

\n

 same code posted above, except that it does not read solution and predictions from the files, but keeps solution in memory, so fast evaluation is possible.

\n

[/quote]

\n

A faster implementation using cPickle and pandas :)

", "votes": null, "replies": [] }, { "id": 52196, "author_name": "datanovice111981", "author_url": "", "post_date": "08/19/2014 18:22:29", "content": "

This might be a very stupid question, but I cannot figure out how to use the APatK.py file to compute a score for my solution.

\n

I understand that the predictions.csv file should be the submission file. This is the one that contains item_ids from the avito_test.tsv file, arranged in order from most illicit to least illicit content. 

\n

However, I don't understand how to generate the solutions.tsv file. 

\n\n

The post above says:

\n

"take the submission file and eliminate all ids that are not in the 50% public

\n

take the top 32500 submitted ids from what remains

\n

count the ids that are actually spam"

\n\n

Now, since the avito_test.tsv file does not mention the is_blocked parameter, how do we count the ids that are actually spam?

\n\n

Equivalently, if I look at Peter's post above, where he has attached a version of the APatK.py file,

\n

I don't understand how to get the data_test.target variable in line 6?

", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "49671": "", "49680": "", "49723": "", "49728": "", "49729": "", "49731": "", "49733": "", "49734": "", "49882": "", "50103": "", "50347": "", "50361": "", "50368": "", "50510": "", "50919": "", "50926": "", "50935": "", "50936": "", "50938": "", "50976": "", "51079": "", "51085": "", "51148": "", "51167": "", "51842": "", "52196": "" }, "source": "meta" }