I found the formula for MAP to be inconsistent.
\n\nAt this competition link: https://www.kaggle.com/c/facebook-v-predicting-check-ins/details/evaluation, the formula is: \n$$MAP@3 = \\frac{1}{|U|} \\sum_{u=1}^{|U|} \\sum_{k=1}^{min(3, n)} P(k)$$
\n\nand, at: https://www.kaggle.com/wiki/MeanAveragePrecision, the formula is: \n$$ap@n = \\sum_{k=1}^n P(k) / min(m, n)$$\n$$MAP@n = \\sum_{i=1}^N ap@n_i / N$$
\n\nBasically the earlier formula is not dividing P(k) by min(m, n), like the later one does. I suppose the 2nd link is correct representation of this formula, am I correct ?
\n\n-- had made another thread, but for some reason my post is not showing in that thread.
", "messages": [ { "id": "119993", "postDate": "05/14/2016 13:14:46", "content": "I found the formula for MAP to be inconsistent.
\n\nAt this competition link: https://www.kaggle.com/c/facebook-v-predicting-check-ins/details/evaluation, the formula is: \n$$MAP@3 = \\frac{1}{|U|} \\sum_{u=1}^{|U|} \\sum_{k=1}^{min(3, n)} P(k)$$
\n\nand, at: https://www.kaggle.com/wiki/MeanAveragePrecision, the formula is: \n$$ap@n = \\sum_{k=1}^n P(k) / min(m, n)$$\n$$MAP@n = \\sum_{i=1}^N ap@n_i / N$$
\n\nBasically the earlier formula is not dividing P(k) by min(m, n), like the later one does. I suppose the 2nd link is correct representation of this formula, am I correct ?
\n\n-- had made another thread, but for some reason my post is not showing in that thread.
", "rawMarkdown": "I found the formula for MAP to be inconsistent.\r\n\r\nAt this competition link: https://www.kaggle.com/c/facebook-v-predicting-check-ins/details/evaluation, the formula is: \r\n$$MAP@3 = \\frac{1}{|U|} \\sum_{u=1}^{|U|} \\sum_{k=1}^{min(3, n)} P(k)$$\r\n\r\nand, at: https://www.kaggle.com/wiki/MeanAveragePrecision, the formula is : \r\n\r\n$$ap@n = \\sum_{k=1}^n P(k) / min(m, n)$$\r\n$$MAP@n = \\sum_{i=1}^N ap@n_i / N$$\r\n\r\nBasically the earlier formula is not dividing P(k) by min(m, n), like the later one does. I suppose the 2nd link is correct representation of this formula, am I correct ?\r\n\r\n-- had made another thread, but for some reason my post is not showing in that thread.", "votes": null }, { "id": "120251", "postDate": "05/16/2016 18:38:04", "content": "In this contest m == 1 (since there's only one place_id per row), therefore min(m, n) == 1 and can be ignored.
", "rawMarkdown": "In this contest m == 1 (since there's only one place_id per row), therefore min(m, n) == 1 and can be ignored.", "votes": null }, { "id": "120266", "postDate": "05/16/2016 23:38:02", "content": "One clarification. Since we only have m = 1, all possible cases are (suppose we produce all three recommendations)
\n\n(True, False, False) : ap@3 = 1
\n\n(False, True, False) : ap@3 = 1/2
\n\n(False, False, True) : ap@3 = 1/3
\n\n(False, False, False) : ap@3 = 0
\n\nIt means that there is no advantage to submit only one prediction, which is correct, rather than all three predictions including incorrect answers.
\n\nIs this right?
", "rawMarkdown": "One clarification. Since we only have m = 1, all possible cases are (suppose we produce all three recommendations)\r\n\r\n\r\n----------\r\n\r\n\r\n(True, False, False) : ap@3 = 1\r\n\r\n(False, True, False) : ap@3 = 1/2\r\n\r\n(False, False, True) : ap@3 = 1/3\r\n\r\n(False, False, False) : ap@3 = 0 \r\n\r\n\r\n----------\r\n\r\n\r\nIt means that there is no advantage to submit only one prediction, which is correct, rather than all three predictions including incorrect answers. \r\n\r\nIs this right?", "votes": null }, { "id": "120267", "postDate": "05/16/2016 23:49:53", "content": "[quote=Seungbum Park;120266]
\n\nIt means that there is no advantage to submit only one prediction, which is correct, rather than all three predictions including incorrect answers.
\n\nIs this right?
\n\n[/quote]
\n\nYou still want your correct prediction to be first in the list: [right, wrong1, wrong2], so you get AP score of 1.0 and not 0.25 or 0.33.
", "rawMarkdown": "[quote=Seungbum Park;120266]\r\n\r\nIt means that there is no advantage to submit only one prediction, which is correct, rather than all three predictions including incorrect answers. \r\n\r\nIs this right?\r\n\r\n[/quote]\r\n\r\nYou still want your correct prediction to be first in the list: [right, wrong1, wrong2], so you get AP score of 1.0 and not 0.25 or 0.33.", "votes": null }, { "id": "123063", "postDate": "06/09/2016 13:26:20", "content": "What if I submit 3 times the same prediction for an example and that this prediction is correct ?\nIn another words what is:
\n\n(True, True, True) : ap@3
\n\nIs it?
\n\n(True, True, True) : ap@3 = 1 + 1/2 + 1/3
", "rawMarkdown": "What if I submit 3 times the same prediction for an example and that this prediction is correct ?\r\nIn another words what is:\r\n\r\n(True, True, True) : ap@3\r\n\r\nIs it?\r\n\r\n(True, True, True) : ap@3 = 1 + 1/2 + 1/3", "votes": null }, { "id": "123099", "postDate": "06/09/2016 17:55:18", "content": "According to "our C# Production Implementation", you don't get score for the same answer more then once
\n\nif (solTokens.Contains(subTokens[i], _StringComparer) && !alreadyRecommended.Contains(subTokens[i], _StringComparer)) \n{\n alreadyRecommended.Add(subTokens[i]);\n count++;\n precisionAtK += count / (i + 1);\n}\nPay attention to alreadyRecommended. i starts from zero.
The evaluation formula for this competition isn't well-defined. I'd like clear mathematical formulas, where every term is defined, that explicitly show how a score is computed. We don't have that right now.
", "rawMarkdown": "The evaluation formula for this competition isn't well-defined. I'd like clear mathematical formulas, where every term is defined, that explicitly show how a score is computed. We don't have that right now.", "votes": null }, { "id": "124130", "postDate": "06/15/2016 18:39:46", "content": "@Adrien, You can see a MAP@K demo script here. In other words, your score doesn't change if you predict the same answer multiple times.
\n\n@Rory, I'm happy to answer questions, can you tell me which terms are not defined? Have you clicked on the link to Kaggle's wiki from our evaluation page? We have the formula, a couple of use cases, and code in different languages (python, R, C# production code) there.
", "rawMarkdown": "Adrien, You can see a MAP@K demo script [here][1]. In other words, your score doesn't change if you predict the same answer multiple times. \r\n\r\n@Rory, I'm happy to answer questions, can you tell me which terms are not defined? Have you clicked on the [link][2] to Kaggle's wiki from our [evaluation][3] page? We have the formula, a couple of use cases, and code in different languages (python, R, C# production code) there. \r\n\r\n\r\n [1]: https://www.kaggle.com/wendykan/expedia-hotel-recommendations/map-k-demo\r\n [2]: https://www.kaggle.com/wiki/MeanAveragePrecision\r\n [3]: https://www.kaggle.com/c/facebook-v-predicting-check-ins/details/evaluation", "votes": null }, { "id": "124156", "postDate": "06/15/2016 20:44:58", "content": "@Wendy Here are some questions I have: the formula sums over u but P(k) doesn't depend on u? What is U? The formula is a mean of an average but I don't see two averages being taken. Where's the definition of P(k) and shouldn't it be the Average Precision AP(k)?
\n\nI'm being pedantic but I think being pedantic can help sometimes (this being one of those times). Thanks.
", "rawMarkdown": "Wendy Here are some questions I have: the [formula](https://www.kaggle.com/c/facebook-v-predicting-check-ins/details/evaluation) sums over u but P(k) doesn't depend on u? What is U? The formula is a mean of an average but I don't see two averages being taken. Where's the definition of P(k) and shouldn't it be the Average Precision AP(k)? \r\n\r\nI'm being pedantic but I think being pedantic can help sometimes (this being one of those times). Thanks.", "votes": null }, { "id": "124157", "postDate": "06/15/2016 20:59:32", "content": "[quote=Rory;124156]
\n\n@Wendy Here are some questions I have: the formula sums over u but P(k) doesn't depend on u? What is U? The formula is a mean of an average but I don't see two averages being taken. Where's the definition of P(k) and shouldn't it be the Average Precision AP(k)?
\n\nI'm being pedantic but I think being pedantic can help sometimes (this being one of those times). Thanks.
\n\n[/quote]
\n\nI agree to some extent that the metric could be simplified for this competition, but if you just look at the metric it is pretty clear what it does. The problem is that the description of MAP@K while correct in the general use is overkill for its use in this competition.
\n\nFor this competition here is a straightforward explanation of the metric: If your 1st prediction is correct you get 1.0 for that row, which is the maximum score for any given row. If your 2nd prediction is correct you get 0.5 for that row. If your 3rd prediction is correct you get 0.33333 (1/3) for that row. You then sum up your score for all the rows, and divide by the total number of rows. Higher scores are better.
", "rawMarkdown": "[quote=Rory;124156]\r\n\r\n@Wendy Here are some questions I have: the [formula](https://www.kaggle.com/c/facebook-v-predicting-check-ins/details/evaluation) sums over u but P(k) doesn't depend on u? What is U? The formula is a mean of an average but I don't see two averages being taken. Where's the definition of P(k) and shouldn't it be the Average Precision AP(k)? \r\n\r\nI'm being pedantic but I think being pedantic can help sometimes (this being one of those times). Thanks.\r\n\r\n[/quote]\r\n\r\nI agree to some extent that the metric could be simplified for this competition, but if you just look at the metric it is pretty clear what it does. The problem is that the description of MAP@K while correct in the general use is overkill for its use in this competition.\r\n\r\nFor this competition here is a straightforward explanation of the metric: If your 1st prediction is correct you get 1.0 for that row, which is the maximum score for any given row. If your 2nd prediction is correct you get 0.5 for that row. If your 3rd prediction is correct you get 0.33333 (1/3) for that row. You then sum up your score for all the rows, and divide by the total number of rows. Higher scores are better.", "votes": null }, { "id": "124161", "postDate": "06/15/2016 21:23:11", "content": "[quote=Rory;124156]
\n\n@Wendy Here are some questions I have: the formula sums over u but P(k) doesn't depend on u? What is U? The formula is a mean of an average but I don't see two averages being taken. Where's the definition of P(k) and shouldn't it be the Average Precision AP(k)?
\n\nI'm being pedantic but I think being pedantic can help sometimes (this being one of those times). Thanks.
\n\n[/quote]
\n\n@Rory, Thanks for an opportunity for me to explain these :)
\n\nU is the number of check-in events. The first summation takes the mean of all the rows (all the check-in events). P(k) is the precision at k, and it's being averaged by the second summation.
\n\nTo understand P(k), think of it as the precision at your current guess. If you had guessed 3, you would have P(1), P(2), P(3), and we take the average of those. Therefore, for each row, you have 3 guesses, you would have 3 precisions. We take the average of those, and each row will have a AP (average precision).
\n\nFinally, I had always liked this video about Mean Average Precision. I like it so much that I use this screenshot (attached) to explain the metric every time MAP@K is used.
", "rawMarkdown": "[quote=Rory;124156]\r\n\r\n@Wendy Here are some questions I have: the [formula](https://www.kaggle.com/c/facebook-v-predicting-check-ins/details/evaluation) sums over u but P(k) doesn't depend on u? What is U? The formula is a mean of an average but I don't see two averages being taken. Where's the definition of P(k) and shouldn't it be the Average Precision AP(k)? \r\n\r\nI'm being pedantic but I think being pedantic can help sometimes (this being one of those times). Thanks.\r\n\r\n[/quote]\r\n\r\n@Rory, Thanks for an opportunity for me to explain these :) \r\n\r\nU is the number of check-in events. The first summation takes the mean of all the rows (all the check-in events). P(k) is the precision at k, and it's being averaged by the second summation. \r\n\r\nTo understand P(k), think of it as the precision at your current guess. If you had guessed 3, you would have P(1), P(2), P(3), and we take the average of those. Therefore, for each row, you have 3 guesses, you would have 3 precisions. We take the average of those, and each row will have a AP (average precision). \r\n\r\nFinally, I had always liked this [video][1] about Mean Average Precision. I like it so much that I use this screenshot (attached) to explain the metric every time MAP@K is used. \r\n\r\n\r\n [1]: https://www.youtube.com/watch?v=pM6DJ0ZZee0", "votes": null }, { "id": "124171", "postDate": "06/16/2016 00:07:27", "content": "", "rawMarkdown": "", "votes": null }, { "id": "124173", "postDate": "06/16/2016 00:51:18", "content": "[quote=Chris Maki;124171]
\n\nFor instance, if you get all 3 correct in a row, ....
\n\n[/quote]
\n\nIn this problem, it's impossible to have 3 correct predictions for one check in, as each check in only corresponds to one truth place_id. And if you put same predictions in a row, only the first one will be considered.
", "rawMarkdown": "[quote=Chris Maki;124171]\r\n\r\nFor instance, if you get all 3 correct in a row, ....\r\n\r\n[/quote]\r\n\r\nIn this problem, it's impossible to have 3 correct predictions for one check in, as each check in only corresponds to one truth place_id. And if you put same predictions in a row, only the first one will be considered.", "votes": null }, { "id": "124174", "postDate": "06/16/2016 01:00:00", "content": "@Leon
\n\nThanks! Now I get it :-) I am just getting started with this comp and totally misread it.
", "rawMarkdown": "Leon \r\n\r\nThanks! Now I get it :-) I am just getting started with this comp and totally misread it.", "votes": null }, { "id": "124177", "postDate": "06/16/2016 01:32:19", "content": "@Leon
\n\nI think I might be confused still and I appreciate your help :-)
\n\nIf I am understanding this correctly there are only 3 possible values returned for each row, 1, 1/2, and 1/3?
", "rawMarkdown": "Leon\r\n\r\nI think I might be confused still and I appreciate your help :-)\r\n\r\nIf I am understanding this correctly there are only **3 possible values** returned **for each row**, **1, 1/2, and 1/3?**", "votes": null }, { "id": "124178", "postDate": "06/16/2016 01:33:59", "content": "[quote=Chris Maki;124177]
\n\n@Leon
\n\nI think I might be confused still and I appreciate your help :-)
\n\nIf I am understanding this correctly there are only 3 possible values returned for each row, 1, 1/2, and 1/3?
\n\n[/quote]
\n\nDon't forget about 0, if none of your predictions are correct. Otherwise you will only ever get one of those 3 values per row.
", "rawMarkdown": "[quote=Chris Maki;124177]\r\n\r\n@Leon\r\n\r\nI think I might be confused still and I appreciate your help :-)\r\n\r\nIf I am understanding this correctly there are only **3 possible values** returned **for each row**, **1, 1/2, and 1/3?** \r\n\r\n\r\n\r\n[/quote]\r\n\r\nDon't forget about 0, if none of your predictions are correct. Otherwise you will only ever get one of those 3 values per row.", "votes": null }, { "id": "124179", "postDate": "06/16/2016 01:38:17", "content": "@Jared
\n\nThanks! If I am understanding this correctly it looks like if you use Ben's python MAP implementation you can speed up your local CV considerably by adding in a break statement around line 35. Since there is only 1 actual per row in this comp.
", "rawMarkdown": "Jared\r\n\r\nThanks! If I am understanding this correctly it looks like if you use Ben's python MAP implementation you can speed up your local CV considerably by adding in a break statement around line 35. Since there is only 1 actual per row in this comp.", "votes": null } ], "comments": [ { "id": 120251, "author_name": "pinguinson", "author_url": "", "post_date": "05/16/2016 18:38:04", "content": "In this contest m == 1 (since there's only one place_id per row), therefore min(m, n) == 1 and can be ignored.
", "votes": null, "replies": [] }, { "id": 120266, "author_name": "purblue", "author_url": "", "post_date": "05/16/2016 23:38:02", "content": "One clarification. Since we only have m = 1, all possible cases are (suppose we produce all three recommendations)
\n\n(True, False, False) : ap@3 = 1
\n\n(False, True, False) : ap@3 = 1/2
\n\n(False, False, True) : ap@3 = 1/3
\n\n(False, False, False) : ap@3 = 0
\n\nIt means that there is no advantage to submit only one prediction, which is correct, rather than all three predictions including incorrect answers.
\n\nIs this right?
", "votes": null, "replies": [] }, { "id": 120267, "author_name": "pinguinson", "author_url": "", "post_date": "05/16/2016 23:49:53", "content": "[quote=Seungbum Park;120266]
\n\nIt means that there is no advantage to submit only one prediction, which is correct, rather than all three predictions including incorrect answers.
\n\nIs this right?
\n\n[/quote]
\n\nYou still want your correct prediction to be first in the list: [right, wrong1, wrong2], so you get AP score of 1.0 and not 0.25 or 0.33.
", "votes": null, "replies": [] }, { "id": 123063, "author_name": "adrienrenaud", "author_url": "", "post_date": "06/09/2016 13:26:20", "content": "What if I submit 3 times the same prediction for an example and that this prediction is correct ?\nIn another words what is:
\n\n(True, True, True) : ap@3
\n\nIs it?
\n\n(True, True, True) : ap@3 = 1 + 1/2 + 1/3
", "votes": null, "replies": [] }, { "id": 123099, "author_name": "hitriy", "author_url": "", "post_date": "06/09/2016 17:55:18", "content": "According to "our C# Production Implementation", you don't get score for the same answer more then once
\n\nif (solTokens.Contains(subTokens[i], _StringComparer) && !alreadyRecommended.Contains(subTokens[i], _StringComparer)) \n{\n alreadyRecommended.Add(subTokens[i]);\n count++;\n precisionAtK += count / (i + 1);\n}\nPay attention to alreadyRecommended. i starts from zero.
The evaluation formula for this competition isn't well-defined. I'd like clear mathematical formulas, where every term is defined, that explicitly show how a score is computed. We don't have that right now.
", "votes": null, "replies": [] }, { "id": 124130, "author_name": "wendykan", "author_url": "", "post_date": "06/15/2016 18:39:46", "content": "@Adrien, You can see a MAP@K demo script here. In other words, your score doesn't change if you predict the same answer multiple times.
\n\n@Rory, I'm happy to answer questions, can you tell me which terms are not defined? Have you clicked on the link to Kaggle's wiki from our evaluation page? We have the formula, a couple of use cases, and code in different languages (python, R, C# production code) there.
", "votes": null, "replies": [] }, { "id": 124156, "author_name": "roryhr", "author_url": "", "post_date": "06/15/2016 20:44:58", "content": "@Wendy Here are some questions I have: the formula sums over u but P(k) doesn't depend on u? What is U? The formula is a mean of an average but I don't see two averages being taken. Where's the definition of P(k) and shouldn't it be the Average Precision AP(k)?
\n\nI'm being pedantic but I think being pedantic can help sometimes (this being one of those times). Thanks.
", "votes": null, "replies": [] }, { "id": 124157, "author_name": "jturkewitz", "author_url": "", "post_date": "06/15/2016 20:59:32", "content": "[quote=Rory;124156]
\n\n@Wendy Here are some questions I have: the formula sums over u but P(k) doesn't depend on u? What is U? The formula is a mean of an average but I don't see two averages being taken. Where's the definition of P(k) and shouldn't it be the Average Precision AP(k)?
\n\nI'm being pedantic but I think being pedantic can help sometimes (this being one of those times). Thanks.
\n\n[/quote]
\n\nI agree to some extent that the metric could be simplified for this competition, but if you just look at the metric it is pretty clear what it does. The problem is that the description of MAP@K while correct in the general use is overkill for its use in this competition.
\n\nFor this competition here is a straightforward explanation of the metric: If your 1st prediction is correct you get 1.0 for that row, which is the maximum score for any given row. If your 2nd prediction is correct you get 0.5 for that row. If your 3rd prediction is correct you get 0.33333 (1/3) for that row. You then sum up your score for all the rows, and divide by the total number of rows. Higher scores are better.
", "votes": null, "replies": [] }, { "id": 124161, "author_name": "wendykan", "author_url": "", "post_date": "06/15/2016 21:23:11", "content": "[quote=Rory;124156]
\n\n@Wendy Here are some questions I have: the formula sums over u but P(k) doesn't depend on u? What is U? The formula is a mean of an average but I don't see two averages being taken. Where's the definition of P(k) and shouldn't it be the Average Precision AP(k)?
\n\nI'm being pedantic but I think being pedantic can help sometimes (this being one of those times). Thanks.
\n\n[/quote]
\n\n@Rory, Thanks for an opportunity for me to explain these :)
\n\nU is the number of check-in events. The first summation takes the mean of all the rows (all the check-in events). P(k) is the precision at k, and it's being averaged by the second summation.
\n\nTo understand P(k), think of it as the precision at your current guess. If you had guessed 3, you would have P(1), P(2), P(3), and we take the average of those. Therefore, for each row, you have 3 guesses, you would have 3 precisions. We take the average of those, and each row will have a AP (average precision).
\n\nFinally, I had always liked this video about Mean Average Precision. I like it so much that I use this screenshot (attached) to explain the metric every time MAP@K is used.
", "votes": null, "replies": [] }, { "id": 124171, "author_name": "neuralnetworks", "author_url": "", "post_date": "06/16/2016 00:07:27", "content": "", "votes": null, "replies": [] }, { "id": 124173, "author_name": "leonlu", "author_url": "", "post_date": "06/16/2016 00:51:18", "content": "[quote=Chris Maki;124171]
\n\nFor instance, if you get all 3 correct in a row, ....
\n\n[/quote]
\n\nIn this problem, it's impossible to have 3 correct predictions for one check in, as each check in only corresponds to one truth place_id. And if you put same predictions in a row, only the first one will be considered.
", "votes": null, "replies": [] }, { "id": 124174, "author_name": "neuralnetworks", "author_url": "", "post_date": "06/16/2016 01:00:00", "content": "@Leon
\n\nThanks! Now I get it :-) I am just getting started with this comp and totally misread it.
", "votes": null, "replies": [] }, { "id": 124177, "author_name": "neuralnetworks", "author_url": "", "post_date": "06/16/2016 01:32:19", "content": "@Leon
\n\nI think I might be confused still and I appreciate your help :-)
\n\nIf I am understanding this correctly there are only 3 possible values returned for each row, 1, 1/2, and 1/3?
", "votes": null, "replies": [] }, { "id": 124178, "author_name": "jturkewitz", "author_url": "", "post_date": "06/16/2016 01:33:59", "content": "[quote=Chris Maki;124177]
\n\n@Leon
\n\nI think I might be confused still and I appreciate your help :-)
\n\nIf I am understanding this correctly there are only 3 possible values returned for each row, 1, 1/2, and 1/3?
\n\n[/quote]
\n\nDon't forget about 0, if none of your predictions are correct. Otherwise you will only ever get one of those 3 values per row.
", "votes": null, "replies": [] }, { "id": 124179, "author_name": "neuralnetworks", "author_url": "", "post_date": "06/16/2016 01:38:17", "content": "@Jared
\n\nThanks! If I am understanding this correctly it looks like if you use Ben's python MAP implementation you can speed up your local CV considerably by adding in a break statement around line 35. Since there is only 1 actual per row in this comp.
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "119993": "I found the formula for MAP to be inconsistent.\r\n\r\nAt this competition link: https://www.kaggle.com/c/facebook-v-predicting-check-ins/details/evaluation, the formula is: \r\n$$MAP@3 = \\frac{1}{|U|} \\sum_{u=1}^{|U|} \\sum_{k=1}^{min(3, n)} P(k)$$\r\n\r\nand, at: https://www.kaggle.com/wiki/MeanAveragePrecision, the formula is : \r\n\r\n$$ap@n = \\sum_{k=1}^n P(k) / min(m, n)$$\r\n$$MAP@n = \\sum_{i=1}^N ap@n_i / N$$\r\n\r\nBasically the earlier formula is not dividing P(k) by min(m, n), like the later one does. I suppose the 2nd link is correct representation of this formula, am I correct ?\r\n\r\n-- had made another thread, but for some reason my post is not showing in that thread.", "120251": "In this contest m == 1 (since there's only one place_id per row), therefore min(m, n) == 1 and can be ignored.", "120266": "One clarification. Since we only have m = 1, all possible cases are (suppose we produce all three recommendations)\r\n\r\n\r\n----------\r\n\r\n\r\n(True, False, False) : ap@3 = 1\r\n\r\n(False, True, False) : ap@3 = 1/2\r\n\r\n(False, False, True) : ap@3 = 1/3\r\n\r\n(False, False, False) : ap@3 = 0 \r\n\r\n\r\n----------\r\n\r\n\r\nIt means that there is no advantage to submit only one prediction, which is correct, rather than all three predictions including incorrect answers. \r\n\r\nIs this right?", "120267": "[quote=Seungbum Park;120266]\r\n\r\nIt means that there is no advantage to submit only one prediction, which is correct, rather than all three predictions including incorrect answers. \r\n\r\nIs this right?\r\n\r\n[/quote]\r\n\r\nYou still want your correct prediction to be first in the list: [right, wrong1, wrong2], so you get AP score of 1.0 and not 0.25 or 0.33.", "123063": "What if I submit 3 times the same prediction for an example and that this prediction is correct ?\r\nIn another words what is:\r\n\r\n(True, True, True) : ap@3\r\n\r\nIs it?\r\n\r\n(True, True, True) : ap@3 = 1 + 1/2 + 1/3", "123099": "According to \"our C# [Production Implementation][1]\", you don't get score for the same answer more then once\r\n\r\n if (solTokens.Contains(subTokens[i], _StringComparer) && !alreadyRecommended.Contains(subTokens[i], _StringComparer)) \r\n {\r\n alreadyRecommended.Add(subTokens[i]);\r\n count++;\r\n precisionAtK += count / (i + 1);\r\n }\r\n\r\nPay attention to `alreadyRecommended`. `i` starts from zero.\r\n\r\n [1]: https://www.kaggle.com/wiki/ProductionImplementation", "124109": "The evaluation formula for this competition isn't well-defined. I'd like clear mathematical formulas, where every term is defined, that explicitly show how a score is computed. We don't have that right now.", "124130": "Adrien, You can see a MAP@K demo script [here][1]. In other words, your score doesn't change if you predict the same answer multiple times. \r\n\r\n@Rory, I'm happy to answer questions, can you tell me which terms are not defined? Have you clicked on the [link][2] to Kaggle's wiki from our [evaluation][3] page? We have the formula, a couple of use cases, and code in different languages (python, R, C# production code) there. \r\n\r\n\r\n [1]: https://www.kaggle.com/wendykan/expedia-hotel-recommendations/map-k-demo\r\n [2]: https://www.kaggle.com/wiki/MeanAveragePrecision\r\n [3]: https://www.kaggle.com/c/facebook-v-predicting-check-ins/details/evaluation", "124156": "Wendy Here are some questions I have: the [formula](https://www.kaggle.com/c/facebook-v-predicting-check-ins/details/evaluation) sums over u but P(k) doesn't depend on u? What is U? The formula is a mean of an average but I don't see two averages being taken. Where's the definition of P(k) and shouldn't it be the Average Precision AP(k)? \r\n\r\nI'm being pedantic but I think being pedantic can help sometimes (this being one of those times). Thanks.", "124157": "[quote=Rory;124156]\r\n\r\n@Wendy Here are some questions I have: the [formula](https://www.kaggle.com/c/facebook-v-predicting-check-ins/details/evaluation) sums over u but P(k) doesn't depend on u? What is U? The formula is a mean of an average but I don't see two averages being taken. Where's the definition of P(k) and shouldn't it be the Average Precision AP(k)? \r\n\r\nI'm being pedantic but I think being pedantic can help sometimes (this being one of those times). Thanks.\r\n\r\n[/quote]\r\n\r\nI agree to some extent that the metric could be simplified for this competition, but if you just look at the metric it is pretty clear what it does. The problem is that the description of MAP@K while correct in the general use is overkill for its use in this competition.\r\n\r\nFor this competition here is a straightforward explanation of the metric: If your 1st prediction is correct you get 1.0 for that row, which is the maximum score for any given row. If your 2nd prediction is correct you get 0.5 for that row. If your 3rd prediction is correct you get 0.33333 (1/3) for that row. You then sum up your score for all the rows, and divide by the total number of rows. Higher scores are better.", "124161": "[quote=Rory;124156]\r\n\r\n@Wendy Here are some questions I have: the [formula](https://www.kaggle.com/c/facebook-v-predicting-check-ins/details/evaluation) sums over u but P(k) doesn't depend on u? What is U? The formula is a mean of an average but I don't see two averages being taken. Where's the definition of P(k) and shouldn't it be the Average Precision AP(k)? \r\n\r\nI'm being pedantic but I think being pedantic can help sometimes (this being one of those times). Thanks.\r\n\r\n[/quote]\r\n\r\n@Rory, Thanks for an opportunity for me to explain these :) \r\n\r\nU is the number of check-in events. The first summation takes the mean of all the rows (all the check-in events). P(k) is the precision at k, and it's being averaged by the second summation. \r\n\r\nTo understand P(k), think of it as the precision at your current guess. If you had guessed 3, you would have P(1), P(2), P(3), and we take the average of those. Therefore, for each row, you have 3 guesses, you would have 3 precisions. We take the average of those, and each row will have a AP (average precision). \r\n\r\nFinally, I had always liked this [video][1] about Mean Average Precision. I like it so much that I use this screenshot (attached) to explain the metric every time MAP@K is used. \r\n\r\n\r\n [1]: https://www.youtube.com/watch?v=pM6DJ0ZZee0", "124171": "", "124173": "[quote=Chris Maki;124171]\r\n\r\nFor instance, if you get all 3 correct in a row, ....\r\n\r\n[/quote]\r\n\r\nIn this problem, it's impossible to have 3 correct predictions for one check in, as each check in only corresponds to one truth place_id. And if you put same predictions in a row, only the first one will be considered.", "124174": "Leon \r\n\r\nThanks! Now I get it :-) I am just getting started with this comp and totally misread it.", "124177": "Leon\r\n\r\nI think I might be confused still and I appreciate your help :-)\r\n\r\nIf I am understanding this correctly there are only **3 possible values** returned **for each row**, **1, 1/2, and 1/3?**", "124178": "[quote=Chris Maki;124177]\r\n\r\n@Leon\r\n\r\nI think I might be confused still and I appreciate your help :-)\r\n\r\nIf I am understanding this correctly there are only **3 possible values** returned **for each row**, **1, 1/2, and 1/3?** \r\n\r\n\r\n\r\n[/quote]\r\n\r\nDon't forget about 0, if none of your predictions are correct. Otherwise you will only ever get one of those 3 values per row.", "124179": "Jared\r\n\r\nThanks! If I am understanding this correctly it looks like if you use Ben's python MAP implementation you can speed up your local CV considerably by adding in a break statement around line 35. Since there is only 1 actual per row in this comp." }, "source": "meta" }