{ "id": 3654, "title": "MAP high values", "url": "/competitions/event-recommendation-engine-challenge/discussion/3654", "author_name": "", "post_date": "2013-01-21T21:17:42.377Z", "votes": null, "comment_count": 10, "views": 2843, "content": "

Hi,

\r\n

I follow a standard procedure, where I divide my dataset into two blocks - one for training and for validation. I select rows in such way, that validation sets has unseen users. For metrics I took the code from here:

\r\n

https://github.com/benhamner/Metrics/blob/master/R/R/metrics.r#L145

\r\n

I understand how does it works, but I can't get why validation gives my high values such as >.95

\r\n

Any idea what can be wrong or any insigths how to order dataset for validation?

\r\n

 

\r\n

Thanks,

\r\n

Dzidas

", "messages": [ { "id": "19564", "postDate": "01/21/2013 21:17:42", "content": "

Hi,

\r\n

I follow a standard procedure, where I divide my dataset into two blocks - one for training and for validation. I select rows in such way, that validation sets has unseen users. For metrics I took the code from here:

\r\n

https://github.com/benhamner/Metrics/blob/master/R/R/metrics.r#L145

\r\n

I understand how does it works, but I can't get why validation gives my high values such as >.95

\r\n

Any idea what can be wrong or any insigths how to order dataset for validation?

\r\n

 

\r\n

Thanks,

\r\n

Dzidas

", "rawMarkdown": "", "votes": null }, { "id": "19706", "postDate": "01/26/2013 09:49:32", "content": "

Hi,

\r\n

 

\r\n

I have the same problem (also using Rmetrics).

\r\n

 

\r\n

Example:

\r\n
>apk(200, c(1918771225, 1502284248, 2529072432, 3072478280, 1390707377, 1532377761), c(3072478280, 1532377761, 1502284248, 1918771225, 1390707377, 2529072432))
\r\n

[1] 1

\r\n

 

\r\n

is this correct?

\r\n


\r\n

", "rawMarkdown": "", "votes": null }, { "id": "19707", "postDate": "01/26/2013 10:04:22", "content": "

Yes, it's correct from the definition of apk.

\r\n

 

\r\n

When you evaluate your score, you should pass only "interested" list to actual list.

", "rawMarkdown": "", "votes": null }, { "id": "19708", "postDate": "01/26/2013 10:07:14", "content": "

Thank you n_m. By the way, i am also living in Japan (tokyo). Where are you located?

", "rawMarkdown": "", "votes": null }, { "id": "19709", "postDate": "01/26/2013 10:13:04", "content": "

The same, Tokyo.

", "rawMarkdown": "", "votes": null }, { "id": "19712", "postDate": "01/26/2013 15:31:41", "content": "

n_m,

\r\n

yes,I found that latter.
\r\nI pass only "interested" column and now I have very low score at validation stage 0.28 vs public>.65

\r\n

Do you have similar deviation on your validation set or is it close to public score?

", "rawMarkdown": "", "votes": null }, { "id": "19714", "postDate": "01/26/2013 15:41:41", "content": "

Quite close to leaderboard score though cv scores have a little bit large sd (0.01 ~ 0.02). So I suppose your code have some bug in it. What exactly your input (like class) to mapk? Could you post codes?

", "rawMarkdown": "", "votes": null }, { "id": "19725", "postDate": "01/26/2013 20:57:54", "content": "

A very high accuracy score almost always means a target leak. Your model building process is incorporating information from the held-out data, perhaps in an indirect way, which it shouldn't know.

\r\n

For example, one of my first attempts was to build a simple CF model over all the data. The result was nearly 100% accuracy. I quickly realized that the model had also read the data held out as validation data, so of course it got the right answers every\r\n time.

", "rawMarkdown": "", "votes": null }, { "id": "19736", "postDate": "01/27/2013 08:37:13", "content": "

n_m,
\r\nthanks. My input had trailing spaces. After fixing that my cv score is +0.02 than public, which is good.\r\n

", "rawMarkdown": "", "votes": null }, { "id": "19737", "postDate": "01/27/2013 08:43:03", "content": "

At first, I thought about leaking something as well. However the problem was how did I build the outcome. As n_m pointed, you need to take into account only "interested" score. Need to say, I had a bug in my code, but it actually depresed my cv score.

", "rawMarkdown": "", "votes": null }, { "id": "19738", "postDate": "01/27/2013 14:46:13", "content": "

Dzidas,

\r\n

you're welcome. I'm quite surprised to hear that you're achieving good score w/o accurate validation:) You must be using great features.

", "rawMarkdown": "", "votes": null } ], "comments": [ { "id": 19706, "author_name": "ildefons", "author_url": "", "post_date": "01/26/2013 09:49:32", "content": "

Hi,

\r\n

 

\r\n

I have the same problem (also using Rmetrics).

\r\n

 

\r\n

Example:

\r\n
>apk(200, c(1918771225, 1502284248, 2529072432, 3072478280, 1390707377, 1532377761), c(3072478280, 1532377761, 1502284248, 1918771225, 1390707377, 2529072432))
\r\n

[1] 1

\r\n

 

\r\n

is this correct?

\r\n


\r\n

", "votes": null, "replies": [] }, { "id": 19707, "author_name": "naokazumizuta", "author_url": "", "post_date": "01/26/2013 10:04:22", "content": "

Yes, it's correct from the definition of apk.

\r\n

 

\r\n

When you evaluate your score, you should pass only "interested" list to actual list.

", "votes": null, "replies": [] }, { "id": 19708, "author_name": "ildefons", "author_url": "", "post_date": "01/26/2013 10:07:14", "content": "

Thank you n_m. By the way, i am also living in Japan (tokyo). Where are you located?

", "votes": null, "replies": [] }, { "id": 19709, "author_name": "naokazumizuta", "author_url": "", "post_date": "01/26/2013 10:13:04", "content": "

The same, Tokyo.

", "votes": null, "replies": [] }, { "id": 19712, "author_name": "dzidas", "author_url": "", "post_date": "01/26/2013 15:31:41", "content": "

n_m,

\r\n

yes,I found that latter.
\r\nI pass only "interested" column and now I have very low score at validation stage 0.28 vs public>.65

\r\n

Do you have similar deviation on your validation set or is it close to public score?

", "votes": null, "replies": [] }, { "id": 19714, "author_name": "naokazumizuta", "author_url": "", "post_date": "01/26/2013 15:41:41", "content": "

Quite close to leaderboard score though cv scores have a little bit large sd (0.01 ~ 0.02). So I suppose your code have some bug in it. What exactly your input (like class) to mapk? Could you post codes?

", "votes": null, "replies": [] }, { "id": 19725, "author_name": "", "author_url": "", "post_date": "01/26/2013 20:57:54", "content": "

A very high accuracy score almost always means a target leak. Your model building process is incorporating information from the held-out data, perhaps in an indirect way, which it shouldn't know.

\r\n

For example, one of my first attempts was to build a simple CF model over all the data. The result was nearly 100% accuracy. I quickly realized that the model had also read the data held out as validation data, so of course it got the right answers every\r\n time.

", "votes": null, "replies": [] }, { "id": 19736, "author_name": "dzidas", "author_url": "", "post_date": "01/27/2013 08:37:13", "content": "

n_m,
\r\nthanks. My input had trailing spaces. After fixing that my cv score is +0.02 than public, which is good.\r\n

", "votes": null, "replies": [] }, { "id": 19737, "author_name": "dzidas", "author_url": "", "post_date": "01/27/2013 08:43:03", "content": "

At first, I thought about leaking something as well. However the problem was how did I build the outcome. As n_m pointed, you need to take into account only "interested" score. Need to say, I had a bug in my code, but it actually depresed my cv score.

", "votes": null, "replies": [] }, { "id": 19738, "author_name": "naokazumizuta", "author_url": "", "post_date": "01/27/2013 14:46:13", "content": "

Dzidas,

\r\n

you're welcome. I'm quite surprised to hear that you're achieving good score w/o accurate validation:) You must be using great features.

", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "19564": "", "19706": "", "19707": "", "19708": "", "19709": "", "19712": "", "19714": "", "19725": "", "19736": "", "19737": "", "19738": "" }, "source": "meta" }