{ "id": 3608, "title": "Training data", "url": "/competitions/event-recommendation-engine-challenge/discussion/3608", "author_name": "", "post_date": "2013-01-15T22:55:55.423Z", "votes": null, "comment_count": 13, "views": 3969, "content": "

Hey folks, I'm a little confused as to what our training data solution is. Are the events in the train.csv in order of what that user would consider to be the most interesting all the way down to the least interesting?

", "messages": [ { "id": "19350", "postDate": "01/15/2013 22:55:55", "content": "

Hey folks, I'm a little confused as to what our training data solution is. Are the events in the train.csv in order of what that user would consider to be the most interesting all the way down to the least interesting?

", "rawMarkdown": "", "votes": null }, { "id": "19353", "postDate": "01/16/2013 00:25:03", "content": "

train.csv is more about whether the person indicated interest in an event or not. Look at event_attendees.csv for whether the person actually went to the event.

", "rawMarkdown": "", "votes": null }, { "id": "19356", "postDate": "01/16/2013 00:51:48", "content": "

I understand that part, but whether they went to an event or not doesn't really indicate how interested they are in the event over the other events. Besides, the lists of users given in event_attendees doesn't include any users from either train.csv or test.csv--they\r\n were specifically removed from the data before it was given to us.

", "rawMarkdown": "", "votes": null }, { "id": "19357", "postDate": "01/16/2013 00:56:12", "content": "

In train.csv the majority of users are not invented to the events, but sometimes they are "not interested" in these events. How is it possible? How could they click "not interested" if they hadn't been invented? In some other strings there are zeros for\r\n "invented", "interested", "not interested". Why are these people included in this file? Does it contain the full matrix of users-events? Obviously, not.

", "rawMarkdown": "", "votes": null }, { "id": "19364", "postDate": "01/16/2013 13:57:49", "content": "

@ Brett.  To score your train.csv predictions I believe you need events marked "interested" in your actual vector in any order.  Here's an example in R using Ben's apk(k, actual, predicted) function from the kaggle wiki:

\r\n

Say you have 5 events to rank (k=5): 1,2,3,4,5 and Events 1 and 2 have been marked interested.  Consider the following cases,

\r\n

Case A: apk(5, c(1,2), c(1,2,5,4,3)) = 1

\r\n

Case B: apk(5, c(1,2), c(2,1,5,4,3)) = 1

\r\n

Case C: apk(5, c(1,2), c(1,4,5,2,3)) = .75

\r\n

Case D: apk(5, c(1,2), c(1,4,2,5,3)) = 0.8333

\r\n

 

\r\n

You can see from Cases A and B a max score is reached regardless of how you order Events 1 and 2 so long as they are the first two events.  If you look at Cases C and D you will find D scores better because Event 2 has been ranked a spot higher.  I hope\r\n this helps!

\r\n

Note: k can be greater than 5 in this example and the scores will remain the same meaning you wont need to count the events for each user.  Just use a large k for everything.

", "rawMarkdown": "", "votes": null }, { "id": "19377", "postDate": "01/16/2013 23:23:48", "content": "

Thanks Chase, now I'm moving again. I assume that if an event is marked as "interested" goes to the top of the list, the events marked as "not interested" go to the bottom?

", "rawMarkdown": "", "votes": null }, { "id": "19412", "postDate": "01/17/2013 21:31:33", "content": "

[quote=raiden;19353]

\r\n

train.csv is more about whether the person indicated interest in an event or not. Look at event_attendees.csv for whether the person actually went to the event.

\r\n

[/quote]

\r\n

 

\r\n

 

\r\n

this is quite wrong. You won't find users from train or test in event_attendees.

", "rawMarkdown": "", "votes": null }, { "id": "19413", "postDate": "01/17/2013 21:47:20", "content": "

Chase sorry but I still don't get the ordered events from train.
\r\nLet's assume that three events have been marked as interested. How will they be ordered? 

", "rawMarkdown": "", "votes": null }, { "id": "19418", "postDate": "01/17/2013 23:21:29", "content": "

Hi Piggy.  The order of 'interested' events does not matter.  In my example I used c(1,2) for the actual ordering of interested events but you will find c(2,1) gives the same score.  This is my first time using the MAP metric so any intuition comes from\r\n tinkering with toy scenarios.  Maybe someone with more experience could provide a better explanation.

", "rawMarkdown": "", "votes": null }, { "id": "19441", "postDate": "01/18/2013 07:29:35", "content": "

Hi Piggy, Chase,

\r\n

I'm new to MAP, but here is how I see it.

\r\n

 

\r\n

Let say you have to arrange 5 items into 5 sorted buckets.

\r\n

Bucket1 has 1/1 multiplier

\r\n

Bucket2 has 1/2 multiplier

\r\n

Bucket3 has 1/3 multiplier

\r\n

Bucket4 has 1/4 multiplier

\r\n

Bucket5 has 1/5 multiplier

\r\n

 

\r\n

if the answer has 2 hits: think as if you have 2 cards to put into buckets. card_1 has a weight of 1. card_2 has a weight of 2. (yes, the later card has more weight)

\r\n

as you examine from bucket 1 to bucket 5,

\r\n

if it is a hit, put a card into that bucket.

\r\n

if it is a miss, then skip to the next bucket.

\r\n

The catch is that you have to put card_1 before card_2.

\r\n

 

\r\n

Multiply the card_weight to the bucket_multiplier.

\r\n

and add them up.

\r\n

 

\r\n

Then, you divide the score with the number of the answer.(normalize)

\r\n

 

\r\n

 

\r\n

IMHO, the score maximizing strategy should focus on the earlier buckets.

\r\n

The difference between bucket1 and bucket2, is much greater than the difference between bucket4 and bucket5.

\r\n

hope that helps.

\r\n

 

\r\n

 

\r\n

 

\r\n

 

", "rawMarkdown": "", "votes": null }, { "id": "19443", "postDate": "01/18/2013 10:02:20", "content": "

Thanks John. I've been reading a bunch of things about MAP and it is clear now. 
\r\nHow is this information (the correctly chosen events) packed in train.csv? Let's pick up the first user, for instance. He's interested in 1 event.\r\n
\r\nHow are the other four ordered (for which he didn't click anything)? 

", "rawMarkdown": "", "votes": null }, { "id": "19498", "postDate": "01/19/2013 15:16:15", "content": "

@Piggy,

\r\n

A user's interest in an event would be one of 3 categories, interested, 'did not click', not interested.

\r\n

My guess is that the events should be sorted according to those categories. The 'interested' events first, 'not interested' last, and 'did not click' in between. If the rest of the events fall into the same category, the order shouldn't matter. 

", "rawMarkdown": "", "votes": null }, { "id": "19509", "postDate": "01/19/2013 19:29:11", "content": "

makes sense. 

\r\n

thanks

", "rawMarkdown": "", "votes": null }, { "id": "19545", "postDate": "01/21/2013 03:24:26", "content": "

Logic would seem to say that we should catagorize events as "interested", "didn't_click", and "not_interested," but the Mean Average Precision is a binary metric--from what I can tell, all it cares about is that the "interested" events are before everything\r\n else.

", "rawMarkdown": "", "votes": null } ], "comments": [ { "id": 19353, "author_name": "raiden", "author_url": "", "post_date": "01/16/2013 00:25:03", "content": "

train.csv is more about whether the person indicated interest in an event or not. Look at event_attendees.csv for whether the person actually went to the event.

", "votes": null, "replies": [] }, { "id": 19356, "author_name": "brett7", "author_url": "", "post_date": "01/16/2013 00:51:48", "content": "

I understand that part, but whether they went to an event or not doesn't really indicate how interested they are in the event over the other events. Besides, the lists of users given in event_attendees doesn't include any users from either train.csv or test.csv--they\r\n were specifically removed from the data before it was given to us.

", "votes": null, "replies": [] }, { "id": 19357, "author_name": "sergeydemyanov", "author_url": "", "post_date": "01/16/2013 00:56:12", "content": "

In train.csv the majority of users are not invented to the events, but sometimes they are "not interested" in these events. How is it possible? How could they click "not interested" if they hadn't been invented? In some other strings there are zeros for\r\n "invented", "interested", "not interested". Why are these people included in this file? Does it contain the full matrix of users-events? Obviously, not.

", "votes": null, "replies": [] }, { "id": 19364, "author_name": "chaseos", "author_url": "", "post_date": "01/16/2013 13:57:49", "content": "

@ Brett.  To score your train.csv predictions I believe you need events marked "interested" in your actual vector in any order.  Here's an example in R using Ben's apk(k, actual, predicted) function from the kaggle wiki:

\r\n

Say you have 5 events to rank (k=5): 1,2,3,4,5 and Events 1 and 2 have been marked interested.  Consider the following cases,

\r\n

Case A: apk(5, c(1,2), c(1,2,5,4,3)) = 1

\r\n

Case B: apk(5, c(1,2), c(2,1,5,4,3)) = 1

\r\n

Case C: apk(5, c(1,2), c(1,4,5,2,3)) = .75

\r\n

Case D: apk(5, c(1,2), c(1,4,2,5,3)) = 0.8333

\r\n

 

\r\n

You can see from Cases A and B a max score is reached regardless of how you order Events 1 and 2 so long as they are the first two events.  If you look at Cases C and D you will find D scores better because Event 2 has been ranked a spot higher.  I hope\r\n this helps!

\r\n

Note: k can be greater than 5 in this example and the scores will remain the same meaning you wont need to count the events for each user.  Just use a large k for everything.

", "votes": null, "replies": [] }, { "id": 19377, "author_name": "brett7", "author_url": "", "post_date": "01/16/2013 23:23:48", "content": "

Thanks Chase, now I'm moving again. I assume that if an event is marked as "interested" goes to the top of the list, the events marked as "not interested" go to the bottom?

", "votes": null, "replies": [] }, { "id": 19412, "author_name": "worldofpiggy", "author_url": "", "post_date": "01/17/2013 21:31:33", "content": "

[quote=raiden;19353]

\r\n

train.csv is more about whether the person indicated interest in an event or not. Look at event_attendees.csv for whether the person actually went to the event.

\r\n

[/quote]

\r\n

 

\r\n

 

\r\n

this is quite wrong. You won't find users from train or test in event_attendees.

", "votes": null, "replies": [] }, { "id": 19413, "author_name": "worldofpiggy", "author_url": "", "post_date": "01/17/2013 21:47:20", "content": "

Chase sorry but I still don't get the ordered events from train.
\r\nLet's assume that three events have been marked as interested. How will they be ordered? 

", "votes": null, "replies": [] }, { "id": 19418, "author_name": "chaseos", "author_url": "", "post_date": "01/17/2013 23:21:29", "content": "

Hi Piggy.  The order of 'interested' events does not matter.  In my example I used c(1,2) for the actual ordering of interested events but you will find c(2,1) gives the same score.  This is my first time using the MAP metric so any intuition comes from\r\n tinkering with toy scenarios.  Maybe someone with more experience could provide a better explanation.

", "votes": null, "replies": [] }, { "id": 19441, "author_name": "sigint", "author_url": "", "post_date": "01/18/2013 07:29:35", "content": "

Hi Piggy, Chase,

\r\n

I'm new to MAP, but here is how I see it.

\r\n

 

\r\n

Let say you have to arrange 5 items into 5 sorted buckets.

\r\n

Bucket1 has 1/1 multiplier

\r\n

Bucket2 has 1/2 multiplier

\r\n

Bucket3 has 1/3 multiplier

\r\n

Bucket4 has 1/4 multiplier

\r\n

Bucket5 has 1/5 multiplier

\r\n

 

\r\n

if the answer has 2 hits: think as if you have 2 cards to put into buckets. card_1 has a weight of 1. card_2 has a weight of 2. (yes, the later card has more weight)

\r\n

as you examine from bucket 1 to bucket 5,

\r\n

if it is a hit, put a card into that bucket.

\r\n

if it is a miss, then skip to the next bucket.

\r\n

The catch is that you have to put card_1 before card_2.

\r\n

 

\r\n

Multiply the card_weight to the bucket_multiplier.

\r\n

and add them up.

\r\n

 

\r\n

Then, you divide the score with the number of the answer.(normalize)

\r\n

 

\r\n

 

\r\n

IMHO, the score maximizing strategy should focus on the earlier buckets.

\r\n

The difference between bucket1 and bucket2, is much greater than the difference between bucket4 and bucket5.

\r\n

hope that helps.

\r\n

 

\r\n

 

\r\n

 

\r\n

 

", "votes": null, "replies": [] }, { "id": 19443, "author_name": "worldofpiggy", "author_url": "", "post_date": "01/18/2013 10:02:20", "content": "

Thanks John. I've been reading a bunch of things about MAP and it is clear now. 
\r\nHow is this information (the correctly chosen events) packed in train.csv? Let's pick up the first user, for instance. He's interested in 1 event.\r\n
\r\nHow are the other four ordered (for which he didn't click anything)? 

", "votes": null, "replies": [] }, { "id": 19498, "author_name": "shark8me", "author_url": "", "post_date": "01/19/2013 15:16:15", "content": "

@Piggy,

\r\n

A user's interest in an event would be one of 3 categories, interested, 'did not click', not interested.

\r\n

My guess is that the events should be sorted according to those categories. The 'interested' events first, 'not interested' last, and 'did not click' in between. If the rest of the events fall into the same category, the order shouldn't matter. 

", "votes": null, "replies": [] }, { "id": 19509, "author_name": "worldofpiggy", "author_url": "", "post_date": "01/19/2013 19:29:11", "content": "

makes sense. 

\r\n

thanks

", "votes": null, "replies": [] }, { "id": 19545, "author_name": "brett7", "author_url": "", "post_date": "01/21/2013 03:24:26", "content": "

Logic would seem to say that we should catagorize events as "interested", "didn't_click", and "not_interested," but the Mean Average Precision is a binary metric--from what I can tell, all it cares about is that the "interested" events are before everything\r\n else.

", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "19350": "", "19353": "", "19356": "", "19357": "", "19364": "", "19377": "", "19412": "", "19413": "", "19418": "", "19441": "", "19443": "", "19498": "", "19509": "", "19545": "" }, "source": "meta" }