{ "id": 3729, "title": "Update: Some rows in the test set no longer count to your score", "url": "/competitions/event-recommendation-engine-challenge/discussion/3729", "author_name": "", "post_date": "2013-01-30T19:57:39.960Z", "votes": null, "comment_count": 4, "views": 3039, "content": "

Hi all,

\r\n

As r0u1i pointed out, there is a bit of data leakage in the test set due to the timestamp column. In cases where there is a group of events sharing the same timestamp and then a single event with a timestamp\r\n a little bit later for a given user, the single event has a high probability of being marked "interested". This is likely due to a sampling issue when the competition host pulled the data for this competition.

\r\n

As a result, we've decided the best way to move forward is to ignore all test cases where this leakage may be present in the evaluation. Specifically, we are now only scoring test rows where all events had exactly the same timestamp for\r\n a particular user. This leaves 787 users in the entire test set (meaning we are now not scoring 570).

\r\n

We are not removing these rows from the test set, so this isn't a breaking change to the submission format and there is no need to re-download the files.

\r\n

Thanks for your enthusiasm so far in this competition, and we apologize for the leakage and corresponding slight leaderboard shuffle in handling it.

\r\n

Ben

", "messages": [ { "id": "19861", "postDate": "01/30/2013 19:57:39", "content": "

Hi all,

\r\n

As r0u1i pointed out, there is a bit of data leakage in the test set due to the timestamp column. In cases where there is a group of events sharing the same timestamp and then a single event with a timestamp\r\n a little bit later for a given user, the single event has a high probability of being marked "interested". This is likely due to a sampling issue when the competition host pulled the data for this competition.

\r\n

As a result, we've decided the best way to move forward is to ignore all test cases where this leakage may be present in the evaluation. Specifically, we are now only scoring test rows where all events had exactly the same timestamp for\r\n a particular user. This leaves 787 users in the entire test set (meaning we are now not scoring 570).

\r\n

We are not removing these rows from the test set, so this isn't a breaking change to the submission format and there is no need to re-download the files.

\r\n

Thanks for your enthusiasm so far in this competition, and we apologize for the leakage and corresponding slight leaderboard shuffle in handling it.

\r\n

Ben

", "rawMarkdown": "", "votes": null }, { "id": "19865", "postDate": "01/30/2013 21:17:29", "content": "

Ben,

\r\n

Can you explain the "all user-event pairs had exactly the same timestamp" in a little more detail?

\r\n

Thank you.

", "rawMarkdown": "", "votes": null }, { "id": "19866", "postDate": "01/30/2013 21:33:34", "content": "

Here are some lines from test.csv:

\r\n
user,event,invited,timestamp
7514340,507707719,0,2012-11-10 08:22:42.330000+00:00
7514340,3841392405,0,2012-11-10 08:22:42.330000+00:00
7514340,4134671058,0,2012-11-10 08:22:42.330000+00:00
7514340,854823005,0,2012-11-10 08:22:42.330000+00:00
7514340,2696649330,0,2012-11-10 08:22:42.330000+00:00
7514340,901144614,0,2012-11-10 08:22:42.330000+00:00
15390083,907302600,0,2012-10-25 13:08:26.153000+00:00
15390083,2643833505,0,2012-10-25 13:08:22.098000+00:00
15390083,1361307272,0,2012-10-25 13:08:22.098000+00:00
15390083,955398943,0,2012-10-25 13:08:22.098000+00:00
15390083,771676713,0,2012-10-25 13:08:22.098000+00:00
15390083,633659090,0,2012-10-25 13:08:22.098000+00:00
15390083,2529072432,0,2012-10-25 13:08:22.098000+00:00
\r\n

Your predictions will be evaluated for user 7514340, since that user has all events with the same time stamp. Your predictions will not be evaluated for user 15390083, since that user does not have all events with the same timestamp.

", "rawMarkdown": "", "votes": null }, { "id": "19920", "postDate": "02/01/2013 07:07:48", "content": "

Ben,

\r\n

The timestamp in test data represents the UAT time, when user saw the event on his system (+/- 2 Hours).

\r\n

I am unable to understand what difference it will make if the time stamps are different for different events (for same user).

\r\n

Can you explain it in more detail?

\r\n

 

\r\n

Thanks,

", "rawMarkdown": "", "votes": null }, { "id": "19939", "postDate": "02/01/2013 21:59:36", "content": "

So do the models that we build now require that either data with small timestamp differences are either made the same or excluded?

\r\n

 

\r\n

By likely in the sampling , and  leakage may be present. Has this been verified with the sponsor? My testing on the data is not showing this. I wonder if when you log on to a page in their site ad type pop ups might appear for highly popular or paid for\r\n by event owner events. 4 seconds seems about the right time for a pop up just to get in the way of the paragraph you're half through

\r\n

 

\r\n

D3

", "rawMarkdown": "", "votes": null } ], "comments": [ { "id": 19865, "author_name": "abundrew", "author_url": "", "post_date": "01/30/2013 21:17:29", "content": "

Ben,

\r\n

Can you explain the "all user-event pairs had exactly the same timestamp" in a little more detail?

\r\n

Thank you.

", "votes": null, "replies": [] }, { "id": 19866, "author_name": "benhamner", "author_url": "", "post_date": "01/30/2013 21:33:34", "content": "

Here are some lines from test.csv:

\r\n
user,event,invited,timestamp
7514340,507707719,0,2012-11-10 08:22:42.330000+00:00
7514340,3841392405,0,2012-11-10 08:22:42.330000+00:00
7514340,4134671058,0,2012-11-10 08:22:42.330000+00:00
7514340,854823005,0,2012-11-10 08:22:42.330000+00:00
7514340,2696649330,0,2012-11-10 08:22:42.330000+00:00
7514340,901144614,0,2012-11-10 08:22:42.330000+00:00
15390083,907302600,0,2012-10-25 13:08:26.153000+00:00
15390083,2643833505,0,2012-10-25 13:08:22.098000+00:00
15390083,1361307272,0,2012-10-25 13:08:22.098000+00:00
15390083,955398943,0,2012-10-25 13:08:22.098000+00:00
15390083,771676713,0,2012-10-25 13:08:22.098000+00:00
15390083,633659090,0,2012-10-25 13:08:22.098000+00:00
15390083,2529072432,0,2012-10-25 13:08:22.098000+00:00
\r\n

Your predictions will be evaluated for user 7514340, since that user has all events with the same time stamp. Your predictions will not be evaluated for user 15390083, since that user does not have all events with the same timestamp.

", "votes": null, "replies": [] }, { "id": 19920, "author_name": "adityasd", "author_url": "", "post_date": "02/01/2013 07:07:48", "content": "

Ben,

\r\n

The timestamp in test data represents the UAT time, when user saw the event on his system (+/- 2 Hours).

\r\n

I am unable to understand what difference it will make if the time stamps are different for different events (for same user).

\r\n

Can you explain it in more detail?

\r\n

 

\r\n

Thanks,

", "votes": null, "replies": [] }, { "id": 19939, "author_name": "d3potf", "author_url": "", "post_date": "02/01/2013 21:59:36", "content": "

So do the models that we build now require that either data with small timestamp differences are either made the same or excluded?

\r\n

 

\r\n

By likely in the sampling , and  leakage may be present. Has this been verified with the sponsor? My testing on the data is not showing this. I wonder if when you log on to a page in their site ad type pop ups might appear for highly popular or paid for\r\n by event owner events. 4 seconds seems about the right time for a pop up just to get in the way of the paragraph you're half through

\r\n

 

\r\n

D3

", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "19861": "", "19865": "", "19866": "", "19920": "", "19939": "" }, "source": "meta" }