{ "id": 5790, "title": "Has anyone managed to achieve a high leaderboard score using legitimate features only?", "url": "/competitions/accelerometer-biometric-competition/discussion/5790", "author_name": "", "post_date": "2013-09-17T16:32:12.067Z", "votes": 1, "comment_count": 11, "views": 5239, "content": "


You are invited to make submissions that do not exploit the existing leakage. (See thread “Preparing fresh data without leakage – Request for comments”)

\n


The organizers will compensate (out of band to this competition) contestants that can demonstrate high leaderboard scores using legitimate features only.

\n

 

", "messages": [ { "id": "31053", "postDate": "09/17/2013 16:32:12", "content": "


You are invited to make submissions that do not exploit the existing leakage. (See thread “Preparing fresh data without leakage – Request for comments”)

\n


The organizers will compensate (out of band to this competition) contestants that can demonstrate high leaderboard scores using legitimate features only.

\n

 

", "rawMarkdown": "", "votes": null }, { "id": "31064", "postDate": "09/17/2013 21:50:52", "content": "

Hi Geoff

\n

I like your idea, and am looking forward to read more about how you want to implement it concretely.

\n

What demonstration do you have in mind? In general, you might need to take a look at model code and algorithm description in order to decide whether it fits your criteria. This is done by other Kaggle competitions, but I guess it takes time and effort.

\n

 

\n

Personally, I think that my solution fits the legitimate criteria - without revealing too much I can say I use a classical supervised learning paradigm - construct features of the train set 4D signals, train a ML method, predict on the test data. Would this be legitimate for your criteria and constitute a solution useful for the competition organizers?

\n

Sometimes it is tricky to decide for some technical implementation details - e.g. usually using the training set distribution in a clever way belongs to good data science practice, but in this competition one of the problems is supposed to be the same distribution of train and test set.

\n

However, I have a feeling this is one of the smallest leaks, less serious than the time stamp issues and other mysterious leaks to which the top players hint at :-)

", "rawMarkdown": "", "votes": null }, { "id": "31082", "postDate": "09/18/2013 13:39:31", "content": "

Hi Dieselboy,

\n

Thanks for responding to this invitation. I'm pleased to see a high scoring submission that didn't attempt to exploit the leakage. We would want to review your code & maybe talk to ascertain that results are not impacted by leakage. If we can determine that score above 90%  is attributable to legitimate features only, you will be rewarded. Please contact me directly to geoffkle@gmail.com, to discuss progressing this further.

\n

 

", "rawMarkdown": "", "votes": null }, { "id": "31364", "postDate": "09/23/2013 00:50:30", "content": "

Newbie here. Would love to understand what qualifies as a high score ? Are AUCs > .9 achievable using only legitimate features ?

", "rawMarkdown": "", "votes": null }, { "id": "31375", "postDate": "09/23/2013 05:48:23", "content": "

AUC > 0.9

\n

This is what we would like to find out :)

", "rawMarkdown": "", "votes": null }, { "id": "31504", "postDate": "09/24/2013 16:19:31", "content": "

I use the same approach as Dieselboy, I think it is not too difficult to get score >0.9 without using any leakages. I think moreover it is possible to get AUC>0.9 even without time feature.

", "rawMarkdown": "", "votes": null }, { "id": "31570", "postDate": "09/25/2013 21:33:26", "content": "

Hi diaman, would love to review your model, please contact me by email.

", "rawMarkdown": "", "votes": null }, { "id": "31704", "postDate": "09/28/2013 12:20:50", "content": "

[quote=Geoff;31082]

\n

If we can determine that score above 90%  is attributable to legitimate features only, you will be rewarded. [/quote]

\n

 

\n

Hi ! I just run my models without leakage and I got 0.90924 on leaderboard score. I used T the same way as the other variables, the only difference is that, instead using T I used diff(T).

\n

I wonder what kind of reward could we expect if we share you our leakage free models scoring more than 0.90 ?

", "rawMarkdown": "", "votes": null }, { "id": "31711", "postDate": "09/28/2013 13:46:54", "content": "

Hi demytt, Do you have an estimate of how much using the diff(T) variable contributes to your score? Differences in sampling frequencies across device types was one of the major sources of leakage encountered.

\n

I was thinking of ~$150 reward (after review that no leakage). 

\n

 

\n

 

\n

 

", "rawMarkdown": "", "votes": null }, { "id": "31721", "postDate": "09/28/2013 19:18:43", "content": "

I'll run my model without T and let you know the result. However, I think that the use of T shouldn't be considered as cheating in some cases, whereas it's obviously cheating in some other cases.

\n

For example, let's imagine you want to know if QuizDevice A corresponds to device 1, and let's assume we return the following score :

\n

isTrue_{A} = (mean(diff(T_A)) - mean(diff(T_1)))^2 + (mean(X_A) - mean(X_1))^2 + (mean(Y_A) - mean(Y_1))^2 + (mean(Z_A) - mean(Z_1))^2

\n

Do you consider this model as acceptable, or do you only want a model without any use of T ?

", "rawMarkdown": "", "votes": null }, { "id": "32250", "postDate": "10/11/2013 13:48:12", "content": "

My model has only mean(diff(T)) and std(diff(T)) and i'm not so bad. mean(diff(T)) let me improve my score of 0.03

\n

Use diff(T) is really leak exploit ? cause it could be specific to device in real life... no?

\n

 

\n

I have a lot of other features with only X,Y or Z.

", "rawMarkdown": "", "votes": null }, { "id": "32426", "postDate": "10/18/2013 22:54:31", "content": "

The problem is that everything leaks: X, Y, Z, timestamp interval, professed device, and the test data generation methodology. I have a FFT-based method that I believe should not be susceptible to leaks, and it does OK (~0.85), but I can't know for sure without a leak-free data set.

\n

 

", "rawMarkdown": "", "votes": null } ], "comments": [ { "id": 31064, "author_name": "dieselboy", "author_url": "", "post_date": "09/17/2013 21:50:52", "content": "

Hi Geoff

\n

I like your idea, and am looking forward to read more about how you want to implement it concretely.

\n

What demonstration do you have in mind? In general, you might need to take a look at model code and algorithm description in order to decide whether it fits your criteria. This is done by other Kaggle competitions, but I guess it takes time and effort.

\n

 

\n

Personally, I think that my solution fits the legitimate criteria - without revealing too much I can say I use a classical supervised learning paradigm - construct features of the train set 4D signals, train a ML method, predict on the test data. Would this be legitimate for your criteria and constitute a solution useful for the competition organizers?

\n

Sometimes it is tricky to decide for some technical implementation details - e.g. usually using the training set distribution in a clever way belongs to good data science practice, but in this competition one of the problems is supposed to be the same distribution of train and test set.

\n

However, I have a feeling this is one of the smallest leaks, less serious than the time stamp issues and other mysterious leaks to which the top players hint at :-)

", "votes": null, "replies": [] }, { "id": 31082, "author_name": "geoff3", "author_url": "", "post_date": "09/18/2013 13:39:31", "content": "

Hi Dieselboy,

\n

Thanks for responding to this invitation. I'm pleased to see a high scoring submission that didn't attempt to exploit the leakage. We would want to review your code & maybe talk to ascertain that results are not impacted by leakage. If we can determine that score above 90%  is attributable to legitimate features only, you will be rewarded. Please contact me directly to geoffkle@gmail.com, to discuss progressing this further.

\n

 

", "votes": null, "replies": [] }, { "id": 31364, "author_name": "sharadbanka", "author_url": "", "post_date": "09/23/2013 00:50:30", "content": "

Newbie here. Would love to understand what qualifies as a high score ? Are AUCs > .9 achievable using only legitimate features ?

", "votes": null, "replies": [] }, { "id": 31375, "author_name": "geoff3", "author_url": "", "post_date": "09/23/2013 05:48:23", "content": "

AUC > 0.9

\n

This is what we would like to find out :)

", "votes": null, "replies": [] }, { "id": 31504, "author_name": "dselivanov", "author_url": "", "post_date": "09/24/2013 16:19:31", "content": "

I use the same approach as Dieselboy, I think it is not too difficult to get score >0.9 without using any leakages. I think moreover it is possible to get AUC>0.9 even without time feature.

", "votes": null, "replies": [] }, { "id": 31570, "author_name": "geoff3", "author_url": "", "post_date": "09/25/2013 21:33:26", "content": "

Hi diaman, would love to review your model, please contact me by email.

", "votes": null, "replies": [] }, { "id": 31704, "author_name": "ademyttenaere", "author_url": "", "post_date": "09/28/2013 12:20:50", "content": "

[quote=Geoff;31082]

\n

If we can determine that score above 90%  is attributable to legitimate features only, you will be rewarded. [/quote]

\n

 

\n

Hi ! I just run my models without leakage and I got 0.90924 on leaderboard score. I used T the same way as the other variables, the only difference is that, instead using T I used diff(T).

\n

I wonder what kind of reward could we expect if we share you our leakage free models scoring more than 0.90 ?

", "votes": null, "replies": [] }, { "id": 31711, "author_name": "geoff3", "author_url": "", "post_date": "09/28/2013 13:46:54", "content": "

Hi demytt, Do you have an estimate of how much using the diff(T) variable contributes to your score? Differences in sampling frequencies across device types was one of the major sources of leakage encountered.

\n

I was thinking of ~$150 reward (after review that no leakage). 

\n

 

\n

 

\n

 

", "votes": null, "replies": [] }, { "id": 31721, "author_name": "ademyttenaere", "author_url": "", "post_date": "09/28/2013 19:18:43", "content": "

I'll run my model without T and let you know the result. However, I think that the use of T shouldn't be considered as cheating in some cases, whereas it's obviously cheating in some other cases.

\n

For example, let's imagine you want to know if QuizDevice A corresponds to device 1, and let's assume we return the following score :

\n

isTrue_{A} = (mean(diff(T_A)) - mean(diff(T_1)))^2 + (mean(X_A) - mean(X_1))^2 + (mean(Y_A) - mean(Y_1))^2 + (mean(Z_A) - mean(Z_1))^2

\n

Do you consider this model as acceptable, or do you only want a model without any use of T ?

", "votes": null, "replies": [] }, { "id": 32250, "author_name": "nnnico", "author_url": "", "post_date": "10/11/2013 13:48:12", "content": "

My model has only mean(diff(T)) and std(diff(T)) and i'm not so bad. mean(diff(T)) let me improve my score of 0.03

\n

Use diff(T) is really leak exploit ? cause it could be specific to device in real life... no?

\n

 

\n

I have a lot of other features with only X,Y or Z.

", "votes": null, "replies": [] }, { "id": 32426, "author_name": "solorzano", "author_url": "", "post_date": "10/18/2013 22:54:31", "content": "

The problem is that everything leaks: X, Y, Z, timestamp interval, professed device, and the test data generation methodology. I have a FFT-based method that I believe should not be susceptible to leaks, and it does OK (~0.85), but I can't know for sure without a leak-free data set.

\n

 

", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "31053": "", "31064": "", "31082": "", "31364": "", "31375": "", "31504": "", "31570": "", "31704": "", "31711": "", "31721": "", "32250": "", "32426": "" }, "source": "meta" }