I'm very curious to know what scores people are getting without using any leakage. Perhaps shortly, you could also tell me something about your algorithm. The best I've been able to get is just over .86. Has anybody achieved > .9?
\nThanks!
", "messages": [ { "id": "35020", "postDate": "11/22/2013 00:40:15", "content": "I'm very curious to know what scores people are getting without using any leakage. Perhaps shortly, you could also tell me something about your algorithm. The best I've been able to get is just over .86. Has anybody achieved > .9?
\nThanks!
", "rawMarkdown": "", "votes": null }, { "id": "35026", "postDate": "11/22/2013 03:12:32", "content": "You might want to check out this prior post:
\nhttp://www.kaggle.com/c/accelerometer-biometric-competition/forums/t/5790/has-anyone-managed-to-achieve-a-high-leaderboard-score-using-legitimate
\nIt still might be tough to tell if you have no leakage. I got 0.86673 without using T,X,Y,Z. Yes, it uses leakage.
", "rawMarkdown": "", "votes": null }, { "id": "35027", "postDate": "11/22/2013 03:38:08", "content": "[quote=Forbin;35026]
\nI got 0.86673 without using T,X,Y,Z.
\n[/quote]
\n\n
Well that's an interesting claim. After removing T,X,Y,Z, what data is left? Just the number of samples? I'll understand if you don't want to reveal that, but do share in 24 hours your technique :)
", "rawMarkdown": "", "votes": null }, { "id": "35028", "postDate": "11/22/2013 06:44:02", "content": "I have 0.89510. I transformed input in such a way that does not contains anymore traces of data leakage and my model seems to be insensitive to any possible trace. I know that I left outside consistent information which could lead me to get easily over 0.9.
However I want to see if I can get over 0.9 (that would be really interesting and basically this is my goal for this contest).
", "rawMarkdown": "", "votes": null }, { "id": "35039", "postDate": "11/22/2013 13:55:33", "content": "
Aurelian,
\nI find that very impressive. After the contest is done, is there any way that you could describe your model to me? Every time I thought of a more advanced way to prepare my data it dropped the AUC score down to .80.
\nSo did you use the $T column for anything at all in this model?
", "rawMarkdown": "", "votes": null }, { "id": "35043", "postDate": "11/22/2013 15:48:50", "content": "We'll see once the leak-free competition launches, but I'm guessing with blending 0.90 is achievable. The problem right now is that even if you're not deliberately exploiting leaks, you might still be exploiting leaks. For example, if you have a classifier that determines the class distribution of X, Y and Z at a very granular level, you could be exploiting the fact that samples are discrete (and classifying for device type does help somewhat in this competition.)
\n", "rawMarkdown": "", "votes": null }, { "id": "35061", "postDate": "11/22/2013 20:44:19", "content": "
Yes, that's very interesting. I too think .9 is achievable. I've read that any usage of the time stamps is considered leakage. I don't feel that using the time of day or day in the week should be considered leakage. These seem to be legitimate ways to analyze usage. Is this officially considered leakage?
", "rawMarkdown": "", "votes": null }, { "id": "35066", "postDate": "11/22/2013 23:06:07", "content": "[quote=r0u1i;35027]
\nAfter removing T,X,Y,Z, what data is left? Just the number of samples?
\n[/quote]
\nThe number of samples divided by the number of questions.
", "rawMarkdown": "", "votes": null }, { "id": "35067", "postDate": "11/22/2013 23:24:02", "content": "I think it is very hard to know if you are exploiting leakage. For example, consider the median of the magnitude of x*x+y*y+z*z. For a given device this should be constant (equal to gravity), but it can vary a bit between devices due to their calibration. The same goes for many other parameters such as sensor noise at rest. How is one to avoid all such leakages in their algorithm? I get probably 0.86 or something (it's been a while since I checked) without overtly exploiting leakage, but that doesn't mean I avoid it entirely.
\nThe time of day is correlated with the time zone of the user. This is leakage because the point of the contest is to tell if a phone is stolen. The thief would likely be awake during the same hours as the legitimate user (except for cat burglars I suppose).
", "rawMarkdown": "", "votes": null }, { "id": "35068", "postDate": "11/22/2013 23:24:41", "content": "Excellent!
\nI feel stupid now :)
", "rawMarkdown": "", "votes": null }, { "id": "35069", "postDate": "11/22/2013 23:40:08", "content": "[quote=Dan Stahlke;35067]
\nThe time of day is correlated with the time zone of the user. This is leakage because the point of the contest is to tell if a phone is stolen. The thief would likely be awake during the same hours as the legitimate user (except for cat burglars I suppose).
\n[/quote]
\nI disagree that this is not an important aspect of the data to analyze. The time of day you're using the phone is extremely important. For instance, consistently between classes, I talk on my phone while walking. I also don't use my phone until after noon. At 5 o'clock after work, I consistently talk on my phone while driving.
\nThis is to say, I feel that a specific person will use his/her phone for specific reasons at different times which is unique to the user. That is why I don't feel that my use the the time of day should be considered leakage. But...I'm not the competition leader.
", "rawMarkdown": "", "votes": null }, { "id": "35070", "postDate": "11/22/2013 23:48:43", "content": "Yes, you are right in this case. My point is that it is hard to know what's cheating and what's not. The only way to avoid leakages would be for the organizers to "steal" the participant's phones every now and then and have us detect when that happened. But I suspect that this would be too much work to organize. Comparing user A on phone A versus user B on phone B (in timezone B, etc) is going to be easier, or at least different, than the case that the organizers care about (user B stealing phone A).
", "rawMarkdown": "", "votes": null } ], "comments": [ { "id": 35026, "author_name": "forbin", "author_url": "", "post_date": "11/22/2013 03:12:32", "content": "You might want to check out this prior post:
\nhttp://www.kaggle.com/c/accelerometer-biometric-competition/forums/t/5790/has-anyone-managed-to-achieve-a-high-leaderboard-score-using-legitimate
\nIt still might be tough to tell if you have no leakage. I got 0.86673 without using T,X,Y,Z. Yes, it uses leakage.
", "votes": null, "replies": [] }, { "id": 35027, "author_name": "roulinir", "author_url": "", "post_date": "11/22/2013 03:38:08", "content": "[quote=Forbin;35026]
\nI got 0.86673 without using T,X,Y,Z.
\n[/quote]
\n\n
Well that's an interesting claim. After removing T,X,Y,Z, what data is left? Just the number of samples? I'll understand if you don't want to reveal that, but do share in 24 hours your technique :)
", "votes": null, "replies": [] }, { "id": 35028, "author_name": "rapaio", "author_url": "", "post_date": "11/22/2013 06:44:02", "content": "I have 0.89510. I transformed input in such a way that does not contains anymore traces of data leakage and my model seems to be insensitive to any possible trace. I know that I left outside consistent information which could lead me to get easily over 0.9.
However I want to see if I can get over 0.9 (that would be really interesting and basically this is my goal for this contest).
", "votes": null, "replies": [] }, { "id": 35039, "author_name": "ryanjones", "author_url": "", "post_date": "11/22/2013 13:55:33", "content": "
Aurelian,
\nI find that very impressive. After the contest is done, is there any way that you could describe your model to me? Every time I thought of a more advanced way to prepare my data it dropped the AUC score down to .80.
\nSo did you use the $T column for anything at all in this model?
", "votes": null, "replies": [] }, { "id": 35043, "author_name": "solorzano", "author_url": "", "post_date": "11/22/2013 15:48:50", "content": "We'll see once the leak-free competition launches, but I'm guessing with blending 0.90 is achievable. The problem right now is that even if you're not deliberately exploiting leaks, you might still be exploiting leaks. For example, if you have a classifier that determines the class distribution of X, Y and Z at a very granular level, you could be exploiting the fact that samples are discrete (and classifying for device type does help somewhat in this competition.)
\n", "votes": null, "replies": [] }, { "id": 35061, "author_name": "ryanjones", "author_url": "", "post_date": "11/22/2013 20:44:19", "content": "
Yes, that's very interesting. I too think .9 is achievable. I've read that any usage of the time stamps is considered leakage. I don't feel that using the time of day or day in the week should be considered leakage. These seem to be legitimate ways to analyze usage. Is this officially considered leakage?
", "votes": null, "replies": [] }, { "id": 35066, "author_name": "nedelko", "author_url": "", "post_date": "11/22/2013 23:06:07", "content": "[quote=r0u1i;35027]
\nAfter removing T,X,Y,Z, what data is left? Just the number of samples?
\n[/quote]
\nThe number of samples divided by the number of questions.
", "votes": null, "replies": [] }, { "id": 35067, "author_name": "dstahlke", "author_url": "", "post_date": "11/22/2013 23:24:02", "content": "I think it is very hard to know if you are exploiting leakage. For example, consider the median of the magnitude of x*x+y*y+z*z. For a given device this should be constant (equal to gravity), but it can vary a bit between devices due to their calibration. The same goes for many other parameters such as sensor noise at rest. How is one to avoid all such leakages in their algorithm? I get probably 0.86 or something (it's been a while since I checked) without overtly exploiting leakage, but that doesn't mean I avoid it entirely.
\nThe time of day is correlated with the time zone of the user. This is leakage because the point of the contest is to tell if a phone is stolen. The thief would likely be awake during the same hours as the legitimate user (except for cat burglars I suppose).
", "votes": null, "replies": [] }, { "id": 35068, "author_name": "roulinir", "author_url": "", "post_date": "11/22/2013 23:24:41", "content": "Excellent!
\nI feel stupid now :)
", "votes": null, "replies": [] }, { "id": 35069, "author_name": "ryanjones", "author_url": "", "post_date": "11/22/2013 23:40:08", "content": "[quote=Dan Stahlke;35067]
\nThe time of day is correlated with the time zone of the user. This is leakage because the point of the contest is to tell if a phone is stolen. The thief would likely be awake during the same hours as the legitimate user (except for cat burglars I suppose).
\n[/quote]
\nI disagree that this is not an important aspect of the data to analyze. The time of day you're using the phone is extremely important. For instance, consistently between classes, I talk on my phone while walking. I also don't use my phone until after noon. At 5 o'clock after work, I consistently talk on my phone while driving.
\nThis is to say, I feel that a specific person will use his/her phone for specific reasons at different times which is unique to the user. That is why I don't feel that my use the the time of day should be considered leakage. But...I'm not the competition leader.
", "votes": null, "replies": [] }, { "id": 35070, "author_name": "dstahlke", "author_url": "", "post_date": "11/22/2013 23:48:43", "content": "Yes, you are right in this case. My point is that it is hard to know what's cheating and what's not. The only way to avoid leakages would be for the organizers to "steal" the participant's phones every now and then and have us detect when that happened. But I suspect that this would be too much work to organize. Comparing user A on phone A versus user B on phone B (in timezone B, etc) is going to be easier, or at least different, than the case that the organizers care about (user B stealing phone A).
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "35020": "", "35026": "", "35027": "", "35028": "", "35039": "", "35043": "", "35061": "", "35066": "", "35067": "", "35068": "", "35069": "", "35070": "" }, "source": "meta" }