I created a python notebook to take a look at the top-3 most checked-in places and the relationship between the accuracy measure and the std deviation for x,y.
\n\n\n\nI've noticed that the histogram of the accuracy values show roughly the same pattern across each of these locations and that the std deviation of y is lowest for accuracy (50-60) and is highest for accuracy (0-10).
\n\nThis suggests to me that accuracy is not a higher is better or lower is better. Accuracy is some measure that happens to result in a lower standard deviation or higher standard deviation.
\n\nSo far, I've only checked the top 3 most frequent check-ins. I'm not sure what it would be measuring. Probably one of the measures mentioned here.
\n\nWhat have others found out with regard to this topic?
", "messages": [ { "id": "124847", "postDate": "06/22/2016 18:41:00", "content": "I created a python notebook to take a look at the top-3 most checked-in places and the relationship between the accuracy measure and the std deviation for x,y.
\n\n\n\nI've noticed that the histogram of the accuracy values show roughly the same pattern across each of these locations and that the std deviation of y is lowest for accuracy (50-60) and is highest for accuracy (0-10).
\n\nThis suggests to me that accuracy is not a higher is better or lower is better. Accuracy is some measure that happens to result in a lower standard deviation or higher standard deviation.
\n\nSo far, I've only checked the top 3 most frequent check-ins. I'm not sure what it would be measuring. Probably one of the measures mentioned here.
\n\nWhat have others found out with regard to this topic?
", "rawMarkdown": "I [created][2] a python notebook to take a look at the top-3 most checked-in places and the relationship between the accuracy measure and the std deviation for x,y.\r\n\r\n[ipython notebook link][2]\r\n\r\nI've noticed that the histogram of the accuracy values show roughly the same pattern across each of these locations and that the std deviation of y is lowest for accuracy (50-60) and is highest for accuracy (0-10).\r\n\r\nThis suggests to me that accuracy is not a higher is better or lower is better. Accuracy is some measure that happens to result in a lower standard deviation or higher standard deviation. \r\n\r\nSo far, I've only checked the top 3 most frequent check-ins. I'm not sure what it would be measuring. Probably one of the measures mentioned [here][3].\r\n\r\nWhat have others found out with regard to this topic? \r\n\r\n\r\n\r\n [1]: https://www.kaggle.com/larryfreeman/facebook-v-predicting-check-ins/accuracy-and-std-dev-of-x-y\r\n [2]: https://www.kaggle.com/larryfreeman/facebook-v-predicting-check-ins/accuracy-and-std-dev-of-x-y/notebook\r\n [3]: http://www.igage.com/mp/GPSAccuracy.htm", "votes": null }, { "id": "124851", "postDate": "06/22/2016 18:59:20", "content": "My current thinking is that accuracy is an estimation of how accurate is the device used to perform the checking - and NOT a measurement of how accurate the x,y coordinates are.
\n\nIf you look at some of the publics scripts written lately re:accuracy, you'll see that there are about 4-5 different accuracy clusters (with peaks at 5, 10, 30, 65 and 165 if I recall correctly). A possible interpretation of it would be to have 4-5 different kind of devices making the checkings and - as you just explained- the higher the accuracy, the lower the possible deviation of the x,y coordinates (although nothing prevents a device with a low accuracy score to be 100% accuracy).
\n\nThis would also explain why using log10(accuracy) as a feature is giving good results in other public scripts: Somehow, it is identifying which kind of device has been used in each case. And also explains the shift from the 5-10 accuracy cluster towards the 30 and 65 ones as time goes by (people get more accurate devices over times).
\n\nStill, it would be nice if we could get some answers from the host about this, after the end of the competition :)
", "rawMarkdown": "My current thinking is that accuracy is an estimation of how accurate is the device used to perform the checking - and NOT a measurement of how accurate the x,y coordinates are.\r\n\r\nIf you look at some of the publics scripts written lately re:accuracy, you'll see that there are about 4-5 different accuracy clusters (with peaks at 5, 10, 30, 65 and 165 if I recall correctly). A possible interpretation of it would be to have 4-5 different kind of devices making the checkings and - as you just explained- the higher the accuracy, the lower the possible deviation of the x,y coordinates (although nothing prevents a device with a low accuracy score to be 100% accuracy).\r\n\r\nThis would also explain why using log10(accuracy) as a feature is giving good results in other public scripts: Somehow, it is identifying which kind of device has been used in each case. And also explains the shift from the 5-10 accuracy cluster towards the 30 and 65 ones as time goes by (people get more accurate devices over times).\r\n\r\nStill, it would be nice if we could get some answers from the host about this, after the end of the competition :)", "votes": null }, { "id": "124878", "postDate": "06/23/2016 02:50:26", "content": "@CarrDelling \nCould you explain more about accuracy is an estimation of how accurate is the device used to perform the checking ?
", "rawMarkdown": "CarrDelling \r\nCould you explain more about accuracy is an estimation of how accurate is the device used to perform the checking ?", "votes": null }, { "id": "124885", "postDate": "06/23/2016 04:46:11", "content": "@CarrDelling Thanks for your information. It seems that the log-transformation of "accuracy" is important if you want to gain better prediction, but I've not figure out why this transformation is need, could you give me an intuitive explanation?
", "rawMarkdown": "CarrDelling Thanks for your information. It seems that the log-transformation of \"accuracy\" is important if you want to gain better prediction, but I've not figure out why this transformation is need, could you give me an intuitive explanation?", "votes": null }, { "id": "124945", "postDate": "06/23/2016 19:11:03", "content": "@ArthurLu: Have a look at the last part of Nigel's script:
\n\nhttps://www.kaggle.com/nigelhenry/facebook-v-predicting-check-ins/accuracy-explained/comments
\n\nThere you could see a very good point about having different 'modes' (the five peaks) that changes from place to place.
\n\nNow, join that with the last part of BeyondBeneath's script:
\n\nhttps://www.kaggle.com/beyondbeneath/facebook-v-predicting-check-ins/testing
\n\nAnd you will see how the distribution of the peaks is changing over time. And what could make a shift from a peak to another as time goes on (in a year and a half)? There might be different reasons, but to me a good one could people just ditching all mobiles and getting new ones.
\n\nAnd finally join that with the conclusion of presented by Larry in this thread: "Accuracy is some measure that happens to result in a lower standard deviation or higher standard deviation".
\n\nIt is just a hunch, and I will probably be wrong, but to me it makes sense as a hypothesis.
\n\n@Jiahuan: Assuming that I am right, we might be having 4-5 different types of devices in play here. With the raw accuracy, the problem is that the ranges of values are quite different (device 0 would go from 0 to 7.5, device 1 from 7.5 to 15... whereas device N would go from 125 to 1000). When you apply a logarithm to that range, you are somehow shortening the ranges for the higher devices, and increasing the ranges for the lower ones. Hence, it makes the problem of discerning between peaks (devices?) easier for the classifier.
", "rawMarkdown": "ArthurLu: Have a look at the last part of Nigel's script:\r\n\r\nhttps://www.kaggle.com/nigelhenry/facebook-v-predicting-check-ins/accuracy-explained/comments\r\n\r\nThere you could see a very good point about having different 'modes' (the five peaks) that changes from place to place.\r\n\r\nNow, join that with the last part of BeyondBeneath's script:\r\n\r\nhttps://www.kaggle.com/beyondbeneath/facebook-v-predicting-check-ins/testing\r\n\r\nAnd you will see how the distribution of the peaks is changing over time. And what could make a shift from a peak to another as time goes on (in a year and a half)? There might be different reasons, but to me a good one could people just ditching all mobiles and getting new ones.\r\n\r\nAnd finally join that with the conclusion of presented by Larry in this thread: \"Accuracy is some measure that happens to result in a lower standard deviation or higher standard deviation\". \r\n\r\nIt is just a hunch, and I will probably be wrong, but to me it makes sense as a hypothesis.\r\n\r\n@Jiahuan: Assuming that I am right, we might be having 4-5 different types of devices in play here. With the raw accuracy, the problem is that the ranges of values are quite different (device 0 would go from 0 to 7.5, device 1 from 7.5 to 15... whereas device N would go from 125 to 1000). When you apply a logarithm to that range, you are somehow shortening the ranges for the higher devices, and increasing the ranges for the lower ones. Hence, it makes the problem of discerning between peaks (devices?) easier for the classifier.", "votes": null }, { "id": "124947", "postDate": "06/23/2016 19:17:59", "content": "@CarrDelling Thanks! I think that makes sense.
", "rawMarkdown": "CarrDelling Thanks! I think that makes sense.", "votes": null }, { "id": "124948", "postDate": "06/23/2016 19:43:32", "content": "I'm wondering for tree classifier. Data transformation is unnecessary?
", "rawMarkdown": "I'm wondering for tree classifier. Data transformation is unnecessary?", "votes": null }, { "id": "124950", "postDate": "06/23/2016 19:54:37", "content": "@FengLi What kinds of transformation have you tried?
", "rawMarkdown": "FengLi What kinds of transformation have you tried?", "votes": null }, { "id": "124956", "postDate": "06/23/2016 20:33:38", "content": "@Jiahuan He. I didn't try any transformation. Do you think tree classifier is sensitive to data transformation?
", "rawMarkdown": "Jiahuan He. I didn't try any transformation. Do you think tree classifier is sensitive to data transformation?", "votes": null }, { "id": "124958", "postDate": "06/23/2016 20:40:50", "content": "[quote=FengLi;124956]
\n\n@Jiahuan He. I didn't try any transformation. Do you think tree classifier is sensitive to data transformation?
\n\n[/quote]
\n\nI think it is worth trying ;)
", "rawMarkdown": "[quote=FengLi;124956]\r\n\r\n@Jiahuan He. I didn't try any transformation. Do you think tree classifier is sensitive to data transformation? \r\n\r\n[/quote]\r\n\r\nI think it is worth trying ;)", "votes": null }, { "id": "125019", "postDate": "06/24/2016 18:01:41", "content": "@CarrDelling Thanks for details :)
", "rawMarkdown": "CarrDelling Thanks for details :)", "votes": null } ], "comments": [ { "id": 124851, "author_name": "carrdelling", "author_url": "", "post_date": "06/22/2016 18:59:20", "content": "My current thinking is that accuracy is an estimation of how accurate is the device used to perform the checking - and NOT a measurement of how accurate the x,y coordinates are.
\n\nIf you look at some of the publics scripts written lately re:accuracy, you'll see that there are about 4-5 different accuracy clusters (with peaks at 5, 10, 30, 65 and 165 if I recall correctly). A possible interpretation of it would be to have 4-5 different kind of devices making the checkings and - as you just explained- the higher the accuracy, the lower the possible deviation of the x,y coordinates (although nothing prevents a device with a low accuracy score to be 100% accuracy).
\n\nThis would also explain why using log10(accuracy) as a feature is giving good results in other public scripts: Somehow, it is identifying which kind of device has been used in each case. And also explains the shift from the 5-10 accuracy cluster towards the 30 and 65 ones as time goes by (people get more accurate devices over times).
\n\nStill, it would be nice if we could get some answers from the host about this, after the end of the competition :)
", "votes": null, "replies": [] }, { "id": 124878, "author_name": "arthurlu", "author_url": "", "post_date": "06/23/2016 02:50:26", "content": "@CarrDelling \nCould you explain more about accuracy is an estimation of how accurate is the device used to perform the checking ?
", "votes": null, "replies": [] }, { "id": 124885, "author_name": "gospursgo", "author_url": "", "post_date": "06/23/2016 04:46:11", "content": "@CarrDelling Thanks for your information. It seems that the log-transformation of "accuracy" is important if you want to gain better prediction, but I've not figure out why this transformation is need, could you give me an intuitive explanation?
", "votes": null, "replies": [] }, { "id": 124945, "author_name": "carrdelling", "author_url": "", "post_date": "06/23/2016 19:11:03", "content": "@ArthurLu: Have a look at the last part of Nigel's script:
\n\nhttps://www.kaggle.com/nigelhenry/facebook-v-predicting-check-ins/accuracy-explained/comments
\n\nThere you could see a very good point about having different 'modes' (the five peaks) that changes from place to place.
\n\nNow, join that with the last part of BeyondBeneath's script:
\n\nhttps://www.kaggle.com/beyondbeneath/facebook-v-predicting-check-ins/testing
\n\nAnd you will see how the distribution of the peaks is changing over time. And what could make a shift from a peak to another as time goes on (in a year and a half)? There might be different reasons, but to me a good one could people just ditching all mobiles and getting new ones.
\n\nAnd finally join that with the conclusion of presented by Larry in this thread: "Accuracy is some measure that happens to result in a lower standard deviation or higher standard deviation".
\n\nIt is just a hunch, and I will probably be wrong, but to me it makes sense as a hypothesis.
\n\n@Jiahuan: Assuming that I am right, we might be having 4-5 different types of devices in play here. With the raw accuracy, the problem is that the ranges of values are quite different (device 0 would go from 0 to 7.5, device 1 from 7.5 to 15... whereas device N would go from 125 to 1000). When you apply a logarithm to that range, you are somehow shortening the ranges for the higher devices, and increasing the ranges for the lower ones. Hence, it makes the problem of discerning between peaks (devices?) easier for the classifier.
", "votes": null, "replies": [] }, { "id": 124947, "author_name": "gospursgo", "author_url": "", "post_date": "06/23/2016 19:17:59", "content": "@CarrDelling Thanks! I think that makes sense.
", "votes": null, "replies": [] }, { "id": 124948, "author_name": "beedata", "author_url": "", "post_date": "06/23/2016 19:43:32", "content": "I'm wondering for tree classifier. Data transformation is unnecessary?
", "votes": null, "replies": [] }, { "id": 124950, "author_name": "gospursgo", "author_url": "", "post_date": "06/23/2016 19:54:37", "content": "@FengLi What kinds of transformation have you tried?
", "votes": null, "replies": [] }, { "id": 124956, "author_name": "beedata", "author_url": "", "post_date": "06/23/2016 20:33:38", "content": "@Jiahuan He. I didn't try any transformation. Do you think tree classifier is sensitive to data transformation?
", "votes": null, "replies": [] }, { "id": 124958, "author_name": "gospursgo", "author_url": "", "post_date": "06/23/2016 20:40:50", "content": "[quote=FengLi;124956]
\n\n@Jiahuan He. I didn't try any transformation. Do you think tree classifier is sensitive to data transformation?
\n\n[/quote]
\n\nI think it is worth trying ;)
", "votes": null, "replies": [] }, { "id": 125019, "author_name": "arthurlu", "author_url": "", "post_date": "06/24/2016 18:01:41", "content": "@CarrDelling Thanks for details :)
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "124847": "I [created][2] a python notebook to take a look at the top-3 most checked-in places and the relationship between the accuracy measure and the std deviation for x,y.\r\n\r\n[ipython notebook link][2]\r\n\r\nI've noticed that the histogram of the accuracy values show roughly the same pattern across each of these locations and that the std deviation of y is lowest for accuracy (50-60) and is highest for accuracy (0-10).\r\n\r\nThis suggests to me that accuracy is not a higher is better or lower is better. Accuracy is some measure that happens to result in a lower standard deviation or higher standard deviation. \r\n\r\nSo far, I've only checked the top 3 most frequent check-ins. I'm not sure what it would be measuring. Probably one of the measures mentioned [here][3].\r\n\r\nWhat have others found out with regard to this topic? \r\n\r\n\r\n\r\n [1]: https://www.kaggle.com/larryfreeman/facebook-v-predicting-check-ins/accuracy-and-std-dev-of-x-y\r\n [2]: https://www.kaggle.com/larryfreeman/facebook-v-predicting-check-ins/accuracy-and-std-dev-of-x-y/notebook\r\n [3]: http://www.igage.com/mp/GPSAccuracy.htm", "124851": "My current thinking is that accuracy is an estimation of how accurate is the device used to perform the checking - and NOT a measurement of how accurate the x,y coordinates are.\r\n\r\nIf you look at some of the publics scripts written lately re:accuracy, you'll see that there are about 4-5 different accuracy clusters (with peaks at 5, 10, 30, 65 and 165 if I recall correctly). A possible interpretation of it would be to have 4-5 different kind of devices making the checkings and - as you just explained- the higher the accuracy, the lower the possible deviation of the x,y coordinates (although nothing prevents a device with a low accuracy score to be 100% accuracy).\r\n\r\nThis would also explain why using log10(accuracy) as a feature is giving good results in other public scripts: Somehow, it is identifying which kind of device has been used in each case. And also explains the shift from the 5-10 accuracy cluster towards the 30 and 65 ones as time goes by (people get more accurate devices over times).\r\n\r\nStill, it would be nice if we could get some answers from the host about this, after the end of the competition :)", "124878": "CarrDelling \r\nCould you explain more about accuracy is an estimation of how accurate is the device used to perform the checking ?", "124885": "CarrDelling Thanks for your information. It seems that the log-transformation of \"accuracy\" is important if you want to gain better prediction, but I've not figure out why this transformation is need, could you give me an intuitive explanation?", "124945": "ArthurLu: Have a look at the last part of Nigel's script:\r\n\r\nhttps://www.kaggle.com/nigelhenry/facebook-v-predicting-check-ins/accuracy-explained/comments\r\n\r\nThere you could see a very good point about having different 'modes' (the five peaks) that changes from place to place.\r\n\r\nNow, join that with the last part of BeyondBeneath's script:\r\n\r\nhttps://www.kaggle.com/beyondbeneath/facebook-v-predicting-check-ins/testing\r\n\r\nAnd you will see how the distribution of the peaks is changing over time. And what could make a shift from a peak to another as time goes on (in a year and a half)? There might be different reasons, but to me a good one could people just ditching all mobiles and getting new ones.\r\n\r\nAnd finally join that with the conclusion of presented by Larry in this thread: \"Accuracy is some measure that happens to result in a lower standard deviation or higher standard deviation\". \r\n\r\nIt is just a hunch, and I will probably be wrong, but to me it makes sense as a hypothesis.\r\n\r\n@Jiahuan: Assuming that I am right, we might be having 4-5 different types of devices in play here. With the raw accuracy, the problem is that the ranges of values are quite different (device 0 would go from 0 to 7.5, device 1 from 7.5 to 15... whereas device N would go from 125 to 1000). When you apply a logarithm to that range, you are somehow shortening the ranges for the higher devices, and increasing the ranges for the lower ones. Hence, it makes the problem of discerning between peaks (devices?) easier for the classifier.", "124947": "CarrDelling Thanks! I think that makes sense.", "124948": "I'm wondering for tree classifier. Data transformation is unnecessary?", "124950": "FengLi What kinds of transformation have you tried?", "124956": "Jiahuan He. I didn't try any transformation. Do you think tree classifier is sensitive to data transformation?", "124958": "[quote=FengLi;124956]\r\n\r\n@Jiahuan He. I didn't try any transformation. Do you think tree classifier is sensitive to data transformation? \r\n\r\n[/quote]\r\n\r\nI think it is worth trying ;)", "125019": "CarrDelling Thanks for details :)" }, "source": "meta" }