I think I am getting a prediction accuracy more than 70%, but stuck with the problem of computing KDEs on the go for 8 million test checkin records. One way is to precompute the KDEs for these 100K+ places. But, my machine (8GB, i7, with SSD) is unable to do it... anyone facing similar issue OR have ideas on how to scale it? \nAzure is not helping much... they don't support ks package. (I am using R)
", "messages": [ { "id": "124272", "postDate": "06/16/2016 17:24:13", "content": "I think I am getting a prediction accuracy more than 70%, but stuck with the problem of computing KDEs on the go for 8 million test checkin records. One way is to precompute the KDEs for these 100K+ places. But, my machine (8GB, i7, with SSD) is unable to do it... anyone facing similar issue OR have ideas on how to scale it? \nAzure is not helping much... they don't support ks package. (I am using R)
", "rawMarkdown": "I think I am getting a prediction accuracy more than 70%, but stuck with the problem of computing KDEs on the go for 8 million test checkin records. One way is to precompute the KDEs for these 100K+ places. But, my machine (8GB, i7, with SSD) is unable to do it... anyone facing similar issue OR have ideas on how to scale it? \r\nAzure is not helping much... they don't support ks package. (I am using R)", "votes": null }, { "id": "124278", "postDate": "06/16/2016 18:24:19", "content": "Wow 70% seems like a great improvement in comparison to current leaders in LB. May I ask what models/tools do you use to get from KDE to predictions?
", "rawMarkdown": "Wow 70% seems like a great improvement in comparison to current leaders in LB. May I ask what models/tools do you use to get from KDE to predictions?", "votes": null }, { "id": "124279", "postDate": "06/16/2016 18:42:37", "content": "Well. The 70% is on my limited validation set I derived out of the train.csv.
\n\nI followed the below approach:
\n\nCompute (rough) place_id coordinates using any classical mechanical approach (I used the center of mass concept for example - in this case I treated accuracy as the mass or 'weightage' for the point (x,y))
Next, given a test check-in event, find the subset of places the check-in can possibly fall in. I used the centroids from step 1 with some std. dev. for this, assuming the spread is normal. This turns out to be a good assumption. I was getting about 90+ accuracy on this. I was getting about 100 places the check-in possibly falls under.
From that list of 100 places, compute KDEs on (x,y,accuracy -> call this accuracy_kde) and (week, week day and hour -> call this time_kde) separately. For some reason combining them into single KDE is yielding a lower accuracy.
Derive the probabilities (you get two probabilities - one for accuracy and the other for time KDE). You can be creative here on how to order the probabilities. The places associated with top three probabilities give you the answer...
Point 4 is the most interesting. I though you use some optimization to determine the best way to compute probabilities. I asked because i tested a similar approach but it yielded very low scores for me. Your value of 70% got me really surprised. Thank you for your answer.
", "rawMarkdown": "Point 4 is the most interesting. I though you use some optimization to determine the best way to compute probabilities. I asked because i tested a similar approach but it yielded very low scores for me. Your value of 70% got me really surprised. Thank you for your answer.", "votes": null }, { "id": "124281", "postDate": "06/16/2016 18:57:08", "content": "If memory is your only concern, you can just chunk the dataset into grids, look at some of the scripts for how to do this. The fact that you are saying accuracy as opposed to MAP@3 though makes me think you are overstating your results, as well as the 70% number seeming to indicate overfitting. On the other hand this approach seems interesting.
", "rawMarkdown": "If memory is your only concern, you can just chunk the dataset into grids, look at some of the scripts for how to do this. The fact that you are saying accuracy as opposed to MAP@3 though makes me think you are overstating your results, as well as the 70% number seeming to indicate overfitting. On the other hand this approach seems interesting.", "votes": null }, { "id": "124282", "postDate": "06/16/2016 19:08:37", "content": "Thanks both for your comments.
\n\nIt is very well possible that I may be misreading the results. \nFurther, I am using the term accuracy in a loose sense, here is why: \nI am resorting to bootstrapping (drawing a sample with replacement and repeating it say a 1000 times) to do the validation. This help me do some visual inspection of the results to do some evaluation. If I predict the correct place_id in the top 3, I am marking the prediction as correct else incorrect - kind of very crude way to do...
\n\nOtherwise, if I don't have an easy 'cloud' way to do this at one shot, I am going to split/chunk and do it.
", "rawMarkdown": "Thanks both for your comments. \r\n\r\nIt is very well possible that I may be misreading the results. \r\nFurther, I am using the term accuracy in a loose sense, here is why: \r\nI am resorting to bootstrapping (drawing a sample with replacement and repeating it say a 1000 times) to do the validation. This help me do some visual inspection of the results to do some evaluation. If I predict the correct place_id in the top 3, I am marking the prediction as correct else incorrect - kind of very crude way to do...\r\n\r\nOtherwise, if I don't have an easy 'cloud' way to do this at one shot, I am going to split/chunk and do it.", "votes": null }, { "id": "124283", "postDate": "06/16/2016 19:16:01", "content": "[quote=Kumar Mankala;124282]
\n\nI am resorting to bootstrapping (drawing a sample with replacement and repeating it say a 1000 times) to do the validation.
\n\n[/quote]
\n\nSeems like you were not separating the training/validation sets by time. If this was the case, it's very likely you overestimated the accuracy.
\n\nIn addition, you can actually assign 1, 1/2, 1/3, or 0, instead of just 1 or 0, depending on which position of top 3 is the correct prediction at. In this way you get estimation of MAP@3 score instead a crude accuracy score.
", "rawMarkdown": "[quote=Kumar Mankala;124282]\r\n\r\n\r\nI am resorting to bootstrapping (drawing a sample with replacement and repeating it say a 1000 times) to do the validation. \r\n\r\n[/quote]\r\n\r\nSeems like you were not separating the training/validation sets by time. If this was the case, it's very likely you overestimated the accuracy.\r\n\r\nIn addition, you can actually assign 1, 1/2, 1/3, or 0, instead of just 1 or 0, depending on which position of top 3 is the correct prediction at. In this way you get estimation of MAP@3 score instead a crude accuracy score.", "votes": null }, { "id": "124285", "postDate": "06/16/2016 19:28:47", "content": "Thanks Leon. You triggered an idea to cross-validate my results. \nI will try to subset an area (x,y) between say a square between (4,4) & (6,6), and split it 90% Train and 10% as Val by time.\nLet me try to approach on the above set and hopefully I should be able to repost the results today.
", "rawMarkdown": "Thanks Leon. You triggered an idea to cross-validate my results. \r\nI will try to subset an area (x,y) between say a square between (4,4) & (6,6), and split it 90% Train and 10% as Val by time.\r\nLet me try to approach on the above set and hopefully I should be able to repost the results today.", "votes": null }, { "id": "124286", "postDate": "06/16/2016 19:32:04", "content": "[quote=Kumar Mankala;124282]
\n\nThanks both for your comments.
\n\nIt is very well possible that I may be misreading the results. \nFurther, I am using the term accuracy in a loose sense, here is why: \nI am resorting to bootstrapping (drawing a sample with replacement and repeating it say a 1000 times) to do the validation. This help me do some visual inspection of the results to do some evaluation. If I predict the correct place_id in the top 3, I am marking the prediction as correct else incorrect - kind of very crude way to do...
\n\nOtherwise, if I don't have an easy 'cloud' way to do this at one shot, I am going to split/chunk and do it.
\n\n[/quote]
\n\nI don't think a top-3 accuracy of 70% would mean much improvement if you reevaluate with MAP@3.
", "rawMarkdown": "[quote=Kumar Mankala;124282]\r\n\r\nThanks both for your comments. \r\n\r\nIt is very well possible that I may be misreading the results. \r\nFurther, I am using the term accuracy in a loose sense, here is why: \r\nI am resorting to bootstrapping (drawing a sample with replacement and repeating it say a 1000 times) to do the validation. This help me do some visual inspection of the results to do some evaluation. If I predict the correct place_id in the top 3, I am marking the prediction as correct else incorrect - kind of very crude way to do...\r\n\r\nOtherwise, if I don't have an easy 'cloud' way to do this at one shot, I am going to split/chunk and do it. \r\n\r\n[/quote]\r\n\r\nI don't think a top-3 accuracy of 70% would mean much improvement if you reevaluate with MAP@3.", "votes": null }, { "id": "124383", "postDate": "06/17/2016 21:25:41", "content": "Actually I have try to fit gaussian distribution and GMM for the checkin points for each place_id. It does not work well, because the checkin points do not seem to follow gaussian distribution. You may look at my post exploring the checkin points distribution below: \nhttps://www.kaggle.com/c/facebook-v-predicting-check-ins/forums/t/21328/2-d-check-in-scatter-plot-of-several-place-id/121933#post121933
\n\nI have tried another way to fitting place. The basic idea is for each place, finding a rectangle that covers 90% of the checkin points. The threshold needs tuning to remove outliers. When predicting a test point, we first find which rectangles the test points falls in, then use some criteria (rectangle area, density, etc.) to get top 3 matches. But the LB score of this idea is only around 0.4. Maybe I did not find the correct criteria and parameters.
\n\nI hope this could help you.
\n\n[quote=Kumar Mankala;124279]
\n\nWell. The 70% is on my limited validation set I derived out of the train.csv.
\n\nI followed the below approach:
\n\nCompute (rough) place_id coordinates using any classical mechanical approach (I used the center of mass concept for example - in this case I treated accuracy as the mass or 'weightage' for the point (x,y))
Next, given a test check-in event, find the subset of places the check-in can possibly fall in. I used the centroids from step 1 with some std. dev. for this, assuming the spread is normal. This turns out to be a good assumption. I was getting about 90+ accuracy on this. I was getting about 100 places the check-in possibly falls under.
From that list of 100 places, compute KDEs on (x,y,accuracy -> call this accuracy_kde) and (week, week day and hour -> call this time_kde) separately. For some reason combining them into single KDE is yielding a lower accuracy.
Derive the probabilities (you get two probabilities - one for accuracy and the other for time KDE). You can be creative here on how to order the probabilities. The places associated with top three probabilities give you the answer...
[/quote]
", "rawMarkdown": "Actually I have try to fit gaussian distribution and GMM for the checkin points for each place_id. It does not work well, because the checkin points do not seem to follow gaussian distribution. You may look at my post exploring the checkin points distribution below: \r\nhttps://www.kaggle.com/c/facebook-v-predicting-check-ins/forums/t/21328/2-d-check-in-scatter-plot-of-several-place-id/121933#post121933 \r\n\r\nI have tried another way to fitting place. The basic idea is for each place, finding a rectangle that covers 90% of the checkin points. The threshold needs tuning to remove outliers. When predicting a test point, we first find which rectangles the test points falls in, then use some criteria (rectangle area, density, etc.) to get top 3 matches. But the LB score of this idea is only around 0.4. Maybe I did not find the correct criteria and parameters.\r\n\r\nI hope this could help you.\r\n\r\n[quote=Kumar Mankala;124279]\r\n\r\nWell. The 70% is on my limited validation set I derived out of the train.csv.\r\n\r\nI followed the below approach:\r\n\r\n1. Compute (rough) place_id coordinates using any classical mechanical approach (I used the center of mass concept for example - in this case I treated accuracy as the mass or 'weightage' for the point (x,y))\r\n\r\n2. Next, given a test check-in event, find the subset of places the check-in can possibly fall in. I used the centroids from step 1 with some std. dev. for this, assuming the spread is normal. This turns out to be a good assumption. I was getting about 90+ accuracy on this. I was getting about 100 places the check-in possibly falls under.\r\n\r\n4. From that list of 100 places, compute KDEs on (x,y,accuracy -> call this accuracy_kde) and (week, week day and hour -> call this time_kde) separately. For some reason combining them into single KDE is yielding a lower accuracy.\r\n\r\n5. Derive the probabilities (you get two probabilities - one for accuracy and the other for time KDE). You can be creative here on how to order the probabilities. The places associated with top three probabilities give you the answer...\r\n\r\n[/quote]", "votes": null }, { "id": "124663", "postDate": "06/21/2016 01:00:38", "content": "Also posted a small script on computing place centroids in the scripts.\nhttps://www.kaggle.com/kumarmankala/facebook-v-predicting-check-ins/computing-place-centroids/
", "rawMarkdown": "Also posted a small script on computing place centroids in the scripts.\r\nhttps://www.kaggle.com/kumarmankala/facebook-v-predicting-check-ins/computing-place-centroids/", "votes": null }, { "id": "124757", "postDate": "06/22/2016 02:15:04", "content": "Could someone tell me what KDE is? Thank you.
", "rawMarkdown": "Could someone tell me what KDE is? Thank you.", "votes": null }, { "id": "124759", "postDate": "06/22/2016 02:36:58", "content": "Kernel Density Estimate - a non-parametric way of determining probability distribution function
", "rawMarkdown": "Kernel Density Estimate - a non-parametric way of determining probability distribution function", "votes": null }, { "id": "124972", "postDate": "06/24/2016 03:37:22", "content": "hey Kumar,\nI've been working on a very similar approach for some time now. I haven't gotten it to run completely yet, but I think I will get there. You can get the 100,000+ KDE's done using Amazon EC2s with some multiprocessing code and being creative about work sharing across about few m4.10x machines so that the job completes in a reasonable time. you would need to figure out how to split the work between the machines.
\n\nBest!
", "rawMarkdown": "hey Kumar,\r\nI've been working on a very similar approach for some time now. I haven't gotten it to run completely yet, but I think I will get there. You can get the 100,000+ KDE's done using Amazon EC2s with some multiprocessing code and being creative about work sharing across about few m4.10x machines so that the job completes in a reasonable time. you would need to figure out how to split the work between the machines. \r\n\r\nBest!", "votes": null }, { "id": "125205", "postDate": "06/27/2016 16:35:49", "content": "[quote=Predictions, Forecasts & Lies;124972]
\n\nThanks Bimal. Yeah. I am trying it... In the skies the AWS's cloud looks best for now...
\n\n[/quote]
", "rawMarkdown": "[quote=Predictions, Forecasts & Lies;124972]\r\n\r\nThanks Bimal. Yeah. I am trying it... In the skies the AWS's cloud looks best for now...\r\n\r\n[/quote]", "votes": null } ], "comments": [ { "id": 124278, "author_name": "mwilam", "author_url": "", "post_date": "06/16/2016 18:24:19", "content": "Wow 70% seems like a great improvement in comparison to current leaders in LB. May I ask what models/tools do you use to get from KDE to predictions?
", "votes": null, "replies": [] }, { "id": 124279, "author_name": "kumarmankala", "author_url": "", "post_date": "06/16/2016 18:42:37", "content": "Well. The 70% is on my limited validation set I derived out of the train.csv.
\n\nI followed the below approach:
\n\nCompute (rough) place_id coordinates using any classical mechanical approach (I used the center of mass concept for example - in this case I treated accuracy as the mass or 'weightage' for the point (x,y))
Next, given a test check-in event, find the subset of places the check-in can possibly fall in. I used the centroids from step 1 with some std. dev. for this, assuming the spread is normal. This turns out to be a good assumption. I was getting about 90+ accuracy on this. I was getting about 100 places the check-in possibly falls under.
From that list of 100 places, compute KDEs on (x,y,accuracy -> call this accuracy_kde) and (week, week day and hour -> call this time_kde) separately. For some reason combining them into single KDE is yielding a lower accuracy.
Derive the probabilities (you get two probabilities - one for accuracy and the other for time KDE). You can be creative here on how to order the probabilities. The places associated with top three probabilities give you the answer...
Point 4 is the most interesting. I though you use some optimization to determine the best way to compute probabilities. I asked because i tested a similar approach but it yielded very low scores for me. Your value of 70% got me really surprised. Thank you for your answer.
", "votes": null, "replies": [] }, { "id": 124281, "author_name": "jturkewitz", "author_url": "", "post_date": "06/16/2016 18:57:08", "content": "If memory is your only concern, you can just chunk the dataset into grids, look at some of the scripts for how to do this. The fact that you are saying accuracy as opposed to MAP@3 though makes me think you are overstating your results, as well as the 70% number seeming to indicate overfitting. On the other hand this approach seems interesting.
", "votes": null, "replies": [] }, { "id": 124282, "author_name": "kumarmankala", "author_url": "", "post_date": "06/16/2016 19:08:37", "content": "Thanks both for your comments.
\n\nIt is very well possible that I may be misreading the results. \nFurther, I am using the term accuracy in a loose sense, here is why: \nI am resorting to bootstrapping (drawing a sample with replacement and repeating it say a 1000 times) to do the validation. This help me do some visual inspection of the results to do some evaluation. If I predict the correct place_id in the top 3, I am marking the prediction as correct else incorrect - kind of very crude way to do...
\n\nOtherwise, if I don't have an easy 'cloud' way to do this at one shot, I am going to split/chunk and do it.
", "votes": null, "replies": [] }, { "id": 124283, "author_name": "leonlu", "author_url": "", "post_date": "06/16/2016 19:16:01", "content": "[quote=Kumar Mankala;124282]
\n\nI am resorting to bootstrapping (drawing a sample with replacement and repeating it say a 1000 times) to do the validation.
\n\n[/quote]
\n\nSeems like you were not separating the training/validation sets by time. If this was the case, it's very likely you overestimated the accuracy.
\n\nIn addition, you can actually assign 1, 1/2, 1/3, or 0, instead of just 1 or 0, depending on which position of top 3 is the correct prediction at. In this way you get estimation of MAP@3 score instead a crude accuracy score.
", "votes": null, "replies": [] }, { "id": 124285, "author_name": "kumarmankala", "author_url": "", "post_date": "06/16/2016 19:28:47", "content": "Thanks Leon. You triggered an idea to cross-validate my results. \nI will try to subset an area (x,y) between say a square between (4,4) & (6,6), and split it 90% Train and 10% as Val by time.\nLet me try to approach on the above set and hopefully I should be able to repost the results today.
", "votes": null, "replies": [] }, { "id": 124286, "author_name": "haothecabbage", "author_url": "", "post_date": "06/16/2016 19:32:04", "content": "[quote=Kumar Mankala;124282]
\n\nThanks both for your comments.
\n\nIt is very well possible that I may be misreading the results. \nFurther, I am using the term accuracy in a loose sense, here is why: \nI am resorting to bootstrapping (drawing a sample with replacement and repeating it say a 1000 times) to do the validation. This help me do some visual inspection of the results to do some evaluation. If I predict the correct place_id in the top 3, I am marking the prediction as correct else incorrect - kind of very crude way to do...
\n\nOtherwise, if I don't have an easy 'cloud' way to do this at one shot, I am going to split/chunk and do it.
\n\n[/quote]
\n\nI don't think a top-3 accuracy of 70% would mean much improvement if you reevaluate with MAP@3.
", "votes": null, "replies": [] }, { "id": 124383, "author_name": "chu1990", "author_url": "", "post_date": "06/17/2016 21:25:41", "content": "Actually I have try to fit gaussian distribution and GMM for the checkin points for each place_id. It does not work well, because the checkin points do not seem to follow gaussian distribution. You may look at my post exploring the checkin points distribution below: \nhttps://www.kaggle.com/c/facebook-v-predicting-check-ins/forums/t/21328/2-d-check-in-scatter-plot-of-several-place-id/121933#post121933
\n\nI have tried another way to fitting place. The basic idea is for each place, finding a rectangle that covers 90% of the checkin points. The threshold needs tuning to remove outliers. When predicting a test point, we first find which rectangles the test points falls in, then use some criteria (rectangle area, density, etc.) to get top 3 matches. But the LB score of this idea is only around 0.4. Maybe I did not find the correct criteria and parameters.
\n\nI hope this could help you.
\n\n[quote=Kumar Mankala;124279]
\n\nWell. The 70% is on my limited validation set I derived out of the train.csv.
\n\nI followed the below approach:
\n\nCompute (rough) place_id coordinates using any classical mechanical approach (I used the center of mass concept for example - in this case I treated accuracy as the mass or 'weightage' for the point (x,y))
Next, given a test check-in event, find the subset of places the check-in can possibly fall in. I used the centroids from step 1 with some std. dev. for this, assuming the spread is normal. This turns out to be a good assumption. I was getting about 90+ accuracy on this. I was getting about 100 places the check-in possibly falls under.
From that list of 100 places, compute KDEs on (x,y,accuracy -> call this accuracy_kde) and (week, week day and hour -> call this time_kde) separately. For some reason combining them into single KDE is yielding a lower accuracy.
Derive the probabilities (you get two probabilities - one for accuracy and the other for time KDE). You can be creative here on how to order the probabilities. The places associated with top three probabilities give you the answer...
[/quote]
", "votes": null, "replies": [] }, { "id": 124663, "author_name": "kumarmankala", "author_url": "", "post_date": "06/21/2016 01:00:38", "content": "Also posted a small script on computing place centroids in the scripts.\nhttps://www.kaggle.com/kumarmankala/facebook-v-predicting-check-ins/computing-place-centroids/
", "votes": null, "replies": [] }, { "id": 124757, "author_name": "beedata", "author_url": "", "post_date": "06/22/2016 02:15:04", "content": "Could someone tell me what KDE is? Thank you.
", "votes": null, "replies": [] }, { "id": 124759, "author_name": "kumarmankala", "author_url": "", "post_date": "06/22/2016 02:36:58", "content": "Kernel Density Estimate - a non-parametric way of determining probability distribution function
", "votes": null, "replies": [] }, { "id": 124972, "author_name": "bimaldatascientist", "author_url": "", "post_date": "06/24/2016 03:37:22", "content": "hey Kumar,\nI've been working on a very similar approach for some time now. I haven't gotten it to run completely yet, but I think I will get there. You can get the 100,000+ KDE's done using Amazon EC2s with some multiprocessing code and being creative about work sharing across about few m4.10x machines so that the job completes in a reasonable time. you would need to figure out how to split the work between the machines.
\n\nBest!
", "votes": null, "replies": [] }, { "id": 125205, "author_name": "kumarmankala", "author_url": "", "post_date": "06/27/2016 16:35:49", "content": "[quote=Predictions, Forecasts & Lies;124972]
\n\nThanks Bimal. Yeah. I am trying it... In the skies the AWS's cloud looks best for now...
\n\n[/quote]
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "124272": "I think I am getting a prediction accuracy more than 70%, but stuck with the problem of computing KDEs on the go for 8 million test checkin records. One way is to precompute the KDEs for these 100K+ places. But, my machine (8GB, i7, with SSD) is unable to do it... anyone facing similar issue OR have ideas on how to scale it? \r\nAzure is not helping much... they don't support ks package. (I am using R)", "124278": "Wow 70% seems like a great improvement in comparison to current leaders in LB. May I ask what models/tools do you use to get from KDE to predictions?", "124279": "Well. The 70% is on my limited validation set I derived out of the train.csv.\r\n\r\nI followed the below approach:\r\n\r\n1. Compute (rough) place_id coordinates using any classical mechanical approach (I used the center of mass concept for example - in this case I treated accuracy as the mass or 'weightage' for the point (x,y))\r\n\r\n2. Next, given a test check-in event, find the subset of places the check-in can possibly fall in. I used the centroids from step 1 with some std. dev. for this, assuming the spread is normal. This turns out to be a good assumption. I was getting about 90+ accuracy on this. I was getting about 100 places the check-in possibly falls under.\r\n\r\n4. From that list of 100 places, compute KDEs on (x,y,accuracy -> call this accuracy_kde) and (week, week day and hour -> call this time_kde) separately. For some reason combining them into single KDE is yielding a lower accuracy.\r\n\r\n5. Derive the probabilities (you get two probabilities - one for accuracy and the other for time KDE). You can be creative here on how to order the probabilities. The places associated with top three probabilities give you the answer...", "124280": "Point 4 is the most interesting. I though you use some optimization to determine the best way to compute probabilities. I asked because i tested a similar approach but it yielded very low scores for me. Your value of 70% got me really surprised. Thank you for your answer.", "124281": "If memory is your only concern, you can just chunk the dataset into grids, look at some of the scripts for how to do this. The fact that you are saying accuracy as opposed to MAP@3 though makes me think you are overstating your results, as well as the 70% number seeming to indicate overfitting. On the other hand this approach seems interesting.", "124282": "Thanks both for your comments. \r\n\r\nIt is very well possible that I may be misreading the results. \r\nFurther, I am using the term accuracy in a loose sense, here is why: \r\nI am resorting to bootstrapping (drawing a sample with replacement and repeating it say a 1000 times) to do the validation. This help me do some visual inspection of the results to do some evaluation. If I predict the correct place_id in the top 3, I am marking the prediction as correct else incorrect - kind of very crude way to do...\r\n\r\nOtherwise, if I don't have an easy 'cloud' way to do this at one shot, I am going to split/chunk and do it.", "124283": "[quote=Kumar Mankala;124282]\r\n\r\n\r\nI am resorting to bootstrapping (drawing a sample with replacement and repeating it say a 1000 times) to do the validation. \r\n\r\n[/quote]\r\n\r\nSeems like you were not separating the training/validation sets by time. If this was the case, it's very likely you overestimated the accuracy.\r\n\r\nIn addition, you can actually assign 1, 1/2, 1/3, or 0, instead of just 1 or 0, depending on which position of top 3 is the correct prediction at. In this way you get estimation of MAP@3 score instead a crude accuracy score.", "124285": "Thanks Leon. You triggered an idea to cross-validate my results. \r\nI will try to subset an area (x,y) between say a square between (4,4) & (6,6), and split it 90% Train and 10% as Val by time.\r\nLet me try to approach on the above set and hopefully I should be able to repost the results today.", "124286": "[quote=Kumar Mankala;124282]\r\n\r\nThanks both for your comments. \r\n\r\nIt is very well possible that I may be misreading the results. \r\nFurther, I am using the term accuracy in a loose sense, here is why: \r\nI am resorting to bootstrapping (drawing a sample with replacement and repeating it say a 1000 times) to do the validation. This help me do some visual inspection of the results to do some evaluation. If I predict the correct place_id in the top 3, I am marking the prediction as correct else incorrect - kind of very crude way to do...\r\n\r\nOtherwise, if I don't have an easy 'cloud' way to do this at one shot, I am going to split/chunk and do it. \r\n\r\n[/quote]\r\n\r\nI don't think a top-3 accuracy of 70% would mean much improvement if you reevaluate with MAP@3.", "124383": "Actually I have try to fit gaussian distribution and GMM for the checkin points for each place_id. It does not work well, because the checkin points do not seem to follow gaussian distribution. You may look at my post exploring the checkin points distribution below: \r\nhttps://www.kaggle.com/c/facebook-v-predicting-check-ins/forums/t/21328/2-d-check-in-scatter-plot-of-several-place-id/121933#post121933 \r\n\r\nI have tried another way to fitting place. The basic idea is for each place, finding a rectangle that covers 90% of the checkin points. The threshold needs tuning to remove outliers. When predicting a test point, we first find which rectangles the test points falls in, then use some criteria (rectangle area, density, etc.) to get top 3 matches. But the LB score of this idea is only around 0.4. Maybe I did not find the correct criteria and parameters.\r\n\r\nI hope this could help you.\r\n\r\n[quote=Kumar Mankala;124279]\r\n\r\nWell. The 70% is on my limited validation set I derived out of the train.csv.\r\n\r\nI followed the below approach:\r\n\r\n1. Compute (rough) place_id coordinates using any classical mechanical approach (I used the center of mass concept for example - in this case I treated accuracy as the mass or 'weightage' for the point (x,y))\r\n\r\n2. Next, given a test check-in event, find the subset of places the check-in can possibly fall in. I used the centroids from step 1 with some std. dev. for this, assuming the spread is normal. This turns out to be a good assumption. I was getting about 90+ accuracy on this. I was getting about 100 places the check-in possibly falls under.\r\n\r\n4. From that list of 100 places, compute KDEs on (x,y,accuracy -> call this accuracy_kde) and (week, week day and hour -> call this time_kde) separately. For some reason combining them into single KDE is yielding a lower accuracy.\r\n\r\n5. Derive the probabilities (you get two probabilities - one for accuracy and the other for time KDE). You can be creative here on how to order the probabilities. The places associated with top three probabilities give you the answer...\r\n\r\n[/quote]", "124663": "Also posted a small script on computing place centroids in the scripts.\r\nhttps://www.kaggle.com/kumarmankala/facebook-v-predicting-check-ins/computing-place-centroids/", "124757": "Could someone tell me what KDE is? Thank you.", "124759": "Kernel Density Estimate - a non-parametric way of determining probability distribution function", "124972": "hey Kumar,\r\nI've been working on a very similar approach for some time now. I haven't gotten it to run completely yet, but I think I will get there. You can get the 100,000+ KDE's done using Amazon EC2s with some multiprocessing code and being creative about work sharing across about few m4.10x machines so that the job completes in a reasonable time. you would need to figure out how to split the work between the machines. \r\n\r\nBest!", "125205": "[quote=Predictions, Forecasts & Lies;124972]\r\n\r\nThanks Bimal. Yeah. I am trying it... In the skies the AWS's cloud looks best for now...\r\n\r\n[/quote]" }, "source": "meta" }