My solution is a combination of K nearest neighbor and kernel density estimation. The idea is firstly using KNN to pick 10+ possible candidates and then reranking the candidates with KDE. You can find my source code here:\nhttps://github.com/linrongc/facebook_check_in_prediction/
\n\nAs for KNN, I applied 2 tricks: (1) the number of neighbor is linearly increasing over time
\n\ndef var_k(record):\n return int(40 + 80 * (record[3] - Setting.LASTTIME)/400000.)\n(2) the neighbors have voting weights which are measured by the distance:
\n\nweight = np.exp(-0.5*dis**2/10.78)\nThese two tricks boosted my score from 0.565 to 0.588 at LB.
\n\nAs for KDE, I use [x, y, accuracy, day_hour, week_day, year, month_day, day_of_year] as the features (they are float numbers). Different from custom KDE solutions, I use a kdestimator to fit the distribution for each dimension (now I think it could be problematic). For example the uploaded image shows the fitted distribution for week_day.
\n\nThen the final score function is just a linear combination of the log probability of place_popularity(#check_in), KNN weights, the predicted probability by the kdestimator for each dimension. Since the number of dimension is small, I could manually tune the parameters.
\n\nThe tricks here are: (1) way to compute the probability of recurrent features(weekday, day_hours)
\n\n values = kde.score_samples([[value], [value+period], [value-period]])\n total = np.logaddexp(values[0], values[1])\n total = np.logaddexp(total, values[2])\n(2) use a smooth_func so that places with fewer checkins will have higher band_width:
\n\ndef smooth_func(self, size, factor):\n return np.log(factor)/np.log(size + 1)\nThis part boosted my score from 0.588 to 0.604. There are multi_thread supporting interfaces in my source code and you can run the code in a cluster or multi_core computers.
\n\nBefore the end of the competition, I finally realized that X, Y are highly denpendent. So a better approach should be use only one KDE for X and Y. I believe that will be an improvement. But no time remains for me to modify the solution.
\n\nActually I didn't start this competition until June 17th. The time was too tight for a kaggle newbie to make his first challenge. But thanks for all of your scripts and insightful comments in the forum, I am able to make it done.
\n\nI really enjoy my first competition in kaggle. Also if lucky enough, I hope I can get an interview from facebook and have a job.
\n\nCongratulations to all the kagglers in this competition!!!
", "messages": [ { "id": "126245", "postDate": "07/07/2016 05:27:43", "content": "My solution is a combination of K nearest neighbor and kernel density estimation. The idea is firstly using KNN to pick 10+ possible candidates and then reranking the candidates with KDE. You can find my source code here:\nhttps://github.com/linrongc/facebook_check_in_prediction/
\n\nAs for KNN, I applied 2 tricks: (1) the number of neighbor is linearly increasing over time
\n\ndef var_k(record):\n return int(40 + 80 * (record[3] - Setting.LASTTIME)/400000.)\n(2) the neighbors have voting weights which are measured by the distance:
\n\nweight = np.exp(-0.5*dis**2/10.78)\nThese two tricks boosted my score from 0.565 to 0.588 at LB.
\n\nAs for KDE, I use [x, y, accuracy, day_hour, week_day, year, month_day, day_of_year] as the features (they are float numbers). Different from custom KDE solutions, I use a kdestimator to fit the distribution for each dimension (now I think it could be problematic). For example the uploaded image shows the fitted distribution for week_day.
\n\nThen the final score function is just a linear combination of the log probability of place_popularity(#check_in), KNN weights, the predicted probability by the kdestimator for each dimension. Since the number of dimension is small, I could manually tune the parameters.
\n\nThe tricks here are: (1) way to compute the probability of recurrent features(weekday, day_hours)
\n\n values = kde.score_samples([[value], [value+period], [value-period]])\n total = np.logaddexp(values[0], values[1])\n total = np.logaddexp(total, values[2])\n(2) use a smooth_func so that places with fewer checkins will have higher band_width:
\n\ndef smooth_func(self, size, factor):\n return np.log(factor)/np.log(size + 1)\nThis part boosted my score from 0.588 to 0.604. There are multi_thread supporting interfaces in my source code and you can run the code in a cluster or multi_core computers.
\n\nBefore the end of the competition, I finally realized that X, Y are highly denpendent. So a better approach should be use only one KDE for X and Y. I believe that will be an improvement. But no time remains for me to modify the solution.
\n\nActually I didn't start this competition until June 17th. The time was too tight for a kaggle newbie to make his first challenge. But thanks for all of your scripts and insightful comments in the forum, I am able to make it done.
\n\nI really enjoy my first competition in kaggle. Also if lucky enough, I hope I can get an interview from facebook and have a job.
\n\nCongratulations to all the kagglers in this competition!!!
", "rawMarkdown": "My solution is a combination of K nearest neighbor and kernel density estimation. The idea is firstly using KNN to pick 10+ possible candidates and then reranking the candidates with KDE. You can find my source code here:\r\nhttps://github.com/linrongc/facebook_check_in_prediction/\r\n\r\nAs for KNN, I applied 2 tricks: (1) the number of neighbor is linearly increasing over time\r\n\r\n def var_k(record):\r\n return int(40 + 80 * (record[3] - Setting.LASTTIME)/400000.)\r\n\r\n(2) the neighbors have voting weights which are measured by the distance:\r\n\r\n weight = np.exp(-0.5*dis**2/10.78)\r\n\r\nThese two tricks boosted my score from 0.565 to 0.588 at LB.\r\n\r\nAs for KDE, I use [x, y, accuracy, day_hour, week_day, year, month_day, day_of_year] as the features (they are float numbers). Different from custom KDE solutions, I use a kdestimator to fit the distribution for each dimension (now I think it could be problematic). For example the uploaded image shows the fitted distribution for week_day.\r\n\r\nThen the final score function is just a linear combination of the log probability of place_popularity(#check_in), KNN weights, the predicted probability by the kdestimator for each dimension. Since the number of dimension is small, I could manually tune the parameters.\r\n\r\nThe tricks here are: (1) way to compute the probability of recurrent features(weekday, day_hours)\r\n\r\n values = kde.score_samples([[value], [value+period], [value-period]])\r\n total = np.logaddexp(values[0], values[1])\r\n total = np.logaddexp(total, values[2])\r\n\r\n(2) use a smooth_func so that places with fewer checkins will have higher band_width:\r\n\r\n def smooth_func(self, size, factor):\r\n return np.log(factor)/np.log(size + 1)\r\n\r\nThis part boosted my score from 0.588 to 0.604. There are multi_thread supporting interfaces in my source code and you can run the code in a cluster or multi_core computers.\r\n\r\nBefore the end of the competition, I finally realized that X, Y are highly denpendent. So a better approach should be use only one KDE for X and Y. I believe that will be an improvement. But no time remains for me to modify the solution. \r\n\r\nActually I didn't start this competition until June 17th. The time was too tight for a kaggle newbie to make his first challenge. But thanks for all of your scripts and insightful comments in the forum, I am able to make it done. \r\n\r\nI really enjoy my first competition in kaggle. Also if lucky enough, I hope I can get an interview from facebook and have a job.\r\n\r\nCongratulations to all the kagglers in this competition!!!", "votes": null }, { "id": "126249", "postDate": "07/07/2016 05:36:02", "content": "@Rongcheng Lin, thank you for sharing, I appreciate it. Surprise that number of neighbor is linearly proportional to the time, a thing that didn't get caught by hundred of shared KNN scripts. Good luck for the job interview.
", "rawMarkdown": "Rongcheng Lin, thank you for sharing, I appreciate it. Surprise that number of neighbor is linearly proportional to the time, a thing that didn't get caught by hundred of shared KNN scripts. Good luck for the job interview.", "votes": null }, { "id": "126251", "postDate": "07/07/2016 05:41:52", "content": "@aldente I just thought the best neighbor number should vary from place to place. So it should be a function that is determined by time or position or the diversity around. And I didn't find a way to come up with a function with position. And thank you.
", "rawMarkdown": "aldente I just thought the best neighbor number should vary from place to place. So it should be a function that is determined by time or position or the diversity around. And I didn't find a way to come up with a function with position. And thank you.", "votes": null }, { "id": "126255", "postDate": "07/07/2016 06:17:53", "content": "Btw, you said "I hope I can get an interview from facebook and have a job." If I recall correctly, in the previous Facebook competitions, after LB cleaning up, interested participants had to submit a resume for the interview.
\n\nIf this competition has the same steps, then since you have high rank, an interview is highly likely 99% for you :)
", "rawMarkdown": "Btw, you said \"*I **hope** I can get an interview from facebook and have a job.*\" If I recall correctly, in the [previous Facebook competitions][1], after LB cleaning up, interested participants had to submit a resume for the interview. \r\n\r\nIf this competition has the same steps, then since you have high rank, an interview is highly likely 99% for you :) \r\n\r\n\r\n [1]: https://www.kaggle.com/c/facebook-recruiting-iv-human-or-bot", "votes": null }, { "id": "126256", "postDate": "07/07/2016 06:28:06", "content": "It is very nice of you to let me know that. I wlll try my best, since my advisor is leaving the school and I need a job as soon as possible.
", "rawMarkdown": "It is very nice of you to let me know that. I wlll try my best, since my advisor is leaving the school and I need a job as soon as possible.", "votes": null }, { "id": "126259", "postDate": "07/07/2016 06:34:41", "content": "The number of kdes are #place * #dimension
", "rawMarkdown": "The number of kdes are #place * #dimension", "votes": null }, { "id": "126274", "postDate": "07/07/2016 08:51:10", "content": "Thanks. For your edited question, it's to remove cheaters (if any).
\n\n[quote=Rongcheng Lin;126259]
\n\n@aldente BTW, what do you mean by "LB cleaning up"? this is my first time in kaggle.
\n\n[/quote]
", "rawMarkdown": "Thanks. For your edited question, it's to remove cheaters (if any). \r\n\r\n[quote=Rongcheng Lin;126259]\r\n\r\n@aldente BTW, what do you mean by \"LB cleaning up\"? this is my first time in kaggle.\r\n\r\n[/quote]", "votes": null }, { "id": "126291", "postDate": "07/07/2016 13:49:53", "content": "Hi @Rongcheng Lin, congratulations on getting a decent result in your first competition! I'm wondering what kde estimator you used, is it fft/ifft? If so, what library did you use?
", "rawMarkdown": "Hi @Rongcheng Lin, congratulations on getting a decent result in your first competition! I'm wondering what kde estimator you used, is it fft/ifft? If so, what library did you use?", "votes": null }, { "id": "126312", "postDate": "07/07/2016 16:34:22", "content": "@Hao The Cabbage I just use the KernelDensity in sklearn.Neighbor. You can refer to my source code ( in pyhton). I suppose it is based on the Nearest Neighbor and the Kernel Function, althought the result may be similar to fft/ifft, it may be just searching for neighbors and suming the probability up.
", "rawMarkdown": "Hao The Cabbage I just use the KernelDensity in sklearn.Neighbor. You can refer to my source code ( in pyhton). I suppose it is based on the Nearest Neighbor and the Kernel Function, althought the result may be similar to fft/ifft, it may be just searching for neighbors and suming the probability up.", "votes": null }, { "id": "126313", "postDate": "07/07/2016 16:35:25", "content": "@aldete Thanks for your replay. I found the answer in previous facebook challenge, so I edit the question. Thank you all the same.
", "rawMarkdown": "aldete Thanks for your replay. I found the answer in previous facebook challenge, so I edit the question. Thank you all the same.", "votes": null } ], "comments": [ { "id": 126249, "author_name": "aldente", "author_url": "", "post_date": "07/07/2016 05:36:02", "content": "@Rongcheng Lin, thank you for sharing, I appreciate it. Surprise that number of neighbor is linearly proportional to the time, a thing that didn't get caught by hundred of shared KNN scripts. Good luck for the job interview.
", "votes": null, "replies": [] }, { "id": 126251, "author_name": "phoenixl", "author_url": "", "post_date": "07/07/2016 05:41:52", "content": "@aldente I just thought the best neighbor number should vary from place to place. So it should be a function that is determined by time or position or the diversity around. And I didn't find a way to come up with a function with position. And thank you.
", "votes": null, "replies": [] }, { "id": 126255, "author_name": "aldente", "author_url": "", "post_date": "07/07/2016 06:17:53", "content": "Btw, you said "I hope I can get an interview from facebook and have a job." If I recall correctly, in the previous Facebook competitions, after LB cleaning up, interested participants had to submit a resume for the interview.
\n\nIf this competition has the same steps, then since you have high rank, an interview is highly likely 99% for you :)
", "votes": null, "replies": [] }, { "id": 126256, "author_name": "phoenixl", "author_url": "", "post_date": "07/07/2016 06:28:06", "content": "It is very nice of you to let me know that. I wlll try my best, since my advisor is leaving the school and I need a job as soon as possible.
", "votes": null, "replies": [] }, { "id": 126259, "author_name": "phoenixl", "author_url": "", "post_date": "07/07/2016 06:34:41", "content": "The number of kdes are #place * #dimension
", "votes": null, "replies": [] }, { "id": 126274, "author_name": "aldente", "author_url": "", "post_date": "07/07/2016 08:51:10", "content": "Thanks. For your edited question, it's to remove cheaters (if any).
\n\n[quote=Rongcheng Lin;126259]
\n\n@aldente BTW, what do you mean by "LB cleaning up"? this is my first time in kaggle.
\n\n[/quote]
", "votes": null, "replies": [] }, { "id": 126291, "author_name": "haothecabbage", "author_url": "", "post_date": "07/07/2016 13:49:53", "content": "Hi @Rongcheng Lin, congratulations on getting a decent result in your first competition! I'm wondering what kde estimator you used, is it fft/ifft? If so, what library did you use?
", "votes": null, "replies": [] }, { "id": 126312, "author_name": "phoenixl", "author_url": "", "post_date": "07/07/2016 16:34:22", "content": "@Hao The Cabbage I just use the KernelDensity in sklearn.Neighbor. You can refer to my source code ( in pyhton). I suppose it is based on the Nearest Neighbor and the Kernel Function, althought the result may be similar to fft/ifft, it may be just searching for neighbors and suming the probability up.
", "votes": null, "replies": [] }, { "id": 126313, "author_name": "phoenixl", "author_url": "", "post_date": "07/07/2016 16:35:25", "content": "@aldete Thanks for your replay. I found the answer in previous facebook challenge, so I edit the question. Thank you all the same.
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "126245": "My solution is a combination of K nearest neighbor and kernel density estimation. The idea is firstly using KNN to pick 10+ possible candidates and then reranking the candidates with KDE. You can find my source code here:\r\nhttps://github.com/linrongc/facebook_check_in_prediction/\r\n\r\nAs for KNN, I applied 2 tricks: (1) the number of neighbor is linearly increasing over time\r\n\r\n def var_k(record):\r\n return int(40 + 80 * (record[3] - Setting.LASTTIME)/400000.)\r\n\r\n(2) the neighbors have voting weights which are measured by the distance:\r\n\r\n weight = np.exp(-0.5*dis**2/10.78)\r\n\r\nThese two tricks boosted my score from 0.565 to 0.588 at LB.\r\n\r\nAs for KDE, I use [x, y, accuracy, day_hour, week_day, year, month_day, day_of_year] as the features (they are float numbers). Different from custom KDE solutions, I use a kdestimator to fit the distribution for each dimension (now I think it could be problematic). For example the uploaded image shows the fitted distribution for week_day.\r\n\r\nThen the final score function is just a linear combination of the log probability of place_popularity(#check_in), KNN weights, the predicted probability by the kdestimator for each dimension. Since the number of dimension is small, I could manually tune the parameters.\r\n\r\nThe tricks here are: (1) way to compute the probability of recurrent features(weekday, day_hours)\r\n\r\n values = kde.score_samples([[value], [value+period], [value-period]])\r\n total = np.logaddexp(values[0], values[1])\r\n total = np.logaddexp(total, values[2])\r\n\r\n(2) use a smooth_func so that places with fewer checkins will have higher band_width:\r\n\r\n def smooth_func(self, size, factor):\r\n return np.log(factor)/np.log(size + 1)\r\n\r\nThis part boosted my score from 0.588 to 0.604. There are multi_thread supporting interfaces in my source code and you can run the code in a cluster or multi_core computers.\r\n\r\nBefore the end of the competition, I finally realized that X, Y are highly denpendent. So a better approach should be use only one KDE for X and Y. I believe that will be an improvement. But no time remains for me to modify the solution. \r\n\r\nActually I didn't start this competition until June 17th. The time was too tight for a kaggle newbie to make his first challenge. But thanks for all of your scripts and insightful comments in the forum, I am able to make it done. \r\n\r\nI really enjoy my first competition in kaggle. Also if lucky enough, I hope I can get an interview from facebook and have a job.\r\n\r\nCongratulations to all the kagglers in this competition!!!", "126249": "Rongcheng Lin, thank you for sharing, I appreciate it. Surprise that number of neighbor is linearly proportional to the time, a thing that didn't get caught by hundred of shared KNN scripts. Good luck for the job interview.", "126251": "aldente I just thought the best neighbor number should vary from place to place. So it should be a function that is determined by time or position or the diversity around. And I didn't find a way to come up with a function with position. And thank you.", "126255": "Btw, you said \"*I **hope** I can get an interview from facebook and have a job.*\" If I recall correctly, in the [previous Facebook competitions][1], after LB cleaning up, interested participants had to submit a resume for the interview. \r\n\r\nIf this competition has the same steps, then since you have high rank, an interview is highly likely 99% for you :) \r\n\r\n\r\n [1]: https://www.kaggle.com/c/facebook-recruiting-iv-human-or-bot", "126256": "It is very nice of you to let me know that. I wlll try my best, since my advisor is leaving the school and I need a job as soon as possible.", "126259": "The number of kdes are #place * #dimension", "126274": "Thanks. For your edited question, it's to remove cheaters (if any). \r\n\r\n[quote=Rongcheng Lin;126259]\r\n\r\n@aldente BTW, what do you mean by \"LB cleaning up\"? this is my first time in kaggle.\r\n\r\n[/quote]", "126291": "Hi @Rongcheng Lin, congratulations on getting a decent result in your first competition! I'm wondering what kde estimator you used, is it fft/ifft? If so, what library did you use?", "126312": "Hao The Cabbage I just use the KernelDensity in sklearn.Neighbor. You can refer to my source code ( in pyhton). I suppose it is based on the Nearest Neighbor and the Kernel Function, althought the result may be similar to fft/ifft, it may be just searching for neighbors and suming the probability up.", "126313": "aldete Thanks for your replay. I found the answer in previous facebook challenge, so I edit the question. Thank you all the same." }, "source": "meta" }