This is an amazing competition and thank all the discussion and scripts made by fellow kaggler. I joined two weeks before the end of this competition. The main reason for me to start this competition is because:
\n\nThe features provided in this competition are very simple (only x, y, time, accuracy). But the data set is big and contains more than 100,000 labels. This gives me chance to think about how to divide the problem into small sub-problems.
Get into top 10 in order to reach master tier.
I learned a lot during this competition. Especially in the final week, the competition become quite intense among ranking 5th to 10th. The fear of fall out of top 10 making me work very hard on trying and learning new method, including naive bayes, support vector machine and kernel density estimation. Special thank to @Qingchen, it is always very educative to talk with him.
\n\nThe size of the data set is very large. In order to use my time efficiently, my model and hyperparameter selection are done on 1/100 of the total data set. It is data in range 5 < x < 6, 5 < y < 6. Then I use the last 171360 minutes as validation set. By using this subset, I am able to know the performance of one model within 20 minutes.
\n\nAccording to the admin, the real test set has removed unseen place_id. But I didn't do so on my validation set. So my local map3 will be lower than public leaderboard.
\n\nn_cell_x, int\n number of cells divided in x for test and validation set\nn_cell_y, int\n number of cells divided in y for test and validation set\nx_border, float\n border on the edge of the cell for larger range of training set\ny_border, float\n border on the edge of the cell for larger range of training set\nth, int\n cutoff low frequency labels\nThose parameters as well as parameters of different models are considered as the hyperparameters. I write my own random grid search code to find the optimal hyperparameters of each classifier. This code will append all the hypermeters and corresponding local validation score to a record file. It also guarantee new generated hyperparameters will not duplicate with previous one in record. Thus, I can easily adjust the range and density of each hyperparameters in the selection process.
\n\nEach single model will predict top 10 place_id with probabilities for each row_id. The are stored in csv as (row_id, place_id, proba). This output file for a validation set is about 3.3Mb and 351Mb for test set. I ensemble probabilities of different models using python defaultdict and output the top 3 place_id as prediction. I will first ensemble the validation set to see which combination of models has better performance. Then, I will run an ensemble on the whole data set with this combination. The idea of finding the combination is:
\n\nTry to increase the diversity of the model. This includes different number of cells and different models.
Then I list model candidates in different category and sorted them by local map3 score.
Do greedy search to find the combination with highest local validation score.
My final submission is an unweighted sum of single models. I also tried to find weight by bayes optimization. There is some improvement on the 5th digit so I just ignore it.
Then run the ensemble on the whole dataset.
My final submission is a soft-voting of 13 single models, including k-nearest neighbors, random forest, extra-trees, gradient boosting trees, naive bayes, kernel density estimation.
\n\nsklearn.neighbors.KNeighborsClassifier/knn_ps2_n1, local map3=0.504975\n\nsklearn.ensemble.RandomForestClassifier/rf_opt1, local map3=0.509508\n\nsklearn.ensemble.ExtraTreesClassifier/et_opt1, local map3=0.497024\n\nXGBoost/xgb0_n4, local map3=0.518815\nXGBoost/xgb0_n2, local map3=0.514897\nXGBoost/xgb0_n4_th13_border5, local map3=0.521114\nXGBoost/xgb150opt, local map3=0.523222\nXGBoost/xgb150opt2, local map3=0.523853\n\nsklearn.naive_bayes.GaussianNB/nb_xyat_weight1_n9, local map3=0.482681\n\nsklearn.neighbors.KernelDensity/kde_opt1, local map3=0.478387\nscipy.stats.gaussian_kde/kde_opt2c, local map3=0.491391\nscipy.stats.gaussian_kde/kde_opt3c, local map3=0.493110\nKDE-for-SciPy/kde_opt4, local map3=0.483254\nThe structure of this dataset is very interesting. My best XGBoost model only has max_depth=3, which is unusual for my experience in previous competitions. This indicates features are almost independent to each other in this dataset. This can be the reason why my Naive Bayes is only 4% worse than my knn.
\n\nNaive Bayes is a first assumption of the data distribution. When using the sklearn.naive_bayes.GaussianNB, it also assumes the distribution of each feature in cell is unimodal. But doing a simple visualization of the x, y for given place_id will tell you that assumption is not very good. One way to improve the Naive Bayes is to use kernel density estimation (KDE) rather than one single gaussian to get the probability density function. I recommend this blog (http://www.mglerner.com/blog/?p=28) for nice explanation of KDE.
\n\nThe performance of KDE is determined by the kernel and kernel parameters (bandwidth for gaussian kernel). sklearn.neighbors.KernelDensity provides you plenty of kernels to choose. My hyperparameter selection code told me gaussian and exponential are optimal choice while tophat, linear, cosine works badly. sklearn does not provide rule of thumb estimation of bandwidth, so the bandwidth is selected by cross-validation. I also tried the scipy.stats.gaussian_kde because it provides two common rule of thumb estimations of bandwidth by statistician, "scott" and "silverman". They can give fast estimation but are designed for unimodal distribution. In order to get better bandwidth estimation, I also used Daniel B Smith's implement of KDE (https://github.com/Daniel-B-Smith/KDE-for-SciPy). KDE-for-SciPy is based on this paper
\n\n\n\n\nZ. I. Botev, J. F. Grotowski, and D. P. Kroese. Kernel density\n estimation via diffusion. The Annals of Statistics, 38(5):2916-2957, 2010.
\n
This estimation is more time consuming but can give good estimation on multimodal distribution. My final submission include KDE from all those implements.
\n\nI feel it is very important to have your own framework to do the model selections and ensemble. The quality of your own code determine how easy it is to tuning the model and test new idea. Thus, this competition is also a good programing exercise for me.
\n\nMy code is on GitHub now.
\n\nPlease press "helpful" button if you like my post.
\n\nThanks,\nLi
", "messages": [ { "id": "126376", "postDate": "07/08/2016 00:52:25", "content": "This is an amazing competition and thank all the discussion and scripts made by fellow kaggler. I joined two weeks before the end of this competition. The main reason for me to start this competition is because:
\n\nThe features provided in this competition are very simple (only x, y, time, accuracy). But the data set is big and contains more than 100,000 labels. This gives me chance to think about how to divide the problem into small sub-problems.
Get into top 10 in order to reach master tier.
I learned a lot during this competition. Especially in the final week, the competition become quite intense among ranking 5th to 10th. The fear of fall out of top 10 making me work very hard on trying and learning new method, including naive bayes, support vector machine and kernel density estimation. Special thank to @Qingchen, it is always very educative to talk with him.
\n\nThe size of the data set is very large. In order to use my time efficiently, my model and hyperparameter selection are done on 1/100 of the total data set. It is data in range 5 < x < 6, 5 < y < 6. Then I use the last 171360 minutes as validation set. By using this subset, I am able to know the performance of one model within 20 minutes.
\n\nAccording to the admin, the real test set has removed unseen place_id. But I didn't do so on my validation set. So my local map3 will be lower than public leaderboard.
\n\nn_cell_x, int\n number of cells divided in x for test and validation set\nn_cell_y, int\n number of cells divided in y for test and validation set\nx_border, float\n border on the edge of the cell for larger range of training set\ny_border, float\n border on the edge of the cell for larger range of training set\nth, int\n cutoff low frequency labels\nThose parameters as well as parameters of different models are considered as the hyperparameters. I write my own random grid search code to find the optimal hyperparameters of each classifier. This code will append all the hypermeters and corresponding local validation score to a record file. It also guarantee new generated hyperparameters will not duplicate with previous one in record. Thus, I can easily adjust the range and density of each hyperparameters in the selection process.
\n\nEach single model will predict top 10 place_id with probabilities for each row_id. The are stored in csv as (row_id, place_id, proba). This output file for a validation set is about 3.3Mb and 351Mb for test set. I ensemble probabilities of different models using python defaultdict and output the top 3 place_id as prediction. I will first ensemble the validation set to see which combination of models has better performance. Then, I will run an ensemble on the whole data set with this combination. The idea of finding the combination is:
\n\nTry to increase the diversity of the model. This includes different number of cells and different models.
Then I list model candidates in different category and sorted them by local map3 score.
Do greedy search to find the combination with highest local validation score.
My final submission is an unweighted sum of single models. I also tried to find weight by bayes optimization. There is some improvement on the 5th digit so I just ignore it.
Then run the ensemble on the whole dataset.
My final submission is a soft-voting of 13 single models, including k-nearest neighbors, random forest, extra-trees, gradient boosting trees, naive bayes, kernel density estimation.
\n\nsklearn.neighbors.KNeighborsClassifier/knn_ps2_n1, local map3=0.504975\n\nsklearn.ensemble.RandomForestClassifier/rf_opt1, local map3=0.509508\n\nsklearn.ensemble.ExtraTreesClassifier/et_opt1, local map3=0.497024\n\nXGBoost/xgb0_n4, local map3=0.518815\nXGBoost/xgb0_n2, local map3=0.514897\nXGBoost/xgb0_n4_th13_border5, local map3=0.521114\nXGBoost/xgb150opt, local map3=0.523222\nXGBoost/xgb150opt2, local map3=0.523853\n\nsklearn.naive_bayes.GaussianNB/nb_xyat_weight1_n9, local map3=0.482681\n\nsklearn.neighbors.KernelDensity/kde_opt1, local map3=0.478387\nscipy.stats.gaussian_kde/kde_opt2c, local map3=0.491391\nscipy.stats.gaussian_kde/kde_opt3c, local map3=0.493110\nKDE-for-SciPy/kde_opt4, local map3=0.483254\nThe structure of this dataset is very interesting. My best XGBoost model only has max_depth=3, which is unusual for my experience in previous competitions. This indicates features are almost independent to each other in this dataset. This can be the reason why my Naive Bayes is only 4% worse than my knn.
\n\nNaive Bayes is a first assumption of the data distribution. When using the sklearn.naive_bayes.GaussianNB, it also assumes the distribution of each feature in cell is unimodal. But doing a simple visualization of the x, y for given place_id will tell you that assumption is not very good. One way to improve the Naive Bayes is to use kernel density estimation (KDE) rather than one single gaussian to get the probability density function. I recommend this blog (http://www.mglerner.com/blog/?p=28) for nice explanation of KDE.
\n\nThe performance of KDE is determined by the kernel and kernel parameters (bandwidth for gaussian kernel). sklearn.neighbors.KernelDensity provides you plenty of kernels to choose. My hyperparameter selection code told me gaussian and exponential are optimal choice while tophat, linear, cosine works badly. sklearn does not provide rule of thumb estimation of bandwidth, so the bandwidth is selected by cross-validation. I also tried the scipy.stats.gaussian_kde because it provides two common rule of thumb estimations of bandwidth by statistician, "scott" and "silverman". They can give fast estimation but are designed for unimodal distribution. In order to get better bandwidth estimation, I also used Daniel B Smith's implement of KDE (https://github.com/Daniel-B-Smith/KDE-for-SciPy). KDE-for-SciPy is based on this paper
\n\n\n\n\nZ. I. Botev, J. F. Grotowski, and D. P. Kroese. Kernel density\n estimation via diffusion. The Annals of Statistics, 38(5):2916-2957, 2010.
\n
This estimation is more time consuming but can give good estimation on multimodal distribution. My final submission include KDE from all those implements.
\n\nI feel it is very important to have your own framework to do the model selections and ensemble. The quality of your own code determine how easy it is to tuning the model and test new idea. Thus, this competition is also a good programing exercise for me.
\n\nMy code is on GitHub now.
\n\nPlease press "helpful" button if you like my post.
\n\nThanks,\nLi
", "rawMarkdown": "This is an amazing competition and thank all the discussion and scripts made by fellow kaggler. I joined two weeks before the end of this competition. The main reason for me to start this competition is because:\r\n\r\n* The features provided in this competition are very simple (only x, y, time, accuracy). But the data set is big and contains more than 100,000 labels. This gives me chance to think about how to divide the problem into small sub-problems.\r\n\r\n* Get into top 10 in order to reach master tier.\r\n\r\nI learned a lot during this competition. Especially in the final week, the competition become quite intense among ranking 5th to 10th. The fear of fall out of top 10 making me work very hard on trying and learning new method, including naive bayes, support vector machine and kernel density estimation. Special thank to @Qingchen, it is always very educative to talk with him.\r\n\r\n## validation set\r\nThe size of the data set is very large. In order to use my time efficiently, my model and hyperparameter selection are done on 1/100 of the total data set. It is data in range 5 < x < 6, 5 < y < 6. Then I use the last 171360 minutes as validation set. By using this subset, I am able to know the performance of one model within 20 minutes.\r\n\r\nAccording to the admin, the real test set has removed unseen place_id. But I didn't do so on my validation set. So my local map3 will be lower than public leaderboard.\r\n\r\n## strategy of hyperparameter selections\r\nn_cell_x, int\r\n number of cells divided in x for test and validation set\r\nn_cell_y, int\r\n number of cells divided in y for test and validation set\r\nx_border, float\r\n border on the edge of the cell for larger range of training set\r\ny_border, float\r\n border on the edge of the cell for larger range of training set\r\nth, int\r\n cutoff low frequency labels\r\nThose parameters as well as parameters of different models are considered as the hyperparameters. I write my own random grid search code to find the optimal hyperparameters of each classifier. This code will append all the hypermeters and corresponding local validation score to a record file. It also guarantee new generated hyperparameters will not duplicate with previous one in record. Thus, I can easily adjust the range and density of each hyperparameters in the selection process.\r\n\r\n## ensemble\r\nEach single model will predict top 10 place_id with probabilities for each row_id. The are stored in csv as (row_id, place_id, proba). This output file for a validation set is about 3.3Mb and 351Mb for test set. I ensemble probabilities of different models using python defaultdict and output the top 3 place_id as prediction. I will first ensemble the validation set to see which combination of models has better performance. Then, I will run an ensemble on the whole data set with this combination. The idea of finding the combination is:\r\n\r\n* Try to increase the diversity of the model. This includes different number of cells and different models.\r\n\r\n* Then I list model candidates in different category and sorted them by local map3 score.\r\n\r\n* Do greedy search to find the combination with highest local validation score.\r\n\r\n* My final submission is an unweighted sum of single models. I also tried to find weight by bayes optimization. There is some improvement on the 5th digit so I just ignore it.\r\n\r\n* Then run the ensemble on the whole dataset.\r\n\r\n## models in my ensemble\r\nMy final submission is a soft-voting of 13 single models, including k-nearest neighbors, random forest, extra-trees, gradient boosting trees, naive bayes, kernel density estimation.\r\n\r\n sklearn.neighbors.KNeighborsClassifier/knn_ps2_n1, local map3=0.504975\r\n \r\n sklearn.ensemble.RandomForestClassifier/rf_opt1, local map3=0.509508\r\n \r\n sklearn.ensemble.ExtraTreesClassifier/et_opt1, local map3=0.497024\r\n \r\n XGBoost/xgb0_n4, local map3=0.518815\r\n XGBoost/xgb0_n2, local map3=0.514897\r\n XGBoost/xgb0_n4_th13_border5, local map3=0.521114\r\n XGBoost/xgb150opt, local map3=0.523222\r\n XGBoost/xgb150opt2, local map3=0.523853\r\n \r\n sklearn.naive_bayes.GaussianNB/nb_xyat_weight1_n9, local map3=0.482681\r\n \r\n sklearn.neighbors.KernelDensity/kde_opt1, local map3=0.478387\r\n scipy.stats.gaussian_kde/kde_opt2c, local map3=0.491391\r\n scipy.stats.gaussian_kde/kde_opt3c, local map3=0.493110\r\n KDE-for-SciPy/kde_opt4, local map3=0.483254\r\n\r\n## Insights\r\nThe structure of this dataset is very interesting. My best XGBoost model only has max_depth=3, which is unusual for my experience in previous competitions. This indicates features are almost independent to each other in this dataset. This can be the reason why my Naive Bayes is only 4% worse than my knn.\r\n\r\nNaive Bayes is a first assumption of the data distribution. When using the sklearn.naive_bayes.GaussianNB, it also assumes the distribution of each feature in cell is unimodal. But doing a simple visualization of the x, y for given place_id will tell you that assumption is not very good. One way to improve the Naive Bayes is to use kernel density estimation (KDE) rather than one single gaussian to get the probability density function. I recommend this blog (http://www.mglerner.com/blog/?p=28) for nice explanation of KDE.\r\n\r\nThe performance of KDE is determined by the kernel and kernel parameters (bandwidth for gaussian kernel). **sklearn.neighbors.KernelDensity** provides you plenty of kernels to choose. My hyperparameter selection code told me gaussian and exponential are optimal choice while tophat, linear, cosine works badly. sklearn does not provide rule of thumb estimation of bandwidth, so the bandwidth is selected by cross-validation. I also tried the **scipy.stats.gaussian_kde** because it provides two common rule of thumb estimations of bandwidth by statistician, \"scott\" and \"silverman\". They can give fast estimation but are designed for unimodal distribution. In order to get better bandwidth estimation, I also used Daniel B Smith's implement of KDE (https://github.com/Daniel-B-Smith/KDE-for-SciPy). **KDE-for-SciPy** is based on this paper\r\n > Z. I. Botev, J. F. Grotowski, and D. P. Kroese. Kernel density\r\n > estimation via diffusion. The Annals of Statistics, 38(5):2916-2957, 2010.\r\n\r\nThis estimation is more time consuming but can give good estimation on multimodal distribution. My final submission include KDE from all those implements.\r\n\r\n-----------------\r\nI feel it is very important to have your own framework to do the model selections and ensemble. The quality of your own code determine how easy it is to tuning the model and test new idea. Thus, this competition is also a good programing exercise for me.\r\n\r\nMy code is on [GitHub](https://github.com/aikinogard/5th_place_solution_facebook_check_ins) now.\r\n\r\nPlease press \"helpful\" button if you like my post.\r\n\r\nThanks,\r\nLi", "votes": null }, { "id": "126378", "postDate": "07/08/2016 01:21:44", "content": "Thanks for sharing Li Li, and congratulations on such a fine finish and reaching your goal to become a Master!
\n\nWhat was your local map3 for your ensemble?
", "rawMarkdown": "Thanks for sharing Li Li, and congratulations on such a fine finish and reaching your goal to become a Master!\r\n\r\nWhat was your local map3 for your ensemble?", "votes": null }, { "id": "126379", "postDate": "07/08/2016 01:27:22", "content": "@Dave Hurst, thank you!\nmy best submission \npublic / private / local\n0.61194 / 0.61171 / 0.53810
", "rawMarkdown": "Dave Hurst, thank you!\r\nmy best submission \r\npublic / private / local\r\n0.61194 / 0.61171 / 0.53810", "votes": null }, { "id": "126385", "postDate": "07/08/2016 03:33:37", "content": "Wow, how do you overcome the “fearness” of local map3 being 0.07~0.08 smaller than the LB score. I am typically interested in how people select the local cv, and is it important at all for this competition.
", "rawMarkdown": "Wow, how do you overcome the “fearness” of local map3 being 0.07~0.08 smaller than the LB score. I am typically interested in how people select the local cv, and is it important at all for this competition.", "votes": null }, { "id": "126386", "postDate": "07/08/2016 03:51:21", "content": "@YijunMa, based on my experience in previous competitions, I don't worry about lower local score. Actually, if the local score is higher than the LB, I will have concern on overfitting. As I mentioned in this post, I didn't remove unseen place_id in the validation set. It means some of the place_id can never be predicted correctly. This effect lower the local score. Since I just use one validation set, I would rather call my local score as validation score instead of cv score.
\n\nIn my opinion, as long as the local score improvement is proportional to the LB improvement, I am safe.
", "rawMarkdown": "YijunMa, based on my experience in previous competitions, I don't worry about lower local score. Actually, if the local score is higher than the LB, I will have concern on overfitting. As I mentioned in this post, I didn't remove unseen place_id in the validation set. It means some of the place_id can never be predicted correctly. This effect lower the local score. Since I just use one validation set, I would rather call my local score as validation score instead of cv score.\r\n\r\nIn my opinion, as long as the local score improvement is proportional to the LB improvement, I am safe.", "votes": null }, { "id": "126388", "postDate": "07/08/2016 04:07:04", "content": "@Li Li, thanks for correction, cv -> validation.
\n\nFor me, it's just counter intuitive that the validation set have a such lower score than LB. My experience told me cleaning the unseen place_ids can improve about 0.02~0.04, which still cannot explain the phenomena.
\n\nAnother thing I noticed is that as my local score improve from 0.55 to 0.57, the LB score whould be just slightly better from 0.57 to 0.58. It's like suddenly don't know what to compare and believe.
", "rawMarkdown": "Li Li, thanks for correction, cv -> validation.\r\n\r\nFor me, it's just counter intuitive that the validation set have a such lower score than LB. My experience told me cleaning the unseen place_ids can improve about 0.02~0.04, which still cannot explain the phenomena.\r\n\r\nAnother thing I noticed is that as my local score improve from 0.55 to 0.57, the LB score whould be just slightly better from 0.57 to 0.58. It's like suddenly don't know what to compare and believe.", "votes": null }, { "id": "126391", "postDate": "07/08/2016 05:02:13", "content": "@YijunMa, How do you split your validation set? Is it by time? In my test, if you do random split instead of time split, you will get higher local score, but I don't think that is reliable.
", "rawMarkdown": "YijunMa, How do you split your validation set? Is it by time? In my test, if you do random split instead of time split, you will get higher local score, but I don't think that is reliable.", "votes": null }, { "id": "126393", "postDate": "07/08/2016 05:38:07", "content": "Of course split by time.
\n\nI didn't pick 5 < x < 6, 5 < y < 6. Sometime it is a random choose of 50 ~ 100 grids spread all over the place, and sometime it's the entire 10km*10km map. I did some exploration on this, but didn't find a good way. I end up using the entire map, and split the time based on the train:test ratio.
\n\nTurns out yours is a much faster and easier way and wouldn't influence too much on the final result. But it's hard for me at that time to persuade myself just using the center of the map.
", "rawMarkdown": "Of course split by time. \r\n\r\nI didn't pick 5 < x < 6, 5 < y < 6. Sometime it is a random choose of 50 ~ 100 grids spread all over the place, and sometime it's the entire 10km*10km map. I did some exploration on this, but didn't find a good way. I end up using the entire map, and split the time based on the train:test ratio.\r\n\r\nTurns out yours is a much faster and easier way and wouldn't influence too much on the final result. But it's hard for me at that time to persuade myself just using the center of the map.", "votes": null }, { "id": "126396", "postDate": "07/08/2016 05:50:00", "content": "The reason for me to fix my validation set in 5 < x < 6, 5 < y < 6 is because:
\n\nThe density of data in the 1/100 small area is close to the density of the whole dataset. If randomly choosing over all the place, the validation set will be sparser than the whole dataset. Some hyperparameters are density related. Random sampling over the whole dataset may not give you optimal hyperparameters on the whole dataset.
Keep staying in 5 < x < 6, 5 < y < 6 will allow me to compare different model and ensemble performance on the same page.
I really take advantage of this validation set for fast model selection and tuning. It also relies on the geographical natural of the data. I tried to split data in the Expedia competition, but fail.
", "rawMarkdown": "The reason for me to fix my validation set in 5 < x < 6, 5 < y < 6 is because:\r\n\r\n* The density of data in the 1/100 small area is close to the density of the whole dataset. If randomly choosing over all the place, the validation set will be **sparser** than the whole dataset. Some hyperparameters are density related. Random sampling over the whole dataset may not give you optimal hyperparameters on the whole dataset.\r\n\r\n* Keep staying in 5 < x < 6, 5 < y < 6 will allow me to compare different model and ensemble performance on the same page.\r\n\r\nI really take advantage of this validation set for fast model selection and tuning. It also relies on the geographical natural of the data. I tried to split data in the Expedia competition, but fail.", "votes": null }, { "id": "126402", "postDate": "07/08/2016 07:21:21", "content": "Congratulations Li Li ! And thanks for sharing the information on different KDE. I am just wondering if there is any specific reason why different KDE models in your ensemble give quite different performance. Are they of different kernels ?
", "rawMarkdown": "Congratulations Li Li ! And thanks for sharing the information on different KDE. I am just wondering if there is any specific reason why different KDE models in your ensemble give quite different performance. Are they of different kernels ?", "votes": null }, { "id": "126424", "postDate": "07/08/2016 09:30:51", "content": "@Li Li, my miniature model is x < 1, its CV (~0.060) and LB (~0.059) is very close. Thanks a lot for sharing, looking forward to learn the code, +1.
", "rawMarkdown": "Li Li, my miniature model is x < 1, its CV (~0.060) and LB (~0.059) is very close. Thanks a lot for sharing, looking forward to learn the code, +1.", "votes": null }, { "id": "126442", "postDate": "07/08/2016 13:30:45", "content": "@Rongting Zhang, thank you.\nThey are all gaussian kernel. I tested other kernel but they do not give good single model score.\nKDE's performance is related to the bandwidth. Different method give different estimation on bandwidth. That's why I have different performance with different KDE models. Because of the diversity, the ensemble of them works much better.
", "rawMarkdown": "Rongting Zhang, thank you.\r\nThey are all gaussian kernel. I tested other kernel but they do not give good single model score.\r\nKDE's performance is related to the bandwidth. Different method give different estimation on bandwidth. That's why I have different performance with different KDE models. Because of the diversity, the ensemble of them works much better.", "votes": null }, { "id": "126443", "postDate": "07/08/2016 13:31:50", "content": "@aldente, did you set a range for y in your miniature model?
", "rawMarkdown": "aldente, did you set a range for y in your miniature model?", "votes": null }, { "id": "126455", "postDate": "07/08/2016 15:06:41", "content": "Big Congrats! @LiLi
\n\nIs the source code going to be published? Thank you very much!
", "rawMarkdown": "Big Congrats! @LiLi\r\n\r\nIs the source code going to be published? Thank you very much!", "votes": null }, { "id": "126464", "postDate": "07/08/2016 16:07:07", "content": "@rcarson Thanks. I will publish the code later. Just busy on work now. I am a big fan of your microsoft malware competition. I hope we can have more chance to discuss and collaborate in the future.
", "rawMarkdown": "rcarson Thanks. I will publish the code later. Just busy on work now. I am a big fan of your microsoft malware competition. I hope we can have more chance to discuss and collaborate in the future.", "votes": null }, { "id": "126465", "postDate": "07/08/2016 16:10:47", "content": "[quote=Li Li;126464]
\n\n@rcarson Thanks. I will publish the code later. Just busy on work now. I am a big fan of your microsoft malware competition. I hope we can have more chance to discuss and collaborate in the future.
\n\n[/quote]
\n\n:D sure thing!!
", "rawMarkdown": "[quote=Li Li;126464]\r\n\r\n@rcarson Thanks. I will publish the code later. Just busy on work now. I am a big fan of your microsoft malware competition. I hope we can have more chance to discuss and collaborate in the future. \r\n\r\n[/quote]\r\n\r\n:D sure thing!!", "votes": null }, { "id": "126482", "postDate": "07/08/2016 19:33:46", "content": "[quote=Li Li;126443]
\n\n@aldente, did you set a range for y in your miniature model?
\n\n[/quote]
\n\nNope, I used all range of y, I stole it from a post in the forum by @Jared Turkewitz. I didn't know smaller range can do well ;) Thanks.
", "rawMarkdown": "[quote=Li Li;126443]\r\n\r\n@aldente, did you set a range for y in your miniature model?\r\n\r\n[/quote]\r\n\r\nNope, I used all range of y, I stole it from a post in the forum by @Jared Turkewitz. I didn't know smaller range can do well ;) Thanks.", "votes": null }, { "id": "126490", "postDate": "07/08/2016 20:39:49", "content": "@aldente, I think that is the reason.\nSince your validation set is 10 times larger than me, it should be more stable and closer to the LB.
", "rawMarkdown": "aldente, I think that is the reason.\r\nSince your validation set is 10 times larger than me, it should be more stable and closer to the LB.", "votes": null }, { "id": "126495", "postDate": "07/08/2016 21:12:21", "content": "Code is on GitHub now.
", "rawMarkdown": "Code is on [GitHub](https://github.com/aikinogard/5th_place_solution_facebook_check_ins) now.", "votes": null }, { "id": "126503", "postDate": "07/08/2016 23:13:48", "content": "@aldente, wouldn't x<1 have some boundary effect? I remember that the number of checkins drops when getting closer to the edge.
", "rawMarkdown": "aldente, wouldn't x<1 have some boundary effect? I remember that the number of checkins drops when getting closer to the edge.", "votes": null }, { "id": "126507", "postDate": "07/09/2016 00:38:08", "content": "@Li Li, right, however it needed longer time (while time is valuable in this case). Thank you for sharing the code, wonderful.
\n\n@YijunMa, yes it is, however in this "miniature" case we only need to compare CV and LB scores of the same small area. I only needed to get a rough CV-LB comparison, to find out whether my model is overfitting or not. Maybe the boundary effect affects when it is scaled to larger area, but again we only compare the same small area.
\n\nBesides, when I did it by considering big data and limited time (I started late), so I just chose what's shared in the forum, because it's already tested by poster (@Jared).
\n\nThanks for great sharing and discussions, this competition is indeed most engaged I've ever had ;)
", "rawMarkdown": "Li Li, right, however it needed longer time (while time is valuable in this case). Thank you for sharing the code, wonderful. \r\n\r\n@YijunMa, yes it is, however in this \"miniature\" case we only need to compare CV and LB scores of the same small area. I only needed to get a rough CV-LB comparison, to find out whether my model is overfitting or not. Maybe the boundary effect affects when it is scaled to larger area, but again we only compare the same small area. \r\n\r\nBesides, when I did it by considering big data and limited time (I started late), so I just chose what's shared in the forum, because it's already tested by poster (@Jared). \r\n\r\nThanks for great sharing and discussions, this competition is indeed most engaged I've ever had ;)", "votes": null }, { "id": "126511", "postDate": "07/09/2016 01:15:43", "content": "@Li Li That code is awesome. Thanks for sharing.
", "rawMarkdown": "Li Li That code is awesome. Thanks for sharing.", "votes": null }, { "id": "126516", "postDate": "07/09/2016 04:20:01", "content": "Hi Li Li, the way you use configuration and model_list is very flexible for massive parameter tuning. I really benefit a lot by reading your code. Thanks for sharing.
", "rawMarkdown": "Hi Li Li, the way you use configuration and model_list is very flexible for massive parameter tuning. I really benefit a lot by reading your code. Thanks for sharing.", "votes": null }, { "id": "126517", "postDate": "07/09/2016 04:26:32", "content": "@Rongcheng Lin @chu_cs_nmsu
\n\nI am glad you like my code.
", "rawMarkdown": "Rongcheng Lin @chu_cs_nmsu\r\n\r\nI am glad you like my code.", "votes": null }, { "id": "197812", "postDate": "06/30/2017 08:54:58", "content": "@LiLi thx for your sharing,i run your code on my PC python3, but the model xgb only get 0.053,i don't why, lookfowrd your help :)
", "rawMarkdown": "LiLi thx for your sharing,i run your code on my PC python3, but the model xgb only get 0.053,i don't why, lookfowrd your help :)", "votes": null } ], "comments": [ { "id": 126378, "author_name": "datadave", "author_url": "", "post_date": "07/08/2016 01:21:44", "content": "Thanks for sharing Li Li, and congratulations on such a fine finish and reaching your goal to become a Master!
\n\nWhat was your local map3 for your ensemble?
", "votes": null, "replies": [] }, { "id": 126379, "author_name": "aikinogard", "author_url": "", "post_date": "07/08/2016 01:27:22", "content": "@Dave Hurst, thank you!\nmy best submission \npublic / private / local\n0.61194 / 0.61171 / 0.53810
", "votes": null, "replies": [] }, { "id": 126385, "author_name": "ma350365879", "author_url": "", "post_date": "07/08/2016 03:33:37", "content": "Wow, how do you overcome the “fearness” of local map3 being 0.07~0.08 smaller than the LB score. I am typically interested in how people select the local cv, and is it important at all for this competition.
", "votes": null, "replies": [] }, { "id": 126386, "author_name": "aikinogard", "author_url": "", "post_date": "07/08/2016 03:51:21", "content": "@YijunMa, based on my experience in previous competitions, I don't worry about lower local score. Actually, if the local score is higher than the LB, I will have concern on overfitting. As I mentioned in this post, I didn't remove unseen place_id in the validation set. It means some of the place_id can never be predicted correctly. This effect lower the local score. Since I just use one validation set, I would rather call my local score as validation score instead of cv score.
\n\nIn my opinion, as long as the local score improvement is proportional to the LB improvement, I am safe.
", "votes": null, "replies": [] }, { "id": 126388, "author_name": "ma350365879", "author_url": "", "post_date": "07/08/2016 04:07:04", "content": "@Li Li, thanks for correction, cv -> validation.
\n\nFor me, it's just counter intuitive that the validation set have a such lower score than LB. My experience told me cleaning the unseen place_ids can improve about 0.02~0.04, which still cannot explain the phenomena.
\n\nAnother thing I noticed is that as my local score improve from 0.55 to 0.57, the LB score whould be just slightly better from 0.57 to 0.58. It's like suddenly don't know what to compare and believe.
", "votes": null, "replies": [] }, { "id": 126391, "author_name": "aikinogard", "author_url": "", "post_date": "07/08/2016 05:02:13", "content": "@YijunMa, How do you split your validation set? Is it by time? In my test, if you do random split instead of time split, you will get higher local score, but I don't think that is reliable.
", "votes": null, "replies": [] }, { "id": 126393, "author_name": "ma350365879", "author_url": "", "post_date": "07/08/2016 05:38:07", "content": "Of course split by time.
\n\nI didn't pick 5 < x < 6, 5 < y < 6. Sometime it is a random choose of 50 ~ 100 grids spread all over the place, and sometime it's the entire 10km*10km map. I did some exploration on this, but didn't find a good way. I end up using the entire map, and split the time based on the train:test ratio.
\n\nTurns out yours is a much faster and easier way and wouldn't influence too much on the final result. But it's hard for me at that time to persuade myself just using the center of the map.
", "votes": null, "replies": [] }, { "id": 126396, "author_name": "aikinogard", "author_url": "", "post_date": "07/08/2016 05:50:00", "content": "The reason for me to fix my validation set in 5 < x < 6, 5 < y < 6 is because:
\n\nThe density of data in the 1/100 small area is close to the density of the whole dataset. If randomly choosing over all the place, the validation set will be sparser than the whole dataset. Some hyperparameters are density related. Random sampling over the whole dataset may not give you optimal hyperparameters on the whole dataset.
Keep staying in 5 < x < 6, 5 < y < 6 will allow me to compare different model and ensemble performance on the same page.
I really take advantage of this validation set for fast model selection and tuning. It also relies on the geographical natural of the data. I tried to split data in the Expedia competition, but fail.
", "votes": null, "replies": [] }, { "id": 126402, "author_name": "tonyzhangrt", "author_url": "", "post_date": "07/08/2016 07:21:21", "content": "Congratulations Li Li ! And thanks for sharing the information on different KDE. I am just wondering if there is any specific reason why different KDE models in your ensemble give quite different performance. Are they of different kernels ?
", "votes": null, "replies": [] }, { "id": 126424, "author_name": "aldente", "author_url": "", "post_date": "07/08/2016 09:30:51", "content": "@Li Li, my miniature model is x < 1, its CV (~0.060) and LB (~0.059) is very close. Thanks a lot for sharing, looking forward to learn the code, +1.
", "votes": null, "replies": [] }, { "id": 126442, "author_name": "aikinogard", "author_url": "", "post_date": "07/08/2016 13:30:45", "content": "@Rongting Zhang, thank you.\nThey are all gaussian kernel. I tested other kernel but they do not give good single model score.\nKDE's performance is related to the bandwidth. Different method give different estimation on bandwidth. That's why I have different performance with different KDE models. Because of the diversity, the ensemble of them works much better.
", "votes": null, "replies": [] }, { "id": 126443, "author_name": "aikinogard", "author_url": "", "post_date": "07/08/2016 13:31:50", "content": "@aldente, did you set a range for y in your miniature model?
", "votes": null, "replies": [] }, { "id": 126455, "author_name": "jiweiliu", "author_url": "", "post_date": "07/08/2016 15:06:41", "content": "Big Congrats! @LiLi
\n\nIs the source code going to be published? Thank you very much!
", "votes": null, "replies": [] }, { "id": 126464, "author_name": "aikinogard", "author_url": "", "post_date": "07/08/2016 16:07:07", "content": "@rcarson Thanks. I will publish the code later. Just busy on work now. I am a big fan of your microsoft malware competition. I hope we can have more chance to discuss and collaborate in the future.
", "votes": null, "replies": [] }, { "id": 126465, "author_name": "jiweiliu", "author_url": "", "post_date": "07/08/2016 16:10:47", "content": "[quote=Li Li;126464]
\n\n@rcarson Thanks. I will publish the code later. Just busy on work now. I am a big fan of your microsoft malware competition. I hope we can have more chance to discuss and collaborate in the future.
\n\n[/quote]
\n\n:D sure thing!!
", "votes": null, "replies": [] }, { "id": 126482, "author_name": "aldente", "author_url": "", "post_date": "07/08/2016 19:33:46", "content": "[quote=Li Li;126443]
\n\n@aldente, did you set a range for y in your miniature model?
\n\n[/quote]
\n\nNope, I used all range of y, I stole it from a post in the forum by @Jared Turkewitz. I didn't know smaller range can do well ;) Thanks.
", "votes": null, "replies": [] }, { "id": 126490, "author_name": "aikinogard", "author_url": "", "post_date": "07/08/2016 20:39:49", "content": "@aldente, I think that is the reason.\nSince your validation set is 10 times larger than me, it should be more stable and closer to the LB.
", "votes": null, "replies": [] }, { "id": 126495, "author_name": "aikinogard", "author_url": "", "post_date": "07/08/2016 21:12:21", "content": "Code is on GitHub now.
", "votes": null, "replies": [] }, { "id": 126503, "author_name": "ma350365879", "author_url": "", "post_date": "07/08/2016 23:13:48", "content": "@aldente, wouldn't x<1 have some boundary effect? I remember that the number of checkins drops when getting closer to the edge.
", "votes": null, "replies": [] }, { "id": 126507, "author_name": "aldente", "author_url": "", "post_date": "07/09/2016 00:38:08", "content": "@Li Li, right, however it needed longer time (while time is valuable in this case). Thank you for sharing the code, wonderful.
\n\n@YijunMa, yes it is, however in this "miniature" case we only need to compare CV and LB scores of the same small area. I only needed to get a rough CV-LB comparison, to find out whether my model is overfitting or not. Maybe the boundary effect affects when it is scaled to larger area, but again we only compare the same small area.
\n\nBesides, when I did it by considering big data and limited time (I started late), so I just chose what's shared in the forum, because it's already tested by poster (@Jared).
\n\nThanks for great sharing and discussions, this competition is indeed most engaged I've ever had ;)
", "votes": null, "replies": [] }, { "id": 126511, "author_name": "phoenixl", "author_url": "", "post_date": "07/09/2016 01:15:43", "content": "@Li Li That code is awesome. Thanks for sharing.
", "votes": null, "replies": [] }, { "id": 126516, "author_name": "chu1990", "author_url": "", "post_date": "07/09/2016 04:20:01", "content": "Hi Li Li, the way you use configuration and model_list is very flexible for massive parameter tuning. I really benefit a lot by reading your code. Thanks for sharing.
", "votes": null, "replies": [] }, { "id": 126517, "author_name": "aikinogard", "author_url": "", "post_date": "07/09/2016 04:26:32", "content": "@Rongcheng Lin @chu_cs_nmsu
\n\nI am glad you like my code.
", "votes": null, "replies": [] }, { "id": 197812, "author_name": "w625009392", "author_url": "", "post_date": "06/30/2017 08:54:58", "content": "@LiLi thx for your sharing,i run your code on my PC python3, but the model xgb only get 0.053,i don't why, lookfowrd your help :)
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "126376": "This is an amazing competition and thank all the discussion and scripts made by fellow kaggler. I joined two weeks before the end of this competition. The main reason for me to start this competition is because:\r\n\r\n* The features provided in this competition are very simple (only x, y, time, accuracy). But the data set is big and contains more than 100,000 labels. This gives me chance to think about how to divide the problem into small sub-problems.\r\n\r\n* Get into top 10 in order to reach master tier.\r\n\r\nI learned a lot during this competition. Especially in the final week, the competition become quite intense among ranking 5th to 10th. The fear of fall out of top 10 making me work very hard on trying and learning new method, including naive bayes, support vector machine and kernel density estimation. Special thank to @Qingchen, it is always very educative to talk with him.\r\n\r\n## validation set\r\nThe size of the data set is very large. In order to use my time efficiently, my model and hyperparameter selection are done on 1/100 of the total data set. It is data in range 5 < x < 6, 5 < y < 6. Then I use the last 171360 minutes as validation set. By using this subset, I am able to know the performance of one model within 20 minutes.\r\n\r\nAccording to the admin, the real test set has removed unseen place_id. But I didn't do so on my validation set. So my local map3 will be lower than public leaderboard.\r\n\r\n## strategy of hyperparameter selections\r\nn_cell_x, int\r\n number of cells divided in x for test and validation set\r\nn_cell_y, int\r\n number of cells divided in y for test and validation set\r\nx_border, float\r\n border on the edge of the cell for larger range of training set\r\ny_border, float\r\n border on the edge of the cell for larger range of training set\r\nth, int\r\n cutoff low frequency labels\r\nThose parameters as well as parameters of different models are considered as the hyperparameters. I write my own random grid search code to find the optimal hyperparameters of each classifier. This code will append all the hypermeters and corresponding local validation score to a record file. It also guarantee new generated hyperparameters will not duplicate with previous one in record. Thus, I can easily adjust the range and density of each hyperparameters in the selection process.\r\n\r\n## ensemble\r\nEach single model will predict top 10 place_id with probabilities for each row_id. The are stored in csv as (row_id, place_id, proba). This output file for a validation set is about 3.3Mb and 351Mb for test set. I ensemble probabilities of different models using python defaultdict and output the top 3 place_id as prediction. I will first ensemble the validation set to see which combination of models has better performance. Then, I will run an ensemble on the whole data set with this combination. The idea of finding the combination is:\r\n\r\n* Try to increase the diversity of the model. This includes different number of cells and different models.\r\n\r\n* Then I list model candidates in different category and sorted them by local map3 score.\r\n\r\n* Do greedy search to find the combination with highest local validation score.\r\n\r\n* My final submission is an unweighted sum of single models. I also tried to find weight by bayes optimization. There is some improvement on the 5th digit so I just ignore it.\r\n\r\n* Then run the ensemble on the whole dataset.\r\n\r\n## models in my ensemble\r\nMy final submission is a soft-voting of 13 single models, including k-nearest neighbors, random forest, extra-trees, gradient boosting trees, naive bayes, kernel density estimation.\r\n\r\n sklearn.neighbors.KNeighborsClassifier/knn_ps2_n1, local map3=0.504975\r\n \r\n sklearn.ensemble.RandomForestClassifier/rf_opt1, local map3=0.509508\r\n \r\n sklearn.ensemble.ExtraTreesClassifier/et_opt1, local map3=0.497024\r\n \r\n XGBoost/xgb0_n4, local map3=0.518815\r\n XGBoost/xgb0_n2, local map3=0.514897\r\n XGBoost/xgb0_n4_th13_border5, local map3=0.521114\r\n XGBoost/xgb150opt, local map3=0.523222\r\n XGBoost/xgb150opt2, local map3=0.523853\r\n \r\n sklearn.naive_bayes.GaussianNB/nb_xyat_weight1_n9, local map3=0.482681\r\n \r\n sklearn.neighbors.KernelDensity/kde_opt1, local map3=0.478387\r\n scipy.stats.gaussian_kde/kde_opt2c, local map3=0.491391\r\n scipy.stats.gaussian_kde/kde_opt3c, local map3=0.493110\r\n KDE-for-SciPy/kde_opt4, local map3=0.483254\r\n\r\n## Insights\r\nThe structure of this dataset is very interesting. My best XGBoost model only has max_depth=3, which is unusual for my experience in previous competitions. This indicates features are almost independent to each other in this dataset. This can be the reason why my Naive Bayes is only 4% worse than my knn.\r\n\r\nNaive Bayes is a first assumption of the data distribution. When using the sklearn.naive_bayes.GaussianNB, it also assumes the distribution of each feature in cell is unimodal. But doing a simple visualization of the x, y for given place_id will tell you that assumption is not very good. One way to improve the Naive Bayes is to use kernel density estimation (KDE) rather than one single gaussian to get the probability density function. I recommend this blog (http://www.mglerner.com/blog/?p=28) for nice explanation of KDE.\r\n\r\nThe performance of KDE is determined by the kernel and kernel parameters (bandwidth for gaussian kernel). **sklearn.neighbors.KernelDensity** provides you plenty of kernels to choose. My hyperparameter selection code told me gaussian and exponential are optimal choice while tophat, linear, cosine works badly. sklearn does not provide rule of thumb estimation of bandwidth, so the bandwidth is selected by cross-validation. I also tried the **scipy.stats.gaussian_kde** because it provides two common rule of thumb estimations of bandwidth by statistician, \"scott\" and \"silverman\". They can give fast estimation but are designed for unimodal distribution. In order to get better bandwidth estimation, I also used Daniel B Smith's implement of KDE (https://github.com/Daniel-B-Smith/KDE-for-SciPy). **KDE-for-SciPy** is based on this paper\r\n > Z. I. Botev, J. F. Grotowski, and D. P. Kroese. Kernel density\r\n > estimation via diffusion. The Annals of Statistics, 38(5):2916-2957, 2010.\r\n\r\nThis estimation is more time consuming but can give good estimation on multimodal distribution. My final submission include KDE from all those implements.\r\n\r\n-----------------\r\nI feel it is very important to have your own framework to do the model selections and ensemble. The quality of your own code determine how easy it is to tuning the model and test new idea. Thus, this competition is also a good programing exercise for me.\r\n\r\nMy code is on [GitHub](https://github.com/aikinogard/5th_place_solution_facebook_check_ins) now.\r\n\r\nPlease press \"helpful\" button if you like my post.\r\n\r\nThanks,\r\nLi", "126378": "Thanks for sharing Li Li, and congratulations on such a fine finish and reaching your goal to become a Master!\r\n\r\nWhat was your local map3 for your ensemble?", "126379": "Dave Hurst, thank you!\r\nmy best submission \r\npublic / private / local\r\n0.61194 / 0.61171 / 0.53810", "126385": "Wow, how do you overcome the “fearness” of local map3 being 0.07~0.08 smaller than the LB score. I am typically interested in how people select the local cv, and is it important at all for this competition.", "126386": "YijunMa, based on my experience in previous competitions, I don't worry about lower local score. Actually, if the local score is higher than the LB, I will have concern on overfitting. As I mentioned in this post, I didn't remove unseen place_id in the validation set. It means some of the place_id can never be predicted correctly. This effect lower the local score. Since I just use one validation set, I would rather call my local score as validation score instead of cv score.\r\n\r\nIn my opinion, as long as the local score improvement is proportional to the LB improvement, I am safe.", "126388": "Li Li, thanks for correction, cv -> validation.\r\n\r\nFor me, it's just counter intuitive that the validation set have a such lower score than LB. My experience told me cleaning the unseen place_ids can improve about 0.02~0.04, which still cannot explain the phenomena.\r\n\r\nAnother thing I noticed is that as my local score improve from 0.55 to 0.57, the LB score whould be just slightly better from 0.57 to 0.58. It's like suddenly don't know what to compare and believe.", "126391": "YijunMa, How do you split your validation set? Is it by time? In my test, if you do random split instead of time split, you will get higher local score, but I don't think that is reliable.", "126393": "Of course split by time. \r\n\r\nI didn't pick 5 < x < 6, 5 < y < 6. Sometime it is a random choose of 50 ~ 100 grids spread all over the place, and sometime it's the entire 10km*10km map. I did some exploration on this, but didn't find a good way. I end up using the entire map, and split the time based on the train:test ratio.\r\n\r\nTurns out yours is a much faster and easier way and wouldn't influence too much on the final result. But it's hard for me at that time to persuade myself just using the center of the map.", "126396": "The reason for me to fix my validation set in 5 < x < 6, 5 < y < 6 is because:\r\n\r\n* The density of data in the 1/100 small area is close to the density of the whole dataset. If randomly choosing over all the place, the validation set will be **sparser** than the whole dataset. Some hyperparameters are density related. Random sampling over the whole dataset may not give you optimal hyperparameters on the whole dataset.\r\n\r\n* Keep staying in 5 < x < 6, 5 < y < 6 will allow me to compare different model and ensemble performance on the same page.\r\n\r\nI really take advantage of this validation set for fast model selection and tuning. It also relies on the geographical natural of the data. I tried to split data in the Expedia competition, but fail.", "126402": "Congratulations Li Li ! And thanks for sharing the information on different KDE. I am just wondering if there is any specific reason why different KDE models in your ensemble give quite different performance. Are they of different kernels ?", "126424": "Li Li, my miniature model is x < 1, its CV (~0.060) and LB (~0.059) is very close. Thanks a lot for sharing, looking forward to learn the code, +1.", "126442": "Rongting Zhang, thank you.\r\nThey are all gaussian kernel. I tested other kernel but they do not give good single model score.\r\nKDE's performance is related to the bandwidth. Different method give different estimation on bandwidth. That's why I have different performance with different KDE models. Because of the diversity, the ensemble of them works much better.", "126443": "aldente, did you set a range for y in your miniature model?", "126455": "Big Congrats! @LiLi\r\n\r\nIs the source code going to be published? Thank you very much!", "126464": "rcarson Thanks. I will publish the code later. Just busy on work now. I am a big fan of your microsoft malware competition. I hope we can have more chance to discuss and collaborate in the future.", "126465": "[quote=Li Li;126464]\r\n\r\n@rcarson Thanks. I will publish the code later. Just busy on work now. I am a big fan of your microsoft malware competition. I hope we can have more chance to discuss and collaborate in the future. \r\n\r\n[/quote]\r\n\r\n:D sure thing!!", "126482": "[quote=Li Li;126443]\r\n\r\n@aldente, did you set a range for y in your miniature model?\r\n\r\n[/quote]\r\n\r\nNope, I used all range of y, I stole it from a post in the forum by @Jared Turkewitz. I didn't know smaller range can do well ;) Thanks.", "126490": "aldente, I think that is the reason.\r\nSince your validation set is 10 times larger than me, it should be more stable and closer to the LB.", "126495": "Code is on [GitHub](https://github.com/aikinogard/5th_place_solution_facebook_check_ins) now.", "126503": "aldente, wouldn't x<1 have some boundary effect? I remember that the number of checkins drops when getting closer to the edge.", "126507": "Li Li, right, however it needed longer time (while time is valuable in this case). Thank you for sharing the code, wonderful. \r\n\r\n@YijunMa, yes it is, however in this \"miniature\" case we only need to compare CV and LB scores of the same small area. I only needed to get a rough CV-LB comparison, to find out whether my model is overfitting or not. Maybe the boundary effect affects when it is scaled to larger area, but again we only compare the same small area. \r\n\r\nBesides, when I did it by considering big data and limited time (I started late), so I just chose what's shared in the forum, because it's already tested by poster (@Jared). \r\n\r\nThanks for great sharing and discussions, this competition is indeed most engaged I've ever had ;)", "126511": "Li Li That code is awesome. Thanks for sharing.", "126516": "Hi Li Li, the way you use configuration and model_list is very flexible for massive parameter tuning. I really benefit a lot by reading your code. Thanks for sharing.", "126517": "Rongcheng Lin @chu_cs_nmsu\r\n\r\nI am glad you like my code.", "197812": "LiLi thx for your sharing,i run your code on my PC python3, but the model xgb only get 0.053,i don't why, lookfowrd your help :)" }, "source": "meta" }