First of all I would like to thank all participants for the amazing interaction. It was my first serious competition attempt and I was extremely impressed with the expert code sharing and forum discussions. Special thanks go to @AlexandruPapiu for his interactive random forest analysis and @msjgriffiths for his exploratory data analysis script. In the end, @Markus was indeed very close and I look forward to find out if he followed a similar approach.
\n\nAlso, thank you very much to everyone at Facebook who contributed to this challenging analysis task and to Kaggle for hosting the competition.
\n\nEDIT: Link to GitHub of the cleaned code. The extended story is now also available, please feel free to ask further questions!
\n\nI started my effort by inspecting many aspects of the data interactively. The published shiny app was the result of this stage. My major finding was the absence of a clear spatial (x-y) pattern in all researched relations. I also noted a high level of noise and variability in the patterns. Consequently I decided to use a single overall model rather than many local grid based models. This approach allowed me to turn the problem into a feature engineering competition and consider many features (430 in the final model) and use a two stage model. Two stages of candidate selection are performed in order to narrow down the number of considered classes in the first level models (all XGBoost). I intended to use a blend of models in the second tier but ended up using only XGBoost models. I will discuss the problem definition, candidate selection, the major features and the two model layers below. Further details on the steps and data handling will be provided in a blog post, hopefully this weekend. The code is now available on GitHub (all in R). This post is intended to share the main ideas and will either skip or simplify less important aspects of the analysis.
\n\nProblem definition\nAt both model stages I consider the problem as a binary classification problem for all selected candidates: model the probability that an observation belongs to each of the candidates.
\n\nThe local train and validation data was chosen as the last time part of the given train data such that the local train/validation to the local non train/validation data has a similar proportional size as the given test data to the given train data. The first part of the given train data was used to extract place summary features and the second part of the given train data, the local train/validation data, was used to build and validate the models. The entire given train data is used to calculate the place summary features which are used to generate test predictions.
\n\nCandidate selection\nThe first candidate selection step reduces the number of potential classes from >100K to 100 by considering nearest neighbors of the observations. I considered the neighbor counts of the 2500 nearest neighbors where y variations are 2.5 times more important than x variations. Ties in the neighbor counts are resolved by the mean time difference since the observations (keep most recent place_ids).
\n\nThe second candidate selection step reduces the number of potential classes from 100 to 20 by using the predictions of an average blend of first level models. These first level model predictions are not used as features in the second level models.
\n\nMajor features\nThe major share of my 430 features is based on nearest neighbors features. I counted the neighbors for different Ks (1, 5, 10, 20, 50, 100, 250, 500, 1000, 2500) and different x-y ratio constants (1, 2.5, 4, 5.5, 7, 12, 30) and also included the mean time difference for all 70 combinations. 10 more features were added by considering the distance between the Kth features and the observations for a ratio constant of 2.5 (measures the spatial density). 40 more features were added around the most significant features (K around 100 and a distance constant around 2.5) for the neighbor counts and 40 more for the distance to these neighbors.
\n\nI further added some additional features that consider the (smoothed) grid densities.
\n\nThe second largest share of the features set belongs to time features. Here I converted all time period counts to period density counts in order to handle the two drops in the time frequency. Periods include 27 two-week periods prior to the end of the train data and 27 1-week periods prior to the end of the train data. I also included features that look at the two-week densities looking back between 75 and 1 weeks from the observations. Additional features were added for the clear yearly pattern of some places.
\n\nHour, day and week features were calculated using the historical densities with and without cyclical smoothing and several relaxation constants (relaxed based on the historical count)
\n\nOther time features include extrapolated weekly densities using various time series models.
\n\nAccuracy was very interesting and I noticed that the location of the observations is sometimes related to the accuracy group. So... I added more features that were likely to capture this pattern. Accuracy was split in three groups: 1-44; 45-84 and 85-Inf. See the second plot for the median deviation of observations from their median as a function of time and accuracy.
\n\nI also calculated a bunch of robust Z scores for anything I could think of (x-median(x))/mad(x)
\n\nAll features are rescaled if needed in order to result in similar interpretations for the train and test features.
\n\nFirst level models\n100 base learners were fitted on different random parts of the training period. Deep trees gave me the best results here (depth 11) and the eta constant was set to (11 or 12)/500 for 500 rounds. Colsampling also helped (0.6) and subsampling the observations (0.5) did not hurt but of course resulted in a fitting speed increase. I included either all 430 features or a random pick of the ordered features by importance in a desirable feature count range (100-285 and 180-240).
\n\nSecond level models\n30 xgboost models were learnt to combine the predictions of the 100 first level models + 21 additional base features. The final predicted probability is the average of those 30 models. The predicted place ids are based on the order of the average prediction. Ties are resolved by considering the predicted probability of the second candidate selection step.
\n\nPlease feel free to ask questions or suggest improvements!
", "messages": [ { "id": 126233, "postDate": "2016-07-07T04:39:07.060Z", "content": "First of all I would like to thank all participants for the amazing interaction. It was my first serious competition attempt and I was extremely impressed with the expert code sharing and forum discussions. Special thanks go to @AlexandruPapiu for his interactive random forest analysis and @msjgriffiths for his exploratory data analysis script. In the end, @Markus was indeed very close and I look forward to find out if he followed a similar approach.
\n\nAlso, thank you very much to everyone at Facebook who contributed to this challenging analysis task and to Kaggle for hosting the competition.
\n\nEDIT: Link to GitHub of the cleaned code. The extended story is now also available, please feel free to ask further questions!
\n\nI started my effort by inspecting many aspects of the data interactively. The published shiny app was the result of this stage. My major finding was the absence of a clear spatial (x-y) pattern in all researched relations. I also noted a high level of noise and variability in the patterns. Consequently I decided to use a single overall model rather than many local grid based models. This approach allowed me to turn the problem into a feature engineering competition and consider many features (430 in the final model) and use a two stage model. Two stages of candidate selection are performed in order to narrow down the number of considered classes in the first level models (all XGBoost). I intended to use a blend of models in the second tier but ended up using only XGBoost models. I will discuss the problem definition, candidate selection, the major features and the two model layers below. Further details on the steps and data handling will be provided in a blog post, hopefully this weekend. The code is now available on GitHub (all in R). This post is intended to share the main ideas and will either skip or simplify less important aspects of the analysis.
\n\nProblem definition\nAt both model stages I consider the problem as a binary classification problem for all selected candidates: model the probability that an observation belongs to each of the candidates.
\n\nThe local train and validation data was chosen as the last time part of the given train data such that the local train/validation to the local non train/validation data has a similar proportional size as the given test data to the given train data. The first part of the given train data was used to extract place summary features and the second part of the given train data, the local train/validation data, was used to build and validate the models. The entire given train data is used to calculate the place summary features which are used to generate test predictions.
\n\nCandidate selection\nThe first candidate selection step reduces the number of potential classes from >100K to 100 by considering nearest neighbors of the observations. I considered the neighbor counts of the 2500 nearest neighbors where y variations are 2.5 times more important than x variations. Ties in the neighbor counts are resolved by the mean time difference since the observations (keep most recent place_ids).
\n\nThe second candidate selection step reduces the number of potential classes from 100 to 20 by using the predictions of an average blend of first level models. These first level model predictions are not used as features in the second level models.
\n\nMajor features\nThe major share of my 430 features is based on nearest neighbors features. I counted the neighbors for different Ks (1, 5, 10, 20, 50, 100, 250, 500, 1000, 2500) and different x-y ratio constants (1, 2.5, 4, 5.5, 7, 12, 30) and also included the mean time difference for all 70 combinations. 10 more features were added by considering the distance between the Kth features and the observations for a ratio constant of 2.5 (measures the spatial density). 40 more features were added around the most significant features (K around 100 and a distance constant around 2.5) for the neighbor counts and 40 more for the distance to these neighbors.
\n\nI further added some additional features that consider the (smoothed) grid densities.
\n\nThe second largest share of the features set belongs to time features. Here I converted all time period counts to period density counts in order to handle the two drops in the time frequency. Periods include 27 two-week periods prior to the end of the train data and 27 1-week periods prior to the end of the train data. I also included features that look at the two-week densities looking back between 75 and 1 weeks from the observations. Additional features were added for the clear yearly pattern of some places.
\n\nHour, day and week features were calculated using the historical densities with and without cyclical smoothing and several relaxation constants (relaxed based on the historical count)
\n\nOther time features include extrapolated weekly densities using various time series models.
\n\nAccuracy was very interesting and I noticed that the location of the observations is sometimes related to the accuracy group. So... I added more features that were likely to capture this pattern. Accuracy was split in three groups: 1-44; 45-84 and 85-Inf. See the second plot for the median deviation of observations from their median as a function of time and accuracy.
\n\nI also calculated a bunch of robust Z scores for anything I could think of (x-median(x))/mad(x)
\n\nAll features are rescaled if needed in order to result in similar interpretations for the train and test features.
\n\nFirst level models\n100 base learners were fitted on different random parts of the training period. Deep trees gave me the best results here (depth 11) and the eta constant was set to (11 or 12)/500 for 500 rounds. Colsampling also helped (0.6) and subsampling the observations (0.5) did not hurt but of course resulted in a fitting speed increase. I included either all 430 features or a random pick of the ordered features by importance in a desirable feature count range (100-285 and 180-240).
\n\nSecond level models\n30 xgboost models were learnt to combine the predictions of the 100 first level models + 21 additional base features. The final predicted probability is the average of those 30 models. The predicted place ids are based on the order of the average prediction. Ties are resolved by considering the predicted probability of the second candidate selection step.
\n\nPlease feel free to ask questions or suggest improvements!
", "rawMarkdown": "First of all I would like to thank all participants for the amazing interaction. It was my first serious competition attempt and I was extremely impressed with the expert code sharing and forum discussions. Special thanks go to @AlexandruPapiu for his interactive random forest analysis and @msjgriffiths for his exploratory data analysis script. In the end, @Markus was indeed very close and I look forward to find out if he followed a similar approach.\r\n\r\nAlso, thank you very much to everyone at Facebook who contributed to this challenging analysis task and to Kaggle for hosting the competition.\r\n\r\nEDIT: Link to [GitHub ][1]of the cleaned code. The [extended story][2] is now also available, please feel free to ask further questions!\r\n\r\n----------------------------------------------------------------------\r\n\r\nI started my effort by inspecting many aspects of the data interactively. The published shiny [app][3] was the result of this stage. My major finding was the absence of a clear spatial (x-y) pattern in all researched relations. I also noted a high level of noise and variability in the patterns. Consequently I decided to use a single overall model rather than many local grid based models. This approach allowed me to turn the problem into a feature engineering competition and consider many features (430 in the final model) and use a two stage model. Two stages of candidate selection are performed in order to narrow down the number of considered classes in the first level models (all XGBoost). I intended to use a blend of models in the second tier but ended up using only XGBoost models. I will discuss the problem definition, candidate selection, the major features and the two model layers below. Further details on the steps and data handling will be provided in a blog post, hopefully this weekend. The code is now available on [GitHub][4] (all in R). This post is intended to share the main ideas and will either skip or simplify less important aspects of the analysis.\r\n\r\n**Problem definition**\r\nAt both model stages I consider the problem as a binary classification problem for all selected candidates: model the probability that an observation belongs to each of the candidates. \r\n\r\nThe local train and validation data was chosen as the last time part of the given train data such that the local train/validation to the local non train/validation data has a similar proportional size as the given test data to the given train data. The first part of the given train data was used to extract place summary features and the second part of the given train data, the local train/validation data, was used to build and validate the models. The entire given train data is used to calculate the place summary features which are used to generate test predictions.\r\n\r\n**Candidate selection**\r\nThe first candidate selection step reduces the number of potential classes from >100K to 100 by considering nearest neighbors of the observations. I considered the neighbor counts of the 2500 nearest neighbors where y variations are 2.5 times more important than x variations. Ties in the neighbor counts are resolved by the mean time difference since the observations (keep most recent place_ids).\r\n\r\nThe second candidate selection step reduces the number of potential classes from 100 to 20 by using the predictions of an average blend of first level models. These first level model predictions are not used as features in the second level models.\r\n\r\n**Major features**\r\nThe major share of my 430 features is based on nearest neighbors features. I counted the neighbors for different Ks (1, 5, 10, 20, 50, 100, 250, 500, 1000, 2500) and different x-y ratio constants (1, 2.5, 4, 5.5, 7, 12, 30) and also included the mean time difference for all 70 combinations. 10 more features were added by considering the distance between the Kth features and the observations for a ratio constant of 2.5 (measures the spatial density). 40 more features were added around the most significant features (K around 100 and a distance constant around 2.5) for the neighbor counts and 40 more for the distance to these neighbors.\r\n\r\nI further added some additional features that consider the (smoothed) grid densities.\r\n\r\nThe second largest share of the features set belongs to time features. Here I converted all time period counts to period density counts in order to handle the two drops in the time frequency. Periods include 27 two-week periods prior to the end of the train data and 27 1-week periods prior to the end of the train data. I also included features that look at the two-week densities looking back between 75 and 1 weeks from the observations. Additional features were added for the clear yearly pattern of some places.\r\n\r\nHour, day and week features were calculated using the historical densities with and without cyclical smoothing and several relaxation constants (relaxed based on the historical count)\r\n\r\nOther time features include extrapolated weekly densities using various time series models.\r\n\r\nAccuracy was very interesting and I noticed that the location of the observations is sometimes related to the accuracy group. So... I added more features that were likely to capture this pattern. Accuracy was split in three groups: 1-44; 45-84 and 85-Inf. See the second plot for the median deviation of observations from their median as a function of time and accuracy.\r\n\r\nI also calculated a bunch of robust Z scores for anything I could think of (x-median(x))/mad(x)\r\n\r\nAll features are rescaled if needed in order to result in similar interpretations for the train and test features.\r\n\r\n**First level models**\r\n100 base learners were fitted on different random parts of the training period. Deep trees gave me the best results here (depth 11) and the eta constant was set to (11 or 12)/500 for 500 rounds. Colsampling also helped (0.6) and subsampling the observations (0.5) did not hurt but of course resulted in a fitting speed increase. I included either all 430 features or a random pick of the ordered features by importance in a desirable feature count range (100-285 and 180-240).\r\n\r\n**Second level models**\r\n30 xgboost models were learnt to combine the predictions of the 100 first level models + 21 additional base features. The final predicted probability is the average of those 30 models. The predicted place ids are based on the order of the average prediction. Ties are resolved by considering the predicted probability of the second candidate selection step.\r\n\r\nPlease feel free to ask questions or suggest improvements!\r\n\r\n\r\n [1]: https://github.com/ttvand/Facebook-V\r\n [2]: https://ttvand.github.io/Winning-approach-of-the-Facebook-V-Kaggle-competition/\r\n [3]: https://www.kaggle.com/c/facebook-v-predicting-check-ins/forums/t/21365/interactive-exploratory-analysis-using-a-shiny-app\r\n [4]: https://github.com/ttvand/Facebook-V", "votes": 64 }, { "id": 126683, "postDate": "2016-07-11T20:18:52.740Z", "content": "The extended story is now available, please feel free to ask further questions!
", "rawMarkdown": "The [extended story][1] is now available, please feel free to ask further questions!\r\n\r\n\r\n [1]: https://ttvand.github.io/Winning-approach-of-the-Facebook-V-Kaggle-competition/", "votes": 5 }, { "id": 126345, "postDate": "2016-07-07T20:39:05.867Z", "content": "The cleaned code is now available on GitHub!
", "rawMarkdown": "The cleaned code is now available on [GitHub][1]!\r\n\r\n\r\n [1]: https://github.com/ttvand/Facebook-V", "votes": 6 }, { "id": 126504, "postDate": "2016-07-08T23:25:30.347Z", "content": "It is very clear explanation, thanks for attending to write this up. Indeed this modeling takes a lot of cpu time but the result is great. One more question just occurred to me is that, I know you have tried different models for the three groups of models and xgb turned out yielding the best results, but have your tried mixing other models with xgb in any of the stages? My surmise is that mixing different models could add to the randomness of the modeling results and might end up voting to a higher score.
", "rawMarkdown": "It is very clear explanation, thanks for attending to write this up. Indeed this modeling takes a lot of cpu time but the result is great. One more question just occurred to me is that, I know you have tried different models for the three groups of models and xgb turned out yielding the best results, but have your tried mixing other models with xgb in any of the stages? My surmise is that mixing different models could add to the randomness of the modeling results and might end up voting to a higher score.", "votes": 3 }, { "id": 126470, "postDate": "2016-07-08T17:35:57.300Z", "content": "Hi Zhenjie Zhang, thank you for your comments. I understand your confusion and will try to clarify your concerns.
\n\nFirst of all IdHi Zhenjie Zhang, thank you for your comments. I understand your confusion and will try to clarify your concerns.
\n\nFirst of all I’ll try to refine my problem definition. Most other binary classification approaches I have studied on the forum model the conditional probability P(place_id|data) where data relates to x, y, time and accuracy directly or through Bayes’s theorem by translating it to P(data|place_id) * P(place_id) / P(data). The last category of approaches varies in how they calculate P(data|place_id) and P(place_id) since P(data) is the same for all place_ids for a given observation. My approach models P(place_match|data) directly for the considered place candidates. This approach does not require the use of prior probabilities. Place_match is the binary outcome variable and I order the places based on P(place_match|data).
\n\nAs described in the “Instructions” file on the GitHub link, there are three groups of models that are generated (all XGBoost)
\n\n - Second candidate selection models (15 models)\n - First level learners (100 models)\n - Second level learners (30 models)\nAll models are trained on different random batches of the train data set (split after cutoff time 587158) and span the entire time range 587158-786239.\nThe confusion arises from the fact that the first two groups are generated using the same model approaches (first level XGBoost models). The only difference between the two groups is the number of considered candidates for each observation and the number of considered features. I used 100 candidates as input to the “Second candidate selection” models, you got that part right. The 15 Second candidate selection models all generate a predicted probability of P(place_match|data), I average those to select the top20 candidates in the second candidate selection step. These 'predicted probabilities' are indeed likelihoods.
\n\nThe 100 first level learnes then have 20 candidates for each observation. The predicted probabilities of P(place_match|data) of the 100 first level learners then serve as features to the second level learners along with 21 additional first level features that were selected manually. The second level learners again try to model P(place_match|data).
\n\nDoes this answer handle all your questions?
", "rawMarkdown": "Hi Zhenjie Zhang, thank you for your comments. I understand your confusion and will try to clarify your concerns.\n \nFirst of all IdHi Zhenjie Zhang, thank you for your comments. I understand your confusion and will try to clarify your concerns.\n \nFirst of all I’ll try to refine my problem definition. Most other binary classification approaches I have studied on the forum model the conditional probability P(place\\_id|data) where data relates to x, y, time and accuracy directly or through Bayes’s theorem by translating it to P(data|place\\_id) * P(place\\_id) / P(data). The last category of approaches varies in how they calculate P(data|place\\_id) and P(place\\_id) since P(data) is the same for all place\\_ids for a given observation. My approach models P(place\\_match|data) directly for the considered place candidates. This approach does not require the use of prior probabilities. Place_match is the binary outcome variable and I order the places based on P(place\\_match|data).\n \nAs described in the “Instructions” file on the GitHub link, there are three groups of models that are generated (all XGBoost)\n\n - Second candidate selection models (15 models)\n - First level learners (100 models)\n - Second level learners (30 models)\n\n \nAll models are trained on different random batches of the train data set (split after cutoff time 587158) and span the entire time range 587158-786239.\nThe confusion arises from the fact that the first two groups are generated using the same model approaches (first level XGBoost models). The only difference between the two groups is the number of considered candidates for each observation and the number of considered features. I used 100 candidates as input to the “Second candidate selection” models, you got that part right. The 15 Second candidate selection models all generate a predicted probability of P(place\\_match|data), I average those to select the top20 candidates in the second candidate selection step. These 'predicted probabilities' are indeed likelihoods.\n \nThe 100 first level learnes then have 20 candidates for each observation. The predicted probabilities of P(place_match|data) of the 100 first level learners then serve as features to the second level learners along with 21 additional first level features that were selected manually. The second level learners again try to model P(place\\_match|data).\n\nDoes this answer handle all your questions?", "votes": 3 }, { "id": 126314, "postDate": "2016-07-07T16:36:33.013Z", "content": "@Gohm - You are right at your interpretation of my approach. I did not perform any downsampling in my final approach since all zeros (not a match between the candidate and true match) contain valuable information. XGBoost is able to handle unbalanced data quite well in my experience. I did however consider to omit observations that didn't contain the true class in the top 100 / top 20 but this resulted in slightly worse validation scores. The reasoning is the same as above: those values contain valuable information!
\n\n@CarrDelling - I started the competition using a modest 8GB laptop but decided to purchase a €1500 workstation two weeks into the competition to speed up the modeling. Starting with limited resources ended up to be an advantage since it forced me to think of ways to optimize the feature generation logic. The major bottleneck of the feature generation is in the nearest neighbors calculation and I wrote a custom Rcpp based package to speed things up. My big friend in this competition was the data.table package.
\n\n@Arcady27 - Those features are counts of the nearest neighbors. For example: if a test observation has 3 of its 5 nearest neighbors of class A and 2 of its 5 nearest neighbors as class B, the candidate A will contain the numeric value of 3 for the K=5 feature, the candidate B will contain the numeric value of 2 for the K=5 feature and all other 18 candidates will contain the value of 0 for that feature.
\n\n@Markus - Thanks for the nice words, I am equally impressed by your work! Maybe we can work together on a similar application for the real world, how cool would that be? I did indeed spend most of my time looking at data and thinking of new potential features. They were gradually added throughout the process and I will discuss the most important features in the full writeup.
\n\nP.S. @aldente - I am not a professor, Tom will do just fine :)
", "rawMarkdown": "@Gohm - You are right at your interpretation of my approach. I did not perform any downsampling in my final approach since all zeros (not a match between the candidate and true match) contain valuable information. XGBoost is able to handle unbalanced data quite well in my experience. I did however consider to omit observations that didn't contain the true class in the top 100 / top 20 but this resulted in slightly worse validation scores. The reasoning is the same as above: those values contain valuable information!\r\n\r\n@CarrDelling - I started the competition using a modest 8GB laptop but decided to purchase a €1500 workstation two weeks into the competition to speed up the modeling. Starting with limited resources ended up to be an advantage since it forced me to think of ways to optimize the feature generation logic. The major bottleneck of the feature generation is in the nearest neighbors calculation and I wrote a custom Rcpp based package to speed things up. My big friend in this competition was the data.table package.\r\n\r\n@Arcady27 - Those features are counts of the nearest neighbors. For example: if a test observation has 3 of its 5 nearest neighbors of class A and 2 of its 5 nearest neighbors as class B, the candidate A will contain the numeric value of 3 for the K=5 feature, the candidate B will contain the numeric value of 2 for the K=5 feature and all other 18 candidates will contain the value of 0 for that feature.\r\n\r\n@Markus - Thanks for the nice words, I am equally impressed by your work! Maybe we can work together on a similar application for the real world, how cool would that be? I did indeed spend most of my time looking at data and thinking of new potential features. They were gradually added throughout the process and I will discuss the most important features in the full writeup.\r\n\r\nP.S. @aldente - I am not a professor, Tom will do just fine :)", "votes": 3 }, { "id": 127857, "postDate": "2016-07-15T18:34:33.783Z", "content": "I am trying to install the package fastOrdeR from the tar.gz file. However I keep getting error in my R console. \nWarning: running command 'make -f "C:/PROGRA~1/R/R-33~1.0/etc/x64/Makeconf" -f "C:/PROGRA~1/R/R-33~1.0/share/make/winshlib.mk" SHLIB_LDFLAGS='$(SHLIB_CXXLDFLAGS)' SHLIB_LD='$(SHLIB_CXXLD)' SHLIB="fastOrdeR.dll" WIN=64 TCLBIN=64 OBJECTS="RcppExports.o fastOrder.o rcpp_hello_world.o"' had status 127\nERROR: compilation failed for package 'fastOrdeR'\n* removing 'D:/Users/kamatchi/Documents/R/win-library/3.3/fastOrdeR'\nWarning messages:\n1: running command '"C:/PROGRA~1/R/R-33~1.0/bin/x64/R" CMD INSTALL -l "D:\\Users\\kamatchi\\Documents\\R\\win-library\\3.3" "D:/Users/kamatchi/Documents/ML_compl_comp/Facebook_Leader_solution/Facebook-V/Candidate selection/fastOrdeR package/fastOrdeR_1.0.tar.gz"' had status 1 \n2: In install.packages(files[tarballs], .libPaths()[1L], repos = NULL, :\n installation of package ‘D:/Users/kamatchi/Documents/ML_compl_comp/Facebook_Leader_solution/Facebook-V/Candidate selection/fastOrdeR package/fastOrdeR_1.0.tar.gz’ had non-zero exit status
\n\nCan you please tell me what I am doing wrong?
\nThanks\nKamatchi Subramanian\nNote: I already installed RcppArmadillo package.
[quote=Tom Van de Wiele;126683]
\n\nThe extended story is now available, please feel free to ask further questions!
\n\n[/quote]\nThank you very much and congratulation!
", "rawMarkdown": "[quote=Tom Van de Wiele;126683]\r\n\r\nThe [extended story][1] is now available, please feel free to ask further questions!\r\n\r\n\r\n [1]: https://ttvand.github.io/Winning-approach-of-the-Facebook-V-Kaggle-competition/\r\n\r\n[/quote]\r\nThank you very much and congratulation! \r\n", "votes": 1 }, { "id": 126563, "postDate": "2016-07-09T20:01:06.453Z", "content": "Thanks for your nice words. I'm happy to pay for the electricity myself, it was very much worth it! It only takes so long because I actually had the time to make it very computationally intensive. This approach got me to the top of the leaderboard three weeks into the competition with a simple 8GB laptop after 11 days of working on the problem so I think it would be unfair to call it a brute force solution (I'm not suggesting that you claim this).
", "rawMarkdown": "Thanks for your nice words. I'm happy to pay for the electricity myself, it was very much worth it! It only takes so long because I actually *had* the time to make it very computationally intensive. This approach got me to the top of the leaderboard three weeks into the competition with a simple 8GB laptop after 11 days of working on the problem so I think it would be unfair to call it a brute force solution (I'm not suggesting that you claim this).", "votes": 1 }, { "id": 126557, "postDate": "2016-07-09T19:27:33.710Z", "content": "[quote=Tom Van de Wiele;126472]
\n\n@CarrDelling - I'm sorry, I forgot to answer your question related to the running time to generate the final submission. On my workstation it would take about a month. That seems like a ridiculously long time but it is explained by the extended computation time of the nearest neighbor features. While calculating the NN features I was continuously working on other parts of the workflow so speeding the NN logic up would not have resulted in a better final score.
\n\nGenerating a ~.621 score could however be achieved in about two weeks by focusing on the most relevant NN features. I would suggest to consider 3 of the 7 distance constants (1, 2.5 and 4) and omit the mid KNN features. Cutting the first level models from 100 to 10 and the second level models from 30 to 5 would also not result in a strong performance decrease (estimated decrease of 0.1% for a LB score of 0.62) and cut the computation time to less than a week. You could of course run the logic on multiple instances and further speed things up.
\n\n[/quote]
\n\nThanks for your answer! Yes, I was expecting something larger than the usual "two days with a laptop", but definitely not a month of running time in a decent workstation. Shame you can't bill Facebook now for the electricity... Anyway, congratulations again!
", "rawMarkdown": "[quote=Tom Van de Wiele;126472]\r\n\r\n@CarrDelling - I'm sorry, I forgot to answer your question related to the running time to generate the final submission. On my workstation it would take about a month. That seems like a ridiculously long time but it is explained by the extended computation time of the nearest neighbor features. While calculating the NN features I was continuously working on other parts of the workflow so speeding the NN logic up would not have resulted in a better final score. \r\n\r\nGenerating a ~.621 score could however be achieved in about two weeks by focusing on the most relevant NN features. I would suggest to consider 3 of the 7 distance constants (1, 2.5 and 4) and omit the mid KNN features. Cutting the first level models from 100 to 10 and the second level models from 30 to 5 would also not result in a strong performance decrease (estimated decrease of 0.1% for a LB score of 0.62) and cut the computation time to less than a week. You could of course run the logic on multiple instances and further speed things up.\r\n\r\n[/quote]\r\n\r\nThanks for your answer! Yes, I was expecting something larger than the usual \"two days with a laptop\", but definitely not a month of running time in a decent workstation. Shame you can't bill Facebook now for the electricity... Anyway, congratulations again!\r\n", "votes": 1 }, { "id": 126521, "postDate": "2016-07-09T05:39:48.173Z", "content": "Thank you @Tom, I will dive into your codes! : )
", "rawMarkdown": "Thank you @Tom, I will dive into your codes! : )", "votes": 1 }, { "id": 126520, "postDate": "2016-07-09T05:31:20.393Z", "content": "If you inspect the code in the "First level learners" folder you will see that the framework was created to handle multiple first level learner types (the second group of models mentioned above). I experimented with the nnet and H2O neural network implementations but those were either too slow in transfering the data (H2O) or too biased (nnet). The way XGBoost handles missing values is another great advantage. Also, the XGBoost models are quite variable since they are trained on different random train batches with differing hyperparameters (eta constant, number of features and the number of considered candidates (either 20 or 100)).
", "rawMarkdown": "If you inspect the code in the \"First level learners\" folder you will see that the framework was created to handle multiple first level learner types (the second group of models mentioned above). I experimented with the nnet and H2O neural network implementations but those were either too slow in transfering the data (H2O) or too biased (nnet). The way XGBoost handles missing values is another great advantage. Also, the XGBoost models are quite variable since they are trained on different random train batches with differing hyperparameters (eta constant, number of features and the number of considered candidates (either 20 or 100)). ", "votes": 1 }, { "id": 126472, "postDate": "2016-07-08T17:51:38.283Z", "content": "@CarrDelling - I'm sorry, I forgot to answer your question related to the running time to generate the final submission. On my workstation it would take about a month. That seems like a ridiculously long time but it is explained by the extended computation time of the nearest neighbor features. While calculating the NN features I was continuously working on other parts of the workflow so speeding the NN logic up would not have resulted in a better final score.
\n\nGenerating a ~.621 score could however be achieved in about two weeks by focusing on the most relevant NN features. I would suggest to consider 3 of the 7 distance constants (1, 2.5 and 4) and omit the mid KNN features. Cutting the first level models from 100 to 10 and the second level models from 30 to 5 would also not result in a strong performance decrease (estimated decrease of 0.1% for a LB score of 0.62) and cut the computation time to less than a week. You could of course run the logic on multiple instances and further speed things up.
", "rawMarkdown": "@CarrDelling - I'm sorry, I forgot to answer your question related to the running time to generate the final submission. On my workstation it would take about a month. That seems like a ridiculously long time but it is explained by the extended computation time of the nearest neighbor features. While calculating the NN features I was continuously working on other parts of the workflow so speeding the NN logic up would not have resulted in a better final score. \r\n\r\nGenerating a ~.621 score could however be achieved in about two weeks by focusing on the most relevant NN features. I would suggest to consider 3 of the 7 distance constants (1, 2.5 and 4) and omit the mid KNN features. Cutting the first level models from 100 to 10 and the second level models from 30 to 5 would also not result in a strong performance decrease (estimated decrease of 0.1% for a LB score of 0.62) and cut the computation time to less than a week. You could of course run the logic on multiple instances and further speed things up.", "votes": 1 }, { "id": 126408, "postDate": "2016-07-08T07:49:46.423Z", "content": "Congratulations to No.1 winner! This is definitely a noteworthy achievement and many thanks for sharing you wisdom!
\n\nI will read your codes in great details when I have a bit more time and I'm sure I will learn a lot. For now, I'm attempting to make a layout of your strategy, please correct whe I get things wrong. Many thanks in advance! (and please bash me with mercy if I got things grossly wrong...)
\n\nYou use knn to find 100 candidate place_ids for each of the samples.
You make 100 first level models, and each of them do the following:\nPick part of the time period for this model. For each of the place_ids included in this random pariod, use a binary classification model to predict the probability of belonging to this place_id, for all the relevant sample. Here, relevant means that this sample picked this place_id in its 100 candidates in step 1.
Use an average of the 100 first level models to narrow down to 20 top candidates for each sample. Question: when you do the averaging, do you mean averaging the probability of a sample belonging to a place_id, predicted by each of the 100 models? If this is true, then these are actually likelyhoods, do you include prior probabilities of each of the relevant place_ids in condideration? If so, did you use global or local prior probabilities?
The second level models part is a bit unclear to me. You said "These first level model predictions are not used as features in the second level models", and then you also said "Second level models: 30 xgboost models were learnt to combine the predictions of the 100 first level models". Do you mean you use the 20 candiate predictions, but not the raw predictions from the first level models? How did you vary the 30 xgb models so they get divergent enough to produce nice combinations that cover larger truth space?
@Tom Thanks for the sharing. I have so much to learn from you. \nAnd congratulations for you again!!!
", "rawMarkdown": "@Tom Thanks for the sharing. I have so much to learn from you. \r\nAnd congratulations for you again!!!", "votes": 1 }, { "id": 126307, "postDate": "2016-07-07T16:06:27.890Z", "content": "Thank you for sharing! I'm impressed by all the data analysis (including the great Shiny app!) and feature engineering you did. I look forward to your full writeup to learn more about the details of the features and which ones were the most important.
", "rawMarkdown": "Thank you for sharing! I'm impressed by all the data analysis (including the great Shiny app!) and feature engineering you did. I look forward to your full writeup to learn more about the details of the features and which ones were the most important.", "votes": 1 }, { "id": 126276, "postDate": "2016-07-07T09:10:45.377Z", "content": "Really interesting solution Tom. Looking forward to read the full write up.
", "rawMarkdown": "Really interesting solution Tom. Looking forward to read the full write up. ", "votes": 1 }, { "id": 126258, "postDate": "2016-07-07T06:32:59.003Z", "content": "@aldente, that's an excellent question! I calculate the features for all 20 candidate records of each observation.
\n\nInstead of doing all the calculations at once I perform all my calculations on random batches of the 3 considered data groups (train, validation, test). The batches are constructed such that they contain about 30,000 observations. This number was picked as high as possible while still allowing for sufficient memory during the modeling phase on my system. I ended up with 219 train batches, 73 validation batches and 287 test batches. Feature generation of a single batch takes about 8 minutes on my system. The feature files are about 800MB for each batch and I saved all the features to an external HD. Generating the 100 base level predictions for a single batch takes about 20 minutes and the second tier predictions take about one minute for each batch. I hope this clears your confusion.
", "rawMarkdown": "@aldente, that's an excellent question! I calculate the features for all 20 candidate records of each observation.\r\n\r\nInstead of doing all the calculations at once I perform all my calculations on random batches of the 3 considered data groups (train, validation, test). The batches are constructed such that they contain about 30,000 observations. This number was picked as high as possible while still allowing for sufficient memory during the modeling phase on my system. I ended up with 219 train batches, 73 validation batches and 287 test batches. Feature generation of a single batch takes about 8 minutes on my system. The feature files are about 800MB for each batch and I saved all the features to an external HD. Generating the 100 base level predictions for a single batch takes about 20 minutes and the second tier predictions take about one minute for each batch. I hope this clears your confusion.", "votes": 1 }, { "id": 126246, "postDate": "2016-07-07T05:28:16.297Z", "content": "@Prof. Tom Van de Wiele, thank you so much for sharing, I appreciate it. I'll save questions if any after you share it in more detail.
\n\nHowever my apologies, I have early (and maybe absurd) question: for the two levels, are your models having only 20 classes from the candidate selection? Or, is it 20 classes for each row (single observation)?
\n\nThe question came from my curiosity, if it's the later (20 per row) then how the huge table of probabilities is saved?
", "rawMarkdown": "@Prof. Tom Van de Wiele, thank you so much for sharing, I appreciate it. I'll save questions if any after you share it in more detail. \r\n\r\nHowever my apologies, I have early (and maybe absurd) question: for the two levels, are your models having only 20 classes from the candidate selection? Or, is it 20 classes for each row (single observation)? \r\n\r\nThe question came from my curiosity, if it's the later (20 per row) then how the huge table of probabilities is saved? \r\n", "votes": 1 }, { "id": 126582, "postDate": "2016-07-10T05:33:38.727Z", "content": "@Jiahuan He - No AWS or expensive hardware here. I only used my 8GB laptop and a modest 48 GB workstation.
", "rawMarkdown": "@Jiahuan He - No AWS or expensive hardware here. I only used my 8GB laptop and a modest 48 GB workstation. ", "votes": 2 }, { "id": 126566, "postDate": "2016-07-09T21:29:28.610Z", "content": "Nah, IMHO brute force in this competition would have been to take the dataset as is, fire up a cluster of 50 x1.32xlarge machines in AWS, and XGBoost the problem to death.
\n\nDeploying a solution on more than 60 scripts, and for which you need several documents in order to provide a proper description, feels like the opposite of brute force. I doubt we find more participants with the same amount of code in this competition, honestly. Don't look only at your position in the leaderboard, there's much more merit in your solution than just the @map3 score :)
", "rawMarkdown": "Nah, IMHO *brute force* in this competition would have been to take the dataset as is, fire up a cluster of 50 x1.32xlarge machines in AWS, and XGBoost the problem to death. \r\n\r\nDeploying a solution on more than 60 scripts, and for which you need several documents in order to provide a proper description, feels like the opposite of brute force. I doubt we find more participants with the same amount of code in this competition, honestly. Don't look only at your position in the leaderboard, there's much more merit in your solution than just the @map3 score :)", "votes": 2 }, { "id": 127873, "postDate": "2016-07-15T20:07:08.673Z", "content": "This is the installation instructions link for Rtools on Windows.
", "rawMarkdown": "This is the installation instructions [link][1] for Rtools on Windows.\r\n\r\n\r\n [1]: https://github.com/stan-dev/rstan/wiki/Install-Rtools-for-Windows" }, { "id": 127865, "postDate": "2016-07-15T19:30:21.453Z", "content": "HI Tom,
\n\nI am using R version 3.3 and when I check for packages there is no package called R tools.
\n\nWhat version of R are you using?
\n\nThanks\nKamatchi
", "rawMarkdown": "HI Tom,\r\n\r\nI am using R version 3.3 and when I check for packages there is no package called R tools. \r\n\r\nWhat version of R are you using?\r\n\r\nThanks\r\nKamatchi\r\n\r\n" }, { "id": 127862, "postDate": "2016-07-15T19:21:56.727Z", "content": "Hi Kamatchi,
\n\nThis sounds like a compiler issue. Your first option is to read through the Rcpp manual (especially section 1.3). Do you have Rtools installed? The alternative is to follow these steps to skip the use of the fastOrder package:
\n\nGreat works! I might need sometime to digest all these;) Just one question first, Did you use AWS or other servers or simply on your own PC?
", "rawMarkdown": "Great works! I might need sometime to digest all these;) Just one question first, Did you use AWS or other servers or simply on your own PC?" }, { "id": 126425, "postDate": "2016-07-08T09:41:01.857Z", "content": "@Tom, sorry Google told me another Prof. Tom VdW ;) Thanks a lot for sharing the code, +1. I admire the winners reflect strong knowledge of data science.
", "rawMarkdown": "@Tom, sorry Google told me another Prof. Tom VdW ;) Thanks a lot for sharing the code, +1. I admire the winners reflect strong knowledge of data science. " }, { "id": 126277, "postDate": "2016-07-07T09:20:11.340Z", "content": "\n\n\nThe major share of my 430 features is based on nearest neighbors features. I counted the neighbors for different Ks (1, 5, 10, 20, 50, 100, 250, 500, 1000, 2500) and different x-y ratio constants (1, 2.5, 4, 5.5, 7, 12, 30) and also included the mean time difference for all 70 combinations.
\n
Excuse me, but I haven't figured out, what are exactly these features? Probabilities of check-in belongs to a candidate place_id calculated by all KNN models along with mean time differences (overall 140 features)?
", "rawMarkdown": "> The major share of my 430 features is based on nearest neighbors features. I counted the neighbors for different Ks (1, 5, 10, 20, 50, 100, 250, 500, 1000, 2500) and different x-y ratio constants (1, 2.5, 4, 5.5, 7, 12, 30) and also included the mean time difference for all 70 combinations. \r\n\r\nExcuse me, but I haven't figured out, what are exactly these features? Probabilities of check-in belongs to a candidate place_id calculated by all KNN models along with mean time differences (overall 140 features)?" }, { "id": 126275, "postDate": "2016-07-07T08:53:03.720Z", "content": "Thank you for the reply, Prof. Tom van de Wiele, smart solution with the batches, looking forward to learn your solution in detail. Congratulations.
", "rawMarkdown": "Thank you for the reply, Prof. Tom van de Wiele, smart solution with the batches, looking forward to learn your solution in detail. Congratulations. " }, { "id": 126269, "postDate": "2016-07-07T08:11:56.870Z", "content": "Awesome solution @Tom! It is surprising to see that a global approach definitely could work for this problem.
\n\nOut of curiosity, what kind of resources (laptop, cluster...) did you use through the competition? And how much time (approx.) would it take to generate the winning solution? (or one close enough...)
", "rawMarkdown": "Awesome solution @Tom! It is surprising to see that a global approach definitely could work for this problem.\r\n\r\nOut of curiosity, what kind of resources (laptop, cluster...) did you use through the competition? And how much time (approx.) would it take to generate the winning solution? (or one close enough...)" }, { "id": 126264, "postDate": "2016-07-07T07:23:42.747Z", "content": "[quote=Tom Van de Wiele;126258]
\n\n@aldente, that's an excellent question! I calculate the features for all 20 candidate records of each observation.
\n\nInstead of doing all the calculations at once I perform all my calculations on random batches of the 3 considered data groups (train, validation, test). The batches are constructed such that they contain about 30,000 observations. This number was picked as high as possible while still allowing for sufficient memory during the modeling phase on my system. I ended up with 219 train batches, 73 validation batches and 287 test batches. Feature generation of a single batch takes about 8 minutes on my system. The feature files are about 800MB for each batch and I saved all the features to an external HD. Generating the 100 base level predictions for a single batch takes about 20 minutes and the second tier predictions take about one minute for each batch. I hope this clears your confusion.
\n\n[/quote]
\n\nThanks for sharing the solution! And congratulations on winning the competition! So if I understand your reply correctly, you calculated the features for all 20 candidate records of each observation, won't you end up having at least 19 negative examples and at most only 1 positive example? And so it is a rare event problem, and did you perform any downsampling (or any smart and efficient downsampling) on the negative class? In your first level model, you have 100 candidates, so the ratio of negative to positive examples is even much higher... You mentioned in your post that you did some sampling, but I am not sure if you just downsampling the negative examples or the entire data set...
", "rawMarkdown": "[quote=Tom Van de Wiele;126258]\r\n\r\n@aldente, that's an excellent question! I calculate the features for all 20 candidate records of each observation.\r\n\r\nInstead of doing all the calculations at once I perform all my calculations on random batches of the 3 considered data groups (train, validation, test). The batches are constructed such that they contain about 30,000 observations. This number was picked as high as possible while still allowing for sufficient memory during the modeling phase on my system. I ended up with 219 train batches, 73 validation batches and 287 test batches. Feature generation of a single batch takes about 8 minutes on my system. The feature files are about 800MB for each batch and I saved all the features to an external HD. Generating the 100 base level predictions for a single batch takes about 20 minutes and the second tier predictions take about one minute for each batch. I hope this clears your confusion.\r\n\r\n[/quote]\r\n\r\nThanks for sharing the solution! And congratulations on winning the competition! So if I understand your reply correctly, you calculated the features for all 20 candidate records of each observation, won't you end up having at least 19 negative examples and at most only 1 positive example? And so it is a rare event problem, and did you perform any downsampling (or any smart and efficient downsampling) on the negative class? In your first level model, you have 100 candidates, so the ratio of negative to positive examples is even much higher... You mentioned in your post that you did some sampling, but I am not sure if you just downsampling the negative examples or the entire data set..." }, { "id": 1015901, "postDate": "2020-09-18T14:09:43.410Z", "rawMarkdown": "", "isDeleted": true }, { "id": 1015898, "postDate": "2020-09-18T14:09:09.860Z", "rawMarkdown": "", "isDeleted": true }, { "id": 126324, "postDate": "2016-07-07T17:29:48.657Z", "content": "thanks and congratulation~
", "rawMarkdown": "thanks and congratulation~", "votes": 1 } ], "comments": [ { "id": 126683, "author_name": "Tom Van de Wiele", "author_url": "", "post_date": "2016-07-11T20:18:52.740000", "content": "The extended story is now available, please feel free to ask further questions!
", "votes": 5, "replies": [] }, { "id": 126345, "author_name": "Tom Van de Wiele", "author_url": "", "post_date": "2016-07-07T20:39:05.867000", "content": "The cleaned code is now available on GitHub!
", "votes": 6, "replies": [] }, { "id": 126504, "author_name": "Zhenjie Zhang", "author_url": "", "post_date": "2016-07-08T23:25:30.347000", "content": "It is very clear explanation, thanks for attending to write this up. Indeed this modeling takes a lot of cpu time but the result is great. One more question just occurred to me is that, I know you have tried different models for the three groups of models and xgb turned out yielding the best results, but have your tried mixing other models with xgb in any of the stages? My surmise is that mixing different models could add to the randomness of the modeling results and might end up voting to a higher score.
", "votes": 3, "replies": [] }, { "id": 126470, "author_name": "Tom Van de Wiele", "author_url": "", "post_date": "2016-07-08T17:35:57.300000", "content": "Hi Zhenjie Zhang, thank you for your comments. I understand your confusion and will try to clarify your concerns.
\n\nFirst of all IdHi Zhenjie Zhang, thank you for your comments. I understand your confusion and will try to clarify your concerns.
\n\nFirst of all I’ll try to refine my problem definition. Most other binary classification approaches I have studied on the forum model the conditional probability P(place_id|data) where data relates to x, y, time and accuracy directly or through Bayes’s theorem by translating it to P(data|place_id) * P(place_id) / P(data). The last category of approaches varies in how they calculate P(data|place_id) and P(place_id) since P(data) is the same for all place_ids for a given observation. My approach models P(place_match|data) directly for the considered place candidates. This approach does not require the use of prior probabilities. Place_match is the binary outcome variable and I order the places based on P(place_match|data).
\n\nAs described in the “Instructions” file on the GitHub link, there are three groups of models that are generated (all XGBoost)
\n\n - Second candidate selection models (15 models)\n - First level learners (100 models)\n - Second level learners (30 models)\nAll models are trained on different random batches of the train data set (split after cutoff time 587158) and span the entire time range 587158-786239.\nThe confusion arises from the fact that the first two groups are generated using the same model approaches (first level XGBoost models). The only difference between the two groups is the number of considered candidates for each observation and the number of considered features. I used 100 candidates as input to the “Second candidate selection” models, you got that part right. The 15 Second candidate selection models all generate a predicted probability of P(place_match|data), I average those to select the top20 candidates in the second candidate selection step. These 'predicted probabilities' are indeed likelihoods.
\n\nThe 100 first level learnes then have 20 candidates for each observation. The predicted probabilities of P(place_match|data) of the 100 first level learners then serve as features to the second level learners along with 21 additional first level features that were selected manually. The second level learners again try to model P(place_match|data).
\n\nDoes this answer handle all your questions?
", "votes": 3, "replies": [] }, { "id": 126314, "author_name": "Tom Van de Wiele", "author_url": "", "post_date": "2016-07-07T16:36:33.013000", "content": "@Gohm - You are right at your interpretation of my approach. I did not perform any downsampling in my final approach since all zeros (not a match between the candidate and true match) contain valuable information. XGBoost is able to handle unbalanced data quite well in my experience. I did however consider to omit observations that didn't contain the true class in the top 100 / top 20 but this resulted in slightly worse validation scores. The reasoning is the same as above: those values contain valuable information!
\n\n@CarrDelling - I started the competition using a modest 8GB laptop but decided to purchase a €1500 workstation two weeks into the competition to speed up the modeling. Starting with limited resources ended up to be an advantage since it forced me to think of ways to optimize the feature generation logic. The major bottleneck of the feature generation is in the nearest neighbors calculation and I wrote a custom Rcpp based package to speed things up. My big friend in this competition was the data.table package.
\n\n@Arcady27 - Those features are counts of the nearest neighbors. For example: if a test observation has 3 of its 5 nearest neighbors of class A and 2 of its 5 nearest neighbors as class B, the candidate A will contain the numeric value of 3 for the K=5 feature, the candidate B will contain the numeric value of 2 for the K=5 feature and all other 18 candidates will contain the value of 0 for that feature.
\n\n@Markus - Thanks for the nice words, I am equally impressed by your work! Maybe we can work together on a similar application for the real world, how cool would that be? I did indeed spend most of my time looking at data and thinking of new potential features. They were gradually added throughout the process and I will discuss the most important features in the full writeup.
\n\nP.S. @aldente - I am not a professor, Tom will do just fine :)
", "votes": 3, "replies": [] }, { "id": 127857, "author_name": "Kamatchi Subramanian", "author_url": "", "post_date": "2016-07-15T18:34:33.783000", "content": "I am trying to install the package fastOrdeR from the tar.gz file. However I keep getting error in my R console. \nWarning: running command 'make -f "C:/PROGRA~1/R/R-33~1.0/etc/x64/Makeconf" -f "C:/PROGRA~1/R/R-33~1.0/share/make/winshlib.mk" SHLIB_LDFLAGS='$(SHLIB_CXXLDFLAGS)' SHLIB_LD='$(SHLIB_CXXLD)' SHLIB="fastOrdeR.dll" WIN=64 TCLBIN=64 OBJECTS="RcppExports.o fastOrder.o rcpp_hello_world.o"' had status 127\nERROR: compilation failed for package 'fastOrdeR'\n* removing 'D:/Users/kamatchi/Documents/R/win-library/3.3/fastOrdeR'\nWarning messages:\n1: running command '"C:/PROGRA~1/R/R-33~1.0/bin/x64/R" CMD INSTALL -l "D:\\Users\\kamatchi\\Documents\\R\\win-library\\3.3" "D:/Users/kamatchi/Documents/ML_compl_comp/Facebook_Leader_solution/Facebook-V/Candidate selection/fastOrdeR package/fastOrdeR_1.0.tar.gz"' had status 1 \n2: In install.packages(files[tarballs], .libPaths()[1L], repos = NULL, :\n installation of package ‘D:/Users/kamatchi/Documents/ML_compl_comp/Facebook_Leader_solution/Facebook-V/Candidate selection/fastOrdeR package/fastOrdeR_1.0.tar.gz’ had non-zero exit status
\n\nCan you please tell me what I am doing wrong?
\nThanks\nKamatchi Subramanian\nNote: I already installed RcppArmadillo package.
[quote=Tom Van de Wiele;126683]
\n\nThe extended story is now available, please feel free to ask further questions!
\n\n[/quote]\nThank you very much and congratulation!
", "votes": 1, "replies": [] }, { "id": 126563, "author_name": "Tom Van de Wiele", "author_url": "", "post_date": "2016-07-09T20:01:06.453000", "content": "Thanks for your nice words. I'm happy to pay for the electricity myself, it was very much worth it! It only takes so long because I actually had the time to make it very computationally intensive. This approach got me to the top of the leaderboard three weeks into the competition with a simple 8GB laptop after 11 days of working on the problem so I think it would be unfair to call it a brute force solution (I'm not suggesting that you claim this).
", "votes": 1, "replies": [] }, { "id": 126557, "author_name": "CarrDelling", "author_url": "", "post_date": "2016-07-09T19:27:33.710000", "content": "[quote=Tom Van de Wiele;126472]
\n\n@CarrDelling - I'm sorry, I forgot to answer your question related to the running time to generate the final submission. On my workstation it would take about a month. That seems like a ridiculously long time but it is explained by the extended computation time of the nearest neighbor features. While calculating the NN features I was continuously working on other parts of the workflow so speeding the NN logic up would not have resulted in a better final score.
\n\nGenerating a ~.621 score could however be achieved in about two weeks by focusing on the most relevant NN features. I would suggest to consider 3 of the 7 distance constants (1, 2.5 and 4) and omit the mid KNN features. Cutting the first level models from 100 to 10 and the second level models from 30 to 5 would also not result in a strong performance decrease (estimated decrease of 0.1% for a LB score of 0.62) and cut the computation time to less than a week. You could of course run the logic on multiple instances and further speed things up.
\n\n[/quote]
\n\nThanks for your answer! Yes, I was expecting something larger than the usual "two days with a laptop", but definitely not a month of running time in a decent workstation. Shame you can't bill Facebook now for the electricity... Anyway, congratulations again!
", "votes": 1, "replies": [] }, { "id": 126521, "author_name": "Zhenjie Zhang", "author_url": "", "post_date": "2016-07-09T05:39:48.173000", "content": "Thank you @Tom, I will dive into your codes! : )
", "votes": 1, "replies": [] }, { "id": 126520, "author_name": "Tom Van de Wiele", "author_url": "", "post_date": "2016-07-09T05:31:20.393000", "content": "If you inspect the code in the "First level learners" folder you will see that the framework was created to handle multiple first level learner types (the second group of models mentioned above). I experimented with the nnet and H2O neural network implementations but those were either too slow in transfering the data (H2O) or too biased (nnet). The way XGBoost handles missing values is another great advantage. Also, the XGBoost models are quite variable since they are trained on different random train batches with differing hyperparameters (eta constant, number of features and the number of considered candidates (either 20 or 100)).
", "votes": 1, "replies": [] }, { "id": 126472, "author_name": "Tom Van de Wiele", "author_url": "", "post_date": "2016-07-08T17:51:38.283000", "content": "@CarrDelling - I'm sorry, I forgot to answer your question related to the running time to generate the final submission. On my workstation it would take about a month. That seems like a ridiculously long time but it is explained by the extended computation time of the nearest neighbor features. While calculating the NN features I was continuously working on other parts of the workflow so speeding the NN logic up would not have resulted in a better final score.
\n\nGenerating a ~.621 score could however be achieved in about two weeks by focusing on the most relevant NN features. I would suggest to consider 3 of the 7 distance constants (1, 2.5 and 4) and omit the mid KNN features. Cutting the first level models from 100 to 10 and the second level models from 30 to 5 would also not result in a strong performance decrease (estimated decrease of 0.1% for a LB score of 0.62) and cut the computation time to less than a week. You could of course run the logic on multiple instances and further speed things up.
", "votes": 1, "replies": [] }, { "id": 126408, "author_name": "Zhenjie Zhang", "author_url": "", "post_date": "2016-07-08T07:49:46.423000", "content": "Congratulations to No.1 winner! This is definitely a noteworthy achievement and many thanks for sharing you wisdom!
\n\nI will read your codes in great details when I have a bit more time and I'm sure I will learn a lot. For now, I'm attempting to make a layout of your strategy, please correct whe I get things wrong. Many thanks in advance! (and please bash me with mercy if I got things grossly wrong...)
\n\nYou use knn to find 100 candidate place_ids for each of the samples.
You make 100 first level models, and each of them do the following:\nPick part of the time period for this model. For each of the place_ids included in this random pariod, use a binary classification model to predict the probability of belonging to this place_id, for all the relevant sample. Here, relevant means that this sample picked this place_id in its 100 candidates in step 1.
Use an average of the 100 first level models to narrow down to 20 top candidates for each sample. Question: when you do the averaging, do you mean averaging the probability of a sample belonging to a place_id, predicted by each of the 100 models? If this is true, then these are actually likelyhoods, do you include prior probabilities of each of the relevant place_ids in condideration? If so, did you use global or local prior probabilities?
The second level models part is a bit unclear to me. You said "These first level model predictions are not used as features in the second level models", and then you also said "Second level models: 30 xgboost models were learnt to combine the predictions of the 100 first level models". Do you mean you use the 20 candiate predictions, but not the raw predictions from the first level models? How did you vary the 30 xgb models so they get divergent enough to produce nice combinations that cover larger truth space?
@Tom Thanks for the sharing. I have so much to learn from you. \nAnd congratulations for you again!!!
", "votes": 1, "replies": [] }, { "id": 126307, "author_name": "Markus Kliegl", "author_url": "", "post_date": "2016-07-07T16:06:27.890000", "content": "Thank you for sharing! I'm impressed by all the data analysis (including the great Shiny app!) and feature engineering you did. I look forward to your full writeup to learn more about the details of the features and which ones were the most important.
", "votes": 1, "replies": [] }, { "id": 126276, "author_name": "Frederik", "author_url": "", "post_date": "2016-07-07T09:10:45.377000", "content": "Really interesting solution Tom. Looking forward to read the full write up.
", "votes": 1, "replies": [] }, { "id": 126258, "author_name": "Tom Van de Wiele", "author_url": "", "post_date": "2016-07-07T06:32:59.003000", "content": "@aldente, that's an excellent question! I calculate the features for all 20 candidate records of each observation.
\n\nInstead of doing all the calculations at once I perform all my calculations on random batches of the 3 considered data groups (train, validation, test). The batches are constructed such that they contain about 30,000 observations. This number was picked as high as possible while still allowing for sufficient memory during the modeling phase on my system. I ended up with 219 train batches, 73 validation batches and 287 test batches. Feature generation of a single batch takes about 8 minutes on my system. The feature files are about 800MB for each batch and I saved all the features to an external HD. Generating the 100 base level predictions for a single batch takes about 20 minutes and the second tier predictions take about one minute for each batch. I hope this clears your confusion.
", "votes": 1, "replies": [] }, { "id": 126246, "author_name": "aldente", "author_url": "", "post_date": "2016-07-07T05:28:16.297000", "content": "@Prof. Tom Van de Wiele, thank you so much for sharing, I appreciate it. I'll save questions if any after you share it in more detail.
\n\nHowever my apologies, I have early (and maybe absurd) question: for the two levels, are your models having only 20 classes from the candidate selection? Or, is it 20 classes for each row (single observation)?
\n\nThe question came from my curiosity, if it's the later (20 per row) then how the huge table of probabilities is saved?
", "votes": 1, "replies": [] }, { "id": 126582, "author_name": "Tom Van de Wiele", "author_url": "", "post_date": "2016-07-10T05:33:38.727000", "content": "@Jiahuan He - No AWS or expensive hardware here. I only used my 8GB laptop and a modest 48 GB workstation.
", "votes": 2, "replies": [] }, { "id": 126566, "author_name": "CarrDelling", "author_url": "", "post_date": "2016-07-09T21:29:28.610000", "content": "Nah, IMHO brute force in this competition would have been to take the dataset as is, fire up a cluster of 50 x1.32xlarge machines in AWS, and XGBoost the problem to death.
\n\nDeploying a solution on more than 60 scripts, and for which you need several documents in order to provide a proper description, feels like the opposite of brute force. I doubt we find more participants with the same amount of code in this competition, honestly. Don't look only at your position in the leaderboard, there's much more merit in your solution than just the @map3 score :)
", "votes": 2, "replies": [] }, { "id": 127873, "author_name": "Tom Van de Wiele", "author_url": "", "post_date": "2016-07-15T20:07:08.673000", "content": "This is the installation instructions link for Rtools on Windows.
", "votes": 0, "replies": [] }, { "id": 127865, "author_name": "Kamatchi Subramanian", "author_url": "", "post_date": "2016-07-15T19:30:21.453000", "content": "HI Tom,
\n\nI am using R version 3.3 and when I check for packages there is no package called R tools.
\n\nWhat version of R are you using?
\n\nThanks\nKamatchi
", "votes": 0, "replies": [] }, { "id": 127862, "author_name": "Tom Van de Wiele", "author_url": "", "post_date": "2016-07-15T19:21:56.727000", "content": "Hi Kamatchi,
\n\nThis sounds like a compiler issue. Your first option is to read through the Rcpp manual (especially section 1.3). Do you have Rtools installed? The alternative is to follow these steps to skip the use of the fastOrder package:
\n\nGreat works! I might need sometime to digest all these;) Just one question first, Did you use AWS or other servers or simply on your own PC?
", "votes": 0, "replies": [] }, { "id": 126425, "author_name": "aldente", "author_url": "", "post_date": "2016-07-08T09:41:01.857000", "content": "@Tom, sorry Google told me another Prof. Tom VdW ;) Thanks a lot for sharing the code, +1. I admire the winners reflect strong knowledge of data science.
", "votes": 0, "replies": [] }, { "id": 126277, "author_name": "Arcady27", "author_url": "", "post_date": "2016-07-07T09:20:11.340000", "content": "\n\n\nThe major share of my 430 features is based on nearest neighbors features. I counted the neighbors for different Ks (1, 5, 10, 20, 50, 100, 250, 500, 1000, 2500) and different x-y ratio constants (1, 2.5, 4, 5.5, 7, 12, 30) and also included the mean time difference for all 70 combinations.
\n
Excuse me, but I haven't figured out, what are exactly these features? Probabilities of check-in belongs to a candidate place_id calculated by all KNN models along with mean time differences (overall 140 features)?
", "votes": 0, "replies": [] }, { "id": 126275, "author_name": "aldente", "author_url": "", "post_date": "2016-07-07T08:53:03.720000", "content": "Thank you for the reply, Prof. Tom van de Wiele, smart solution with the batches, looking forward to learn your solution in detail. Congratulations.
", "votes": 0, "replies": [] }, { "id": 126269, "author_name": "CarrDelling", "author_url": "", "post_date": "2016-07-07T08:11:56.870000", "content": "Awesome solution @Tom! It is surprising to see that a global approach definitely could work for this problem.
\n\nOut of curiosity, what kind of resources (laptop, cluster...) did you use through the competition? And how much time (approx.) would it take to generate the winning solution? (or one close enough...)
", "votes": 0, "replies": [] }, { "id": 126264, "author_name": "Gohm", "author_url": "", "post_date": "2016-07-07T07:23:42.747000", "content": "[quote=Tom Van de Wiele;126258]
\n\n@aldente, that's an excellent question! I calculate the features for all 20 candidate records of each observation.
\n\nInstead of doing all the calculations at once I perform all my calculations on random batches of the 3 considered data groups (train, validation, test). The batches are constructed such that they contain about 30,000 observations. This number was picked as high as possible while still allowing for sufficient memory during the modeling phase on my system. I ended up with 219 train batches, 73 validation batches and 287 test batches. Feature generation of a single batch takes about 8 minutes on my system. The feature files are about 800MB for each batch and I saved all the features to an external HD. Generating the 100 base level predictions for a single batch takes about 20 minutes and the second tier predictions take about one minute for each batch. I hope this clears your confusion.
\n\n[/quote]
\n\nThanks for sharing the solution! And congratulations on winning the competition! So if I understand your reply correctly, you calculated the features for all 20 candidate records of each observation, won't you end up having at least 19 negative examples and at most only 1 positive example? And so it is a rare event problem, and did you perform any downsampling (or any smart and efficient downsampling) on the negative class? In your first level model, you have 100 candidates, so the ratio of negative to positive examples is even much higher... You mentioned in your post that you did some sampling, but I am not sure if you just downsampling the negative examples or the entire data set...
", "votes": 0, "replies": [] }, { "id": 1015901, "author_name": "", "author_url": "", "post_date": "2020-09-18T14:09:43.410000", "content": "", "votes": 0, "replies": [] }, { "id": 1015898, "author_name": "", "author_url": "", "post_date": "2020-09-18T14:09:09.860000", "content": "", "votes": 0, "replies": [] }, { "id": 126324, "author_name": "", "author_url": "", "post_date": "2016-07-07T17:29:48.657000", "content": "", "votes": 1, "replies": [] } ], "raw_markdown_by_id": { "126233": "First of all I would like to thank all participants for the amazing interaction. It was my first serious competition attempt and I was extremely impressed with the expert code sharing and forum discussions. Special thanks go to @AlexandruPapiu for his interactive random forest analysis and @msjgriffiths for his exploratory data analysis script. In the end, @Markus was indeed very close and I look forward to find out if he followed a similar approach.\r\n\r\nAlso, thank you very much to everyone at Facebook who contributed to this challenging analysis task and to Kaggle for hosting the competition.\r\n\r\nEDIT: Link to [GitHub ][1]of the cleaned code. The [extended story][2] is now also available, please feel free to ask further questions!\r\n\r\n----------------------------------------------------------------------\r\n\r\nI started my effort by inspecting many aspects of the data interactively. The published shiny [app][3] was the result of this stage. My major finding was the absence of a clear spatial (x-y) pattern in all researched relations. I also noted a high level of noise and variability in the patterns. Consequently I decided to use a single overall model rather than many local grid based models. This approach allowed me to turn the problem into a feature engineering competition and consider many features (430 in the final model) and use a two stage model. Two stages of candidate selection are performed in order to narrow down the number of considered classes in the first level models (all XGBoost). I intended to use a blend of models in the second tier but ended up using only XGBoost models. I will discuss the problem definition, candidate selection, the major features and the two model layers below. Further details on the steps and data handling will be provided in a blog post, hopefully this weekend. The code is now available on [GitHub][4] (all in R). This post is intended to share the main ideas and will either skip or simplify less important aspects of the analysis.\r\n\r\n**Problem definition**\r\nAt both model stages I consider the problem as a binary classification problem for all selected candidates: model the probability that an observation belongs to each of the candidates. \r\n\r\nThe local train and validation data was chosen as the last time part of the given train data such that the local train/validation to the local non train/validation data has a similar proportional size as the given test data to the given train data. The first part of the given train data was used to extract place summary features and the second part of the given train data, the local train/validation data, was used to build and validate the models. The entire given train data is used to calculate the place summary features which are used to generate test predictions.\r\n\r\n**Candidate selection**\r\nThe first candidate selection step reduces the number of potential classes from >100K to 100 by considering nearest neighbors of the observations. I considered the neighbor counts of the 2500 nearest neighbors where y variations are 2.5 times more important than x variations. Ties in the neighbor counts are resolved by the mean time difference since the observations (keep most recent place_ids).\r\n\r\nThe second candidate selection step reduces the number of potential classes from 100 to 20 by using the predictions of an average blend of first level models. These first level model predictions are not used as features in the second level models.\r\n\r\n**Major features**\r\nThe major share of my 430 features is based on nearest neighbors features. I counted the neighbors for different Ks (1, 5, 10, 20, 50, 100, 250, 500, 1000, 2500) and different x-y ratio constants (1, 2.5, 4, 5.5, 7, 12, 30) and also included the mean time difference for all 70 combinations. 10 more features were added by considering the distance between the Kth features and the observations for a ratio constant of 2.5 (measures the spatial density). 40 more features were added around the most significant features (K around 100 and a distance constant around 2.5) for the neighbor counts and 40 more for the distance to these neighbors.\r\n\r\nI further added some additional features that consider the (smoothed) grid densities.\r\n\r\nThe second largest share of the features set belongs to time features. Here I converted all time period counts to period density counts in order to handle the two drops in the time frequency. Periods include 27 two-week periods prior to the end of the train data and 27 1-week periods prior to the end of the train data. I also included features that look at the two-week densities looking back between 75 and 1 weeks from the observations. Additional features were added for the clear yearly pattern of some places.\r\n\r\nHour, day and week features were calculated using the historical densities with and without cyclical smoothing and several relaxation constants (relaxed based on the historical count)\r\n\r\nOther time features include extrapolated weekly densities using various time series models.\r\n\r\nAccuracy was very interesting and I noticed that the location of the observations is sometimes related to the accuracy group. So... I added more features that were likely to capture this pattern. Accuracy was split in three groups: 1-44; 45-84 and 85-Inf. See the second plot for the median deviation of observations from their median as a function of time and accuracy.\r\n\r\nI also calculated a bunch of robust Z scores for anything I could think of (x-median(x))/mad(x)\r\n\r\nAll features are rescaled if needed in order to result in similar interpretations for the train and test features.\r\n\r\n**First level models**\r\n100 base learners were fitted on different random parts of the training period. Deep trees gave me the best results here (depth 11) and the eta constant was set to (11 or 12)/500 for 500 rounds. Colsampling also helped (0.6) and subsampling the observations (0.5) did not hurt but of course resulted in a fitting speed increase. I included either all 430 features or a random pick of the ordered features by importance in a desirable feature count range (100-285 and 180-240).\r\n\r\n**Second level models**\r\n30 xgboost models were learnt to combine the predictions of the 100 first level models + 21 additional base features. The final predicted probability is the average of those 30 models. The predicted place ids are based on the order of the average prediction. Ties are resolved by considering the predicted probability of the second candidate selection step.\r\n\r\nPlease feel free to ask questions or suggest improvements!\r\n\r\n\r\n [1]: https://github.com/ttvand/Facebook-V\r\n [2]: https://ttvand.github.io/Winning-approach-of-the-Facebook-V-Kaggle-competition/\r\n [3]: https://www.kaggle.com/c/facebook-v-predicting-check-ins/forums/t/21365/interactive-exploratory-analysis-using-a-shiny-app\r\n [4]: https://github.com/ttvand/Facebook-V", "126683": "The [extended story][1] is now available, please feel free to ask further questions!\r\n\r\n\r\n [1]: https://ttvand.github.io/Winning-approach-of-the-Facebook-V-Kaggle-competition/", "126345": "The cleaned code is now available on [GitHub][1]!\r\n\r\n\r\n [1]: https://github.com/ttvand/Facebook-V", "126504": "It is very clear explanation, thanks for attending to write this up. Indeed this modeling takes a lot of cpu time but the result is great. One more question just occurred to me is that, I know you have tried different models for the three groups of models and xgb turned out yielding the best results, but have your tried mixing other models with xgb in any of the stages? My surmise is that mixing different models could add to the randomness of the modeling results and might end up voting to a higher score.", "126470": "Hi Zhenjie Zhang, thank you for your comments. I understand your confusion and will try to clarify your concerns.\n \nFirst of all IdHi Zhenjie Zhang, thank you for your comments. I understand your confusion and will try to clarify your concerns.\n \nFirst of all I’ll try to refine my problem definition. Most other binary classification approaches I have studied on the forum model the conditional probability P(place\\_id|data) where data relates to x, y, time and accuracy directly or through Bayes’s theorem by translating it to P(data|place\\_id) * P(place\\_id) / P(data). The last category of approaches varies in how they calculate P(data|place\\_id) and P(place\\_id) since P(data) is the same for all place\\_ids for a given observation. My approach models P(place\\_match|data) directly for the considered place candidates. This approach does not require the use of prior probabilities. Place_match is the binary outcome variable and I order the places based on P(place\\_match|data).\n \nAs described in the “Instructions” file on the GitHub link, there are three groups of models that are generated (all XGBoost)\n\n - Second candidate selection models (15 models)\n - First level learners (100 models)\n - Second level learners (30 models)\n\n \nAll models are trained on different random batches of the train data set (split after cutoff time 587158) and span the entire time range 587158-786239.\nThe confusion arises from the fact that the first two groups are generated using the same model approaches (first level XGBoost models). The only difference between the two groups is the number of considered candidates for each observation and the number of considered features. I used 100 candidates as input to the “Second candidate selection” models, you got that part right. The 15 Second candidate selection models all generate a predicted probability of P(place\\_match|data), I average those to select the top20 candidates in the second candidate selection step. These 'predicted probabilities' are indeed likelihoods.\n \nThe 100 first level learnes then have 20 candidates for each observation. The predicted probabilities of P(place_match|data) of the 100 first level learners then serve as features to the second level learners along with 21 additional first level features that were selected manually. The second level learners again try to model P(place\\_match|data).\n\nDoes this answer handle all your questions?", "126314": "@Gohm - You are right at your interpretation of my approach. I did not perform any downsampling in my final approach since all zeros (not a match between the candidate and true match) contain valuable information. XGBoost is able to handle unbalanced data quite well in my experience. I did however consider to omit observations that didn't contain the true class in the top 100 / top 20 but this resulted in slightly worse validation scores. The reasoning is the same as above: those values contain valuable information!\r\n\r\n@CarrDelling - I started the competition using a modest 8GB laptop but decided to purchase a €1500 workstation two weeks into the competition to speed up the modeling. Starting with limited resources ended up to be an advantage since it forced me to think of ways to optimize the feature generation logic. The major bottleneck of the feature generation is in the nearest neighbors calculation and I wrote a custom Rcpp based package to speed things up. My big friend in this competition was the data.table package.\r\n\r\n@Arcady27 - Those features are counts of the nearest neighbors. For example: if a test observation has 3 of its 5 nearest neighbors of class A and 2 of its 5 nearest neighbors as class B, the candidate A will contain the numeric value of 3 for the K=5 feature, the candidate B will contain the numeric value of 2 for the K=5 feature and all other 18 candidates will contain the value of 0 for that feature.\r\n\r\n@Markus - Thanks for the nice words, I am equally impressed by your work! Maybe we can work together on a similar application for the real world, how cool would that be? I did indeed spend most of my time looking at data and thinking of new potential features. They were gradually added throughout the process and I will discuss the most important features in the full writeup.\r\n\r\nP.S. @aldente - I am not a professor, Tom will do just fine :)", "127857": "I am trying to install the package fastOrdeR from the tar.gz file. However I keep getting error in my R console. \r\nWarning: running command 'make -f \"C:/PROGRA~1/R/R-33~1.0/etc/x64/Makeconf\" -f \"C:/PROGRA~1/R/R-33~1.0/share/make/winshlib.mk\" SHLIB_LDFLAGS='$(SHLIB_CXXLDFLAGS)' SHLIB_LD='$(SHLIB_CXXLD)' SHLIB=\"fastOrdeR.dll\" WIN=64 TCLBIN=64 OBJECTS=\"RcppExports.o fastOrder.o rcpp_hello_world.o\"' had status 127\r\nERROR: compilation failed for package 'fastOrdeR'\r\n* removing 'D:/Users/kamatchi/Documents/R/win-library/3.3/fastOrdeR'\r\nWarning messages:\r\n1: running command '\"C:/PROGRA~1/R/R-33~1.0/bin/x64/R\" CMD INSTALL -l \"D:\\Users\\kamatchi\\Documents\\R\\win-library\\3.3\" \"D:/Users/kamatchi/Documents/ML_compl_comp/Facebook_Leader_solution/Facebook-V/Candidate selection/fastOrdeR package/fastOrdeR_1.0.tar.gz\"' had status 1 \r\n2: In install.packages(files[tarballs], .libPaths()[1L], repos = NULL, :\r\n installation of package ‘D:/Users/kamatchi/Documents/ML_compl_comp/Facebook_Leader_solution/Facebook-V/Candidate selection/fastOrdeR package/fastOrdeR_1.0.tar.gz’ had non-zero exit status\r\n \r\nCan you please tell me what I am doing wrong? \r\nThanks\r\nKamatchi Subramanian\r\nNote: I already installed RcppArmadillo package.\r\n", "126688": "[quote=Tom Van de Wiele;126683]\r\n\r\nThe [extended story][1] is now available, please feel free to ask further questions!\r\n\r\n\r\n [1]: https://ttvand.github.io/Winning-approach-of-the-Facebook-V-Kaggle-competition/\r\n\r\n[/quote]\r\nThank you very much and congratulation! \r\n", "126563": "Thanks for your nice words. I'm happy to pay for the electricity myself, it was very much worth it! It only takes so long because I actually *had* the time to make it very computationally intensive. This approach got me to the top of the leaderboard three weeks into the competition with a simple 8GB laptop after 11 days of working on the problem so I think it would be unfair to call it a brute force solution (I'm not suggesting that you claim this).", "126557": "[quote=Tom Van de Wiele;126472]\r\n\r\n@CarrDelling - I'm sorry, I forgot to answer your question related to the running time to generate the final submission. On my workstation it would take about a month. That seems like a ridiculously long time but it is explained by the extended computation time of the nearest neighbor features. While calculating the NN features I was continuously working on other parts of the workflow so speeding the NN logic up would not have resulted in a better final score. \r\n\r\nGenerating a ~.621 score could however be achieved in about two weeks by focusing on the most relevant NN features. I would suggest to consider 3 of the 7 distance constants (1, 2.5 and 4) and omit the mid KNN features. Cutting the first level models from 100 to 10 and the second level models from 30 to 5 would also not result in a strong performance decrease (estimated decrease of 0.1% for a LB score of 0.62) and cut the computation time to less than a week. You could of course run the logic on multiple instances and further speed things up.\r\n\r\n[/quote]\r\n\r\nThanks for your answer! Yes, I was expecting something larger than the usual \"two days with a laptop\", but definitely not a month of running time in a decent workstation. Shame you can't bill Facebook now for the electricity... Anyway, congratulations again!\r\n", "126521": "Thank you @Tom, I will dive into your codes! : )", "126520": "If you inspect the code in the \"First level learners\" folder you will see that the framework was created to handle multiple first level learner types (the second group of models mentioned above). I experimented with the nnet and H2O neural network implementations but those were either too slow in transfering the data (H2O) or too biased (nnet). The way XGBoost handles missing values is another great advantage. Also, the XGBoost models are quite variable since they are trained on different random train batches with differing hyperparameters (eta constant, number of features and the number of considered candidates (either 20 or 100)). ", "126472": "@CarrDelling - I'm sorry, I forgot to answer your question related to the running time to generate the final submission. On my workstation it would take about a month. That seems like a ridiculously long time but it is explained by the extended computation time of the nearest neighbor features. While calculating the NN features I was continuously working on other parts of the workflow so speeding the NN logic up would not have resulted in a better final score. \r\n\r\nGenerating a ~.621 score could however be achieved in about two weeks by focusing on the most relevant NN features. I would suggest to consider 3 of the 7 distance constants (1, 2.5 and 4) and omit the mid KNN features. Cutting the first level models from 100 to 10 and the second level models from 30 to 5 would also not result in a strong performance decrease (estimated decrease of 0.1% for a LB score of 0.62) and cut the computation time to less than a week. You could of course run the logic on multiple instances and further speed things up.", "126408": "Congratulations to No.1 winner! This is definitely a noteworthy achievement and many thanks for sharing you wisdom! \r\n\r\nI will read your codes in great details when I have a bit more time and I'm sure I will learn a lot. For now, I'm attempting to make a layout of your strategy, please correct whe I get things wrong. Many thanks in advance! (and please bash me with mercy if I got things grossly wrong...)\r\n\r\n1. You use knn to find 100 candidate place_ids for each of the samples.\r\n\r\n2. You make 100 first level models, and each of them do the following:\r\n\tPick part of the time period for this model. For each of the place_ids included in this random pariod, use a binary classification model to predict the probability of belonging to this place_id, for all the relevant sample. Here, relevant means that this sample picked this place_id in its 100 candidates in step 1.\r\n\r\n3. Use an average of the 100 first level models to narrow down to 20 top candidates for each sample. Question: when you do the averaging, do you mean averaging the probability of a sample belonging to a place_id, predicted by each of the 100 models? If this is true, then these are actually likelyhoods, do you include prior probabilities of each of the relevant place_ids in condideration? If so, did you use global or local prior probabilities?\r\n\r\n\r\n4. The second level models part is a bit unclear to me. You said \"These first level model predictions are not used as features in the second level models\", and then you also said \"Second level models: 30 xgboost models were learnt to combine the predictions of the 100 first level models\". Do you mean you use the 20 candiate predictions, but not the raw predictions from the first level models? How did you vary the 30 xgb models so they get divergent enough to produce nice combinations that cover larger truth space?", "126364": "@Tom Thanks for the sharing. I have so much to learn from you. \r\nAnd congratulations for you again!!!", "126307": "Thank you for sharing! I'm impressed by all the data analysis (including the great Shiny app!) and feature engineering you did. I look forward to your full writeup to learn more about the details of the features and which ones were the most important.", "126276": "Really interesting solution Tom. Looking forward to read the full write up. ", "126258": "@aldente, that's an excellent question! I calculate the features for all 20 candidate records of each observation.\r\n\r\nInstead of doing all the calculations at once I perform all my calculations on random batches of the 3 considered data groups (train, validation, test). The batches are constructed such that they contain about 30,000 observations. This number was picked as high as possible while still allowing for sufficient memory during the modeling phase on my system. I ended up with 219 train batches, 73 validation batches and 287 test batches. Feature generation of a single batch takes about 8 minutes on my system. The feature files are about 800MB for each batch and I saved all the features to an external HD. Generating the 100 base level predictions for a single batch takes about 20 minutes and the second tier predictions take about one minute for each batch. I hope this clears your confusion.", "126246": "@Prof. Tom Van de Wiele, thank you so much for sharing, I appreciate it. I'll save questions if any after you share it in more detail. \r\n\r\nHowever my apologies, I have early (and maybe absurd) question: for the two levels, are your models having only 20 classes from the candidate selection? Or, is it 20 classes for each row (single observation)? \r\n\r\nThe question came from my curiosity, if it's the later (20 per row) then how the huge table of probabilities is saved? \r\n", "126582": "@Jiahuan He - No AWS or expensive hardware here. I only used my 8GB laptop and a modest 48 GB workstation. ", "126566": "Nah, IMHO *brute force* in this competition would have been to take the dataset as is, fire up a cluster of 50 x1.32xlarge machines in AWS, and XGBoost the problem to death. \r\n\r\nDeploying a solution on more than 60 scripts, and for which you need several documents in order to provide a proper description, feels like the opposite of brute force. I doubt we find more participants with the same amount of code in this competition, honestly. Don't look only at your position in the leaderboard, there's much more merit in your solution than just the @map3 score :)", "127873": "This is the installation instructions [link][1] for Rtools on Windows.\r\n\r\n\r\n [1]: https://github.com/stan-dev/rstan/wiki/Install-Rtools-for-Windows", "127865": "HI Tom,\r\n\r\nI am using R version 3.3 and when I check for packages there is no package called R tools. \r\n\r\nWhat version of R are you using?\r\n\r\nThanks\r\nKamatchi\r\n\r\n", "127862": "Hi Kamatchi,\r\n\r\nThis sounds like a compiler issue. Your first option is to read through the Rcpp [manual][1] (especially section 1.3). Do you have Rtools installed? The alternative is to follow these steps to skip the use of the fastOrder package:\r\n\r\n - Remove the library calls to fastOrder in 'candidateSel Tier I NN topCalc' and midKnnWrapper\r\n - Remove the reference at the foreach calls to fastOrder in the getTopKNNDT and midKnnCalc functions\r\n - Replace the fastOrder calls by order calls (one call in both getTopKNNDT and midKnnCalc)\r\n\r\n [1]: http://dirk.eddelbuettel.com/code/rcpp/Rcpp-FAQ.pdf", "126575": "Great works! I might need sometime to digest all these;) Just one question first, Did you use AWS or other servers or simply on your own PC?", "126425": "@Tom, sorry Google told me another Prof. Tom VdW ;) Thanks a lot for sharing the code, +1. I admire the winners reflect strong knowledge of data science. ", "126277": "> The major share of my 430 features is based on nearest neighbors features. I counted the neighbors for different Ks (1, 5, 10, 20, 50, 100, 250, 500, 1000, 2500) and different x-y ratio constants (1, 2.5, 4, 5.5, 7, 12, 30) and also included the mean time difference for all 70 combinations. \r\n\r\nExcuse me, but I haven't figured out, what are exactly these features? Probabilities of check-in belongs to a candidate place_id calculated by all KNN models along with mean time differences (overall 140 features)?", "126275": "Thank you for the reply, Prof. Tom van de Wiele, smart solution with the batches, looking forward to learn your solution in detail. Congratulations. ", "126269": "Awesome solution @Tom! It is surprising to see that a global approach definitely could work for this problem.\r\n\r\nOut of curiosity, what kind of resources (laptop, cluster...) did you use through the competition? And how much time (approx.) would it take to generate the winning solution? (or one close enough...)", "126264": "[quote=Tom Van de Wiele;126258]\r\n\r\n@aldente, that's an excellent question! I calculate the features for all 20 candidate records of each observation.\r\n\r\nInstead of doing all the calculations at once I perform all my calculations on random batches of the 3 considered data groups (train, validation, test). The batches are constructed such that they contain about 30,000 observations. This number was picked as high as possible while still allowing for sufficient memory during the modeling phase on my system. I ended up with 219 train batches, 73 validation batches and 287 test batches. Feature generation of a single batch takes about 8 minutes on my system. The feature files are about 800MB for each batch and I saved all the features to an external HD. Generating the 100 base level predictions for a single batch takes about 20 minutes and the second tier predictions take about one minute for each batch. I hope this clears your confusion.\r\n\r\n[/quote]\r\n\r\nThanks for sharing the solution! And congratulations on winning the competition! So if I understand your reply correctly, you calculated the features for all 20 candidate records of each observation, won't you end up having at least 19 negative examples and at most only 1 positive example? And so it is a rare event problem, and did you perform any downsampling (or any smart and efficient downsampling) on the negative class? In your first level model, you have 100 candidates, so the ratio of negative to positive examples is even much higher... You mentioned in your post that you did some sampling, but I am not sure if you just downsampling the negative examples or the entire data set...", "1015901": "", "1015898": "", "126324": "thanks and congratulation~" } }