Interesting competition in a time when tabular competitions on Kaggle are quite rare and based only on synthetically generated data (oh, but also this one is a synthetic one). In addition, this competition has been somewhat reminiscent of older times for me, because I don't work anymore in insurance, having shifted to finance (instead my teammate, Matteo, still works in the sector).
\nSince now the competition has closed, I can reveal that I found the competition a bit puzzling because CV and public LB were always so different (unless you kept a single outlier in your training set) and now again the private has complete different values. Having seen more than 200 competitions up so far, I don't get much surprised anymore that so often public and private scores are not consistent in respect each other and that some shake-up happens. Another puzzling aspect of the completion was the task to predict on the same period of the training data. Though it is not uncommon to have to predict on the same distribution of the training data, a more realistic task would have been to predict on a subsequent period (for instance in our case on year 2006).
\nAnyway, here are the characteristics of our solution, which is a somehow computational intensive one due to example bootstrapping (bagging), but we dealt with it completely using Kaggle Kernels. Probably we won’t release more of our solution than what you are going to read here, unless we decide to write a paper on it.
\nOur solution is a blending of bagged 2nd level stacking solutions. We built many diverse models (Catboost, Lightgbm, XGBoost, Random Forest, Regularized Greedy Forests, simple neural DNN, Tabnet, Node, K-Nearest Neighbors, GLMs, Support Vector Machines) always using the cross-validation-prediction approach on the same seed (https://scikit-learn.org/stable/modules/generated/sklearn.model_selection.cross_val_predict.html).
\nWe used a custom stratified cross-validation to reproduce the training distribution accurately.
\nWe excluded two outlying observations based on target and features' values.
\nAs for as feature engineering, we used both the original features and some features derived from autoencoders processing with and without multi-head attention.
\nWe also \"cleaned\" key predictive variables by custom rule based functions and using some dedicated models.
\nWe extracted further information from the target using dedicated models and reused it as features.
\nIn the end, we devised quite a few features based on existing ones, especially features based on time since we realized that time mattered (not only both accident and claim but also the order of records).
\nAs for the text, we used the simple raw counting of words and their n-grams and a projection of descriptions to WCIO actuarial tables using GloVe embeddings.
\nEach model has been run multiple times on bootstraps of the training data and then all the models were put into ensemble using a constrained linear model.
\nWe repeated once more all the above operations using the first models in order to feed the second level ones.
\nFinally, we blended the first and second level models coming from different runs based on different random seeds.
\nWe submitted the best CV model and the best LB model. We noticed that \"post-processing\" worked on the submission on LB, that is, you could adjust your predictions using coefficients based on accident time and your LB improved (but our CV worsened, of course). We deducted that such happened as a kind of overfitting on the tails of the public LB and we decided not to use any post-processed submission (correctly, because none of them performed better on the private).
\nBefore submitting, we tested that our model's residuals had no time correlation and that residuals were correctly centered and distributed. We couldn't find a way to reduced the variance of the residuals of accidents from 2001-2002 onward (which was much higher than the previous years) and we believe, given the short distance of our solution to the top ones, that there is little way of doing it due to the some underlying stochastic process or other latent generative process.
", "comments": [ { "id": 1271310, "author_name": "Thac Do", "author_url": "", "post_date": "2021-04-12T13:21:58.070000", "content": "Thanks for sharing!
", "votes": 1, "replies": [ { "id": 1271324, "author_name": "Luca Massaron", "author_url": "", "post_date": "2021-04-12T13:35:49.187000", "content": "You are welcome. For us the keys (in order of importance) for this competition are:
\nLet's see if we missed something else by reading others' solutions :-)
", "votes": 0, "replies": [] } ] }, { "id": 1322976, "author_name": "Jeremy", "author_url": "", "post_date": "2021-05-25T20:55:06.803000", "content": "Hi Luca
\nThank you for participating in the competition and sharing your approach.
\nI am Jeremy, one of the organisers of the competition.
\nWe would very much like to discuss your modelling approach and insights in more detail. (https://www.kaggle.com/c/actuarial-loss-estimation/discussion/234916)
\nIf you are happy to get in touch, please drop me an email at jeremwang@deloitte.co.uk.
\nThanks
\nJeremy
Interesting competition in a time when tabular competitions on Kaggle are quite rare and based only on synthetically generated data (oh, but also this one is a synthetic one). In addition, this competition has been somewhat reminiscent of older times for me, because I don't work anymore in insurance, having shifted to finance (instead my teammate, Matteo, still works in the sector).
\nSince now the competition has closed, I can reveal that I found the competition a bit puzzling because CV and public LB were always so different (unless you kept a single outlier in your training set) and now again the private has complete different values. Having seen more than 200 competitions up so far, I don't get much surprised anymore that so often public and private scores are not consistent in respect each other and that some shake-up happens. Another puzzling aspect of the completion was the task to predict on the same period of the training data. Though it is not uncommon to have to predict on the same distribution of the training data, a more realistic task would have been to predict on a subsequent period (for instance in our case on year 2006).
\nAnyway, here are the characteristics of our solution, which is a somehow computational intensive one due to example bootstrapping (bagging), but we dealt with it completely using Kaggle Kernels. Probably we won’t release more of our solution than what you are going to read here, unless we decide to write a paper on it.
\nOur solution is a blending of bagged 2nd level stacking solutions. We built many diverse models (Catboost, Lightgbm, XGBoost, Random Forest, Regularized Greedy Forests, simple neural DNN, Tabnet, Node, K-Nearest Neighbors, GLMs, Support Vector Machines) always using the cross-validation-prediction approach on the same seed (https://scikit-learn.org/stable/modules/generated/sklearn.model_selection.cross_val_predict.html).
\nWe used a custom stratified cross-validation to reproduce the training distribution accurately.
\nWe excluded two outlying observations based on target and features' values.
\nAs for as feature engineering, we used both the original features and some features derived from autoencoders processing with and without multi-head attention.
\nWe also \"cleaned\" key predictive variables by custom rule based functions and using some dedicated models.
\nWe extracted further information from the target using dedicated models and reused it as features.
\nIn the end, we devised quite a few features based on existing ones, especially features based on time since we realized that time mattered (not only both accident and claim but also the order of records).
\nAs for the text, we used the simple raw counting of words and their n-grams and a projection of descriptions to WCIO actuarial tables using GloVe embeddings.
\nEach model has been run multiple times on bootstraps of the training data and then all the models were put into ensemble using a constrained linear model.
\nWe repeated once more all the above operations using the first models in order to feed the second level ones.
\nFinally, we blended the first and second level models coming from different runs based on different random seeds.
\nWe submitted the best CV model and the best LB model. We noticed that \"post-processing\" worked on the submission on LB, that is, you could adjust your predictions using coefficients based on accident time and your LB improved (but our CV worsened, of course). We deducted that such happened as a kind of overfitting on the tails of the public LB and we decided not to use any post-processed submission (correctly, because none of them performed better on the private).
\nBefore submitting, we tested that our model's residuals had no time correlation and that residuals were correctly centered and distributed. We couldn't find a way to reduced the variance of the residuals of accidents from 2001-2002 onward (which was much higher than the previous years) and we believe, given the short distance of our solution to the top ones, that there is little way of doing it due to the some underlying stochastic process or other latent generative process.
", "rawMarkdown": "Interesting competition in a time when tabular competitions on Kaggle are quite rare and based only on synthetically generated data (oh, but also this one is a synthetic one). In addition, this competition has been somewhat reminiscent of older times for me, because I don't work anymore in insurance, having shifted to finance (instead my teammate, Matteo, still works in the sector).\n\nSince now the competition has closed, I can reveal that I found the competition a bit puzzling because CV and public LB were always so different (unless you kept a single outlier in your training set) and now again the private has complete different values. Having seen more than 200 competitions up so far, I don't get much surprised anymore that so often public and private scores are not consistent in respect each other and that some shake-up happens. Another puzzling aspect of the completion was the task to predict on the same period of the training data. Though it is not uncommon to have to predict on the same distribution of the training data, a more realistic task would have been to predict on a subsequent period (for instance in our case on year 2006). \n\nAnyway, here are the characteristics of our solution, which is a somehow computational intensive one due to example bootstrapping (bagging), but we dealt with it completely using Kaggle Kernels. Probably we won’t release more of our solution than what you are going to read here, unless we decide to write a paper on it.\n\nOur solution is a blending of bagged 2nd level stacking solutions. We built many diverse models (Catboost, Lightgbm, XGBoost, Random Forest, Regularized Greedy Forests, simple neural DNN, Tabnet, Node, K-Nearest Neighbors, GLMs, Support Vector Machines) always using the cross-validation-prediction approach on the same seed (https://scikit-learn.org/stable/modules/generated/sklearn.model_selection.cross_val_predict.html). \n\nWe used a custom stratified cross-validation to reproduce the training distribution accurately. \n\nWe excluded two outlying observations based on target and features' values.\n\nAs for as feature engineering, we used both the original features and some features derived from autoencoders processing with and without multi-head attention. \n\nWe also \"cleaned\" key predictive variables by custom rule based functions and using some dedicated models.\n\nWe extracted further information from the target using dedicated models and reused it as features.\n\nIn the end, we devised quite a few features based on existing ones, especially features based on time since we realized that time mattered (not only both accident and claim but also the order of records). \n\nAs for the text, we used the simple raw counting of words and their n-grams and a projection of descriptions to WCIO actuarial tables using GloVe embeddings. \n\nEach model has been run multiple times on bootstraps of the training data and then all the models were put into ensemble using a constrained linear model.\n\nWe repeated once more all the above operations using the first models in order to feed the second level ones.\n\nFinally, we blended the first and second level models coming from different runs based on different random seeds.\n\nWe submitted the best CV model and the best LB model. We noticed that \"post-processing\" worked on the submission on LB, that is, you could adjust your predictions using coefficients based on accident time and your LB improved (but our CV worsened, of course). We deducted that such happened as a kind of overfitting on the tails of the public LB and we decided not to use any post-processed submission (correctly, because none of them performed better on the private).\n\nBefore submitting, we tested that our model's residuals had no time correlation and that residuals were correctly centered and distributed. We couldn't find a way to reduced the variance of the residuals of accidents from 2001-2002 onward (which was much higher than the previous years) and we believe, given the short distance of our solution to the top ones, that there is little way of doing it due to the some underlying stochastic process or other latent generative process. \n", "votes": 14 }, { "id": 1271310, "postDate": "2021-04-12T13:21:58.070Z", "content": "Thanks for sharing!
", "rawMarkdown": "Thanks for sharing!", "votes": 1, "replies": [ { "id": 1271324, "postDate": "2021-04-12T13:35:49.187Z", "content": "You are welcome. For us the keys (in order of importance) for this competition are:
\nLet's see if we missed something else by reading others' solutions :-)
", "rawMarkdown": "Let's see if we missed something else by reading others' solutions :-)" } ] }, { "id": 1322976, "postDate": "2021-05-25T20:55:06.803Z", "content": "Hi Luca
\nThank you for participating in the competition and sharing your approach.
\nI am Jeremy, one of the organisers of the competition.
\nWe would very much like to discuss your modelling approach and insights in more detail. (https://www.kaggle.com/c/actuarial-loss-estimation/discussion/234916)
\nIf you are happy to get in touch, please drop me an email at jeremwang@deloitte.co.uk.
\nThanks
\nJeremy