{
  "id": 232350,
  "title": "3rd Place Solution",
  "content": "<p>Congratulations to everyone and thanks to the team for organizing this. </p>\n<p>I had fun and learnt a ton. Grateful to share my approach here. I have to note that I come from a traditional actuarial background in the life insurance industry, and am new to the world of machine learning (hoping to branch into this universe on a more fulltime basis). So would really appreciate your feedback and point out the flaws in my thinking! </p>\n<p><strong>Summary of my approach</strong></p>\n<ol>\n<li>spell checks + cosine similarity of words</li>\n<li>separate model on each word </li>\n<li>residual trend, bagging, hyperparameters</li>\n</ol>\n<p><strong>Model</strong><br>\nI mainly only use LightGBM, for a few reasons:</p>\n<ol>\n<li>Speed. I don't have good gears, so the rest (NNs, CatBoost, XGB) are very slow for me to run.</li>\n<li>I read somewhere that the loss-guide tree growing method performs well on datasets that rely on distribution rules or close to synthetic, though I'm not sure why yet. Loss-guide is the default method of LGBM.</li>\n<li>I think synthetically generated data for the purpose of competition would contain lots of random noise. And hence binning of numerical features should be quite important as a tree-based model is very sensitive to noise. This is also done by default in LGBM's histogram binning.</li>\n</ol>\n<p><strong>Local CV vs Public LB</strong><br>\nAs mentioned by others, I too faced the issue of uncorrelated local-cross validated rmse against public leaderboard rmse. Would appreciate it if anyone has any advice on how to approach this issue.</p>\n<p><strong>Things that I think worked and why</strong><br>\nThese are some of the things that I think works, although it's hard to be sure sometimes as it may improve the local cv but hardly on the public lb. </p>\n<ol>\n<li>Spelling correction<br>\nThere are plenty of spelling mistakes, I handled most with \"symspellpy\". However, the rest of the spelling mistakes are more context-based. And I've yet to learn how to do that, appreciate it if anyone can point me out. So I ended up with a very unelegant manual approach of querying for unrecognized words from the spell check runs + eyeballing them, some notable examples, \"nger\" to \"finger', 'ha' to \"hand\", \"whipl\" to \"whiplash\", etc. <br>\nI think the importance of spelling correction should be quite straightforward, as it helps to link up the relationship between rows which would otherwise be delinked. <br>\n(One interesting thing I noted is, the texts seems to be mostly in the context of Australia, though I'm not sure how to make use of this idea.)</li>\n<li>Cosine similarity of words<br>\nI couldn't quite get word embeddings to work with my LGBM model (be it raw or NMF), so I was trying to find other ways to link up words with similar meanings. My attempt is, for each word, using their pre-trained embeddings (BioBert), find the top 2 similar words to it and add them to the sentence. It's far from perfect but I do see many related body parts get added. This way, I think it forms a relationship between rows with similar elements when I do 1-gram TextVectorization later on. </li>\n<li>LGBM on each word<br>\nI think the relationship of the numerical features to the target is very dependent on what kind of claim it is. And I'm not sure how to force LGBM to split on text columns first, it almost always starts and concentrates on InitialClaim. So my attempt to combat this is, with the vectorized text columns, I do LGBM for each text column for only rows that contain that text. And I just simply mean up the predictions across text columns. I think this approach helps LGBM to focus on the individual pattern for each text.</li>\n<li>Residual trend<br>\nI feel tree-based model would leave out quite a lot of the small residual trend given it's more discrete predictions. So as a final step, I fit a Ridge model on the residual with a high alpha. </li>\n<li>Hyperparameter, Bagging, Outlier<br>\nI thought \"Gamma\" objective would work well, given traditional actuarial theory suggesting that for claims size distribution, but I find \"Poisson\" objective works much better, which is rather puzzling to me. As I think synthetic data is very noisy, I find a high \"min data in leaf\" (200) works well to combat overfitting to noise. I do bagging across different seeds of shuffled kfold, feature sampling and max bin. I also clipped the 4m outlier to 800k to avoid it skewing my training too much.</li>\n</ol>\n<p><strong>Things that I couldn't get them to work</strong><br>\nMany things. Some that I feel should work but lack the skills to get it to work are:</p>\n<ul>\n<li>Linear LGBM given the inflationary nature of claim size, works very well locally with some tuning but suffer greatly on public lb. </li>\n<li>De-trending Target before training</li>\n<li>LSTM on Text concat with Dense on Numerical features. I read somewhere that it's hard to train this kind of model, as the LSTM leg has a different gradient/learning rate versus the other leg.</li>\n</ul>\n<p>Hope this is a fun read for you. Looking forward to reading your feedback and other's solution!</p>",
  "comments": [
    {
      "id": 1276535,
      "author_name": "Issac Lee",
      "author_url": "",
      "post_date": "2021-04-17T17:06:34.553000",
      "content": "<p>Congratulation! I have learned a lot from this! Spelling check (very basic but I missed..) and LGBM on each word is knit!</p>",
      "votes": 1,
      "replies": [
        {
          "id": 1276801,
          "author_name": "teehoong",
          "author_url": "",
          "post_date": "2021-04-18T02:11:20.403000",
          "content": "<p>Thank you, Issac. I'm heartened by your comments!</p>",
          "votes": 1,
          "replies": []
        }
      ]
    },
    {
      "id": 1272823,
      "author_name": "aldparis",
      "author_url": "",
      "post_date": "2021-04-13T19:39:20.377000",
      "content": "<p>Congratulation !</p>\n<blockquote>\n  <p><strong>Local CV vs Public LB</strong><br>\n  As mentioned by others, I too faced the issue of uncorrelated local-cross validated rmse against public leaderboard rmse. Would appreciate it if anyone has any advice on how to approach this issue.</p>\n</blockquote>\n<p>I believe the only issue is : don't trust public leaderboard and trust only your CV !</p>",
      "votes": 1,
      "replies": [
        {
          "id": 1273361,
          "author_name": "Luca Massaron",
          "author_url": "",
          "post_date": "2021-04-14T09:18:53.173000",
          "content": "<p>Particularly, in this competition, we also found our local CV being kind of uncorrelated with the private dataset (we checked it in respect of our top solutions). Probably, the standard deviation of the CV remained high in spite of our efforts to stratify and bootstrap in order to obtain more reliable and stable estimates.</p>",
          "votes": 1,
          "replies": []
        }
      ]
    },
    {
      "id": 1272426,
      "author_name": "Luca Massaron",
      "author_url": "",
      "post_date": "2021-04-13T13:17:44.617000",
      "content": "<p>We used Linear LGBM, detrending/ price normalizing target as we were used in Axa, LSTM and also BERT, but none of them were decisive. We instead missed dealing with misspells.</p>",
      "votes": 1,
      "replies": [
        {
          "id": 1272466,
          "author_name": "teehoong",
          "author_url": "",
          "post_date": "2021-04-13T13:49:55.403000",
          "content": "<p>That's interesting. Thanks for sharing. By Axa do you mean the insurer? </p>",
          "votes": 0,
          "replies": []
        },
        {
          "id": 1272500,
          "author_name": "Luca Massaron",
          "author_url": "",
          "post_date": "2021-04-13T14:25:03.467000",
          "content": "<p>Yes, I worked there previously. We had specific procedures for price normalization and apart from that at the time I worked on projects modelling price normalizing parameters as latent variables using machine learning algorithms.</p>",
          "votes": 1,
          "replies": []
        },
        {
          "id": 1276800,
          "author_name": "teehoong",
          "author_url": "",
          "post_date": "2021-04-18T02:10:04.227000",
          "content": "<p>That's insightful and inspiring to see the practice of using ML in the actuarial field. Thanks for sharing.<br>\nIn contrast, I've not really seen any in my life insurance career so far. </p>",
          "votes": 0,
          "replies": []
        }
      ]
    },
    {
      "id": 1272419,
      "author_name": "Luca Massaron",
      "author_url": "",
      "post_date": "2021-04-13T13:11:01.937000",
      "content": "<p>LGBM on each word? How many LGBM models have you trained for that?</p>",
      "votes": 1,
      "replies": [
        {
          "id": 1272458,
          "author_name": "teehoong",
          "author_url": "",
          "post_date": "2021-04-13T13:43:58.267000",
          "content": "<p>Hello Luca. Yeap, there's actually not that many distinct words in this dataset, I recall is less than 2k. And I find that training just the top n most frequent words work better than training on all. I've thus picked the top 100/200/300/500 most frequent words, so that many LGBM models. Thankfully LGBM is very fast! </p>",
          "votes": 2,
          "replies": []
        }
      ]
    },
    {
      "id": 1287012,
      "author_name": "Jeremy",
      "author_url": "",
      "post_date": "2021-04-28T16:06:24.787000",
      "content": "<p>Hi teehoong</p>\n<p>Congratulations on finishing in 3rd place! I am Jeremy, one of the organisers of the competition. </p>\n<p>We would very much like to discuss your modelling approach and insights. If you are happy to get in touch, please drop me an email at jeremwang@deloitte.co.uk.</p>\n<p>Thanks<br>\nJeremy </p>",
      "votes": 0,
      "replies": []
    }
  ],
  "comment_count": 11,
  "messages": [
    {
      "id": 1272274,
      "postDate": "2021-04-13T11:27:00.970Z",
      "content": "<p>Congratulations to everyone and thanks to the team for organizing this. </p>\n<p>I had fun and learnt a ton. Grateful to share my approach here. I have to note that I come from a traditional actuarial background in the life insurance industry, and am new to the world of machine learning (hoping to branch into this universe on a more fulltime basis). So would really appreciate your feedback and point out the flaws in my thinking! </p>\n<p><strong>Summary of my approach</strong></p>\n<ol>\n<li>spell checks + cosine similarity of words</li>\n<li>separate model on each word </li>\n<li>residual trend, bagging, hyperparameters</li>\n</ol>\n<p><strong>Model</strong><br>\nI mainly only use LightGBM, for a few reasons:</p>\n<ol>\n<li>Speed. I don't have good gears, so the rest (NNs, CatBoost, XGB) are very slow for me to run.</li>\n<li>I read somewhere that the loss-guide tree growing method performs well on datasets that rely on distribution rules or close to synthetic, though I'm not sure why yet. Loss-guide is the default method of LGBM.</li>\n<li>I think synthetically generated data for the purpose of competition would contain lots of random noise. And hence binning of numerical features should be quite important as a tree-based model is very sensitive to noise. This is also done by default in LGBM's histogram binning.</li>\n</ol>\n<p><strong>Local CV vs Public LB</strong><br>\nAs mentioned by others, I too faced the issue of uncorrelated local-cross validated rmse against public leaderboard rmse. Would appreciate it if anyone has any advice on how to approach this issue.</p>\n<p><strong>Things that I think worked and why</strong><br>\nThese are some of the things that I think works, although it's hard to be sure sometimes as it may improve the local cv but hardly on the public lb. </p>\n<ol>\n<li>Spelling correction<br>\nThere are plenty of spelling mistakes, I handled most with \"symspellpy\". However, the rest of the spelling mistakes are more context-based. And I've yet to learn how to do that, appreciate it if anyone can point me out. So I ended up with a very unelegant manual approach of querying for unrecognized words from the spell check runs + eyeballing them, some notable examples, \"nger\" to \"finger', 'ha' to \"hand\", \"whipl\" to \"whiplash\", etc. <br>\nI think the importance of spelling correction should be quite straightforward, as it helps to link up the relationship between rows which would otherwise be delinked. <br>\n(One interesting thing I noted is, the texts seems to be mostly in the context of Australia, though I'm not sure how to make use of this idea.)</li>\n<li>Cosine similarity of words<br>\nI couldn't quite get word embeddings to work with my LGBM model (be it raw or NMF), so I was trying to find other ways to link up words with similar meanings. My attempt is, for each word, using their pre-trained embeddings (BioBert), find the top 2 similar words to it and add them to the sentence. It's far from perfect but I do see many related body parts get added. This way, I think it forms a relationship between rows with similar elements when I do 1-gram TextVectorization later on. </li>\n<li>LGBM on each word<br>\nI think the relationship of the numerical features to the target is very dependent on what kind of claim it is. And I'm not sure how to force LGBM to split on text columns first, it almost always starts and concentrates on InitialClaim. So my attempt to combat this is, with the vectorized text columns, I do LGBM for each text column for only rows that contain that text. And I just simply mean up the predictions across text columns. I think this approach helps LGBM to focus on the individual pattern for each text.</li>\n<li>Residual trend<br>\nI feel tree-based model would leave out quite a lot of the small residual trend given it's more discrete predictions. So as a final step, I fit a Ridge model on the residual with a high alpha. </li>\n<li>Hyperparameter, Bagging, Outlier<br>\nI thought \"Gamma\" objective would work well, given traditional actuarial theory suggesting that for claims size distribution, but I find \"Poisson\" objective works much better, which is rather puzzling to me. As I think synthetic data is very noisy, I find a high \"min data in leaf\" (200) works well to combat overfitting to noise. I do bagging across different seeds of shuffled kfold, feature sampling and max bin. I also clipped the 4m outlier to 800k to avoid it skewing my training too much.</li>\n</ol>\n<p><strong>Things that I couldn't get them to work</strong><br>\nMany things. Some that I feel should work but lack the skills to get it to work are:</p>\n<ul>\n<li>Linear LGBM given the inflationary nature of claim size, works very well locally with some tuning but suffer greatly on public lb. </li>\n<li>De-trending Target before training</li>\n<li>LSTM on Text concat with Dense on Numerical features. I read somewhere that it's hard to train this kind of model, as the LSTM leg has a different gradient/learning rate versus the other leg.</li>\n</ul>\n<p>Hope this is a fun read for you. Looking forward to reading your feedback and other's solution!</p>",
      "rawMarkdown": "Congratulations to everyone and thanks to the team for organizing this. \n\nI had fun and learnt a ton. Grateful to share my approach here. I have to note that I come from a traditional actuarial background in the life insurance industry, and am new to the world of machine learning (hoping to branch into this universe on a more fulltime basis). So would really appreciate your feedback and point out the flaws in my thinking! \n\n**Summary of my approach**\n1. spell checks + cosine similarity of words\n2. separate model on each word \n3. residual trend, bagging, hyperparameters\n\n**Model**\nI mainly only use LightGBM, for a few reasons:\n1. Speed. I don't have good gears, so the rest (NNs, CatBoost, XGB) are very slow for me to run.\n2. I read somewhere that the loss-guide tree growing method performs well on datasets that rely on distribution rules or close to synthetic, though I'm not sure why yet. Loss-guide is the default method of LGBM.\n3. I think synthetically generated data for the purpose of competition would contain lots of random noise. And hence binning of numerical features should be quite important as a tree-based model is very sensitive to noise. This is also done by default in LGBM's histogram binning.\n \n**Local CV vs Public LB**\nAs mentioned by others, I too faced the issue of uncorrelated local-cross validated rmse against public leaderboard rmse. Would appreciate it if anyone has any advice on how to approach this issue.\n\n**Things that I think worked and why**\nThese are some of the things that I think works, although it's hard to be sure sometimes as it may improve the local cv but hardly on the public lb. \n1.  Spelling correction\nThere are plenty of spelling mistakes, I handled most with \"symspellpy\". However, the rest of the spelling mistakes are more context-based. And I've yet to learn how to do that, appreciate it if anyone can point me out. So I ended up with a very unelegant manual approach of querying for unrecognized words from the spell check runs + eyeballing them, some notable examples, \"nger\" to \"finger', 'ha' to \"hand\", \"whipl\" to \"whiplash\", etc. \nI think the importance of spelling correction should be quite straightforward, as it helps to link up the relationship between rows which would otherwise be delinked. \n(One interesting thing I noted is, the texts seems to be mostly in the context of Australia, though I'm not sure how to make use of this idea.)\n2. Cosine similarity of words\nI couldn't quite get word embeddings to work with my LGBM model (be it raw or NMF), so I was trying to find other ways to link up words with similar meanings. My attempt is, for each word, using their pre-trained embeddings (BioBert), find the top 2 similar words to it and add them to the sentence. It's far from perfect but I do see many related body parts get added. This way, I think it forms a relationship between rows with similar elements when I do 1-gram TextVectorization later on. \n3. LGBM on each word\nI think the relationship of the numerical features to the target is very dependent on what kind of claim it is. And I'm not sure how to force LGBM to split on text columns first, it almost always starts and concentrates on InitialClaim. So my attempt to combat this is, with the vectorized text columns, I do LGBM for each text column for only rows that contain that text. And I just simply mean up the predictions across text columns. I think this approach helps LGBM to focus on the individual pattern for each text.\n4. Residual trend\nI feel tree-based model would leave out quite a lot of the small residual trend given it's more discrete predictions. So as a final step, I fit a Ridge model on the residual with a high alpha. \n5. Hyperparameter, Bagging, Outlier\nI thought \"Gamma\" objective would work well, given traditional actuarial theory suggesting that for claims size distribution, but I find \"Poisson\" objective works much better, which is rather puzzling to me. As I think synthetic data is very noisy, I find a high \"min data in leaf\" (200) works well to combat overfitting to noise. I do bagging across different seeds of shuffled kfold, feature sampling and max bin. I also clipped the 4m outlier to 800k to avoid it skewing my training too much.\n\n**Things that I couldn't get them to work**\nMany things. Some that I feel should work but lack the skills to get it to work are:\n- Linear LGBM given the inflationary nature of claim size, works very well locally with some tuning but suffer greatly on public lb. \n- De-trending Target before training\n- LSTM on Text concat with Dense on Numerical features. I read somewhere that it's hard to train this kind of model, as the LSTM leg has a different gradient/learning rate versus the other leg.\n\nHope this is a fun read for you. Looking forward to reading your feedback and other's solution!\n",
      "votes": 10
    },
    {
      "id": 1276535,
      "postDate": "2021-04-17T17:06:34.553Z",
      "content": "<p>Congratulation! I have learned a lot from this! Spelling check (very basic but I missed..) and LGBM on each word is knit!</p>",
      "rawMarkdown": "Congratulation! I have learned a lot from this! Spelling check (very basic but I missed..) and LGBM on each word is knit!",
      "votes": 1,
      "replies": [
        {
          "id": 1276801,
          "postDate": "2021-04-18T02:11:20.403Z",
          "content": "<p>Thank you, Issac. I'm heartened by your comments!</p>",
          "rawMarkdown": "Thank you, Issac. I'm heartened by your comments!",
          "votes": 1
        }
      ]
    },
    {
      "id": 1272823,
      "postDate": "2021-04-13T19:39:20.377Z",
      "content": "<p>Congratulation !</p>\n<blockquote>\n  <p><strong>Local CV vs Public LB</strong><br>\n  As mentioned by others, I too faced the issue of uncorrelated local-cross validated rmse against public leaderboard rmse. Would appreciate it if anyone has any advice on how to approach this issue.</p>\n</blockquote>\n<p>I believe the only issue is : don't trust public leaderboard and trust only your CV !</p>",
      "rawMarkdown": "Congratulation !\n> **Local CV vs Public LB**\nAs mentioned by others, I too faced the issue of uncorrelated local-cross validated rmse against public leaderboard rmse. Would appreciate it if anyone has any advice on how to approach this issue.\n\nI believe the only issue is : don't trust public leaderboard and trust only your CV !",
      "votes": 1,
      "replies": [
        {
          "id": 1273361,
          "postDate": "2021-04-14T09:18:53.173Z",
          "content": "<p>Particularly, in this competition, we also found our local CV being kind of uncorrelated with the private dataset (we checked it in respect of our top solutions). Probably, the standard deviation of the CV remained high in spite of our efforts to stratify and bootstrap in order to obtain more reliable and stable estimates.</p>",
          "rawMarkdown": "Particularly, in this competition, we also found our local CV being kind of uncorrelated with the private dataset (we checked it in respect of our top solutions). Probably, the standard deviation of the CV remained high in spite of our efforts to stratify and bootstrap in order to obtain more reliable and stable estimates.",
          "votes": 1
        }
      ]
    },
    {
      "id": 1272426,
      "postDate": "2021-04-13T13:17:44.617Z",
      "content": "<p>We used Linear LGBM, detrending/ price normalizing target as we were used in Axa, LSTM and also BERT, but none of them were decisive. We instead missed dealing with misspells.</p>",
      "rawMarkdown": "We used Linear LGBM, detrending/ price normalizing target as we were used in Axa, LSTM and also BERT, but none of them were decisive. We instead missed dealing with misspells.",
      "votes": 1,
      "replies": [
        {
          "id": 1272466,
          "postDate": "2021-04-13T13:49:55.403Z",
          "content": "<p>That's interesting. Thanks for sharing. By Axa do you mean the insurer? </p>",
          "rawMarkdown": "That's interesting. Thanks for sharing. By Axa do you mean the insurer? "
        },
        {
          "id": 1272500,
          "postDate": "2021-04-13T14:25:03.467Z",
          "content": "<p>Yes, I worked there previously. We had specific procedures for price normalization and apart from that at the time I worked on projects modelling price normalizing parameters as latent variables using machine learning algorithms.</p>",
          "rawMarkdown": "Yes, I worked there previously. We had specific procedures for price normalization and apart from that at the time I worked on projects modelling price normalizing parameters as latent variables using machine learning algorithms.",
          "votes": 1
        },
        {
          "id": 1276800,
          "postDate": "2021-04-18T02:10:04.227Z",
          "content": "<p>That's insightful and inspiring to see the practice of using ML in the actuarial field. Thanks for sharing.<br>\nIn contrast, I've not really seen any in my life insurance career so far. </p>",
          "rawMarkdown": "That's insightful and inspiring to see the practice of using ML in the actuarial field. Thanks for sharing.\nIn contrast, I've not really seen any in my life insurance career so far. "
        }
      ]
    },
    {
      "id": 1272419,
      "postDate": "2021-04-13T13:11:01.937Z",
      "content": "<p>LGBM on each word? How many LGBM models have you trained for that?</p>",
      "rawMarkdown": "LGBM on each word? How many LGBM models have you trained for that?",
      "votes": 1,
      "replies": [
        {
          "id": 1272458,
          "postDate": "2021-04-13T13:43:58.267Z",
          "content": "<p>Hello Luca. Yeap, there's actually not that many distinct words in this dataset, I recall is less than 2k. And I find that training just the top n most frequent words work better than training on all. I've thus picked the top 100/200/300/500 most frequent words, so that many LGBM models. Thankfully LGBM is very fast! </p>",
          "rawMarkdown": "Hello Luca. Yeap, there's actually not that many distinct words in this dataset, I recall is less than 2k. And I find that training just the top n most frequent words work better than training on all. I've thus picked the top 100/200/300/500 most frequent words, so that many LGBM models. Thankfully LGBM is very fast! ",
          "votes": 2
        }
      ]
    },
    {
      "id": 1287012,
      "postDate": "2021-04-28T16:06:24.787Z",
      "content": "<p>Hi teehoong</p>\n<p>Congratulations on finishing in 3rd place! I am Jeremy, one of the organisers of the competition. </p>\n<p>We would very much like to discuss your modelling approach and insights. If you are happy to get in touch, please drop me an email at jeremwang@deloitte.co.uk.</p>\n<p>Thanks<br>\nJeremy </p>",
      "rawMarkdown": "Hi teehoong\n\nCongratulations on finishing in 3rd place! I am Jeremy, one of the organisers of the competition. \n\nWe would very much like to discuss your modelling approach and insights. If you are happy to get in touch, please drop me an email at jeremwang@deloitte.co.uk.\n\nThanks\nJeremy \n\n"
    }
  ],
  "complete": true,
  "source": "live",
  "fetched_at": "2026-10-19T19:14:02Z",
  "competition_id": 24764,
  "competition_slug": "actuarial-loss-estimation",
  "team_id": 6136817,
  "solution_url": "/competitions/actuarial-loss-estimation/writeups/teehoong-3rd-place-solution"
}