# 3rd Place Solution

Congratulations to everyone and thanks to the team for organizing this.
I had fun and learnt a ton. Grateful to share my approach here. I have to note that I come from a traditional actuarial background in the life insurance industry, and am new to the world of machine learning (hoping to branch into this universe on a more fulltime basis). So would really appreciate your feedback and point out the flaws in my thinking!
Summary of my approach
spell checks + cosine similarity of words
separate model on each word
residual trend, bagging, hyperparameters
Model
I mainly only use LightGBM, for a few reasons:
Speed. I don't have good gears, so the rest (NNs, CatBoost, XGB) are very slow for me to run.
I read somewhere that the loss-guide tree growing method performs well on datasets that rely on distribution rules or close to synthetic, though I'm not sure why yet. Loss-guide is the default method of LGBM.
I think synthetically generated data for the purpose of competition would contain lots of random noise. And hence binning of numerical features should be quite important as a tree-based model is very sensitive to noise. This is also done by default in LGBM's histogram binning.
Local CV vs Public LB
As mentioned by others, I too faced the issue of uncorrelated local-cross validated rmse against public leaderboard rmse. Would appreciate it if anyone has any advice on how to approach this issue.
Things that I think worked and why
These are some of the things that I think works, although it's hard to be sure sometimes as it may improve the local cv but hardly on the public lb.
Spelling correction
There are plenty of spelling mistakes, I handled most with "symspellpy". However, the rest of the spelling mistakes are more context-based. And I've yet to learn how to do that, appreciate it if anyone can point me out. So I ended up with a very unelegant manual approach of querying for unrecognized words from the spell check runs + eyeballing them, some notable examples, "nger" to "finger', 'ha' to "hand", "whipl" to "whiplash", etc.
I think the importance of spelling correction should be quite straightforward, as it helps to link up the relationship between rows which would otherwise be delinked.
(One interesting thing I noted is, the texts seems to be mostly in the context of Australia, though I'm not sure how to make use of this idea.)
Cosine similarity of words
I couldn't quite get word embeddings to work with my LGBM model (be it raw or NMF), so I was trying to find other ways to link up words with similar meanings. My attempt is, for each word, using their pre-trained embeddings (BioBert), find the top 2 similar words to it and add them to the sentence. It's far from perfect but I do see many related body parts get added. This way, I think it forms a relationship between rows with similar elements when I do 1-gram TextVectorization later on.
LGBM on each word
I think the relationship of the numerical features to the target is very dependent on what kind of claim it is. And I'm not sure how to force LGBM to split on text columns first, it almost always starts and concentrates on InitialClaim. So my attempt to combat this is, with the vectorized text columns, I do LGBM for each text column for only rows that contain that text. And I just simply mean up the predictions across text columns. I think this approach helps LGBM to focus on the individual pattern for each text.
Residual trend
I feel tree-based model would leave out quite a lot of the small residual trend given it's more discrete predictions. So as a final step, I fit a Ridge model on the residual with a high alpha.
Hyperparameter, Bagging, Outlier
I thought "Gamma" objective would work well, given traditional actuarial theory suggesting that for claims size distribution, but I find "Poisson" objective works much better, which is rather puzzling to me. As I think synthetic data is very noisy, I find a high "min data in leaf" (200) works well to combat overfitting to noise. I do bagging across different seeds of shuffled kfold, feature sampling and max bin. I also clipped the 4m outlier to 800k to avoid it skewing my training too much.
Things that I couldn't get them to work
Many things. Some that I feel should work but lack the skills to get it to work are:
Linear LGBM given the inflationary nature of claim size, works very well locally with some tuning but suffer greatly on public lb.
De-trending Target before training
LSTM on Text concat with Dense on Numerical features. I read somewhere that it's hard to train this kind of model, as the LSTM leg has a different gradient/learning rate versus the other leg.
Hope this is a fun read for you. Looking forward to reading your feedback and other's solution!