{ "id": 453809, "title": "2nd-place Solution : LLM Decoders, Assignment Optimization, and Customize Post-Process", "content": "

In this post I would like to present our team’s solution write-up that finished ….. place (public 2nd place) in the “h2o.ai Predict the LLM” competition on Kaggle. We would like to express our thanks to the Kaggle team, to h2o.ai (@robikscube, @philippsinger), and to all fellow competitors for such a short-lived joyful experience during the 2 weeks of this state-of-the-art hackathon.
\nPersonally, I also would like to thank my teammate (@phanisrikanth Phani, aka Binga) for our short but fruitful collaboration. Hope we can work together in another competition soon.
\nIn short, here are the 2 major ideas that mainly contributed to our success:

\n\n

1. THE LLM MODELS: (Binga)

\n

On Kaggle, it’s established that encoders, such as DeBERTa, have ruled NLP competitions on Kaggle for years. To brush up my NLP skills, I joined this competition on day 1 and started with encoders. I realized that simple DeBERTa models scored in the range of 1.6, 1.4 on mlogloss metric.
\n1.1 Model Selection
\nGiven that it’s 2023 and we have new decoders coming out almost each day, I decided to use a decoder model out of sheer curiosity. Now, which models do we begin with? I went with the most popular Mistral model for 2 reasons: 1) Performance. It outperforms Llama2-13b on some benchmarks, and 2) Apache 2.0 License.
\n1.2 Experiments
\nEach of the experiments were done on H2O-LLMStudio. It’s a fantastic no-code tool to quickly run experiments with many LLMs. Highly recommend having this tool in your data science toolbox. 🙂
\nComing to experiments, with Mistral-Instruct-v0.1, my local validation loss dramatically reduced and the leaderboard score fell from 1.6 to 1.063. This seemed like the right direction as psi’s benchmark was 0.88 and we’re getting there. A little bit of tuning with epochs and I got 0.925 on LB. I tried modifying int4/int8/bfloat16 finetuning, LoRA parameters and always saw an adverse effect. Details in my logbook. One of the key parameters that improved my score was prompt length, answer length. The more they are, the better the logloss. This was easy to explain as some of the sentences (specifically answers) were longer than 2000 in length.
\nOne of the interesting parameters is the seed. Fixing the seed led to deterministic results but a random seed resulted in a different logloss and LB score. The variance is 0.05 which is extremely high in this competition. This led to the conclusion that averaging multiple models was the way to go to fight variance.
\nAlong the way, I clustered the data to visually explore it using cluestar library and found that GroupKFold was the way to go. This was a moment of personal learning that “explore the data first before modeling”.
\n1.3 Open Models
\nWith huggingface hosting so many models, the open_llm_leaderboard shows top models based on benchmark scores. Motivated to further improve the score, I tried multiple models finetuned on top of Mistral and while some worked, some did not. Ablation studies are in the logbook.
\n1.4 Averaging
\nLastly, these models were not perfectly correlated and that led to nice performance on averaging. So, a straight average of predictions (not oof predictions) from the 9 models below, with different seeds, led to 0.63 on the leaderboard. Here is the architecture.
\n5 days prior to the competition end, Kha and I decided to team up and his optimization work (details in the next section) was a big part of this solution going from 0.63 to 0.50 on the public leaderboard.
\n\"\"
\n1.5 Infrastructure
\nAll the decoder models were trained on a machine with 8 x V100 GPUs. This meant that each finetuning run approximately took around 60 minutes along with inference on test data.
\n1.6 Note from binga
\nA big thanks to H2O.ai, Kaggle and the organizers for setting up this wonderful competition where I learnt so much about SoTA NLP.
\nThanks to AI practitioners in open source who are behind projects such as Open Hermes, Open Orca etc for making these great models available to us.
\nSpecial thanks to Kha for giving me an opportunity to team up with him! I've learnt so much from Kha -- understanding data, doing quick experiments and most importantly a never-say-die attitude in the last 2 days when we were challenged by competitors such as Reacher and that made us push ourselves during the last hours.

\n

2. OPTIMIZATION AND POST-PROCESS (Kha)

\n

This competition’s data has a very special trait, that for each question group, we should expect exactly 1 response for each of 7 label targets. During the normal modeling process, we may not exploit any of this extra information about relation between data samples.
\nThis sparks an idea of using an optimization method to find the best assignment of targets for a group of 7 responses to a specific question. The optimization method takes as input a 7x7 matrix, where each row is the normalized probability (sum to 1) of 1 data sample. We then try to maximize the selected probabilities given the constraint that each target must be assigned exactly one time.
\nThis optimization method alone, however, cannot outperform the original predictions due to its extreme nature (of hard 0/1 prediction) that can't drive the log-loss down. So I need to perform some more extra steps of combining this binary prediction (also a 7x7 matrix) with the original matrix. The combining steps and optimization coefficients were achieved manually by a systematic search to minimize the out-of-fold predictions. The single steps of post-processing are shown in the function below.
\nYou may wonder why there are some highly manual steps. In general, they are just the results of the following ideas:

\n\n
 ():\n    q, _df  df1.groupby(): \n\n       \n       indices = _df.index.values\n       cost_matrix = -pred[indices]\n       row_idx, col_idx = linear_sum_assignment(cost_matrix)\n       orig = pred[indices]\n       hard_pred = np.zeros((,))\n       hard_pred[row_idx, col_idx] = \n       hard_pred_minus = -hard_pred\n\n       orig_selected = np.zeros((,))\n       orig_selected[row_idx, col_idx] = orig[row_idx, col_idx]\n       orig_unselected = orig.copy()\n       orig_unselected[row_idx, col_idx] = \n       soft_pred_1 = orig_selected + (hard_pred-orig_selected)*\n       soft_pred_2 = orig_unselected - orig_unselected**\n       soft_pred = soft_pred_1 + soft_pred_2\n       pred[indices] = soft_pred\n\n       \n        jcol  ():\n           max_idx = np.argmax(soft_pred[:, jcol])\n            (soft_pred[:, jcol]) > :\n               soft_pred[:, jcol] = \n               soft_pred[max_idx, jcol] = \n            (soft_pred[:, jcol]) > :\n               soft_pred[:, jcol] -= soft_pred[:, jcol]*\n               soft_pred[max_idx, jcol] += (-soft_pred[max_idx, jcol])*\n            (soft_pred[:, jcol]) > :\n               soft_pred[:, jcol] -= soft_pred[:, jcol]*\n               soft_pred[max_idx, jcol] += (-soft_pred[max_idx, jcol])*\n\n       pred[indices] = soft_pred\n\n    pred\noptimized_preds = optim(test, probabilities) \n
\n

3. THE OTHER IDEAS THAT ALSO WORKED (Kha)

\n

Sidelining the main 2 major factors presented, I also experimented with some other classical approaches and found out that they worked nicely, but I couldn’t find the time to combine that with Phani’s because Phani cannot produce the out-of-fold predictions on time. My methods highly depend on the train out-of-fold predictions. Even the binary classification idea and post-process was based on my out-of-fold deberta-v3-large predictions, instead of the best Phani’s LLM model. Had the optimization been tuned on the latter, we would have a better final score I’m pretty sure.
\n3.1 Using a tree-based method (LightGBM) to learn a stacking model on top of the NLP predictions.
\nThe crafted features that helped are:

\n", "comments": [ { "id": 2516916, "author_name": "binga", "author_url": "", "post_date": "2023-11-08T04:29:52.013000", "content": "

I'd like to once again thank h2o.ai, Kaggle and organizers for setting up this wonderful competition. While I have learnt a ton of new things working with decoder models, I had a lot of fun :-)

\n

The decoder part of the solution is available here - Link.

\n

Here are a few more resources if you're interested.

\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n
ResourceLink
ML Experiments LogbookLink
H2O LLMStudio Training yamlLink
ML Experiment RunsLink
Beating the Benchmark SolutionLink
Data Exploration via Clustering (which hinted the GroupKFold validation)Link
\n
\n

Congratulations to all the winners! Special congratulations to @ihebch for his persistence. You made us push harder and you came out ahead across the finish line with a very well deserved solution! It was a super exciting finish :-)

", "votes": 4, "replies": [] }, { "id": 2519919, "author_name": "Kha Vo", "author_url": "", "post_date": "2023-11-10T13:09:22.190000", "content": "

Our full inference code is released https://www.kaggle.com/khahuras/2nd-place-solution

", "votes": 1, "replies": [] }, { "id": 2517672, "author_name": "Psi", "author_url": "", "post_date": "2023-11-08T17:06:40.147000", "content": "

Great stuff, thanks for the detailed share and congrats!

", "votes": 1, "replies": [] }, { "id": 2517605, "author_name": "RB", "author_url": "", "post_date": "2023-11-08T16:10:30.923000", "content": "

Thank you for sharing PP @khahuras , it's really effective , I'm going to take a closer look.

\n

Results on single 5 fold Zephyr-7b-beta model -

\n
No PP ->  private LB , public LB  \n PP -> private LB , public LB \n
", "votes": 1, "replies": [ { "id": 2517655, "author_name": "Kha Vo", "author_url": "", "post_date": "2023-11-08T16:54:51.103000", "content": "

Glad to hear that!

", "votes": 1, "replies": [] } ] }, { "id": 2528178, "author_name": "", "author_url": "", "post_date": "2023-11-17T06:48:37.327000", "content": "", "votes": 0, "replies": [] } ], "comment_count": 6, "messages": [ { "id": 2516603, "postDate": "2023-11-07T20:04:55.233Z", "content": "

In this post I would like to present our team’s solution write-up that finished ….. place (public 2nd place) in the “h2o.ai Predict the LLM” competition on Kaggle. We would like to express our thanks to the Kaggle team, to h2o.ai (@robikscube, @philippsinger), and to all fellow competitors for such a short-lived joyful experience during the 2 weeks of this state-of-the-art hackathon.
\nPersonally, I also would like to thank my teammate (@phanisrikanth Phani, aka Binga) for our short but fruitful collaboration. Hope we can work together in another competition soon.
\nIn short, here are the 2 major ideas that mainly contributed to our success:

\n\n

1. THE LLM MODELS: (Binga)

\n

On Kaggle, it’s established that encoders, such as DeBERTa, have ruled NLP competitions on Kaggle for years. To brush up my NLP skills, I joined this competition on day 1 and started with encoders. I realized that simple DeBERTa models scored in the range of 1.6, 1.4 on mlogloss metric.
\n1.1 Model Selection
\nGiven that it’s 2023 and we have new decoders coming out almost each day, I decided to use a decoder model out of sheer curiosity. Now, which models do we begin with? I went with the most popular Mistral model for 2 reasons: 1) Performance. It outperforms Llama2-13b on some benchmarks, and 2) Apache 2.0 License.
\n1.2 Experiments
\nEach of the experiments were done on H2O-LLMStudio. It’s a fantastic no-code tool to quickly run experiments with many LLMs. Highly recommend having this tool in your data science toolbox. 🙂
\nComing to experiments, with Mistral-Instruct-v0.1, my local validation loss dramatically reduced and the leaderboard score fell from 1.6 to 1.063. This seemed like the right direction as psi’s benchmark was 0.88 and we’re getting there. A little bit of tuning with epochs and I got 0.925 on LB. I tried modifying int4/int8/bfloat16 finetuning, LoRA parameters and always saw an adverse effect. Details in my logbook. One of the key parameters that improved my score was prompt length, answer length. The more they are, the better the logloss. This was easy to explain as some of the sentences (specifically answers) were longer than 2000 in length.
\nOne of the interesting parameters is the seed. Fixing the seed led to deterministic results but a random seed resulted in a different logloss and LB score. The variance is 0.05 which is extremely high in this competition. This led to the conclusion that averaging multiple models was the way to go to fight variance.
\nAlong the way, I clustered the data to visually explore it using cluestar library and found that GroupKFold was the way to go. This was a moment of personal learning that “explore the data first before modeling”.
\n1.3 Open Models
\nWith huggingface hosting so many models, the open_llm_leaderboard shows top models based on benchmark scores. Motivated to further improve the score, I tried multiple models finetuned on top of Mistral and while some worked, some did not. Ablation studies are in the logbook.
\n1.4 Averaging
\nLastly, these models were not perfectly correlated and that led to nice performance on averaging. So, a straight average of predictions (not oof predictions) from the 9 models below, with different seeds, led to 0.63 on the leaderboard. Here is the architecture.
\n5 days prior to the competition end, Kha and I decided to team up and his optimization work (details in the next section) was a big part of this solution going from 0.63 to 0.50 on the public leaderboard.
\n\"\"
\n1.5 Infrastructure
\nAll the decoder models were trained on a machine with 8 x V100 GPUs. This meant that each finetuning run approximately took around 60 minutes along with inference on test data.
\n1.6 Note from binga
\nA big thanks to H2O.ai, Kaggle and the organizers for setting up this wonderful competition where I learnt so much about SoTA NLP.
\nThanks to AI practitioners in open source who are behind projects such as Open Hermes, Open Orca etc for making these great models available to us.
\nSpecial thanks to Kha for giving me an opportunity to team up with him! I've learnt so much from Kha -- understanding data, doing quick experiments and most importantly a never-say-die attitude in the last 2 days when we were challenged by competitors such as Reacher and that made us push ourselves during the last hours.

\n

2. OPTIMIZATION AND POST-PROCESS (Kha)

\n

This competition’s data has a very special trait, that for each question group, we should expect exactly 1 response for each of 7 label targets. During the normal modeling process, we may not exploit any of this extra information about relation between data samples.
\nThis sparks an idea of using an optimization method to find the best assignment of targets for a group of 7 responses to a specific question. The optimization method takes as input a 7x7 matrix, where each row is the normalized probability (sum to 1) of 1 data sample. We then try to maximize the selected probabilities given the constraint that each target must be assigned exactly one time.
\nThis optimization method alone, however, cannot outperform the original predictions due to its extreme nature (of hard 0/1 prediction) that can't drive the log-loss down. So I need to perform some more extra steps of combining this binary prediction (also a 7x7 matrix) with the original matrix. The combining steps and optimization coefficients were achieved manually by a systematic search to minimize the out-of-fold predictions. The single steps of post-processing are shown in the function below.
\nYou may wonder why there are some highly manual steps. In general, they are just the results of the following ideas:

\n\n
 ():\n    q, _df  df1.groupby(): \n\n       \n       indices = _df.index.values\n       cost_matrix = -pred[indices]\n       row_idx, col_idx = linear_sum_assignment(cost_matrix)\n       orig = pred[indices]\n       hard_pred = np.zeros((,))\n       hard_pred[row_idx, col_idx] = \n       hard_pred_minus = -hard_pred\n\n       orig_selected = np.zeros((,))\n       orig_selected[row_idx, col_idx] = orig[row_idx, col_idx]\n       orig_unselected = orig.copy()\n       orig_unselected[row_idx, col_idx] = \n       soft_pred_1 = orig_selected + (hard_pred-orig_selected)*\n       soft_pred_2 = orig_unselected - orig_unselected**\n       soft_pred = soft_pred_1 + soft_pred_2\n       pred[indices] = soft_pred\n\n       \n        jcol  ():\n           max_idx = np.argmax(soft_pred[:, jcol])\n            (soft_pred[:, jcol]) > :\n               soft_pred[:, jcol] = \n               soft_pred[max_idx, jcol] = \n            (soft_pred[:, jcol]) > :\n               soft_pred[:, jcol] -= soft_pred[:, jcol]*\n               soft_pred[max_idx, jcol] += (-soft_pred[max_idx, jcol])*\n            (soft_pred[:, jcol]) > :\n               soft_pred[:, jcol] -= soft_pred[:, jcol]*\n               soft_pred[max_idx, jcol] += (-soft_pred[max_idx, jcol])*\n\n       pred[indices] = soft_pred\n\n    pred\noptimized_preds = optim(test, probabilities) \n
\n

3. THE OTHER IDEAS THAT ALSO WORKED (Kha)

\n

Sidelining the main 2 major factors presented, I also experimented with some other classical approaches and found out that they worked nicely, but I couldn’t find the time to combine that with Phani’s because Phani cannot produce the out-of-fold predictions on time. My methods highly depend on the train out-of-fold predictions. Even the binary classification idea and post-process was based on my out-of-fold deberta-v3-large predictions, instead of the best Phani’s LLM model. Had the optimization been tuned on the latter, we would have a better final score I’m pretty sure.
\n3.1 Using a tree-based method (LightGBM) to learn a stacking model on top of the NLP predictions.
\nThe crafted features that helped are:

\n", "rawMarkdown": "\nIn this post I would like to present our team’s solution write-up that finished ..... place (public 2nd place) in the “h2o.ai Predict the LLM” competition on Kaggle. We would like to express our thanks to the Kaggle team, to h2o.ai (@robikscube, @philippsinger), and to all fellow competitors for such a short-lived joyful experience during the 2 weeks of this state-of-the-art hackathon.\n\nPersonally, I also would like to thank my teammate (@phanisrikanth Phani, aka Binga) for our short but fruitful collaboration. Hope we can work together in another competition soon.\n\nIn short, here are the 2 major ideas that mainly contributed to our success:\n\n+ Finetune an LLM decoder with a classification head with the competition’s data.\n\n+ Optimize the prediction (binarizing it using linear assignment optimization method), then combine this binary prediction with the original raw prediction with custom post-processing steps.\n\n\n### 1. THE LLM MODELS: (Binga)\n\nOn Kaggle, it’s established that encoders, such as DeBERTa, have ruled NLP competitions on Kaggle for years. To brush up my NLP skills, I joined this competition on day 1 and started with encoders. I realized that simple DeBERTa models scored in the range of 1.6, 1.4 on mlogloss metric.\n\n**1.1 Model Selection**\n\nGiven that it’s 2023 and we have new decoders coming out almost each day, I decided to use a decoder model out of sheer curiosity. Now, which models do we begin with? I went with the most popular Mistral model for 2 reasons: 1) Performance. It outperforms Llama2-13b on some benchmarks, and 2) Apache 2.0 License.\n\n\n**1.2 Experiments**\n\nEach of the experiments were done on H2O-LLMStudio. It’s a fantastic no-code tool to quickly run experiments with many LLMs. Highly recommend having this tool in your data science toolbox. 🙂\n\nComing to experiments, with Mistral-Instruct-v0.1, my local validation loss dramatically reduced and the leaderboard score fell from 1.6 to 1.063. This seemed like the right direction as psi’s benchmark was 0.88 and we’re getting there. A little bit of tuning with epochs and I got 0.925 on LB. I tried modifying int4/int8/bfloat16 finetuning, LoRA parameters and always saw an adverse effect. Details in my logbook. One of the key parameters that improved my score was prompt length, answer length. The more they are, the better the logloss. This was easy to explain as some of the sentences (specifically answers) were longer than 2000 in length.\n\nOne of the interesting parameters is the seed. Fixing the seed led to deterministic results but a random seed resulted in a different logloss and LB score. The variance is 0.05 which is extremely high in this competition. This led to the conclusion that averaging multiple models was the way to go to fight variance.\n\nAlong the way, I clustered the data to visually explore it using cluestar library and found that GroupKFold was the way to go. This was a moment of personal learning that “explore the data first before modeling”.\n\n**1.3 Open Models**\n\nWith huggingface hosting so many models, the open_llm_leaderboard shows top models based on benchmark scores. Motivated to further improve the score, I tried multiple models finetuned on top of Mistral and while some worked, some did not. Ablation studies are in the logbook.\n\n**1.4 Averaging**\n\nLastly, these models were not perfectly correlated and that led to nice performance on averaging. So, a straight average of predictions (not oof predictions) from the 9 models below, with different seeds, led to 0.63 on the leaderboard. Here is the architecture.\n\n5 days prior to the competition end, Kha and I decided to team up and his optimization work (details in the next section) was a big part of this solution going from 0.63 to 0.50 on the public leaderboard.\n\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F1829450%2F0737d6dd5a2bdcb891196edacdf29bb3%2FScreenshot%202023-11-08%20at%206.59.27%20am.png?generation=1699387188619974&alt=media)\n\n**1.5 Infrastructure**\n\nAll the decoder models were trained on a machine with 8 x V100 GPUs. This meant that each finetuning run approximately took around 60 minutes along with inference on test data.\n\n**1.6 Note from binga**\n\nA big thanks to H2O.ai, Kaggle and the organizers for setting up this wonderful competition where I learnt so much about SoTA NLP.\n\nThanks to AI practitioners in open source who are behind projects such as Open Hermes, Open Orca etc for making these great models available to us.\n\nSpecial thanks to Kha for giving me an opportunity to team up with him! I've learnt so much from Kha -- understanding data, doing quick experiments and most importantly a never-say-die attitude in the last 2 days when we were challenged by competitors such as Reacher and that made us push ourselves during the last hours.\n\n\n### 2. OPTIMIZATION AND POST-PROCESS (Kha)\n\nThis competition’s data has a very special trait, that for each question group, we should expect exactly 1 response for each of 7 label targets. During the normal modeling process, we may not exploit any of this extra information about relation between data samples.\n\nThis sparks an idea of using an optimization method to find the best assignment of targets for a group of 7 responses to a specific question. The optimization method takes as input a 7x7 matrix, where each row is the normalized probability (sum to 1) of 1 data sample. We then try to maximize the selected probabilities given the constraint that each target must be assigned exactly one time.\n\nThis optimization method alone, however, cannot outperform the original predictions due to its extreme nature (of hard 0/1 prediction) that can't drive the log-loss down. So I need to perform some more extra steps of combining this binary prediction (also a 7x7 matrix) with the original matrix. The combining steps and optimization coefficients were achieved manually by a systematic search to minimize the out-of-fold predictions. The single steps of post-processing are shown in the function below.\n\nYou may wonder why there are some highly manual steps. In general, they are just the results of the following ideas:\n\n- The highest prediction within 1 data sample (a row) should come to 1\n- All the other 6 predictions within that data sample should come to 0\n- The above 2 ideas can only be implemented after the assignment optimization step.\n\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F1829450%2F046d82599c33e7462179efdb8f7efe50%2FScreenshot%202023-11-08%20at%206.46.22%20am.png?generation=1699387078180767&alt=media)\n\nThe optimization function is highly customized and straightforward, that can be applied to any submission! Fellow Kagglers please try this to see how your score will be improved significantly in an instant!\n\n\n```python\ndef optim(df1, pred):\n\n for q, _df in df1.groupby('Question'): \n \n # Stage 1 optim\n indices = _df.index.values\n cost_matrix = -pred[indices]\n row_idx, col_idx = linear_sum_assignment(cost_matrix)\n orig = pred[indices]\n hard_pred = np.zeros((7,7))\n hard_pred[row_idx, col_idx] = 1\n hard_pred_minus = 1-hard_pred\n \n orig_selected = np.zeros((7,7))\n orig_selected[row_idx, col_idx] = orig[row_idx, col_idx]\n orig_unselected = orig.copy()\n orig_unselected[row_idx, col_idx] = 0\n\n soft_pred_1 = orig_selected + (hard_pred-orig_selected)*0.005\n soft_pred_2 = orig_unselected - orig_unselected**1.475\n soft_pred = soft_pred_1 + soft_pred_2\n\n pred[indices] = soft_pred\n \n # Stage 2 optim\n for jcol in range(7):\n max_idx = np.argmax(soft_pred[:, jcol])\n if max(soft_pred[:, jcol]) > 1:\n soft_pred[:, jcol] = 0.01\n soft_pred[max_idx, jcol] = 0.99\n elif max(soft_pred[:, jcol]) > 0.67:\n soft_pred[:, jcol] -= soft_pred[:, jcol]*0.715\n soft_pred[max_idx, jcol] += (1-soft_pred[max_idx, jcol])*0\n elif max(soft_pred[:, jcol]) > 0.245:\n soft_pred[:, jcol] -= soft_pred[:, jcol]*0.56\n soft_pred[max_idx, jcol] += (1-soft_pred[max_idx, jcol])*0\n \n pred[indices] = soft_pred\n \n return pred\n\noptimized_preds = optim(test, probabilities) # test is the test dataframe\n```\n\n\n### 3. THE OTHER IDEAS THAT ALSO WORKED (Kha)\n\nSidelining the main 2 major factors presented, I also experimented with some other classical approaches and found out that they worked nicely, but I couldn’t find the time to combine that with Phani’s because Phani cannot produce the out-of-fold predictions on time. My methods highly depend on the train out-of-fold predictions. Even the binary classification idea and post-process was based on my out-of-fold deberta-v3-large predictions, instead of the best Phani’s LLM model. Had the optimization been tuned on the latter, we would have a better final score I’m pretty sure.\n\n**3.1 Using a tree-based method (LightGBM) to learn a stacking model on top of the NLP predictions.**\n\nThe crafted features that helped are:\n\n- Gpt_rating: asking another LLM to rate the response within a fixed scale.\n- Statistics (mean, median, min, max, std) of some main features (text length, gpt_rating) aggregated by each question’s group\n- Category of each question: asking another LLM to categorize each question into a few categories such as: biology, history, math, idea, recommendation…\n\nThis modeling process boosted my raw deberta-v3-large score of 1.1 to about 0.950.\n\n\n**3.2 Pseudo-labeling with the test prediction retrieved after the optimization method**\n\nUsing the test prediction as the additional training data isn’t a new idea. But in this competition it won’t give any benefits given the log-loss metric (binarize the prediction hurts the score, hence additional data is indeed detrimental).\n\nHowever if the binarized version is retrieved (simply by using an argmax function) after being optimized and post-processed (as described in section 2), pseudo-label will work. However (again), the training process with pseudo-labels is not straightforward, because if we train on full test data in any session, the knowledge cannot improve and the test prediction itself finally is biased to its original version, giving no benefit. \n\nSo I have to train a k-fold on the question (suppose k=5), do the split on the train data, and also do the split on the pseudo data. In a single fold training, the ⅘ parts of train data plus the ⅘ parts of pseudo data will serve as training data, the ⅕ remaining part of train data will serve as validation data, and the ⅕ remaining part of pseudo data will serve as test data (that needs prediction). This strategy can ensure that the test prediction on the ⅕ test part will not have bias towards its existing target knowledge.\n\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F1829450%2F401ab43c42e583a6af29a490a18fc07a%2FScreenshot%202023-11-08%20at%206.58.29%20am.png?generation=1699387132047069&alt=media)\n\nWith pseudo training and LightGBM modeling, I observed a nice-size score boost of about 0.1 to 0.15 on the raw deberta-v3-large submission. But as Phani didn’t generate out-of-fold LLM predictions so I couldn’t implement these two methods although I have the code.\n\n**3.3 Conclusive Remarks**\nWe may have a significantly better score that could break 0.499 if we combine all of the ideas together, like in these steps:\n\n- Step 1: Train an decoder LLM, predict the out-of-fold along with test predictions.\n- Step 2: Train a LightGBM model on top of step 1, with the hand-crafted features.\n- Step 3: Optimize and post-process the predictions from step 2.\n- Step 4: Use the test predictions from step 3 as the additional training data, repeat from step 1\n\nThank you for reading. Our full code is at: https://www.kaggle.com/khahuras/2nd-place-solution\nEnjoy!\n", "votes": 21 }, { "id": 2516916, "postDate": "2023-11-08T04:29:52.013Z", "content": "

I'd like to once again thank h2o.ai, Kaggle and organizers for setting up this wonderful competition. While I have learnt a ton of new things working with decoder models, I had a lot of fun :-)

\n

The decoder part of the solution is available here - Link.

\n

Here are a few more resources if you're interested.

\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n
ResourceLink
ML Experiments LogbookLink
H2O LLMStudio Training yamlLink
ML Experiment RunsLink
Beating the Benchmark SolutionLink
Data Exploration via Clustering (which hinted the GroupKFold validation)Link
\n
\n

Congratulations to all the winners! Special congratulations to @ihebch for his persistence. You made us push harder and you came out ahead across the finish line with a very well deserved solution! It was a super exciting finish :-)

", "rawMarkdown": "I'd like to once again thank h2o.ai, Kaggle and organizers for setting up this wonderful competition. While I have learnt a ton of new things working with decoder models, I had a lot of fun :-)\n\nThe decoder part of the solution is available here - [Link](https://www.kaggle.com/code/phanisrikanth/2nd-place-solution-decoder-inference?scriptVersionId=149798332).\n\nHere are a few more resources if you're interested.\n\n| Resource | Link |\n|--------------------------------------------------------------------------|-------------------------------------------------------------------------------------------------------------|\n| ML Experiments Logbook | [Link](https://github.com/binga/kaggle-predict-the-llm/) |\n| H2O LLMStudio Training yaml | [Link](https://www.kaggle.com/datasets/phanisrikanth/predict-the-llm-h2o-llm-studio-training-yaml) |\n| ML Experiment Runs | [Link](https://app.neptune.ai/binga/predict-the-llm/runs/table?viewId=9a77f72e-07ba-4bc0-be9e-016f76ebacd2) |\n| Beating the Benchmark Solution | [Link](https://www.kaggle.com/code/phanisrikanth/beating-the-benchmark-top-10-solution) |\n| Data Exploration via Clustering (which hinted the GroupKFold validation) | [Link](https://www.kaggle.com/code/phanisrikanth/cluster-explore-data) |\n\n---\n\nCongratulations to all the winners! Special congratulations to @ihebch for his persistence. You made us push harder and you came out ahead across the finish line with a very well deserved solution! It was a super exciting finish :-)", "votes": 4 }, { "id": 2519919, "postDate": "2023-11-10T13:09:22.190Z", "content": "

Our full inference code is released https://www.kaggle.com/khahuras/2nd-place-solution

", "rawMarkdown": "Our full inference code is released https://www.kaggle.com/khahuras/2nd-place-solution", "votes": 1 }, { "id": 2517672, "postDate": "2023-11-08T17:06:40.147Z", "content": "

Great stuff, thanks for the detailed share and congrats!

", "rawMarkdown": "Great stuff, thanks for the detailed share and congrats!", "votes": 1 }, { "id": 2517605, "postDate": "2023-11-08T16:10:30.923Z", "content": "

Thank you for sharing PP @khahuras , it's really effective , I'm going to take a closer look.

\n

Results on single 5 fold Zephyr-7b-beta model -

\n
No PP ->  private LB , public LB  \n PP -> private LB , public LB \n
", "rawMarkdown": "Thank you for sharing PP @khahuras , it's really effective , I'm going to take a closer look. \n\nResults on single 5 fold Zephyr-7b-beta model - \n```python\nNo PP -> private LB 0.649, public LB 0.639 \nwith PP -> private LB 0.533, public LB 0.52\n```", "votes": 1, "replies": [ { "id": 2517655, "postDate": "2023-11-08T16:54:51.103Z", "content": "

Glad to hear that!

", "rawMarkdown": "Glad to hear that!", "votes": 1 } ] }, { "id": 2528178, "postDate": "2023-11-17T06:48:37.327Z", "rawMarkdown": "", "isDeleted": true } ], "complete": true, "source": "live", "fetched_at": "2026-10-19T21:54:11Z", "competition_id": 63021, "competition_slug": "h2oai-predict-the-llm", "team_id": 11130867, "solution_url": "/competitions/h2oai-predict-the-llm/writeups/kha-vo-binga-2nd-place-solution-llm-decoders-assig" }