# First Place Solution

Firstly, I want to  thank  H2o for organizing this great competition, it was a great opportunity  to delve into the world of LLMs and try different things.
Summary
My solution is an ensemble of 3 models (Mistral-7b , Zephyr-7b and Debertav3-large) feed into a 2nd Stage stacking Model with different features .
LLMs
I was really amazed by the performance of decoder only models in classification tasks. Due to hardware limitations (a single 16 Gb gpu) I was only able to experiment with a 4-bit quantized and  Low Rank Adaptation (LoRa).I experimented with Mistral-7b and Zephyr-7b.
Encoder only Models
input: Question + sep + [R_Start] response  [R_end] + other responses.
for pooling I only pool over tokens between [R_Start] and [R_end]  , this worked for my team in
[commonlit](https://www.kaggle.com/competitions/commonlit-evaluate-student-summaries/discussion/446542)
.
![](assets/0f6a45d1003b481373351bb3.png)
Second Stage Model
Our first stage models do not take into consideration that for each question we have the responses of the 7 LLMs.So as features for each row: (question, response) I added predictions of 1st stage models with prediction of other question response pairs along with some interaction features feed into LGBM/XGB, this was the secret sauce of my solution(+0.3/0.2 in cv/lb).
model
cv
public
private
deberta-v3-large
1.08
0.97
0.916
mistral-7b
0.9592
0.754
0.802
Zephyr-7b
0.936
0.787
0.79
+ 2nd stage
0.607
0.497
0.509
What Didn't Work
I tried to predict the target for the 7 responses together with different approaches, I even tried a token classification approach, but this unfortunately didn't work.
Training & Inference code
deberta-v3:
[https://www.kaggle.com/code/ihebch/h2o-ai-predict-the-llm-deberta-v3-large/notebook](https://www.kaggle.com/code/ihebch/h2o-ai-predict-the-llm-deberta-v3-large/notebook)
Mistral-7b:
[https://www.kaggle.com/code/ihebch/h2o-ai-predict-the-llm-mistral-7b-quantized-lora/notebook](https://www.kaggle.com/code/ihebch/h2o-ai-predict-the-llm-mistral-7b-quantized-lora/notebook)
Zephyr-7b:
[https://www.kaggle.com/code/ihebch/h2o-ai-predict-the-llm-zephyr-7b-quantized-lora/notebook](https://www.kaggle.com/code/ihebch/h2o-ai-predict-the-llm-zephyr-7b-quantized-lora/notebook)
2nd stage:
[https://www.kaggle.com/code/ihebch/h2o-ai-predict-the-llm-2nd-stage-stacking-model/notebook](https://www.kaggle.com/code/ihebch/h2o-ai-predict-the-llm-2nd-stage-stacking-model/notebook)