# 3rd place solution

Thank you
[@robikscube](https://www.kaggle.com/robikscube)
for hosting this competition, it was a great learning experience and a good clean dataset is a joy to work with.
Also Thank you
[@lucamassaron](https://www.kaggle.com/lucamassaron)
for joining me on this short project, pleasure working with you.
Our final submissions are:
1)  the weighted average of our single solutions, where my best submission weights 0.4, and Luca’s best submission weights 0.6.  We used the public leaderboard to evaluate our best solutions (also because the LB agreed with our local CVs) and also to establish the 0.4/0.6 ratio we used. Our final inference notebook:
[https://www.kaggle.com/code/lucamassaron/blending-4-h2o-hackaton](https://www.kaggle.com/code/lucamassaron/blending-4-h2o-hackaton)
2) My best submission
Our best model was 5 fold, 3x pseudo labeled Zephyr-7b-beta. Inference for single fold can be found
[here](https://www.kaggle.com/code/rashmibanthia/h2o-ai-predict-llm-inference?scriptVersionId=149634870)
.   5 fold of this model had CV 0.7888. Zephyr-7b-beta was the best one amongst many different LLMs we tried.
Following different LLMs were part of this ensemble :
HuggingFaceH4/zephyr-7b-beta (Many different variations with different parameters of this model)
NousResearch/Nous-Capybara-7B-V1.9
HuggingFaceH4/zephyr-7b-alpha
Open-Orca/Mistral-7B-OpenOrca
Total of 15 LLMs were part of this ensemble, and we used Nelder Mead to find optimal weights for models. Some of the models ended up with 0 weights which I noticed after the deadline. As the ensemble improved with tabular features (explained below), we were able to pseudo label again up to 3 times.  Each 7B model took about 1 hr to fine tune on a single 4090.
Other models that didn’t work - 01-ai/Yi-6B, NousResearch/Yarn-Mistral-7b-64k, NousResearch/Yarn-Mistral-7b-128k, WizardLM/WizardLM-7B-V1.0, teknium/OpenHermes-2-Mistral-7B
Thank you for open sourcing
[H2O LLM Studio](https://github.com/h2oai/h2o-llmstudio)
- it’s very well written without relying on many external libraries. This served as reference for my own (single GPU) training code.
3) Luca's Solution
The solution has taken together:
(a) underperforming (around 1.3-1.4) finetuning approaches as seen in the public notebooks for roberta (1 model), distilroberta-base (1 model), and deepset/roberta-base-squad2 (1 model). All finetuned models were quite disappointing in their LB performances (and local CV, too), but somehow they blended well.
(b) stylometric and readability features derived from packages such as
[lexicalrichness](https://pypi.org/project/lexicalrichness/)
,
[lexical_diversity](https://pypi.org/project/lexical-diversity/)
,
[textstat](https://pypi.org/project/textstat/)
,
[textblob](https://pypi.org/project/textblob/)
.
(c) the idea of using log probabilities from open-source language models taken from the paper Verma, Vivek, et al. "Ghostbuster: Detecting Text Ghostwritten by Large Language Models." arXiv preprint arXiv:2305.15047, 2023 (
[https://arxiv.org/abs/2305.15047](https://arxiv.org/abs/2305.15047)
). As in the paper, the log probabilities are combined together in statistics and added, subtracted, multiplied, and divided together. These proved the most successful features because they could catch similarities in the language models that produced the train and test data based on similar architectures and training data. Here is the list of the log probabilities produced:
12b-databricks-dolly-v2
13b-chavinlo-alpaca
13b-chavinlo-gpt4-x-alpaca
13b-open-orca-platypus2
34b-h2ogpt-aquilachat2
40b-tiiuae-falcon
7b-RedPajama-INCITE-Chat
alpaca_7b
aquila2_7b
aquila_7b
bloom
dolly2_3b
dolly2_7b
dolphin_minstral7b
falcon7b
gpt2_large
gpt2
gpt_neo
koala_7b
llama_13b-hf
llama_7b-hf
mistral_7b
nomic-ai-gpt4all-j
opt_2_7b
opt
pythia
tinyllama
vicuna_7b
wizardlm_7b
xlnet
zephyr_7b
(d) TfidfVectorizer ngrams of both tokenized words (using Llama tokenizer) and pos-tagging. We have used ngrams 1-5 (with a minimum count of 30), and in order to avoid overfitting, we limited the dictionary to only ngrams present both in train and test.
(e) Out-of-fold predictions and submissions from Rashmi’s work.
As a special processing, given that we were asked to evaluate blocks of 7 outputs (a question with 7 answers related each to a different llm), I’ve computed for the features the ranks and deviance in their own block set.
The above 5 points produced 10,144 features that have been left to XGBoost and Boruta to filter out between irrelevant signals and promising features. After 100 iterations, Boruta pointed out 650 confirmed features and 92 tentative ones. We took them all for the following ensembling phase. The selected features are, for the most part, derived from log probabilities.
In the final ensembling part, all these features have been fed to a CV prediction loop using XGboost, LightGBM, and Catboost, all set to a shallow maximum depth (3-4). All the models have been tuned by hand. The ensembling part has been re-run 10 times, and the oofs and predictions averaged together to reduce the variance.
References:
[Feature creation notebook](https://www.kaggle.com/code/lucamassaron/classical-ml)
[Large language model training notebook](https://www.kaggle.com/code/lucamassaron/hf-model)
[Log probabilities computation notebook](https://www.kaggle.com/code/lucamassaron/logprobability-llama-7b)
[Ensembling notebook](https://www.kaggle.com/code/lucamassaron/xgboost-lightgbm-catboost)
[Log probabilities and the finetuned model’s results dataset](https://www.kaggle.com/datasets/lucamassaron/h2o-trained-oof)
[Zephyr-7b-beta Inference](https://www.kaggle.com/code/rashmibanthia/h2o-ai-predict-llm-inference?scriptVersionId=149634870)