Thank you @robikscube for hosting this competition, it was a great learning experience and a good clean dataset is a joy to work with.
\nAlso Thank you @lucamassaron for joining me on this short project, pleasure working with you.
\nOur final submissions are:
\n1) the weighted average of our single solutions, where my best submission weights 0.4, and Luca’s best submission weights 0.6. We used the public leaderboard to evaluate our best solutions (also because the LB agreed with our local CVs) and also to establish the 0.4/0.6 ratio we used. Our final inference notebook: https://www.kaggle.com/code/lucamassaron/blending-4-h2o-hackaton
\n2) My best submission
\nOur best model was 5 fold, 3x pseudo labeled Zephyr-7b-beta. Inference for single fold can be found here. 5 fold of this model had CV 0.7888. Zephyr-7b-beta was the best one amongst many different LLMs we tried.
\nFollowing different LLMs were part of this ensemble :
\nHuggingFaceH4/zephyr-7b-beta (Many different variations with different parameters of this model)
\nNousResearch/Nous-Capybara-7B-V1.9
\nHuggingFaceH4/zephyr-7b-alpha
\nOpen-Orca/Mistral-7B-OpenOrca
Total of 15 LLMs were part of this ensemble, and we used Nelder Mead to find optimal weights for models. Some of the models ended up with 0 weights which I noticed after the deadline. As the ensemble improved with tabular features (explained below), we were able to pseudo label again up to 3 times. Each 7B model took about 1 hr to fine tune on a single 4090.
\nOther models that didn’t work - 01-ai/Yi-6B, NousResearch/Yarn-Mistral-7b-64k, NousResearch/Yarn-Mistral-7b-128k, WizardLM/WizardLM-7B-V1.0, teknium/OpenHermes-2-Mistral-7B
\nThank you for open sourcing H2O LLM Studio - it’s very well written without relying on many external libraries. This served as reference for my own (single GPU) training code.
\n3) Luca's Solution
\nThe solution has taken together:
\n(a) underperforming (around 1.3-1.4) finetuning approaches as seen in the public notebooks for roberta (1 model), distilroberta-base (1 model), and deepset/roberta-base-squad2 (1 model). All finetuned models were quite disappointing in their LB performances (and local CV, too), but somehow they blended well.
(b) stylometric and readability features derived from packages such as lexicalrichness, lexical_diversity, textstat, textblob.
\n(c) the idea of using log probabilities from open-source language models taken from the paper Verma, Vivek, et al. \"Ghostbuster: Detecting Text Ghostwritten by Large Language Models.\" arXiv preprint arXiv:2305.15047, 2023 (https://arxiv.org/abs/2305.15047). As in the paper, the log probabilities are combined together in statistics and added, subtracted, multiplied, and divided together. These proved the most successful features because they could catch similarities in the language models that produced the train and test data based on similar architectures and training data. Here is the list of the log probabilities produced:
\n(d) TfidfVectorizer ngrams of both tokenized words (using Llama tokenizer) and pos-tagging. We have used ngrams 1-5 (with a minimum count of 30), and in order to avoid overfitting, we limited the dictionary to only ngrams present both in train and test.
\n(e) Out-of-fold predictions and submissions from Rashmi’s work.
\nAs a special processing, given that we were asked to evaluate blocks of 7 outputs (a question with 7 answers related each to a different llm), I’ve computed for the features the ranks and deviance in their own block set.
\nThe above 5 points produced 10,144 features that have been left to XGBoost and Boruta to filter out between irrelevant signals and promising features. After 100 iterations, Boruta pointed out 650 confirmed features and 92 tentative ones. We took them all for the following ensembling phase. The selected features are, for the most part, derived from log probabilities.
\nIn the final ensembling part, all these features have been fed to a CV prediction loop using XGboost, LightGBM, and Catboost, all set to a shallow maximum depth (3-4). All the models have been tuned by hand. The ensembling part has been re-run 10 times, and the oofs and predictions averaged together to reduce the variance.
\nReferences:
\n\nLarge language model training notebook
\nLog probabilities computation notebook
\n\nLog probabilities and the finetuned model’s results dataset
\n", "comments": [ { "id": 2519483, "author_name": "Ashvanth s", "author_url": "", "post_date": "2023-11-10T04:45:22.967000", "content": "Interesting results @rashmibanthia , crazy to know how many LLMs were used for the ensemble model. Any particular reason as to why these were used ?
", "votes": 1, "replies": [ { "id": 2519886, "author_name": "RB", "author_url": "", "post_date": "2023-11-10T12:41:19.983000", "content": "for one, there was no limit here 😂 - but I used NelderMead for ensemble which made weights 0 for many models eventually iirc it was 5-6 LLMs with > 0 weights.
", "votes": 0, "replies": [] } ] }, { "id": 2517457, "author_name": "Sitraka_Forler", "author_url": "", "post_date": "2023-11-08T13:48:40.263000", "content": "Dam crazy good results! Bravo :O
", "votes": 1, "replies": [] }, { "id": 2516271, "author_name": "ducnh279", "author_url": "", "post_date": "2023-11-07T15:36:43.613000", "content": "@rashmibanthia Your efforts in using LLMs are really impressive! I will review your team's solution thoroughly to learn more about LLMs. Thanks for sharing!!
", "votes": 1, "replies": [ { "id": 2516377, "author_name": "RB", "author_url": "", "post_date": "2023-11-07T17:17:57.573000", "content": "I was also surprised how well LLMs performed compared to deberta without any data setup or extensive hyperparameter tuning.
", "votes": 1, "replies": [] } ] }, { "id": 2561051, "author_name": "Jabir", "author_url": "", "post_date": "2023-12-14T08:44:42.657000", "content": "Nice Solution @rashmibanthia.I was curious about ensembling the llms.It will be helpful if you could briefly explain about it.Thanks for the writeup
", "votes": 0, "replies": [] }, { "id": 2519763, "author_name": "Samvel Kocharyan", "author_url": "", "post_date": "2023-11-10T10:19:40.690000", "content": "Thank you for sharing @rashmibanthia
", "votes": 1, "replies": [] } ], "comment_count": 7, "messages": [ { "id": 2516118, "postDate": "2023-11-07T13:43:19.623Z", "content": "Thank you @robikscube for hosting this competition, it was a great learning experience and a good clean dataset is a joy to work with.
\nAlso Thank you @lucamassaron for joining me on this short project, pleasure working with you.
\nOur final submissions are:
\n1) the weighted average of our single solutions, where my best submission weights 0.4, and Luca’s best submission weights 0.6. We used the public leaderboard to evaluate our best solutions (also because the LB agreed with our local CVs) and also to establish the 0.4/0.6 ratio we used. Our final inference notebook: https://www.kaggle.com/code/lucamassaron/blending-4-h2o-hackaton
\n2) My best submission
\nOur best model was 5 fold, 3x pseudo labeled Zephyr-7b-beta. Inference for single fold can be found here. 5 fold of this model had CV 0.7888. Zephyr-7b-beta was the best one amongst many different LLMs we tried.
\nFollowing different LLMs were part of this ensemble :
\nHuggingFaceH4/zephyr-7b-beta (Many different variations with different parameters of this model)
\nNousResearch/Nous-Capybara-7B-V1.9
\nHuggingFaceH4/zephyr-7b-alpha
\nOpen-Orca/Mistral-7B-OpenOrca
Total of 15 LLMs were part of this ensemble, and we used Nelder Mead to find optimal weights for models. Some of the models ended up with 0 weights which I noticed after the deadline. As the ensemble improved with tabular features (explained below), we were able to pseudo label again up to 3 times. Each 7B model took about 1 hr to fine tune on a single 4090.
\nOther models that didn’t work - 01-ai/Yi-6B, NousResearch/Yarn-Mistral-7b-64k, NousResearch/Yarn-Mistral-7b-128k, WizardLM/WizardLM-7B-V1.0, teknium/OpenHermes-2-Mistral-7B
\nThank you for open sourcing H2O LLM Studio - it’s very well written without relying on many external libraries. This served as reference for my own (single GPU) training code.
\n3) Luca's Solution
\nThe solution has taken together:
\n(a) underperforming (around 1.3-1.4) finetuning approaches as seen in the public notebooks for roberta (1 model), distilroberta-base (1 model), and deepset/roberta-base-squad2 (1 model). All finetuned models were quite disappointing in their LB performances (and local CV, too), but somehow they blended well.
(b) stylometric and readability features derived from packages such as lexicalrichness, lexical_diversity, textstat, textblob.
\n(c) the idea of using log probabilities from open-source language models taken from the paper Verma, Vivek, et al. \"Ghostbuster: Detecting Text Ghostwritten by Large Language Models.\" arXiv preprint arXiv:2305.15047, 2023 (https://arxiv.org/abs/2305.15047). As in the paper, the log probabilities are combined together in statistics and added, subtracted, multiplied, and divided together. These proved the most successful features because they could catch similarities in the language models that produced the train and test data based on similar architectures and training data. Here is the list of the log probabilities produced:
\n(d) TfidfVectorizer ngrams of both tokenized words (using Llama tokenizer) and pos-tagging. We have used ngrams 1-5 (with a minimum count of 30), and in order to avoid overfitting, we limited the dictionary to only ngrams present both in train and test.
\n(e) Out-of-fold predictions and submissions from Rashmi’s work.
\nAs a special processing, given that we were asked to evaluate blocks of 7 outputs (a question with 7 answers related each to a different llm), I’ve computed for the features the ranks and deviance in their own block set.
\nThe above 5 points produced 10,144 features that have been left to XGBoost and Boruta to filter out between irrelevant signals and promising features. After 100 iterations, Boruta pointed out 650 confirmed features and 92 tentative ones. We took them all for the following ensembling phase. The selected features are, for the most part, derived from log probabilities.
\nIn the final ensembling part, all these features have been fed to a CV prediction loop using XGboost, LightGBM, and Catboost, all set to a shallow maximum depth (3-4). All the models have been tuned by hand. The ensembling part has been re-run 10 times, and the oofs and predictions averaged together to reduce the variance.
\nReferences:
\n\nLarge language model training notebook
\nLog probabilities computation notebook
\n\nLog probabilities and the finetuned model’s results dataset
\n", "rawMarkdown": "Thank you @robikscube for hosting this competition, it was a great learning experience and a good clean dataset is a joy to work with. \n\nAlso Thank you @lucamassaron for joining me on this short project, pleasure working with you.\n\nOur final submissions are:\n\n1) the weighted average of our single solutions, where my best submission weights 0.4, and Luca’s best submission weights 0.6. We used the public leaderboard to evaluate our best solutions (also because the LB agreed with our local CVs) and also to establish the 0.4/0.6 ratio we used. Our final inference notebook: https://www.kaggle.com/code/lucamassaron/blending-4-h2o-hackaton\n\n---\n2) My best submission\n\nOur best model was 5 fold, 3x pseudo labeled Zephyr-7b-beta. Inference for single fold can be found [here](https://www.kaggle.com/code/rashmibanthia/h2o-ai-predict-llm-inference?scriptVersionId=149634870). 5 fold of this model had CV 0.7888. Zephyr-7b-beta was the best one amongst many different LLMs we tried.\n\nFollowing different LLMs were part of this ensemble : \nHuggingFaceH4/zephyr-7b-beta (Many different variations with different parameters of this model)\nNousResearch/Nous-Capybara-7B-V1.9\nHuggingFaceH4/zephyr-7b-alpha\nOpen-Orca/Mistral-7B-OpenOrca \n\nTotal of 15 LLMs were part of this ensemble, and we used Nelder Mead to find optimal weights for models. Some of the models ended up with 0 weights which I noticed after the deadline. As the ensemble improved with tabular features (explained below), we were able to pseudo label again up to 3 times. Each 7B model took about 1 hr to fine tune on a single 4090. \n\nOther models that didn’t work - 01-ai/Yi-6B, NousResearch/Yarn-Mistral-7b-64k, NousResearch/Yarn-Mistral-7b-128k, WizardLM/WizardLM-7B-V1.0, teknium/OpenHermes-2-Mistral-7B \n\nThank you for open sourcing [H2O LLM Studio](https://github.com/h2oai/h2o-llmstudio) - it’s very well written without relying on many external libraries. This served as reference for my own (single GPU) training code. \n\n---\n3) Luca's Solution\n\nThe solution has taken together:\n(a) underperforming (around 1.3-1.4) finetuning approaches as seen in the public notebooks for roberta (1 model), distilroberta-base (1 model), and deepset/roberta-base-squad2 (1 model). All finetuned models were quite disappointing in their LB performances (and local CV, too), but somehow they blended well.\n\n(b) stylometric and readability features derived from packages such as [lexicalrichness](https://pypi.org/project/lexicalrichness/), [lexical_diversity](https://pypi.org/project/lexical-diversity/), [textstat] (https://pypi.org/project/textstat/), [textblob](https://pypi.org/project/textblob/).\n\n(c) the idea of using log probabilities from open-source language models taken from the paper Verma, Vivek, et al. \"Ghostbuster: Detecting Text Ghostwritten by Large Language Models.\" arXiv preprint arXiv:2305.15047, 2023 ([https://arxiv.org/abs/2305.15047](https://arxiv.org/abs/2305.15047)). As in the paper, the log probabilities are combined together in statistics and added, subtracted, multiplied, and divided together. These proved the most successful features because they could catch similarities in the language models that produced the train and test data based on similar architectures and training data. Here is the list of the log probabilities produced:\n\n- 12b-databricks-dolly-v2\n- 13b-chavinlo-alpaca\n- 13b-chavinlo-gpt4-x-alpaca\n- 13b-open-orca-platypus2\n- 34b-h2ogpt-aquilachat2\n- 40b-tiiuae-falcon\n- 7b-RedPajama-INCITE-Chat\n- alpaca_7b\n- aquila2_7b\n- aquila_7b\n- bloom\n- dolly2_3b\n- dolly2_7b\n- dolphin_minstral7b\n- falcon7b\n- gpt2_large\n- gpt2\n- gpt_neo\n- koala_7b\n- llama_13b-hf\n- llama_7b-hf\n- mistral_7b\n- nomic-ai-gpt4all-j\n- opt_2_7b\n- opt\n- pythia\n- tinyllama\n- vicuna_7b\n- wizardlm_7b\n- xlnet\n- zephyr_7b\n\n(d) TfidfVectorizer ngrams of both tokenized words (using Llama tokenizer) and pos-tagging. We have used ngrams 1-5 (with a minimum count of 30), and in order to avoid overfitting, we limited the dictionary to only ngrams present both in train and test.\n\n(e) Out-of-fold predictions and submissions from Rashmi’s work.\n\nAs a special processing, given that we were asked to evaluate blocks of 7 outputs (a question with 7 answers related each to a different llm), I’ve computed for the features the ranks and deviance in their own block set.\n\nThe above 5 points produced 10,144 features that have been left to XGBoost and Boruta to filter out between irrelevant signals and promising features. After 100 iterations, Boruta pointed out 650 confirmed features and 92 tentative ones. We took them all for the following ensembling phase. The selected features are, for the most part, derived from log probabilities. \n\nIn the final ensembling part, all these features have been fed to a CV prediction loop using XGboost, LightGBM, and Catboost, all set to a shallow maximum depth (3-4). All the models have been tuned by hand. The ensembling part has been re-run 10 times, and the oofs and predictions averaged together to reduce the variance.\n\n**References:**\n\n[Feature creation notebook](https://www.kaggle.com/code/lucamassaron/classical-ml)\n\n[Large language model training notebook](https://www.kaggle.com/code/lucamassaron/hf-model)\n\n[Log probabilities computation notebook](https://www.kaggle.com/code/lucamassaron/logprobability-llama-7b)\n\n[Ensembling notebook](https://www.kaggle.com/code/lucamassaron/xgboost-lightgbm-catboost)\n\n[Log probabilities and the finetuned model’s results dataset](https://www.kaggle.com/datasets/lucamassaron/h2o-trained-oof)\n\n[Zephyr-7b-beta Inference](https://www.kaggle.com/code/rashmibanthia/h2o-ai-predict-llm-inference?scriptVersionId=149634870)\n \n\n", "votes": 19 }, { "id": 2519483, "postDate": "2023-11-10T04:45:22.967Z", "content": "Interesting results @rashmibanthia , crazy to know how many LLMs were used for the ensemble model. Any particular reason as to why these were used ?
", "rawMarkdown": "Interesting results @rashmibanthia , crazy to know how many LLMs were used for the ensemble model. Any particular reason as to why these were used ?", "votes": 1, "replies": [ { "id": 2519886, "postDate": "2023-11-10T12:41:19.983Z", "content": "for one, there was no limit here 😂 - but I used NelderMead for ensemble which made weights 0 for many models eventually iirc it was 5-6 LLMs with > 0 weights.
", "rawMarkdown": "for one, there was no limit here 😂 - but I used NelderMead for ensemble which made weights 0 for many models eventually iirc it was 5-6 LLMs with > 0 weights. " } ] }, { "id": 2517457, "postDate": "2023-11-08T13:48:40.263Z", "content": "Dam crazy good results! Bravo :O
", "rawMarkdown": "Dam crazy good results! Bravo :O", "votes": 1 }, { "id": 2516271, "postDate": "2023-11-07T15:36:43.613Z", "content": "@rashmibanthia Your efforts in using LLMs are really impressive! I will review your team's solution thoroughly to learn more about LLMs. Thanks for sharing!!
", "rawMarkdown": "@rashmibanthia Your efforts in using LLMs are really impressive! I will review your team's solution thoroughly to learn more about LLMs. Thanks for sharing!!", "votes": 1, "replies": [ { "id": 2516377, "postDate": "2023-11-07T17:17:57.573Z", "content": "I was also surprised how well LLMs performed compared to deberta without any data setup or extensive hyperparameter tuning.
", "rawMarkdown": "I was also surprised how well LLMs performed compared to deberta without any data setup or extensive hyperparameter tuning. ", "votes": 1 } ] }, { "id": 2561051, "postDate": "2023-12-14T08:44:42.657Z", "content": "Nice Solution @rashmibanthia.I was curious about ensembling the llms.It will be helpful if you could briefly explain about it.Thanks for the writeup
", "rawMarkdown": "Nice Solution @rashmibanthia.I was curious about ensembling the llms.It will be helpful if you could briefly explain about it.Thanks for the writeup" }, { "id": 2519763, "postDate": "2023-11-10T10:19:40.690Z", "content": "Thank you for sharing @rashmibanthia
", "rawMarkdown": "Thank you for sharing @rashmibanthia ", "votes": 1 } ], "complete": true, "source": "live", "fetched_at": "2026-10-19T21:54:21Z", "competition_id": 63021, "competition_slug": "h2oai-predict-the-llm", "team_id": 11130150, "solution_url": "/competitions/h2oai-predict-the-llm/writeups/luca-rb-3rd-place-solution" }