Wrote an article about our team experience in this competition. Not sure if I should to publish it here since it doesn't include something special for the community. But let it be. Thank you.
\nOn the day when Sam Altman, dressed in dark attire against a dark background, announced the launch of GPT-4-Turbo to the world, at the very same moment, a small yet intriguing competition called ‘Predict the LLM’ was concluding on Kaggle. Its goal was to determine the authorship of a text. The texts in the competition were authored by 7 anonymous, large language models…
\nKaggle is a place where people go for knowledge, new ideas, and sometimes for victories, but more and more often, it’s turning into a psychological casino. It’s best not to let enthusiastic data scientists near Kaggle. Ten days ago, while scrolling through the list of competitions with the thought, ‘What can I solve, what can I learn today,’ I stumbled upon a Community Competition by H2O. The description said:
\n\n\n“Delve deep into the world of Language Learning Models (LLMs)! The objective of this competition is to detect which out of 7 possible LLM models produced a particular output. With each model having its unique subtleties and quirks, can you identify which one generated the response?
\nThe challenge of pinpointing the origin LLM of a given output is not only intriguing but is also an area of spirited research. Understanding the nuances between different models allows for advancements in model differentiation, quality assessment, and potential applications in multiple sectors. Dive in, explore the intricacies, and contribute to this emerging area of study!”
\n
Super! Oh, that’s just what I wanted to try.
\nBut there’s more to it. The competition organizers are the guys from H2O, and the guys from H2O are, in general, legends in the world of Kaggle and competitive Data Science.
\nAnd there’s more again. The baseline to beat on the leaderboard is @psi, and Philipp Singer is #1 in the Kaggle ranking.
\nWell, and the personal cherry on top is that I somehow know all three of the people mentioned in the citation a little bit. I watch Rob Mulla on YouTube, I recently read an interview with Dmitry Larko in a cool DS book, and I’ve crossed paths with Gabor at Kaggle meetups in Budapest. Anyway, let’s get to solving.
\nAt the beginning, we have a labeled dataset with three columns: Question, Response, Label. Each of the 7 models presented in the training data was given 568 identical questions on various topics, from movies and music to mathematics and medicine. Fun fact: the questions were also written by LLM. The metric is Log Loss, with 50% of the score on the public leaderboard and 50% in the private section. No tricks with a Code Competition, everything is fair, no shakeups (???) ahead.
\nAs a newbie in the NLP field and a fan of classical ML, I started solving it in a classic way. My initial assumption was that there would be differences in LLM’s answers that would make it easy to classify authorship. And this difference wasn’t in the content of the texts but in the stylistic and some simple semantic-orthographic characteristics. I broke down the text into 130+ predictors using the spaCy framework.
\nRacing through all this quickly with boosting or AutoML and making the first submission was a failure. The score was terrible, second to last, far from Singer.
\nNow there are two of us, Chuk & Gek, forming a team. One in Budapest, the other in Moscow. We’re gaining experience on Kaggle. Both of us have been searching for jobs in data science for a while, but energetic juniors are not in demand, everyone’s looking for unicorns.
\nSo, we move on. We can remove unnecessary predictors, fine-tune model parameters, try a different framework, or create some ensemble models.
\nWe try it all. The score improves, but the psychological milestone of 1, which we still want to reach, is far from being achieved. We’re oscillating in the range of 1.4–1.8, while Singer is at 0.885.
\nWe read the Discussions at Kaggle. They talk about clustering, embeddings, BERT-like models. Our next solution architecture is built on extracting embeddings through various models and operations with them. We feed all of this back into boosting. The score improves a little bit, but we realized that with this architecture, we won’t go below 1.
\nIt’s time to unleash the 30-hour GPU weekly quota on Kaggle and build a solution with less-explored neural networks. Fine-tuning, new heads, endless parameters. The head is spinning from the amount of new information you need to know to make decisions. There are many networks, it takes a long time to compute, and mistakes are easy to make. Time is running out. There are only a few days left in the 14-day competition. Let’s try to simplify the task and create a baseline not manually but using the open-source H2O LLM Studio.
\nFollowing a forum tip from the competition, we’ve chosen the following approach: take the top non-proprietary LLM model from HuggingFace. Fine-tune it on our data and use Softmax to get the desired probabilities. The amount of sleep has been drastically reduced, but we are rewarded. Singer is in 17th place, and we are in 18th. Hooray! We’ve broken the 1 mark, and we’re standing next to a legend with 40+ submissions.
\nThe casino is on. We want to go higher. One team member is digging through the Transformers documentation and looking at top solutions from previous NLP competitions (by the way, one of our favorites is when yurko, Yuri Kashnitsky, taught networks to detect sarcasm in texts). Meanwhile, another team member, with zero experience, is trying to deploy LLM Studio on a powerful rented server. Setting up tunnels, installing GPU drivers, some sudo, make errors from Unix… We don’t make it in time.
\nOur last hope an hour before the final is to simply average our previous predictions, and now we’re in 10th place on the public leaderboard with a score of 0.774 from 50 submissions, while Singer is in 25th place with a score of 1.
\nTime to sleep. In the morning, there will be private leaderboard results, and we hope we won’t end up in the basement on the shakeup.
\nThe results haven’t been announced yet, although solutions from the leaders are starting to appear. Interestingly, the LLMs themselves seem to predict LLMs best :)
\nFine-tuning and customizing large models are topics that are gaining significant momentum. It’s no wonder I started the article with a person in the dark. These guys with thousands of ideas, millions of investment dollars, and billions of neurons in their heads, building amazing networks, are doing something remarkable. And you can be a part of this fantastic Data Science community, even if it’s just on Kaggle.
\nTeam:
\nChuk -> https://www.linkedin.com/in/samvelkoch/
\n&
\nGek -> https://www.linkedin.com/in/aleksey-schukin/
Links:
\nhttps://www.kaggle.com/competitions/h2oai-predict-the-llm/overview
\nhttps://www.kaggle.com/competitions/h2oai-predict-the-llm/leaderboard
\nhttps://www.kaggle.com/code/tanulsingh077/deep-learning-for-nlp-zero-to-transformers-bert
\nhttps://www.kaggle.com/AI-Report-2023
\nhttps://huggingface.co/learn/nlp-course/chapter1/1
\nhttps://www.deeplearning.ai/short-courses/finetuning-large-language-models/
\nhttps://www.coursera.org/specializations/natural-language-processing
\nhttps://docs.cohere.com/docs/llmu
\nhttps://www.coursera.org/learn/generative-ai-with-llms
\nhttps://web.stanford.edu/class/cs224u/index.html
\nhttps://github.com/h2oai/h2o-llmstudio
\nhttps://auto.gluon.ai/stable/index.html
\nhttps://www.youtube.com/@robmulla
\nhttps://www.youtube.com/@H2Oai
\nhttps://www.philippsinger.info
Thanks ! It was informative for us . We started with the basic principles in NLP and saw how with each complication of our solution and our assessment grows.
", "votes": 1, "replies": [] }, { "id": 2516236, "author_name": "ducnh279", "author_url": "", "post_date": "2023-11-07T15:13:28.137000", "content": "I'm sure you have spent significant time and effort on this competition. Congrats on the 10th place! Your adventures around this competition are quite inspiring to me!
", "votes": 1, "replies": [] } ], "comment_count": 2, "messages": [ { "id": 2516146, "postDate": "2023-11-07T14:02:10.010Z", "content": "Wrote an article about our team experience in this competition. Not sure if I should to publish it here since it doesn't include something special for the community. But let it be. Thank you.
\nOn the day when Sam Altman, dressed in dark attire against a dark background, announced the launch of GPT-4-Turbo to the world, at the very same moment, a small yet intriguing competition called ‘Predict the LLM’ was concluding on Kaggle. Its goal was to determine the authorship of a text. The texts in the competition were authored by 7 anonymous, large language models…
\nKaggle is a place where people go for knowledge, new ideas, and sometimes for victories, but more and more often, it’s turning into a psychological casino. It’s best not to let enthusiastic data scientists near Kaggle. Ten days ago, while scrolling through the list of competitions with the thought, ‘What can I solve, what can I learn today,’ I stumbled upon a Community Competition by H2O. The description said:
\n\n\n“Delve deep into the world of Language Learning Models (LLMs)! The objective of this competition is to detect which out of 7 possible LLM models produced a particular output. With each model having its unique subtleties and quirks, can you identify which one generated the response?
\nThe challenge of pinpointing the origin LLM of a given output is not only intriguing but is also an area of spirited research. Understanding the nuances between different models allows for advancements in model differentiation, quality assessment, and potential applications in multiple sectors. Dive in, explore the intricacies, and contribute to this emerging area of study!”
\n
Super! Oh, that’s just what I wanted to try.
\nBut there’s more to it. The competition organizers are the guys from H2O, and the guys from H2O are, in general, legends in the world of Kaggle and competitive Data Science.
\nAnd there’s more again. The baseline to beat on the leaderboard is @psi, and Philipp Singer is #1 in the Kaggle ranking.
\nWell, and the personal cherry on top is that I somehow know all three of the people mentioned in the citation a little bit. I watch Rob Mulla on YouTube, I recently read an interview with Dmitry Larko in a cool DS book, and I’ve crossed paths with Gabor at Kaggle meetups in Budapest. Anyway, let’s get to solving.
\nAt the beginning, we have a labeled dataset with three columns: Question, Response, Label. Each of the 7 models presented in the training data was given 568 identical questions on various topics, from movies and music to mathematics and medicine. Fun fact: the questions were also written by LLM. The metric is Log Loss, with 50% of the score on the public leaderboard and 50% in the private section. No tricks with a Code Competition, everything is fair, no shakeups (???) ahead.
\nAs a newbie in the NLP field and a fan of classical ML, I started solving it in a classic way. My initial assumption was that there would be differences in LLM’s answers that would make it easy to classify authorship. And this difference wasn’t in the content of the texts but in the stylistic and some simple semantic-orthographic characteristics. I broke down the text into 130+ predictors using the spaCy framework.
\nRacing through all this quickly with boosting or AutoML and making the first submission was a failure. The score was terrible, second to last, far from Singer.
\nNow there are two of us, Chuk & Gek, forming a team. One in Budapest, the other in Moscow. We’re gaining experience on Kaggle. Both of us have been searching for jobs in data science for a while, but energetic juniors are not in demand, everyone’s looking for unicorns.
\nSo, we move on. We can remove unnecessary predictors, fine-tune model parameters, try a different framework, or create some ensemble models.
\nWe try it all. The score improves, but the psychological milestone of 1, which we still want to reach, is far from being achieved. We’re oscillating in the range of 1.4–1.8, while Singer is at 0.885.
\nWe read the Discussions at Kaggle. They talk about clustering, embeddings, BERT-like models. Our next solution architecture is built on extracting embeddings through various models and operations with them. We feed all of this back into boosting. The score improves a little bit, but we realized that with this architecture, we won’t go below 1.
\nIt’s time to unleash the 30-hour GPU weekly quota on Kaggle and build a solution with less-explored neural networks. Fine-tuning, new heads, endless parameters. The head is spinning from the amount of new information you need to know to make decisions. There are many networks, it takes a long time to compute, and mistakes are easy to make. Time is running out. There are only a few days left in the 14-day competition. Let’s try to simplify the task and create a baseline not manually but using the open-source H2O LLM Studio.
\nFollowing a forum tip from the competition, we’ve chosen the following approach: take the top non-proprietary LLM model from HuggingFace. Fine-tune it on our data and use Softmax to get the desired probabilities. The amount of sleep has been drastically reduced, but we are rewarded. Singer is in 17th place, and we are in 18th. Hooray! We’ve broken the 1 mark, and we’re standing next to a legend with 40+ submissions.
\nThe casino is on. We want to go higher. One team member is digging through the Transformers documentation and looking at top solutions from previous NLP competitions (by the way, one of our favorites is when yurko, Yuri Kashnitsky, taught networks to detect sarcasm in texts). Meanwhile, another team member, with zero experience, is trying to deploy LLM Studio on a powerful rented server. Setting up tunnels, installing GPU drivers, some sudo, make errors from Unix… We don’t make it in time.
\nOur last hope an hour before the final is to simply average our previous predictions, and now we’re in 10th place on the public leaderboard with a score of 0.774 from 50 submissions, while Singer is in 25th place with a score of 1.
\nTime to sleep. In the morning, there will be private leaderboard results, and we hope we won’t end up in the basement on the shakeup.
\nThe results haven’t been announced yet, although solutions from the leaders are starting to appear. Interestingly, the LLMs themselves seem to predict LLMs best :)
\nFine-tuning and customizing large models are topics that are gaining significant momentum. It’s no wonder I started the article with a person in the dark. These guys with thousands of ideas, millions of investment dollars, and billions of neurons in their heads, building amazing networks, are doing something remarkable. And you can be a part of this fantastic Data Science community, even if it’s just on Kaggle.
\nTeam:
\nChuk -> https://www.linkedin.com/in/samvelkoch/
\n&
\nGek -> https://www.linkedin.com/in/aleksey-schukin/
Links:
\nhttps://www.kaggle.com/competitions/h2oai-predict-the-llm/overview
\nhttps://www.kaggle.com/competitions/h2oai-predict-the-llm/leaderboard
\nhttps://www.kaggle.com/code/tanulsingh077/deep-learning-for-nlp-zero-to-transformers-bert
\nhttps://www.kaggle.com/AI-Report-2023
\nhttps://huggingface.co/learn/nlp-course/chapter1/1
\nhttps://www.deeplearning.ai/short-courses/finetuning-large-language-models/
\nhttps://www.coursera.org/specializations/natural-language-processing
\nhttps://docs.cohere.com/docs/llmu
\nhttps://www.coursera.org/learn/generative-ai-with-llms
\nhttps://web.stanford.edu/class/cs224u/index.html
\nhttps://github.com/h2oai/h2o-llmstudio
\nhttps://auto.gluon.ai/stable/index.html
\nhttps://www.youtube.com/@robmulla
\nhttps://www.youtube.com/@H2Oai
\nhttps://www.philippsinger.info
Thanks ! It was informative for us . We started with the basic principles in NLP and saw how with each complication of our solution and our assessment grows.
", "rawMarkdown": "Thanks ! It was informative for us . We started with the basic principles in NLP and saw how with each complication of our solution and our assessment grows.\n", "votes": 1 }, { "id": 2516236, "postDate": "2023-11-07T15:13:28.137Z", "content": "I'm sure you have spent significant time and effort on this competition. Congrats on the 10th place! Your adventures around this competition are quite inspiring to me!
", "rawMarkdown": "I'm sure you have spent significant time and effort on this competition. Congrats on the 10th place! Your adventures around this competition are quite inspiring to me!", "votes": 1 } ], "complete": true, "source": "live", "fetched_at": "2026-10-19T21:54:39Z", "competition_id": 63021, "competition_slug": "h2oai-predict-the-llm", "team_id": 11131882, "solution_url": "/competitions/h2oai-predict-the-llm/writeups/chuck-gek-8th-public-thrilling-adventures-notes-in" }