First of all, I want to thank H2o.ai for hosting a very interesting competition. This is such a competition where I can apply techniques I learned from the H20 team in past NLP competitions.
\nIn my solution, I used a single deberta-v3-large with a linear head for classification. Input setup is the key to my solution and plays a crucial role in optimizing the performance of Deberta. After cross-validation with 5 folds, I retrained the model on full data with 8 different seeds for stability.
\nThe structure of the data contains 7 unique responses for each question. Thus, I used the input below for Deberta to see the question and all 7 responses at a time. I then applied simple mean pooling for pooling each response that is marked by added special tokens [START] and [END] and ran it through the linear head for classification.
\ninput: Question [SEP] [START] Response_1 [END] [START] Response_2 [END] [START] Response_3 [END] ... [START] Response_7 [END]
Note: Response_1, Response_2 … Response_7 are responses from some question (order does not matter here)
\nI initially began using GroupKFold by question, but I switched to using K-Fold after adopting the input structure described earlier. The reason for this shift is that each training example contains all responses to a specific question, thereby eliminating any potential data leakage across folds by using K-Fold. My local CV is 0.846 with seed=252.
\nNotebook: https://www.kaggle.com/code/ducnh279/h2o-ai-predict-the-llm-4th-solution?scriptVersionId=148539480
", "comments": [ { "id": 2516424, "author_name": "Phaedrus", "author_url": "", "post_date": "2023-11-07T17:54:49.133000", "content": "I was also inclined to play around with pooling options for deberta. Especially wanted to try some of the tricks from commonlit and feedback comps. Hats off for the great solution and intuition. Would you be able to share your training codes?
", "votes": 1, "replies": [ { "id": 2516451, "author_name": "ducnh279", "author_url": "", "post_date": "2023-11-07T18:09:47.107000", "content": "Thank you @pheadrus for your kind words! It means a lot to me when it comes from you!
\nThe code I shared in this post includes both training and inference code with the exception of the full data training after I do cross-validation with 5 folds on my local machine!
\nDo you mean training code is the code with cross-validation?
", "votes": 0, "replies": [ { "id": 2516627, "author_name": "Phaedrus", "author_url": "", "post_date": "2023-11-07T20:30:47.563000", "content": "Sorry I wasn't clear. I meant if you'd be sharing the couple of private datasets in your notebook. Interested to see your implementation and some parameter choices.
", "votes": 0, "replies": [] }, { "id": 2517317, "author_name": "ducnh279", "author_url": "", "post_date": "2023-11-08T11:33:10.697000", "content": "I got you @pheadrus ! I forgot to make the datasets public. It consists of a simple training script that is a bit messy though. It will be a shame to share that with you, however; I created a cleaner version. You now can see my training script here:
\nhttps://www.kaggle.com/code/ducnh279/fork-of-deberta-v3-large-train-infer-sub/input
", "votes": 1, "replies": [] }, { "id": 2517614, "author_name": "Phaedrus", "author_url": "", "post_date": "2023-11-08T16:22:56.747000", "content": "lol. Thanks. :)
", "votes": 0, "replies": [] } ] } ] }, { "id": 2516381, "author_name": "RB", "author_url": "", "post_date": "2023-11-07T17:20:54.123000", "content": "Thank you for sharing your solution. I also wanted to try this approach i.e. running the model for 568 samples rather than 568*7 samples , unfortunately I missed trying this. what loss function did you use for training ?
", "votes": 1, "replies": [ { "id": 2516387, "author_name": "ducnh279", "author_url": "", "post_date": "2023-11-07T17:26:20.397000", "content": "I directly optimize cross-entropy loss (log loss)!
", "votes": 1, "replies": [] } ] } ], "comment_count": 7, "messages": [ { "id": 2515736, "postDate": "2023-11-07T07:20:37.960Z", "content": "First of all, I want to thank H2o.ai for hosting a very interesting competition. This is such a competition where I can apply techniques I learned from the H20 team in past NLP competitions.
\nIn my solution, I used a single deberta-v3-large with a linear head for classification. Input setup is the key to my solution and plays a crucial role in optimizing the performance of Deberta. After cross-validation with 5 folds, I retrained the model on full data with 8 different seeds for stability.
\nThe structure of the data contains 7 unique responses for each question. Thus, I used the input below for Deberta to see the question and all 7 responses at a time. I then applied simple mean pooling for pooling each response that is marked by added special tokens [START] and [END] and ran it through the linear head for classification.
\ninput: Question [SEP] [START] Response_1 [END] [START] Response_2 [END] [START] Response_3 [END] ... [START] Response_7 [END]
Note: Response_1, Response_2 … Response_7 are responses from some question (order does not matter here)
\nI initially began using GroupKFold by question, but I switched to using K-Fold after adopting the input structure described earlier. The reason for this shift is that each training example contains all responses to a specific question, thereby eliminating any potential data leakage across folds by using K-Fold. My local CV is 0.846 with seed=252.
\nNotebook: https://www.kaggle.com/code/ducnh279/h2o-ai-predict-the-llm-4th-solution?scriptVersionId=148539480
", "rawMarkdown": "First of all, I want to thank H2o.ai for hosting a very interesting competition. This is such a competition where I can apply techniques I learned from the H20 team in past NLP competitions. \n\n# TL;DR\nIn my solution, I used a single deberta-v3-large with a linear head for classification. Input setup is the key to my solution and plays a crucial role in optimizing the performance of Deberta. After cross-validation with 5 folds, I retrained the model on full data with 8 different seeds for stability. \n\n# Model\n- Single deberta-v3-large (for me, deberta is still king!!)\n\n# Special tokens added\n- [START] and [END] tokens, primarily utilized for pooling.\n- The newline character (\\n) was introduced to enhance Deberta's understanding of the text.\n\n# Input\nThe structure of the data contains 7 unique responses for each question. Thus, I used the input below for Deberta to see the question and all 7 responses at a time. I then applied simple mean pooling for pooling each response that is marked by added special tokens [START] and [END] and ran it through the linear head for classification.\n\ninput: `Question [SEP] [START] Response_1 [END] [START] Response_2 [END] [START] Response_3 [END] ... [START] Response_7 [END]`\n\n*Note*: Response_1, Response_2 ... Response_7 are responses from some question (order does not matter here)\n\n# Cross-validation\nI initially began using GroupKFold by question, but I switched to using K-Fold after adopting the input structure described earlier. The reason for this shift is that each training example contains all responses to a specific question, thereby eliminating any potential data leakage across folds by using K-Fold. My local CV is 0.846 with seed=252.\n\n# Code\nNotebook: https://www.kaggle.com/code/ducnh279/h2o-ai-predict-the-llm-4th-solution?scriptVersionId=148539480", "votes": 20 }, { "id": 2516424, "postDate": "2023-11-07T17:54:49.133Z", "content": "I was also inclined to play around with pooling options for deberta. Especially wanted to try some of the tricks from commonlit and feedback comps. Hats off for the great solution and intuition. Would you be able to share your training codes?
", "rawMarkdown": "I was also inclined to play around with pooling options for deberta. Especially wanted to try some of the tricks from commonlit and feedback comps. Hats off for the great solution and intuition. Would you be able to share your training codes? ", "votes": 1, "replies": [ { "id": 2516451, "postDate": "2023-11-07T18:09:47.107Z", "content": "Thank you @pheadrus for your kind words! It means a lot to me when it comes from you!
\nThe code I shared in this post includes both training and inference code with the exception of the full data training after I do cross-validation with 5 folds on my local machine!
\nDo you mean training code is the code with cross-validation?
", "rawMarkdown": "Thank you @pheadrus for your kind words! It means a lot to me when it comes from you! \n\nThe code I shared in this post includes both training and inference code with the exception of the full data training after I do cross-validation with 5 folds on my local machine!\n\nDo you mean training code is the code with cross-validation? ", "replies": [ { "id": 2516627, "postDate": "2023-11-07T20:30:47.563Z", "content": "Sorry I wasn't clear. I meant if you'd be sharing the couple of private datasets in your notebook. Interested to see your implementation and some parameter choices.
", "rawMarkdown": "Sorry I wasn't clear. I meant if you'd be sharing the couple of private datasets in your notebook. Interested to see your implementation and some parameter choices. " }, { "id": 2517317, "postDate": "2023-11-08T11:33:10.697Z", "content": "I got you @pheadrus ! I forgot to make the datasets public. It consists of a simple training script that is a bit messy though. It will be a shame to share that with you, however; I created a cleaner version. You now can see my training script here:
\nhttps://www.kaggle.com/code/ducnh279/fork-of-deberta-v3-large-train-infer-sub/input
", "rawMarkdown": "I got you @pheadrus ! I forgot to make the datasets public. It consists of a simple training script that is a bit messy though. It will be a shame to share that with you, however; I created a cleaner version. You now can see my training script here:\n\nhttps://www.kaggle.com/code/ducnh279/fork-of-deberta-v3-large-train-infer-sub/input", "votes": 1 }, { "id": 2517614, "postDate": "2023-11-08T16:22:56.747Z", "content": "lol. Thanks. :)
", "rawMarkdown": "lol. Thanks. :) " } ] } ] }, { "id": 2516381, "postDate": "2023-11-07T17:20:54.123Z", "content": "Thank you for sharing your solution. I also wanted to try this approach i.e. running the model for 568 samples rather than 568*7 samples , unfortunately I missed trying this. what loss function did you use for training ?
", "rawMarkdown": "Thank you for sharing your solution. I also wanted to try this approach i.e. running the model for 568 samples rather than 568*7 samples , unfortunately I missed trying this. what loss function did you use for training ? ", "votes": 1, "replies": [ { "id": 2516387, "postDate": "2023-11-07T17:26:20.397Z", "content": "I directly optimize cross-entropy loss (log loss)!
", "rawMarkdown": "I directly optimize cross-entropy loss (log loss)!", "votes": 1 } ] } ], "complete": true, "source": "live", "fetched_at": "2026-10-19T21:54:30Z", "competition_id": 63021, "competition_slug": "h2oai-predict-the-llm", "team_id": 11129753, "solution_url": "" }