First and foremost, I want to express how much I've enjoyed participating in this competition and how deeply I adore delving into the world of machine learning. However, I must confess that I became quite engrossed in this competition, finding it somewhat addictive to explore the plethora of novel ideas and approaches constantly bubbling up in my mind. I couldn't resist the urge to implement and experiment with them. In all honesty, especially during the last few weeks, I did experience a degree of disappointment. It stemmed from my inability to unearth any substantial correlations, structures, or approaches that significantly contributed to improving my scores. I meticulously crafted features that exhibited a high correlation with the score, but to my chagrin, they often led to worse results. Towards the end, it felt more akin to filling out a lottery ticket than engaging in scientific endeavors; it had become far too random.
\nSince we're encouraged to share our solutions, I'd like to offer my highest-scoring notebook and provide an overview of some of the strategies I pursued with great hope, only to see them fall short in the end.
\nI should mention that from the start, I made a conscious decision to focus exclusively on the Efficiency track, given my passion for constructing computationally efficient solutions. I also refused to simply run one of the high scoring publicly shared notebooks.
\nHere are some of the techniques I experimented with, despite their ultimate lack of impact on my scores:
\nIn my best-performing notebook, I did incorporate BM25 and textblob features. However, I can't say I was entirely confident that these features were responsible for the observed score improvements. There must have been some effect, but I wished I could have established a more concrete relationship between these features and my scores.
\n**My unsuccessful attempt with a BERT model: **
\nI aimed to deploy a BERT model capable of processing data in under 10 minutes using only CPU resources. I experimented with various BERT variants but eventually settled on deberta-v3-xsmall, with a limited input length of only 96 tokens to heavily expedite inference. I trained the model on 30% of the data in a single epoch with a learning rate of 1.5e-5. Subsequently, I froze the layers and added content and wording heads, one for each fold. These heads consisted of simple GRU models connected to the CLS layer, essentially creating an ensemble without incurring additional time costs.
\nWhile the model achieved an RMSE of approximately 0.58 in isolation, it lowered my cv score to around 0.44. However, my public score remained at 0.56, indicating severe overfitting to the training data.
\nIn hindsight, I acknowledge that I might have achieved better results with a longer input length, but given my CPU constraints and my desire to perform well on the efficiency leaderboard, I opted to work with a smaller text portion.
\nI want to emphasize that I'm not a seasoned LLM expert; rather, I'm a learner eager to experiment and get creative in crafting solutions that work effectively and score well. I truly wish my efforts had borne fruit, but that is true for all of us of course.
\n** Regarding my used BERT model: **
\nIn this particular notebook, I employed a deberta-v3-xsmall model trained using the data from the English Language competition. My approach involved selecting the three middle sentences from each text in that competition data to train the model, as this approximated the average text length of the input_text. However, I restricted the maximum input length as well to only 96 tokens.
\nThis managed to boost my score by approximately 0.04.
\nI welcome any feedback on my approach, as there's nothing more valuable than the opportunity to learn and improve.
\nThank you very much.
\nhttps://www.kaggle.com/code/geoffreybeulque/commonlit-efficiency-final-submission
", "comments": [], "comment_count": 0, "messages": [ { "id": 2488834, "postDate": "2023-10-19T14:36:03.313Z", "content": "First and foremost, I want to express how much I've enjoyed participating in this competition and how deeply I adore delving into the world of machine learning. However, I must confess that I became quite engrossed in this competition, finding it somewhat addictive to explore the plethora of novel ideas and approaches constantly bubbling up in my mind. I couldn't resist the urge to implement and experiment with them. In all honesty, especially during the last few weeks, I did experience a degree of disappointment. It stemmed from my inability to unearth any substantial correlations, structures, or approaches that significantly contributed to improving my scores. I meticulously crafted features that exhibited a high correlation with the score, but to my chagrin, they often led to worse results. Towards the end, it felt more akin to filling out a lottery ticket than engaging in scientific endeavors; it had become far too random.
\nSince we're encouraged to share our solutions, I'd like to offer my highest-scoring notebook and provide an overview of some of the strategies I pursued with great hope, only to see them fall short in the end.
\nI should mention that from the start, I made a conscious decision to focus exclusively on the Efficiency track, given my passion for constructing computationally efficient solutions. I also refused to simply run one of the high scoring publicly shared notebooks.
\nHere are some of the techniques I experimented with, despite their ultimate lack of impact on my scores:
\nIn my best-performing notebook, I did incorporate BM25 and textblob features. However, I can't say I was entirely confident that these features were responsible for the observed score improvements. There must have been some effect, but I wished I could have established a more concrete relationship between these features and my scores.
\n**My unsuccessful attempt with a BERT model: **
\nI aimed to deploy a BERT model capable of processing data in under 10 minutes using only CPU resources. I experimented with various BERT variants but eventually settled on deberta-v3-xsmall, with a limited input length of only 96 tokens to heavily expedite inference. I trained the model on 30% of the data in a single epoch with a learning rate of 1.5e-5. Subsequently, I froze the layers and added content and wording heads, one for each fold. These heads consisted of simple GRU models connected to the CLS layer, essentially creating an ensemble without incurring additional time costs.
\nWhile the model achieved an RMSE of approximately 0.58 in isolation, it lowered my cv score to around 0.44. However, my public score remained at 0.56, indicating severe overfitting to the training data.
\nIn hindsight, I acknowledge that I might have achieved better results with a longer input length, but given my CPU constraints and my desire to perform well on the efficiency leaderboard, I opted to work with a smaller text portion.
\nI want to emphasize that I'm not a seasoned LLM expert; rather, I'm a learner eager to experiment and get creative in crafting solutions that work effectively and score well. I truly wish my efforts had borne fruit, but that is true for all of us of course.
\n** Regarding my used BERT model: **
\nIn this particular notebook, I employed a deberta-v3-xsmall model trained using the data from the English Language competition. My approach involved selecting the three middle sentences from each text in that competition data to train the model, as this approximated the average text length of the input_text. However, I restricted the maximum input length as well to only 96 tokens.
\nThis managed to boost my score by approximately 0.04.
\nI welcome any feedback on my approach, as there's nothing more valuable than the opportunity to learn and improve.
\nThank you very much.
\nhttps://www.kaggle.com/code/geoffreybeulque/commonlit-efficiency-final-submission
", "rawMarkdown": "First and foremost, I want to express how much I've enjoyed participating in this competition and how deeply I adore delving into the world of machine learning. However, I must confess that I became quite engrossed in this competition, finding it somewhat addictive to explore the plethora of novel ideas and approaches constantly bubbling up in my mind. I couldn't resist the urge to implement and experiment with them. In all honesty, especially during the last few weeks, I did experience a degree of disappointment. It stemmed from my inability to unearth any substantial correlations, structures, or approaches that significantly contributed to improving my scores. I meticulously crafted features that exhibited a high correlation with the score, but to my chagrin, they often led to worse results. Towards the end, it felt more akin to filling out a lottery ticket than engaging in scientific endeavors; it had become far too random.\n\nSince we're encouraged to share our solutions, I'd like to offer my highest-scoring notebook and provide an overview of some of the strategies I pursued with great hope, only to see them fall short in the end.\n\nI should mention that from the start, I made a conscious decision to focus exclusively on the Efficiency track, given my passion for constructing computationally efficient solutions. I also refused to simply run one of the high scoring publicly shared notebooks.\n\nHere are some of the techniques I experimented with, despite their ultimate lack of impact on my scores:\n\n- Extracting 10 stemmed noun-keywords from both prompt_question+prompt_text and input_text, and then comparing them, both direct similarity as tfidf_vectorized cosine similarity.\n- Extracting the three most crucial sentences from the prompt_text and utilizing them for comparisons with the input_text.\n- Employing metrics such as ROUGE scores, METEOR scores, JACCARD similarity, and BLEU scores.\n\nIn my best-performing notebook, I did incorporate BM25 and textblob features. However, I can't say I was entirely confident that these features were responsible for the observed score improvements. There must have been some effect, but I wished I could have established a more concrete relationship between these features and my scores.\n\n**My unsuccessful attempt with a BERT model: **\n\nI aimed to deploy a BERT model capable of processing data in under 10 minutes using only CPU resources. I experimented with various BERT variants but eventually settled on deberta-v3-xsmall, with a limited input length of only 96 tokens to heavily expedite inference. I trained the model on 30% of the data in a single epoch with a learning rate of 1.5e-5. Subsequently, I froze the layers and added content and wording heads, one for each fold. These heads consisted of simple GRU models connected to the CLS layer, essentially creating an ensemble without incurring additional time costs.\n\nWhile the model achieved an RMSE of approximately 0.58 in isolation, it lowered my cv score to around 0.44. However, my public score remained at 0.56, indicating severe overfitting to the training data.\n\nIn hindsight, I acknowledge that I might have achieved better results with a longer input length, but given my CPU constraints and my desire to perform well on the efficiency leaderboard, I opted to work with a smaller text portion.\n\nI want to emphasize that I'm not a seasoned LLM expert; rather, I'm a learner eager to experiment and get creative in crafting solutions that work effectively and score well. I truly wish my efforts had borne fruit, but that is true for all of us of course.\n\n** Regarding my used BERT model: **\n\nIn this particular notebook, I employed a deberta-v3-xsmall model trained using the data from the English Language competition. My approach involved selecting the three middle sentences from each text in that competition data to train the model, as this approximated the average text length of the input_text. However, I restricted the maximum input length as well to only 96 tokens.\n\nThis managed to boost my score by approximately 0.04.\n\nI welcome any feedback on my approach, as there's nothing more valuable than the opportunity to learn and improve.\n\nThank you very much.\n\nhttps://www.kaggle.com/code/geoffreybeulque/commonlit-efficiency-final-submission", "votes": 3 } ], "complete": true, "source": "live", "fetched_at": "2026-10-19T23:07:47Z", "competition_id": 53482, "competition_slug": "commonlit-evaluate-student-summaries", "team_id": 10794497, "solution_url": "/competitions/commonlit-evaluate-student-summaries/writeups/geoffrey-beulque-22nd-efficiency-my-creative-ideas" }