![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F7523757%2F717c6210085b3728d0a839bb7b35e846%2FCDS.jpg?generation=1697980812788529&alt=media\")
Several transformer models were leveraged, forming a \"stack of BERTs\" to enhance prediction accuracy.
Pre-existing models were harnessed to make out-of-fold predictions on the training data. The training process can be reviewed here.
Functions were defined to facilitate the efficient inference of test data using the trained models.
A customized model architecture was employed, utilizing Huggingface's trainer.
Different pooling strategies were explored for the transformer outputs. WeightedLayerPooling was the chosen method due to its efficacy.
A custom dataset class in PyTorch was designed for test inference, accompanied by its associated function.
Given its resilience against overfitting and robustness to outliers, the HuberRegressor was chosen as the meta model.
Beyond transformer-based features, various intuitive features were conceptualized, like word/token statistics, text readability score, and prompt-essay comparisons.
The BARTScorer class was utilized to compute the BART score, shedding light on the essay's quality.
Several pre-trained models were deployed to derive features that measure the degree of alignment between the prompt and the essay.
A plethora of custom features were engineered to capture the nuances of the text, such as sentence structure, token intersections, and linguistic errors.
Sentence encoders, reward models, and other tools were employed to craft features that capture the essence of the text.
Features that provide insights into the text's structure and coherence, like the count of stop words, n-gram token intersections, and sentence errors, were extracted.
Models trained on previous competition datasets were leveraged to craft additional, insightful features.
All features, including transformer predictions and manually crafted ones, were integrated and readied for the final model.
Cross-validation methods were possibly used to gauge the model's performance on unseen data.
The model was trained with a limited iteration count to prevent overfitting and ensure its general applicability.
Predictions for the test dataset were crafted using the meticulously trained HuberRegressor model.
Predictions were organized into the required format, typically consisting of an ID and its corresponding predicted value.
The structured submission file was uploaded to Kaggle's competition page, marking the culmination of the process.
Wait wait !!! But in 98th place - me!)
", "votes": 0, "replies": [] }, { "id": 2499017, "author_name": "Paul Cummins", "author_url": "", "post_date": "2023-10-25T17:15:11.550000", "content": "Can I check my understanding of your method (my lack of experience in this kaggle lark rather than your lack of explanation)?
\nIn ridiculously simple termsβ¦
\nand bam, final scores?
", "votes": 0, "replies": [] } ], "comment_count": 2, "messages": [ { "id": 2492443, "postDate": "2023-10-22T14:16:56.810Z", "content": "Several transformer models were leveraged, forming a \"stack of BERTs\" to enhance prediction accuracy.
Pre-existing models were harnessed to make out-of-fold predictions on the training data. The training process can be reviewed here.
Functions were defined to facilitate the efficient inference of test data using the trained models.
A customized model architecture was employed, utilizing Huggingface's trainer.
Different pooling strategies were explored for the transformer outputs. WeightedLayerPooling was the chosen method due to its efficacy.
A custom dataset class in PyTorch was designed for test inference, accompanied by its associated function.
Given its resilience against overfitting and robustness to outliers, the HuberRegressor was chosen as the meta model.
Beyond transformer-based features, various intuitive features were conceptualized, like word/token statistics, text readability score, and prompt-essay comparisons.
The BARTScorer class was utilized to compute the BART score, shedding light on the essay's quality.
Several pre-trained models were deployed to derive features that measure the degree of alignment between the prompt and the essay.
A plethora of custom features were engineered to capture the nuances of the text, such as sentence structure, token intersections, and linguistic errors.
Sentence encoders, reward models, and other tools were employed to craft features that capture the essence of the text.
Features that provide insights into the text's structure and coherence, like the count of stop words, n-gram token intersections, and sentence errors, were extracted.
Models trained on previous competition datasets were leveraged to craft additional, insightful features.
All features, including transformer predictions and manually crafted ones, were integrated and readied for the final model.
Cross-validation methods were possibly used to gauge the model's performance on unseen data.
The model was trained with a limited iteration count to prevent overfitting and ensure its general applicability.
Predictions for the test dataset were crafted using the meticulously trained HuberRegressor model.
Predictions were organized into the required format, typically consisting of an ID and its corresponding predicted value.
The structured submission file was uploaded to Kaggle's competition page, marking the culmination of the process.
Wait wait !!! But in 98th place - me!)
", "rawMarkdown": "Wait wait !!! But in 98th place - me!)" }, { "id": 2499017, "postDate": "2023-10-25T17:15:11.550Z", "content": "Can I check my understanding of your method (my lack of experience in this kaggle lark rather than your lack of explanation)?
\nIn ridiculously simple termsβ¦
\nand bam, final scores?
", "rawMarkdown": "Can I check my understanding of your method (my lack of experience in this kaggle lark rather than your lack of explanation)?\n\nIn ridiculously simple terms... \n- you used a battery of BERT models to tokenise the prompts and responses, then used them to predict content and wording scores\n- Engineered a bunch of other features\n- combined the outputs of the BERT predictions and the other features you engineered to train a final model\n\nand bam, final scores?\n\n\n\n\n" } ], "complete": true, "source": "live", "fetched_at": "2026-10-19T23:08:57Z", "competition_id": 53482, "competition_slug": "commonlit-evaluate-student-summaries", "team_id": 10676067, "solution_url": "/competitions/commonlit-evaluate-student-summaries/writeups/sanseymysamba-98th-place-solution-overview" }