{
  "id": 447955,
  "title": "2nd Place Efficiency Solution",
  "content": "<p>Thank you to the sponsors and admins! Absolutely love the impact that these education-focused competitions will have down the road so please keep them coming - these are awesome initiatives.</p>\n<p>Thank you to <a href=\"https://www.kaggle.com/kononenko\" target=\"_blank\">@kononenko</a> and <a href=\"https://www.kaggle.com/andreippv\" target=\"_blank\">@andreippv</a> for providing support to the competition's community to help overcome those license compliancy issues.</p>\n<p>And finally thank you to the heavyweight grandmasters both old and new who didn’t enter the efficiency track. Not sure I would be writing this if they had!</p>\n<h1>2ND PLACE SOLUTION</h1>\n<p>With most competitors focusing on (and clearly succeeding in) pushing the boundaries of transformers, I decided from the onset that I would focus solely on the efficiency track. Not sure if I loved the efficiency leaderboard delay but for some reason it made things a bit more exciting lol - maybe that's just me though.</p>\n<p>Anyway after exhausting readability metrics and prompt/summary overlaps, I turned to exploiting missteps in English writing (e.g. don’t start sentences with subordinating conjunctions, don’t use first-person pronoun I in this context, etc.) and also the human nature of the scorers (e.g. focus on early sections of summary and the lasting first impressions they make, bias towards penalizing for spelling errors). I felt that by doing so I could hone in on the few elements (i.e. scorers and their rubrics/scoring instructions) that were relatively consistent across all prompts no matter the grade or subject - maybe this helped stabilize things when facing the private data.</p>\n<p>Below are some of my solution’s specs:</p>\n<ul>\n<li>Catboost</li>\n<li>CV/PUBLIC/PRIVATE - 0.52/0.452/0.504<ul>\n<li>Used GroupKFold with five different seeds for validation - due to the nature of the public LB I often found myself using it as another fold and re-calculating CV when I was on the fence about a feature. </li></ul></li>\n<li>Hyperparameters by way of Optuna: <ul>\n<li>content_params = {'grow_policy': 'Depthwise', 'iterations': 1490, 'min_data_in_leaf': 93, 'learning_rate': 0.01, 'l2_leaf_reg': 2.0484264500233307, 'colsample_bylevel': 0.42, 'subsample': 0.42, 'max_depth': 6}</li>\n<li>wording_params = {'grow_policy': 'Depthwise', 'iterations': 1230, 'min_data_in_leaf': 7, 'learning_rate': 0.01, 'l2_leaf_reg': 8.403269352596215, 'colsample_bylevel': 0.75, 'subsample': 0.4, 'max_depth': 6}</li></ul></li>\n<li>Features:<ul>\n<li>% of unique words that were misspelled - found that just the count didn’t work while % worked very well</li>\n<li># of quotes in the summary that were found in the prompt</li>\n<li>% of prompt words found in summary</li>\n<li># of bigrams/trigrams that intersect between prompt and summary, also represented these values as a % of the total summary bigrams/trigrams</li>\n<li>% of stop words in summary</li>\n<li># of periods that aren’t followed by a space and capital letter</li>\n<li># of commas that aren’t followed by a space and a letter</li>\n<li># of commas that are preceeded by a space or not typed directly beside a character</li>\n<li># of cardinal numbers + # of existential there + # of co-ordinating conjunctions &lt;- for this I replaced nltk pos_tag with just a list of words/numbers that are captured by pos_tag</li>\n<li># of uppercase characters</li>\n<li>Mean words in each sentence, mean unique words in each sentence, # of sentences, syntax count, # of “and”s, # of characters, # of words, # of difficult words</li>\n<li>% of words in summary that are all uppercase</li>\n<li># of unique verbs and # of unique nouns found in the second sentence of the summary</li>\n<li># of unique word meanings in the summary</li>\n<li>Index of first period, binary flag if summary ends with a period</li>\n<li># of “ i “ or “ I “ - to help indicate if writer used first-person</li>\n<li># of sentences starting with a subordinating conjunction</li>\n<li>Pretty much every readability metric out there except the McAlpine EFLAW score, szigriszt-pazos is the only non-english language metric used</li>\n<li>Difference between Dale-Chall readability scores of prompt and summary, also did this with Spache readability</li>\n<li>Difference between # of stop words found in summary and mean number of stop words across all summaries for the respective prompt_id</li></ul></li>\n<li>Speed-ups (i.e. vrooms):<ul>\n<li>Catboost for faster inference! Much slower to train, but for inference it's great. It was always considerably faster than my LightGBM models, but at the same time slightly worse in terms of CV/PUBLIC LB.</li>\n<li>In almost all cases I used regex patterns instead of word_tokenize and sent_tokenize from nltk - found that they really slowed things down so I avoided them at all costs.</li>\n<li>Used simple list instead of pos_tag from nltk when possible, also used simple list instead of importing stopwords</li>\n<li>Used numpy vectorize as much as possible (groupby features were done with pandas data frame)</li>\n<li>Also tried libraries like daal4py, treelite, and lleaves for faster inference but the compile time didn’t seem to be worth the inference speed-up so I just ended up loading my models normally.</li></ul></li>\n</ul>\n<p><strong>Inference Notebook:</strong> <a href=\"https://www.kaggle.com/code/kpfmma/efficiency-2nd-place-inference-catboost\" target=\"_blank\">https://www.kaggle.com/code/kpfmma/efficiency-2nd-place-inference-catboost</a></p>",
  "comments": [
    {
      "id": 2486510,
      "author_name": "Marília Prata",
      "author_url": "",
      "post_date": "2023-10-18T01:31:41.920000",
      "content": "<p>That was funny: \"finally thanks to the heavyweight grandmasters both old and new who didn’t enter the efficiency track. Not sure I would be writing this if they had!\"</p>\n<p>Old! 🤣</p>",
      "votes": 2,
      "replies": [
        {
          "id": 2486536,
          "author_name": "jagstang",
          "author_url": "",
          "post_date": "2023-10-18T02:31:57.493000",
          "content": "<p>Hahah sorry I should clarify! Meant new/old as in when they received the title 😂</p>",
          "votes": 1,
          "replies": []
        }
      ]
    }
  ],
  "comment_count": 2,
  "messages": [
    {
      "id": 2486436,
      "postDate": "2023-10-17T23:18:01.627Z",
      "content": "<p>Thank you to the sponsors and admins! Absolutely love the impact that these education-focused competitions will have down the road so please keep them coming - these are awesome initiatives.</p>\n<p>Thank you to <a href=\"https://www.kaggle.com/kononenko\" target=\"_blank\">@kononenko</a> and <a href=\"https://www.kaggle.com/andreippv\" target=\"_blank\">@andreippv</a> for providing support to the competition's community to help overcome those license compliancy issues.</p>\n<p>And finally thank you to the heavyweight grandmasters both old and new who didn’t enter the efficiency track. Not sure I would be writing this if they had!</p>\n<h1>2ND PLACE SOLUTION</h1>\n<p>With most competitors focusing on (and clearly succeeding in) pushing the boundaries of transformers, I decided from the onset that I would focus solely on the efficiency track. Not sure if I loved the efficiency leaderboard delay but for some reason it made things a bit more exciting lol - maybe that's just me though.</p>\n<p>Anyway after exhausting readability metrics and prompt/summary overlaps, I turned to exploiting missteps in English writing (e.g. don’t start sentences with subordinating conjunctions, don’t use first-person pronoun I in this context, etc.) and also the human nature of the scorers (e.g. focus on early sections of summary and the lasting first impressions they make, bias towards penalizing for spelling errors). I felt that by doing so I could hone in on the few elements (i.e. scorers and their rubrics/scoring instructions) that were relatively consistent across all prompts no matter the grade or subject - maybe this helped stabilize things when facing the private data.</p>\n<p>Below are some of my solution’s specs:</p>\n<ul>\n<li>Catboost</li>\n<li>CV/PUBLIC/PRIVATE - 0.52/0.452/0.504<ul>\n<li>Used GroupKFold with five different seeds for validation - due to the nature of the public LB I often found myself using it as another fold and re-calculating CV when I was on the fence about a feature. </li></ul></li>\n<li>Hyperparameters by way of Optuna: <ul>\n<li>content_params = {'grow_policy': 'Depthwise', 'iterations': 1490, 'min_data_in_leaf': 93, 'learning_rate': 0.01, 'l2_leaf_reg': 2.0484264500233307, 'colsample_bylevel': 0.42, 'subsample': 0.42, 'max_depth': 6}</li>\n<li>wording_params = {'grow_policy': 'Depthwise', 'iterations': 1230, 'min_data_in_leaf': 7, 'learning_rate': 0.01, 'l2_leaf_reg': 8.403269352596215, 'colsample_bylevel': 0.75, 'subsample': 0.4, 'max_depth': 6}</li></ul></li>\n<li>Features:<ul>\n<li>% of unique words that were misspelled - found that just the count didn’t work while % worked very well</li>\n<li># of quotes in the summary that were found in the prompt</li>\n<li>% of prompt words found in summary</li>\n<li># of bigrams/trigrams that intersect between prompt and summary, also represented these values as a % of the total summary bigrams/trigrams</li>\n<li>% of stop words in summary</li>\n<li># of periods that aren’t followed by a space and capital letter</li>\n<li># of commas that aren’t followed by a space and a letter</li>\n<li># of commas that are preceeded by a space or not typed directly beside a character</li>\n<li># of cardinal numbers + # of existential there + # of co-ordinating conjunctions &lt;- for this I replaced nltk pos_tag with just a list of words/numbers that are captured by pos_tag</li>\n<li># of uppercase characters</li>\n<li>Mean words in each sentence, mean unique words in each sentence, # of sentences, syntax count, # of “and”s, # of characters, # of words, # of difficult words</li>\n<li>% of words in summary that are all uppercase</li>\n<li># of unique verbs and # of unique nouns found in the second sentence of the summary</li>\n<li># of unique word meanings in the summary</li>\n<li>Index of first period, binary flag if summary ends with a period</li>\n<li># of “ i “ or “ I “ - to help indicate if writer used first-person</li>\n<li># of sentences starting with a subordinating conjunction</li>\n<li>Pretty much every readability metric out there except the McAlpine EFLAW score, szigriszt-pazos is the only non-english language metric used</li>\n<li>Difference between Dale-Chall readability scores of prompt and summary, also did this with Spache readability</li>\n<li>Difference between # of stop words found in summary and mean number of stop words across all summaries for the respective prompt_id</li></ul></li>\n<li>Speed-ups (i.e. vrooms):<ul>\n<li>Catboost for faster inference! Much slower to train, but for inference it's great. It was always considerably faster than my LightGBM models, but at the same time slightly worse in terms of CV/PUBLIC LB.</li>\n<li>In almost all cases I used regex patterns instead of word_tokenize and sent_tokenize from nltk - found that they really slowed things down so I avoided them at all costs.</li>\n<li>Used simple list instead of pos_tag from nltk when possible, also used simple list instead of importing stopwords</li>\n<li>Used numpy vectorize as much as possible (groupby features were done with pandas data frame)</li>\n<li>Also tried libraries like daal4py, treelite, and lleaves for faster inference but the compile time didn’t seem to be worth the inference speed-up so I just ended up loading my models normally.</li></ul></li>\n</ul>\n<p><strong>Inference Notebook:</strong> <a href=\"https://www.kaggle.com/code/kpfmma/efficiency-2nd-place-inference-catboost\" target=\"_blank\">https://www.kaggle.com/code/kpfmma/efficiency-2nd-place-inference-catboost</a></p>",
      "rawMarkdown": "Thank you to the sponsors and admins! Absolutely love the impact that these education-focused competitions will have down the road so please keep them coming - these are awesome initiatives.\n\nThank you to @kononenko and @andreippv for providing support to the competition's community to help overcome those license compliancy issues.\n\nAnd finally thank you to the heavyweight grandmasters both old and new who didn’t enter the efficiency track. Not sure I would be writing this if they had!\n\n# 2ND PLACE SOLUTION\n\nWith most competitors focusing on (and clearly succeeding in) pushing the boundaries of transformers, I decided from the onset that I would focus solely on the efficiency track. Not sure if I loved the efficiency leaderboard delay but for some reason it made things a bit more exciting lol - maybe that's just me though.\n\nAnyway after exhausting readability metrics and prompt/summary overlaps, I turned to exploiting missteps in English writing (e.g. don’t start sentences with subordinating conjunctions, don’t use first-person pronoun I in this context, etc.) and also the human nature of the scorers (e.g. focus on early sections of summary and the lasting first impressions they make, bias towards penalizing for spelling errors). I felt that by doing so I could hone in on the few elements (i.e. scorers and their rubrics/scoring instructions) that were relatively consistent across all prompts no matter the grade or subject - maybe this helped stabilize things when facing the private data.\n\nBelow are some of my solution’s specs:\n\n- Catboost\n- CV/PUBLIC/PRIVATE - 0.52/0.452/0.504\n    - Used GroupKFold with five different seeds for validation - due to the nature of the public LB I often found myself using it as another fold and re-calculating CV when I was on the fence about a feature. \n- Hyperparameters by way of Optuna: \n    - content_params = {'grow_policy': 'Depthwise', 'iterations': 1490, 'min_data_in_leaf': 93, 'learning_rate': 0.01, 'l2_leaf_reg': 2.0484264500233307, 'colsample_bylevel': 0.42, 'subsample': 0.42, 'max_depth': 6}\n    - wording_params = {'grow_policy': 'Depthwise', 'iterations': 1230, 'min_data_in_leaf': 7, 'learning_rate': 0.01, 'l2_leaf_reg': 8.403269352596215, 'colsample_bylevel': 0.75, 'subsample': 0.4, 'max_depth': 6}\n- Features:\n    - % of unique words that were misspelled - found that just the count didn’t work while % worked very well\n    - # of quotes in the summary that were found in the prompt\n    - % of prompt words found in summary\n    - # of bigrams/trigrams that intersect between prompt and summary, also represented these values as a % of the total summary bigrams/trigrams\n    - % of stop words in summary\n    - # of periods that aren’t followed by a space and capital letter\n    - # of commas that aren’t followed by a space and a letter\n    - # of commas that are preceeded by a space or not typed directly beside a character\n    - # of cardinal numbers + # of existential there + # of co-ordinating conjunctions <- for this I replaced nltk pos_tag with just a list of words/numbers that are captured by pos_tag\n    - # of uppercase characters\n    - Mean words in each sentence, mean unique words in each sentence, # of sentences, syntax count, # of “and”s, # of characters, # of words, # of difficult words\n    - % of words in summary that are all uppercase\n    - # of unique verbs and # of unique nouns found in the second sentence of the summary\n    - # of unique word meanings in the summary\n    - Index of first period, binary flag if summary ends with a period\n    - # of “ i “ or “ I “ - to help indicate if writer used first-person\n    - # of sentences starting with a subordinating conjunction\n    - Pretty much every readability metric out there except the McAlpine EFLAW score, szigriszt-pazos is the only non-english language metric used\n    - Difference between Dale-Chall readability scores of prompt and summary, also did this with Spache readability\n    - Difference between # of stop words found in summary and mean number of stop words across all summaries for the respective prompt_id\n- Speed-ups (i.e. vrooms):\n    - Catboost for faster inference! Much slower to train, but for inference it's great. It was always considerably faster than my LightGBM models, but at the same time slightly worse in terms of CV/PUBLIC LB.\n    - In almost all cases I used regex patterns instead of word_tokenize and sent_tokenize from nltk - found that they really slowed things down so I avoided them at all costs.\n    - Used simple list instead of pos_tag from nltk when possible, also used simple list instead of importing stopwords\n    - Used numpy vectorize as much as possible (groupby features were done with pandas data frame)\n    - Also tried libraries like daal4py, treelite, and lleaves for faster inference but the compile time didn’t seem to be worth the inference speed-up so I just ended up loading my models normally.\n\n**Inference Notebook:** https://www.kaggle.com/code/kpfmma/efficiency-2nd-place-inference-catboost\n",
      "votes": 18
    },
    {
      "id": 2486510,
      "postDate": "2023-10-18T01:31:41.920Z",
      "content": "<p>That was funny: \"finally thanks to the heavyweight grandmasters both old and new who didn’t enter the efficiency track. Not sure I would be writing this if they had!\"</p>\n<p>Old! 🤣</p>",
      "rawMarkdown": "That was funny: \"finally thanks to the heavyweight grandmasters both old and new who didn’t enter the efficiency track. Not sure I would be writing this if they had!\"\n\nOld! 🤣",
      "votes": 2,
      "replies": [
        {
          "id": 2486536,
          "postDate": "2023-10-18T02:31:57.493Z",
          "content": "<p>Hahah sorry I should clarify! Meant new/old as in when they received the title 😂</p>",
          "rawMarkdown": "Hahah sorry I should clarify! Meant new/old as in when they received the title 😂",
          "votes": 1
        }
      ]
    }
  ],
  "complete": true,
  "source": "live",
  "fetched_at": "2026-10-19T23:07:31Z",
  "competition_id": 53482,
  "competition_slug": "commonlit-evaluate-student-summaries",
  "team_id": 10737880,
  "solution_url": "/competitions/commonlit-evaluate-student-summaries/writeups/jagstang-2nd-place-efficiency-solution"
}