{
  "id": 553573,
  "title": "Efficiency 3rd Place : Public Model + Simple Post Processing",
  "content": "<p>I'd like to thank Kaggle, the organisers and all the winners of this very interesting challenge. I want also to <strong>SPECIALLY THANK <a href=\"https://www.kaggle.com/medali1992\" target=\"_blank\">@medali1992</a></strong>. Without him, I could'nt have made it to this 3rd place. Indeed, I used his public model shared <a href=\"https://www.kaggle.com/datasets/medali1992/mod-gte-base-weights\" target=\"_blank\">here</a>.<br>\nThis thread is intented to share the approach I used to reach efficiency 3rd place. </p>\n<h1>1- Main Idea</h1>\n<p>The main idea was to adjust the similarity matrix derived from model embeddings. The adjustment is based on couple (<code>ConstructName</code>,<code>MisconceptionName</code>). The approach I used, was quite similar to the one used in the beginning of the competition. We create a prompt by concatenating <code>SubjectName</code>,<code>ConstructName</code>,<code>QuestionText</code> and <code>AnswerText</code>. Once we get the embedding of the prompts and the 2587 misconceptions, We use cosine similarity to predict the top 25.</p>\n<p>But I noticed that, they were <strong>blattant inconsistencies</strong> in the top 25 that can be sorted by <strong>precomputed pointers</strong>.  For instance :</p>\n<ul>\n<li>1) for question related to pie charts we can get misconceptions related to simultaneous equations in the top 25.</li>\n<li>2) for question related to gradient we can get misconceptions related triangles in the top 25.</li>\n</ul>\n<h3>Construction &amp; Post Processing rule</h3>\n<p>That's the reason why, I decided to build pointers for the couples (<code>ConstructName</code>,<code>MisconceptionName</code>). I split the misconceptions in two groups. The first group \\(M_{-}\\) contains all the 1623 misconceptions present in the training dataset, the second group \\(M_{+}\\) the 964 remaining misconceptions. Now for a <code>ConstructName</code>  \\(c\\) present in the training dataset, we associate two groups of misconceptions. The first group \\(S_c\\) contains all the misconceptions that can safely be associated with \\(c\\), and the second group \\(P_c\\) all the misconceptions that can potentially be associated with \\(c\\).  For a new misconception \\(c\\) not present in the training dataset, we decided to take \\(S_c=\\emptyset\\)  and \\(P_c\\ = M_{+}\\). It means that for a new misconception, we have no guarantee of a safe association, and we take all the unused misconceptions as potential assocation (to be more inclusive).</p>\n<p>The adjustment rule used two parameters \\(\\alpha\\) and \\(\\beta\\).  For a sample with  prompt embedding \\(x\\), <code>ConstructName</code> \\(c\\) and a misconception \\(m\\) with embedding \\(x_m\\); We adjust the similarity as follow: </p>\n<ul>\n<li>\\(  S(c,m) = x^T.x_m + \\alpha  \\) if \\(m \\in S_c \\)</li>\n<li>\\(  S(c,m) = x^T.x_m + \\beta  \\) if \\(m \\in P_c \\)</li>\n<li>\\(  S(c,m) = x^T.x_m   \\)  else.</li>\n</ul>\n<h3>Limitations</h3>\n<p>They are <strong>two drawbacks</strong> for this post-processing :</p>\n<ul>\n<li>First  for the new <code>ConstructName</code>, we only adjust the <em>unseen</em> <code>MisconceptionName</code>. By doing this, we are assuming that the most probable misconceptions are the unseen ones. So in the case where the true misconception is in the training dataset, it is less likely for us to get it in the top 25.</li>\n<li>The choice of \\(\\alpha\\) and \\(\\beta\\) : For instance if  \\(\\alpha - \\beta\\) is too big, we can constrain the top 25 to almost the same misconceptions. At least, these parameters shoud be chosen by cross-validation. Yet I didn't have time to do it, since I was very active in this challenge only the first month. </li>\n</ul>\n<h1>2- Notebooks</h1>\n<p>There is nothing special here. Since this approach was intended for the efficiency track, we first pre-computed the misconceptions embeddings in a prior notebook (to save time) and then use them in the very inference notebook as well as adjustment pointers.</p>\n<ul>\n<li>The inference notebook can be found <a href=\"https://www.kaggle.com/code/ulrich07/eedi-cpu-starter/notebook\" target=\"_blank\">here</a>.</li>\n<li>The precomputed embeddings can be found <a href=\"https://www.kaggle.com/code/ulrich07/eedi-starter-sent-embs-cpu\" target=\"_blank\">here</a>.</li>\n<li>The similarity adjustment pointers can be found <a href=\"https://www.kaggle.com/datasets/ulrich07/eedi-handy\" target=\"_blank\">here</a> in the picke file <code>dus.pkl</code>.</li>\n</ul>\n<h1>3- Potential Improvements</h1>\n<p>Of course, this work can be improved in diverse ways. But the one that came in my mind, was to use my pre-computed pointers to learn new embeddings both for <code>ConstructName</code> and <code>MisconceptionName</code>. In this way, we could effectively better handle the new  <code>ConstructName</code> not adressed in the pointer file.</p>",
  "comments": [
    {
      "id": 3081644,
      "author_name": "Med Ali Bouchhioua",
      "author_url": "",
      "post_date": "2024-12-27T03:09:58.987000",
      "content": "<p>Congrats, I'm happy for you.</p>",
      "votes": 2,
      "replies": []
    },
    {
      "id": 3083317,
      "author_name": "Romaric APPIAH",
      "author_url": "",
      "post_date": "2024-12-29T09:52:31.523000",
      "content": "<p>Congratulations 👏👏👏</p>",
      "votes": 2,
      "replies": []
    }
  ],
  "comment_count": 2,
  "messages": [
    {
      "id": 3081590,
      "postDate": "2024-12-27T01:11:43.147Z",
      "content": "<p>I'd like to thank Kaggle, the organisers and all the winners of this very interesting challenge. I want also to <strong>SPECIALLY THANK <a href=\"https://www.kaggle.com/medali1992\" target=\"_blank\">@medali1992</a></strong>. Without him, I could'nt have made it to this 3rd place. Indeed, I used his public model shared <a href=\"https://www.kaggle.com/datasets/medali1992/mod-gte-base-weights\" target=\"_blank\">here</a>.<br>\nThis thread is intented to share the approach I used to reach efficiency 3rd place. </p>\n<h1>1- Main Idea</h1>\n<p>The main idea was to adjust the similarity matrix derived from model embeddings. The adjustment is based on couple (<code>ConstructName</code>,<code>MisconceptionName</code>). The approach I used, was quite similar to the one used in the beginning of the competition. We create a prompt by concatenating <code>SubjectName</code>,<code>ConstructName</code>,<code>QuestionText</code> and <code>AnswerText</code>. Once we get the embedding of the prompts and the 2587 misconceptions, We use cosine similarity to predict the top 25.</p>\n<p>But I noticed that, they were <strong>blattant inconsistencies</strong> in the top 25 that can be sorted by <strong>precomputed pointers</strong>.  For instance :</p>\n<ul>\n<li>1) for question related to pie charts we can get misconceptions related to simultaneous equations in the top 25.</li>\n<li>2) for question related to gradient we can get misconceptions related triangles in the top 25.</li>\n</ul>\n<h3>Construction &amp; Post Processing rule</h3>\n<p>That's the reason why, I decided to build pointers for the couples (<code>ConstructName</code>,<code>MisconceptionName</code>). I split the misconceptions in two groups. The first group \\(M_{-}\\) contains all the 1623 misconceptions present in the training dataset, the second group \\(M_{+}\\) the 964 remaining misconceptions. Now for a <code>ConstructName</code>  \\(c\\) present in the training dataset, we associate two groups of misconceptions. The first group \\(S_c\\) contains all the misconceptions that can safely be associated with \\(c\\), and the second group \\(P_c\\) all the misconceptions that can potentially be associated with \\(c\\).  For a new misconception \\(c\\) not present in the training dataset, we decided to take \\(S_c=\\emptyset\\)  and \\(P_c\\ = M_{+}\\). It means that for a new misconception, we have no guarantee of a safe association, and we take all the unused misconceptions as potential assocation (to be more inclusive).</p>\n<p>The adjustment rule used two parameters \\(\\alpha\\) and \\(\\beta\\).  For a sample with  prompt embedding \\(x\\), <code>ConstructName</code> \\(c\\) and a misconception \\(m\\) with embedding \\(x_m\\); We adjust the similarity as follow: </p>\n<ul>\n<li>\\(  S(c,m) = x^T.x_m + \\alpha  \\) if \\(m \\in S_c \\)</li>\n<li>\\(  S(c,m) = x^T.x_m + \\beta  \\) if \\(m \\in P_c \\)</li>\n<li>\\(  S(c,m) = x^T.x_m   \\)  else.</li>\n</ul>\n<h3>Limitations</h3>\n<p>They are <strong>two drawbacks</strong> for this post-processing :</p>\n<ul>\n<li>First  for the new <code>ConstructName</code>, we only adjust the <em>unseen</em> <code>MisconceptionName</code>. By doing this, we are assuming that the most probable misconceptions are the unseen ones. So in the case where the true misconception is in the training dataset, it is less likely for us to get it in the top 25.</li>\n<li>The choice of \\(\\alpha\\) and \\(\\beta\\) : For instance if  \\(\\alpha - \\beta\\) is too big, we can constrain the top 25 to almost the same misconceptions. At least, these parameters shoud be chosen by cross-validation. Yet I didn't have time to do it, since I was very active in this challenge only the first month. </li>\n</ul>\n<h1>2- Notebooks</h1>\n<p>There is nothing special here. Since this approach was intended for the efficiency track, we first pre-computed the misconceptions embeddings in a prior notebook (to save time) and then use them in the very inference notebook as well as adjustment pointers.</p>\n<ul>\n<li>The inference notebook can be found <a href=\"https://www.kaggle.com/code/ulrich07/eedi-cpu-starter/notebook\" target=\"_blank\">here</a>.</li>\n<li>The precomputed embeddings can be found <a href=\"https://www.kaggle.com/code/ulrich07/eedi-starter-sent-embs-cpu\" target=\"_blank\">here</a>.</li>\n<li>The similarity adjustment pointers can be found <a href=\"https://www.kaggle.com/datasets/ulrich07/eedi-handy\" target=\"_blank\">here</a> in the picke file <code>dus.pkl</code>.</li>\n</ul>\n<h1>3- Potential Improvements</h1>\n<p>Of course, this work can be improved in diverse ways. But the one that came in my mind, was to use my pre-computed pointers to learn new embeddings both for <code>ConstructName</code> and <code>MisconceptionName</code>. In this way, we could effectively better handle the new  <code>ConstructName</code> not adressed in the pointer file.</p>",
      "rawMarkdown": "I'd like to thank Kaggle, the organisers and all the winners of this very interesting challenge. I want also to **SPECIALLY THANK @medali1992**. Without him, I could'nt have made it to this 3rd place. Indeed, I used his public model shared [here](https://www.kaggle.com/datasets/medali1992/mod-gte-base-weights).\nThis thread is intented to share the approach I used to reach efficiency 3rd place. \n\n#  1- Main Idea\nThe main idea was to adjust the similarity matrix derived from model embeddings. The adjustment is based on couple (`ConstructName`,`MisconceptionName`). The approach I used, was quite similar to the one used in the beginning of the competition. We create a prompt by concatenating `SubjectName`,`ConstructName`,`QuestionText` and `AnswerText`. Once we get the embedding of the prompts and the 2587 misconceptions, We use cosine similarity to predict the top 25.\n\nBut I noticed that, they were **blattant inconsistencies** in the top 25 that can be sorted by **precomputed pointers**.  For instance :\n* 1) for question related to pie charts we can get misconceptions related to simultaneous equations in the top 25.\n* 2) for question related to gradient we can get misconceptions related triangles in the top 25.\n\n### Construction & Post Processing rule\nThat's the reason why, I decided to build pointers for the couples (`ConstructName`,`MisconceptionName`). I split the misconceptions in two groups. The first group \\\\(M_{-}\\\\) contains all the 1623 misconceptions present in the training dataset, the second group \\\\(M_{+}\\\\) the 964 remaining misconceptions. Now for a `ConstructName`  \\\\(c\\\\) present in the training dataset, we associate two groups of misconceptions. The first group \\\\(S_c\\\\) contains all the misconceptions that can safely be associated with \\\\(c\\\\), and the second group \\\\(P_c\\\\) all the misconceptions that can potentially be associated with \\\\(c\\\\).  For a new misconception \\\\(c\\\\) not present in the training dataset, we decided to take \\\\(S_c\\=\\emptyset\\\\)  and \\\\(P_c\\ = M_{+}\\\\). It means that for a new misconception, we have no guarantee of a safe association, and we take all the unused misconceptions as potential assocation (to be more inclusive).\n\nThe adjustment rule used two parameters \\\\(\\alpha\\\\) and \\\\(\\beta\\\\).  For a sample with  prompt embedding \\\\(x\\\\), `ConstructName` \\\\(c\\\\) and a misconception \\\\(m\\\\) with embedding \\\\(x_m\\\\); We adjust the similarity as follow: \n* \\\\(  S(c,m) = x^T.x_m + \\alpha  \\\\) if \\\\(m \\in S_c \\\\)\n* \\\\(  S(c,m) = x^T.x_m + \\beta  \\\\) if \\\\(m \\in P_c \\\\)\n* \\\\(  S(c,m) = x^T.x_m   \\\\)  else.\n\n### Limitations\nThey are **two drawbacks** for this post-processing :\n* First  for the new `ConstructName`, we only adjust the *unseen* `MisconceptionName`. By doing this, we are assuming that the most probable misconceptions are the unseen ones. So in the case where the true misconception is in the training dataset, it is less likely for us to get it in the top 25.\n* The choice of \\\\(\\alpha\\\\) and \\\\(\\beta\\\\) : For instance if  \\\\(\\alpha - \\beta\\\\) is too big, we can constrain the top 25 to almost the same misconceptions. At least, these parameters shoud be chosen by cross-validation. Yet I didn't have time to do it, since I was very active in this challenge only the first month. \n\n\n\n\n# 2- Notebooks\nThere is nothing special here. Since this approach was intended for the efficiency track, we first pre-computed the misconceptions embeddings in a prior notebook (to save time) and then use them in the very inference notebook as well as adjustment pointers.\n* The inference notebook can be found [here](https://www.kaggle.com/code/ulrich07/eedi-cpu-starter/notebook).\n* The precomputed embeddings can be found [here](https://www.kaggle.com/code/ulrich07/eedi-starter-sent-embs-cpu).\n* The similarity adjustment pointers can be found [here](https://www.kaggle.com/datasets/ulrich07/eedi-handy) in the picke file `dus.pkl`.\n\n# 3- Potential Improvements\nOf course, this work can be improved in diverse ways. But the one that came in my mind, was to use my pre-computed pointers to learn new embeddings both for `ConstructName` and `MisconceptionName`. In this way, we could effectively better handle the new  `ConstructName` not adressed in the pointer file.",
      "votes": 13
    },
    {
      "id": 3081644,
      "postDate": "2024-12-27T03:09:58.987Z",
      "content": "<p>Congrats, I'm happy for you.</p>",
      "rawMarkdown": "Congrats, I'm happy for you.",
      "votes": 2
    },
    {
      "id": 3083317,
      "postDate": "2024-12-29T09:52:31.523Z",
      "content": "<p>Congratulations 👏👏👏</p>",
      "rawMarkdown": "Congratulations 👏👏👏",
      "votes": 2
    }
  ],
  "complete": true,
  "source": "live",
  "fetched_at": "2026-10-20T03:44:20Z",
  "competition_id": 82695,
  "competition_slug": "eedi-mining-misconceptions-in-mathematics",
  "team_id": 12667823,
  "solution_url": "/competitions/eedi-mining-misconceptions-in-mathematics/writeups/ulrich-g-efficiency-3rd-place-public-model-simple-"
}