{
  "id": 343082,
  "title": "what's the trick?",
  "url": "/competitions/AI4Code/discussion/343082",
  "author_name": "",
  "post_date": "2022-08-10T00:48:39.483917300Z",
  "votes": 6,
  "comment_count": 9,
  "views": 0,
  "content": "<p>I'm loooking forward to the top kagglers share tricks after the competition end, for me, I think the most possible tricks maybe</p>\n<ol>\n<li>code sample strategy</li>\n<li>code rank feature</li>\n</ol>\n<p>but pity for me, at last I don't found it, I think the main reasons are do experiment take too long.</p>\n<p>So I'm very curiouse how top players how to solve this problem.</p>\n<p>at last, I think this competition is fun, maybe we could just write markdown and google generate code for us in the future😀</p>",
  "messages": [
    {
      "id": "1892219",
      "postDate": "08/10/2022 00:48:39",
      "content": "<p>I'm loooking forward to the top kagglers share tricks after the competition end, for me, I think the most possible tricks maybe</p>\n<ol>\n<li>code sample strategy</li>\n<li>code rank feature</li>\n</ol>\n<p>but pity for me, at last I don't found it, I think the main reasons are do experiment take too long.</p>\n<p>So I'm very curiouse how top players how to solve this problem.</p>\n<p>at last, I think this competition is fun, maybe we could just write markdown and google generate code for us in the future😀</p>",
      "rawMarkdown": "I'm loooking forward to the top kagglers share tricks after the competition end, for me, I think the most possible tricks maybe\n1. code sample strategy\n2. code rank feature\n\nbut pity for me, at last I don't found it, I think the main reasons are do experiment take too long.\n\nSo I'm very curiouse how top players how to solve this problem.\n\nat last, I think this competition is fun, maybe we could just write markdown and google generate code for us in the future😀",
      "votes": null
    },
    {
      "id": "1892232",
      "postDate": "08/10/2022 01:11:35",
      "content": "<p>Let's discuss after LB close:) Maybe tricks for top2 teams do not need running long.</p>",
      "rawMarkdown": "Let's discuss after LB close:) Maybe tricks for top2 teams do not need running long.",
      "votes": null
    },
    {
      "id": "1893251",
      "postDate": "08/10/2022 16:50:28",
      "content": "<p>I'm eager to learn from you as well 🙏</p>",
      "rawMarkdown": "I'm eager to learn from you as well 🙏",
      "votes": null
    },
    {
      "id": "1893318",
      "postDate": "08/10/2022 17:51:21",
      "content": "<p>Eager to learn from all who put effort on this great game.</p>",
      "rawMarkdown": "Eager to learn from all who put effort on this great game.",
      "votes": null
    },
    {
      "id": "1895071",
      "postDate": "08/12/2022 00:08:55",
      "content": "<p>Game over, just learned from 1 of the top teams, seems large model and large inputs is the key. <br>\nFor example using deberta-v3-large with large number of codes like 100+ as inputs.<br>\nI think top teams used listwise(multiple markdowns infer at once for 1 single instance) to reduce infer time.<br>\n However I used deberta-v3-small all the way with 40 codes as inputs😄 <br>\nThat might be the reason why I found improve so hard for 0.001 while other teams improve so easy..</p>",
      "rawMarkdown": "Game over, just learned from 1 of the top teams, seems large model and large inputs is the key. \nFor example using deberta-v3-large with large number of codes like 100+ as inputs.\nI think top teams used listwise(multiple markdowns infer at once for 1 single instance) to reduce infer time.\n However I used deberta-v3-small all the way with 40 codes as inputs😄 \nThat might be the reason why I found improve so hard for 0.001 while other teams improve so easy..",
      "votes": null
    },
    {
      "id": "1895192",
      "postDate": "08/12/2022 02:36:55",
      "content": "<p>Maybe ArcFace? I have no time to try yet.😳</p>",
      "rawMarkdown": "Maybe ArcFace? I have no time to try yet.😳",
      "votes": null
    },
    {
      "id": "1895274",
      "postDate": "08/12/2022 03:59:48",
      "content": "<p>CosFace seemed to work for pairwise model in the limited experiments I conducted. Somehow Arcface didn't. But my pairwise model was not pretty strong to start out with anyway, so I didnt explore further. I felt Arcface may not have relevance in point wise model (maybe I am wrong).</p>\n<p><a href=\"https://www.kaggle.com/goldenlock\" target=\"_blank\">@goldenlock</a> moving from 512 to 768 (no other change) almost gave me .01 improvement on DebertaV3small. So your observations may be correct. I had plan of using 1536/Debertabase but it would have taken 8 times longer to train and I didnt have time.</p>\n<p>congrats to you both on your strong finishes!!</p>",
      "rawMarkdown": "CosFace seemed to work for pairwise model in the limited experiments I conducted. Somehow Arcface didn't. But my pairwise model was not pretty strong to start out with anyway, so I didnt explore further. I felt Arcface may not have relevance in point wise model (maybe I am wrong).\n\n@goldenlock moving from 512 to 768 (no other change) almost gave me .01 improvement on DebertaV3small. So your observations may be correct. I had plan of using 1536/Debertabase but it would have taken 8 times longer to train and I didnt have time.\n\ncongrats to you both on your strong finishes!!",
      "votes": null
    },
    {
      "id": "1895284",
      "postDate": "08/12/2022 04:06:20",
      "content": "<p>Thanks! Personally I found crossentropy the best for pairwise model, and I'm so suprised senttence transformer with mpnet work much better then deberta-v3-small on pairwise target where on all other models deberta is much better.</p>",
      "rawMarkdown": "Thanks! Personally I found crossentropy the best for pairwise model, and I'm so suprised senttence transformer with mpnet work much better then deberta-v3-small on pairwise target where on all other models deberta is much better.",
      "votes": null
    },
    {
      "id": "1895469",
      "postDate": "08/12/2022 06:49:59",
      "content": "<p>So it is clear now the top trick is data, intesting to see the advantage of external data on new dataset.</p>\n<ol>\n<li>Externel dataset of kaggle public notebooks.  </li>\n<li>More context, more codes as your model inputs.</li>\n</ol>",
      "rawMarkdown": "So it is clear now the top trick is data, intesting to see the advantage of external data on new dataset.\n1. Externel dataset of kaggle public notebooks.  \n2. More context, more codes as your model inputs.",
      "votes": null
    },
    {
      "id": "1895977",
      "postDate": "08/12/2022 13:35:20",
      "content": "<p>It's very nice to see the published solutions are quite different, with still some similar ideas.</p>\n<p>It was an interesting challenge.</p>",
      "rawMarkdown": "It's very nice to see the published solutions are quite different, with still some similar ideas.\n\nIt was an interesting challenge.",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 1892232,
      "author_name": "goldenlock",
      "author_url": "",
      "post_date": "08/10/2022 01:11:35",
      "content": "<p>Let's discuss after LB close:) Maybe tricks for top2 teams do not need running long.</p>",
      "votes": null,
      "replies": [
        {
          "id": 1893251,
          "author_name": "init27",
          "author_url": "",
          "post_date": "08/10/2022 16:50:28",
          "content": "<p>I'm eager to learn from you as well 🙏</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 1893318,
          "author_name": "goldenlock",
          "author_url": "",
          "post_date": "08/10/2022 17:51:21",
          "content": "<p>Eager to learn from all who put effort on this great game.</p>",
          "votes": null,
          "replies": []
        }
      ]
    },
    {
      "id": 1895071,
      "author_name": "goldenlock",
      "author_url": "",
      "post_date": "08/12/2022 00:08:55",
      "content": "<p>Game over, just learned from 1 of the top teams, seems large model and large inputs is the key. <br>\nFor example using deberta-v3-large with large number of codes like 100+ as inputs.<br>\nI think top teams used listwise(multiple markdowns infer at once for 1 single instance) to reduce infer time.<br>\n However I used deberta-v3-small all the way with 40 codes as inputs😄 <br>\nThat might be the reason why I found improve so hard for 0.001 while other teams improve so easy..</p>",
      "votes": null,
      "replies": [
        {
          "id": 1895274,
          "author_name": "allohvk",
          "author_url": "",
          "post_date": "08/12/2022 03:59:48",
          "content": "<p>CosFace seemed to work for pairwise model in the limited experiments I conducted. Somehow Arcface didn't. But my pairwise model was not pretty strong to start out with anyway, so I didnt explore further. I felt Arcface may not have relevance in point wise model (maybe I am wrong).</p>\n<p><a href=\"https://www.kaggle.com/goldenlock\" target=\"_blank\">@goldenlock</a> moving from 512 to 768 (no other change) almost gave me .01 improvement on DebertaV3small. So your observations may be correct. I had plan of using 1536/Debertabase but it would have taken 8 times longer to train and I didnt have time.</p>\n<p>congrats to you both on your strong finishes!!</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 1895284,
          "author_name": "goldenlock",
          "author_url": "",
          "post_date": "08/12/2022 04:06:20",
          "content": "<p>Thanks! Personally I found crossentropy the best for pairwise model, and I'm so suprised senttence transformer with mpnet work much better then deberta-v3-small on pairwise target where on all other models deberta is much better.</p>",
          "votes": null,
          "replies": []
        }
      ]
    },
    {
      "id": 1895192,
      "author_name": "hydantess",
      "author_url": "",
      "post_date": "08/12/2022 02:36:55",
      "content": "<p>Maybe ArcFace? I have no time to try yet.😳</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 1895469,
      "author_name": "goldenlock",
      "author_url": "",
      "post_date": "08/12/2022 06:49:59",
      "content": "<p>So it is clear now the top trick is data, intesting to see the advantage of external data on new dataset.</p>\n<ol>\n<li>Externel dataset of kaggle public notebooks.  </li>\n<li>More context, more codes as your model inputs.</li>\n</ol>",
      "votes": null,
      "replies": []
    },
    {
      "id": 1895977,
      "author_name": "dmytropoplavskiy",
      "author_url": "",
      "post_date": "08/12/2022 13:35:20",
      "content": "<p>It's very nice to see the published solutions are quite different, with still some similar ideas.</p>\n<p>It was an interesting challenge.</p>",
      "votes": null,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "1892219": "I'm loooking forward to the top kagglers share tricks after the competition end, for me, I think the most possible tricks maybe\n1. code sample strategy\n2. code rank feature\n\nbut pity for me, at last I don't found it, I think the main reasons are do experiment take too long.\n\nSo I'm very curiouse how top players how to solve this problem.\n\nat last, I think this competition is fun, maybe we could just write markdown and google generate code for us in the future😀",
    "1892232": "Let's discuss after LB close:) Maybe tricks for top2 teams do not need running long.",
    "1893251": "I'm eager to learn from you as well 🙏",
    "1893318": "Eager to learn from all who put effort on this great game.",
    "1895071": "Game over, just learned from 1 of the top teams, seems large model and large inputs is the key. \nFor example using deberta-v3-large with large number of codes like 100+ as inputs.\nI think top teams used listwise(multiple markdowns infer at once for 1 single instance) to reduce infer time.\n However I used deberta-v3-small all the way with 40 codes as inputs😄 \nThat might be the reason why I found improve so hard for 0.001 while other teams improve so easy..",
    "1895192": "Maybe ArcFace? I have no time to try yet.😳",
    "1895274": "CosFace seemed to work for pairwise model in the limited experiments I conducted. Somehow Arcface didn't. But my pairwise model was not pretty strong to start out with anyway, so I didnt explore further. I felt Arcface may not have relevance in point wise model (maybe I am wrong).\n\n@goldenlock moving from 512 to 768 (no other change) almost gave me .01 improvement on DebertaV3small. So your observations may be correct. I had plan of using 1536/Debertabase but it would have taken 8 times longer to train and I didnt have time.\n\ncongrats to you both on your strong finishes!!",
    "1895284": "Thanks! Personally I found crossentropy the best for pairwise model, and I'm so suprised senttence transformer with mpnet work much better then deberta-v3-small on pairwise target where on all other models deberta is much better.",
    "1895469": "So it is clear now the top trick is data, intesting to see the advantage of external data on new dataset.\n1. Externel dataset of kaggle public notebooks.  \n2. More context, more codes as your model inputs.",
    "1895977": "It's very nice to see the published solutions are quite different, with still some similar ideas.\n\nIt was an interesting challenge."
  },
  "source": "meta"
}