{
  "id": 411999,
  "title": "Matcha: Reducing training times",
  "url": "/competitions/benetech-making-graphs-accessible/discussion/411999",
  "author_name": "Cody_Null",
  "post_date": "2023-05-21T22:30:37.783000",
  "votes": 3,
  "comment_count": 6,
  "views": 0,
  "content": "<p>I was wondering if anyone had any insights as to how to effectively reduce training time with matcha? Currently, training times are quite long and I am thinking there has to be a better way?</p>",
  "messages": [
    {
      "id": 2271236,
      "postDate": "2023-05-23T17:49:57.120Z",
      "content": "<p>There are a few things you could try:</p>\n<ol>\n<li>Mixed precision training (if you are not already using it)</li>\n<li>Freeze encoder first and then finetune everything only a few epochs</li>\n<li>First pre-train on lower resolution (max-patches) and finetune on full res afterwards</li>\n<li>Since Pix2Struct-based models use ViT as encoder you could try <a href=\"https://github.com/facebookresearch/ToMe/tree/main\" target=\"_blank\">ToMe</a></li>\n<li>Never tried myself, but also heard good things about <a href=\"https://github.com/facebookresearch/ToMe/tree/main\" target=\"_blank\">LorA</a></li>\n</ol>",
      "rawMarkdown": "There are a few things you could try:\n1. Mixed precision training (if you are not already using it)\n2. Freeze encoder first and then finetune everything only a few epochs\n3. First pre-train on lower resolution (max-patches) and finetune on full res afterwards\n4. Since Pix2Struct-based models use ViT as encoder you could try [ToMe](https://github.com/facebookresearch/ToMe/tree/main)\n5. Never tried myself, but also heard good things about [LorA](https://github.com/facebookresearch/ToMe/tree/main)",
      "votes": 14,
      "replies": [
        {
          "id": 2271255,
          "postDate": "2023-05-23T18:03:17.633Z",
          "content": "<p>I can absolutely look into this. Thank you for the ideas!</p>",
          "rawMarkdown": "I can absolutely look into this. Thank you for the ideas!",
          "votes": 1
        }
      ]
    },
    {
      "id": 2268669,
      "postDate": "2023-05-21T22:30:37.783Z",
      "content": "<p>I was wondering if anyone had any insights as to how to effectively reduce training time with matcha? Currently, training times are quite long and I am thinking there has to be a better way?</p>",
      "rawMarkdown": "I was wondering if anyone had any insights as to how to effectively reduce training time with matcha? Currently, training times are quite long and I am thinking there has to be a better way?",
      "votes": 3
    },
    {
      "id": 2273571,
      "postDate": "2023-05-25T08:32:59.263Z",
      "content": "<p>Interesting notebook that sums up optimization ideas for Transformers here:<br>\n<a href=\"https://www.kaggle.com/code/rhtsingh/speeding-up-transformer-w-optimization-strategies/notebook\" target=\"_blank\">https://www.kaggle.com/code/rhtsingh/speeding-up-transformer-w-optimization-strategies/notebook</a></p>",
      "rawMarkdown": "Interesting notebook that sums up optimization ideas for Transformers here:\nhttps://www.kaggle.com/code/rhtsingh/speeding-up-transformer-w-optimization-strategies/notebook",
      "votes": 1,
      "replies": [
        {
          "id": 2273889,
          "postDate": "2023-05-25T13:38:38.433Z",
          "content": "<p>Thank you! good to know!</p>",
          "rawMarkdown": "Thank you! good to know!",
          "votes": 1
        }
      ]
    },
    {
      "id": 2270751,
      "postDate": "2023-05-23T11:32:57.190Z",
      "content": "<p>hi Cody, I am struggling for the training  of Matcha model and processor. Could you provide some link?</p>",
      "rawMarkdown": "hi Cody, I am struggling for the training  of Matcha model and processor. Could you provide some link?",
      "votes": 1,
      "replies": [
        {
          "id": 2270898,
          "postDate": "2023-05-23T13:36:25.490Z",
          "rawMarkdown": "",
          "isDeleted": true
        }
      ]
    }
  ],
  "comments": [
    {
      "id": 2271236,
      "author_name": "Eduardo Rocha de Andrade",
      "author_url": "",
      "post_date": "2023-05-23T17:49:57.120000",
      "content": "<p>There are a few things you could try:</p>\n<ol>\n<li>Mixed precision training (if you are not already using it)</li>\n<li>Freeze encoder first and then finetune everything only a few epochs</li>\n<li>First pre-train on lower resolution (max-patches) and finetune on full res afterwards</li>\n<li>Since Pix2Struct-based models use ViT as encoder you could try <a href=\"https://github.com/facebookresearch/ToMe/tree/main\" target=\"_blank\">ToMe</a></li>\n<li>Never tried myself, but also heard good things about <a href=\"https://github.com/facebookresearch/ToMe/tree/main\" target=\"_blank\">LorA</a></li>\n</ol>",
      "votes": 14,
      "replies": [
        {
          "id": 2271255,
          "author_name": "Cody_Null",
          "author_url": "",
          "post_date": "2023-05-23T18:03:17.633000",
          "content": "<p>I can absolutely look into this. Thank you for the ideas!</p>",
          "votes": 1,
          "replies": []
        }
      ]
    },
    {
      "id": 2273571,
      "author_name": "Rafi Hai",
      "author_url": "",
      "post_date": "2023-05-25T08:32:59.263000",
      "content": "<p>Interesting notebook that sums up optimization ideas for Transformers here:<br>\n<a href=\"https://www.kaggle.com/code/rhtsingh/speeding-up-transformer-w-optimization-strategies/notebook\" target=\"_blank\">https://www.kaggle.com/code/rhtsingh/speeding-up-transformer-w-optimization-strategies/notebook</a></p>",
      "votes": 1,
      "replies": [
        {
          "id": 2273889,
          "author_name": "Cody_Null",
          "author_url": "",
          "post_date": "2023-05-25T13:38:38.433000",
          "content": "<p>Thank you! good to know!</p>",
          "votes": 1,
          "replies": []
        }
      ]
    },
    {
      "id": 2270751,
      "author_name": "pksamantaray",
      "author_url": "",
      "post_date": "2023-05-23T11:32:57.190000",
      "content": "<p>hi Cody, I am struggling for the training  of Matcha model and processor. Could you provide some link?</p>",
      "votes": 1,
      "replies": [
        {
          "id": 2270898,
          "author_name": "",
          "author_url": "",
          "post_date": "2023-05-23T13:36:25.490000",
          "content": "",
          "votes": 0,
          "replies": []
        }
      ]
    }
  ],
  "raw_markdown_by_id": {
    "2271236": "There are a few things you could try:\n1. Mixed precision training (if you are not already using it)\n2. Freeze encoder first and then finetune everything only a few epochs\n3. First pre-train on lower resolution (max-patches) and finetune on full res afterwards\n4. Since Pix2Struct-based models use ViT as encoder you could try [ToMe](https://github.com/facebookresearch/ToMe/tree/main)\n5. Never tried myself, but also heard good things about [LorA](https://github.com/facebookresearch/ToMe/tree/main)",
    "2268669": "I was wondering if anyone had any insights as to how to effectively reduce training time with matcha? Currently, training times are quite long and I am thinking there has to be a better way?",
    "2273571": "Interesting notebook that sums up optimization ideas for Transformers here:\nhttps://www.kaggle.com/code/rhtsingh/speeding-up-transformer-w-optimization-strategies/notebook",
    "2270751": "hi Cody, I am struggling for the training  of Matcha model and processor. Could you provide some link?"
  }
}