{
  "id": 417640,
  "title": "For Matcha/Deplot/Pix2Struct: line chart tokenization strategy",
  "url": "/competitions/benetech-making-graphs-accessible/discussion/417640",
  "author_name": "kh",
  "post_date": "2023-06-16T14:10:04.528000",
  "votes": 0,
  "comment_count": 0,
  "views": 0,
  "content": "<p>I bumped into this competition too late to enter, but have tried to develop something similar in the past. Apart from this competition I'm interested in scalable solutions for Matcha/Deplot/Pix2Struct to handle line charts whose table would need a max token sequence length that is a multiple of 512. </p>\n<p>For those using Matcha/Deplot/Pix2struct, it seems Deplot was pre-trained on tokenization max length of 512. Line graphs with a lot of points easily need more, right? I see a few strategies:</p>\n<ul>\n<li><p>Training on pixel integers for numerical axes with some reference points could save some space, but doesn't scale terribly well.</p></li>\n<li><p>Fine-tune on cases that fit within the max_length of 512. On inference generate with max_new_tokens=n&gt;&gt;512 and hope for the best.</p></li>\n<li><p>Fine-tune on a training set with larger cases and a max_length of n &gt;&gt; 512. In any case this would need more mem.</p></li>\n<li><p>Chunk up the training cases. E.g. each image now is n times in the training set, one for each 10 lines of data. They each have a slightly different prompt. After inference you tie everything together. I worry that this would rely too much on a heuristic to know when you have all of the data.</p></li>\n</ul>\n<p>Unclear if these strategies work. I have to say I don't have a lot of experience with transformers / tokenizers. Has anyone run these experiments or could tell me what a good approach would be?</p>\n<p>I understand there are other ways to handle line charts with a lot of points that work well, but I'm hoping to just use 1, more cleaner, architecture.</p>",
  "messages": [
    {
      "id": 2305209,
      "postDate": "2023-06-16T14:10:04.530Z",
      "content": "<p>I bumped into this competition too late to enter, but have tried to develop something similar in the past. Apart from this competition I'm interested in scalable solutions for Matcha/Deplot/Pix2Struct to handle line charts whose table would need a max token sequence length that is a multiple of 512. </p>\n<p>For those using Matcha/Deplot/Pix2struct, it seems Deplot was pre-trained on tokenization max length of 512. Line graphs with a lot of points easily need more, right? I see a few strategies:</p>\n<ul>\n<li><p>Training on pixel integers for numerical axes with some reference points could save some space, but doesn't scale terribly well.</p></li>\n<li><p>Fine-tune on cases that fit within the max_length of 512. On inference generate with max_new_tokens=n&gt;&gt;512 and hope for the best.</p></li>\n<li><p>Fine-tune on a training set with larger cases and a max_length of n &gt;&gt; 512. In any case this would need more mem.</p></li>\n<li><p>Chunk up the training cases. E.g. each image now is n times in the training set, one for each 10 lines of data. They each have a slightly different prompt. After inference you tie everything together. I worry that this would rely too much on a heuristic to know when you have all of the data.</p></li>\n</ul>\n<p>Unclear if these strategies work. I have to say I don't have a lot of experience with transformers / tokenizers. Has anyone run these experiments or could tell me what a good approach would be?</p>\n<p>I understand there are other ways to handle line charts with a lot of points that work well, but I'm hoping to just use 1, more cleaner, architecture.</p>",
      "rawMarkdown": "I bumped into this competition too late to enter, but have tried to develop something similar in the past. Apart from this competition I'm interested in scalable solutions for Matcha/Deplot/Pix2Struct to handle line charts whose table would need a max token sequence length that is a multiple of 512. \n\nFor those using Matcha/Deplot/Pix2struct, it seems Deplot was pre-trained on tokenization max length of 512. Line graphs with a lot of points easily need more, right? I see a few strategies:\n\n- Training on pixel integers for numerical axes with some reference points could save some space, but doesn't scale terribly well.\n\n- Fine-tune on cases that fit within the max_length of 512. On inference generate with max_new_tokens=n>>512 and hope for the best.\n\n- Fine-tune on a training set with larger cases and a max_length of n >> 512. In any case this would need more mem.\n\n- Chunk up the training cases. E.g. each image now is n times in the training set, one for each 10 lines of data. They each have a slightly different prompt. After inference you tie everything together. I worry that this would rely too much on a heuristic to know when you have all of the data.\n\nUnclear if these strategies work. I have to say I don't have a lot of experience with transformers / tokenizers. Has anyone run these experiments or could tell me what a good approach would be?\n\nI understand there are other ways to handle line charts with a lot of points that work well, but I'm hoping to just use 1, more cleaner, architecture."
    }
  ],
  "comments": [],
  "raw_markdown_by_id": {
    "2305209": "I bumped into this competition too late to enter, but have tried to develop something similar in the past. Apart from this competition I'm interested in scalable solutions for Matcha/Deplot/Pix2Struct to handle line charts whose table would need a max token sequence length that is a multiple of 512. \n\nFor those using Matcha/Deplot/Pix2struct, it seems Deplot was pre-trained on tokenization max length of 512. Line graphs with a lot of points easily need more, right? I see a few strategies:\n\n- Training on pixel integers for numerical axes with some reference points could save some space, but doesn't scale terribly well.\n\n- Fine-tune on cases that fit within the max_length of 512. On inference generate with max_new_tokens=n>>512 and hope for the best.\n\n- Fine-tune on a training set with larger cases and a max_length of n >> 512. In any case this would need more mem.\n\n- Chunk up the training cases. E.g. each image now is n times in the training set, one for each 10 lines of data. They each have a slightly different prompt. After inference you tie everything together. I worry that this would rely too much on a heuristic to know when you have all of the data.\n\nUnclear if these strategies work. I have to say I don't have a lot of experience with transformers / tokenizers. Has anyone run these experiments or could tell me what a good approach would be?\n\nI understand there are other ways to handle line charts with a lot of points that work well, but I'm hoping to just use 1, more cleaner, architecture."
  }
}