{
  "id": 418431,
  "title": "Chart-specific scores and finetuning process | solution summary",
  "url": "/competitions/benetech-making-graphs-accessible/discussion/418431",
  "author_name": "",
  "post_date": "2023-06-20T14:43:40.268253200Z",
  "votes": 2,
  "comment_count": 7,
  "views": 0,
  "content": "<p>I think, it would be really interesting to compare chart-specific scores. It might be interesting for the competition host as well. I just used generative models for my solution and missed to tune scatter. However, I think the other scores are quite good. Could you guys share the scores of your final submitted solution?</p>\n<table>\n<thead>\n<tr>\n<th></th>\n<th>Overall</th>\n<th>Scatter</th>\n<th>Dot</th>\n<th>Line</th>\n<th>Vertical Bar</th>\n<th>Horizontal Bar</th>\n</tr>\n</thead>\n<tbody>\n<tr>\n<td>public</td>\n<td>0.79</td>\n<td>0.03</td>\n<td>0</td>\n<td>0.3</td>\n<td>0.38</td>\n<td>0.04</td>\n</tr>\n<tr>\n<td>private</td>\n<td>0.46 🤕</td>\n<td>0.07</td>\n<td>0</td>\n<td>0.11</td>\n<td>0.26</td>\n<td>0.01</td>\n</tr>\n</tbody>\n</table>\n<hr>\n<p><strong>Key steps for my training pipeline</strong></p>\n<ul>\n<li>Autoregressive ensembling within greedy search (2x matcha-base, 2x deplot) (+0.03 in the public lb in comparison to the best single model)</li>\n<li>Pretraining on synthetic data and just finetuning on the competition data: pretrained first epoch with max_patches 1024 and increased in the second epoch to 1536 (also finetuned with 1536) (+0.02 for a single model). I think it works like a warmup phase to generate more general features.</li>\n<li>Using the ICDAR dataset (all charts with one data-series, I also manually adjusted the annotations of line charts to fit the competition conventions) at the finetuning stage (+0.02 for a single model)</li>\n<li>Training different layer of the encoder and decoder with different learning rates (+0.02 for a single model)</li>\n</ul>\n<p><strong>Further ideas</strong></p>\n<ul>\n<li>I thought about \"dynamic ensembling\". It was very important to predict the right amount of data points. I thought it might be interesting to use a much larger ensemble in the case greedy search produces an eos-token. If the eos-token is predicted, go a step back, feed the current sequence to xx models and check if the majority of the model is voting for an eos-token, otherwise go on. That would be an option to utilize more than 4 models. Maybe it might makes sense to do this for the first x tokens as well to point the sequence in the right direction</li>\n<li>I thought about implementing a seperate model for scatterplots like <a href=\"url\" target=\"_blank\">https://github.com/bloomberg/scatteract</a> but didn´t have time left</li>\n</ul>\n<p>It would be really great if you guys can share your results.</p>",
  "messages": [
    {
      "id": "2310682",
      "postDate": "06/20/2023 14:43:40",
      "content": "<p>I think, it would be really interesting to compare chart-specific scores. It might be interesting for the competition host as well. I just used generative models for my solution and missed to tune scatter. However, I think the other scores are quite good. Could you guys share the scores of your final submitted solution?</p>\n<table>\n<thead>\n<tr>\n<th></th>\n<th>Overall</th>\n<th>Scatter</th>\n<th>Dot</th>\n<th>Line</th>\n<th>Vertical Bar</th>\n<th>Horizontal Bar</th>\n</tr>\n</thead>\n<tbody>\n<tr>\n<td>public</td>\n<td>0.79</td>\n<td>0.03</td>\n<td>0</td>\n<td>0.3</td>\n<td>0.38</td>\n<td>0.04</td>\n</tr>\n<tr>\n<td>private</td>\n<td>0.46 🤕</td>\n<td>0.07</td>\n<td>0</td>\n<td>0.11</td>\n<td>0.26</td>\n<td>0.01</td>\n</tr>\n</tbody>\n</table>\n<hr>\n<p><strong>Key steps for my training pipeline</strong></p>\n<ul>\n<li>Autoregressive ensembling within greedy search (2x matcha-base, 2x deplot) (+0.03 in the public lb in comparison to the best single model)</li>\n<li>Pretraining on synthetic data and just finetuning on the competition data: pretrained first epoch with max_patches 1024 and increased in the second epoch to 1536 (also finetuned with 1536) (+0.02 for a single model). I think it works like a warmup phase to generate more general features.</li>\n<li>Using the ICDAR dataset (all charts with one data-series, I also manually adjusted the annotations of line charts to fit the competition conventions) at the finetuning stage (+0.02 for a single model)</li>\n<li>Training different layer of the encoder and decoder with different learning rates (+0.02 for a single model)</li>\n</ul>\n<p><strong>Further ideas</strong></p>\n<ul>\n<li>I thought about \"dynamic ensembling\". It was very important to predict the right amount of data points. I thought it might be interesting to use a much larger ensemble in the case greedy search produces an eos-token. If the eos-token is predicted, go a step back, feed the current sequence to xx models and check if the majority of the model is voting for an eos-token, otherwise go on. That would be an option to utilize more than 4 models. Maybe it might makes sense to do this for the first x tokens as well to point the sequence in the right direction</li>\n<li>I thought about implementing a seperate model for scatterplots like <a href=\"url\" target=\"_blank\">https://github.com/bloomberg/scatteract</a> but didn´t have time left</li>\n</ul>\n<p>It would be really great if you guys can share your results.</p>",
      "rawMarkdown": "I think, it would be really interesting to compare chart-specific scores. It might be interesting for the competition host as well. I just used generative models for my solution and missed to tune scatter. However, I think the other scores are quite good. Could you guys share the scores of your final submitted solution?\n\n|  | Overall | Scatter | Dot | Line | Vertical Bar | Horizontal Bar\n| --- | --- | --- | --- | --- |\n| public  | 0.79 | 0.03| 0 | 0.3 | 0.38 | 0.04\n| private | 0.46 🤕 | 0.07 | 0 | 0.11 | 0.26 | 0.01\n____\n\n**Key steps for my training pipeline**\n- Autoregressive ensembling within greedy search (2x matcha-base, 2x deplot) (+0.03 in the public lb in comparison to the best single model)\n- Pretraining on synthetic data and just finetuning on the competition data: pretrained first epoch with max_patches 1024 and increased in the second epoch to 1536 (also finetuned with 1536) (+0.02 for a single model). I think it works like a warmup phase to generate more general features.\n- Using the ICDAR dataset (all charts with one data-series, I also manually adjusted the annotations of line charts to fit the competition conventions) at the finetuning stage (+0.02 for a single model)\n- Training different layer of the encoder and decoder with different learning rates (+0.02 for a single model)\n\n**Further ideas**\n- I thought about \"dynamic ensembling\". It was very important to predict the right amount of data points. I thought it might be interesting to use a much larger ensemble in the case greedy search produces an eos-token. If the eos-token is predicted, go a step back, feed the current sequence to xx models and check if the majority of the model is voting for an eos-token, otherwise go on. That would be an option to utilize more than 4 models. Maybe it might makes sense to do this for the first x tokens as well to point the sequence in the right direction\n- I thought about implementing a seperate model for scatterplots like [https://github.com/bloomberg/scatteract](url) but didn´t have time left\n\nIt would be really great if you guys can share your results.",
      "votes": null
    },
    {
      "id": "2310686",
      "postDate": "06/20/2023 14:46:31",
      "content": "<p><a href=\"https://www.kaggle.com/crodoc\" target=\"_blank\">@crodoc</a> <a href=\"https://www.kaggle.com/theoviel\" target=\"_blank\">@theoviel</a> </p>\n<p>I added your result here (place 3 solution):</p>\n<table>\n<thead>\n<tr>\n<th></th>\n<th>Overall</th>\n<th>Scatter</th>\n<th>Dot</th>\n<th>Line</th>\n<th>Vertical Bar</th>\n<th>Horizontal Bar</th>\n</tr>\n</thead>\n<tbody>\n<tr>\n<td>public</td>\n<td>0.87</td>\n<td>0.09</td>\n<td>0</td>\n<td>0.33</td>\n<td>0.39</td>\n<td>0.04</td>\n</tr>\n<tr>\n<td>private</td>\n<td>0.71</td>\n<td>0.28</td>\n<td>0.01</td>\n<td>0.13</td>\n<td>0.27</td>\n<td>0.01</td>\n</tr>\n</tbody>\n</table>",
      "rawMarkdown": "crodoc @theoviel \n\nI added your result here (place 3 solution):\n|  | Overall | Scatter | Dot | Line | Vertical Bar | Horizontal Bar\n| --- | --- | --- | --- | --- |\n| public  | 0.87 | 0.09| 0 | 0.33 | 0.39 | 0.04\n| private | 0.71 | 0.28 | 0.01 | 0.13 | 0.27 | 0.01",
      "votes": null
    },
    {
      "id": "2310698",
      "postDate": "06/20/2023 14:55:45",
      "content": "<p>Thanks ! I think you swapped public and private horizontal bar in your write-up :)</p>\n<p>Would love to get input from <a href=\"https://www.kaggle.com/kashiwaba\" target=\"_blank\">@kashiwaba</a> and <a href=\"https://www.kaggle.com/conjuring92\" target=\"_blank\">@conjuring92</a> on this</p>",
      "rawMarkdown": "Thanks ! I think you swapped public and private horizontal bar in your write-up :)\n\nWould love to get input from @kashiwaba and @conjuring92 on this",
      "votes": null
    },
    {
      "id": "2310754",
      "postDate": "06/20/2023 15:28:39",
      "content": "<p>This is certainly an interesting consideration.<br>\nI have not yet been able to check the scores per chart for the final sub.<br>\nI hope to check them in the late sub in the few days and will share the results when they are available!</p>\n<p>By the way, looking at the scores I submitted, the solutions with improved scatter prediction methods had higher Private scores across the board, so I'm pretty sure that scatter is the key to Private scores.</p>",
      "rawMarkdown": "This is certainly an interesting consideration.\nI have not yet been able to check the scores per chart for the final sub.\nI hope to check them in the late sub in the few days and will share the results when they are available!\n\nBy the way, looking at the scores I submitted, the solutions with improved scatter prediction methods had higher Private scores across the board, so I'm pretty sure that scatter is the key to Private scores.",
      "votes": null
    },
    {
      "id": "2310759",
      "postDate": "06/20/2023 15:32:49",
      "content": "<p>I'm pretty sure your scatter is stronger that ours and that's why you went up 7 spots :)</p>",
      "rawMarkdown": "I'm pretty sure your scatter is stronger that ours and that's why you went up 7 spots :)",
      "votes": null
    },
    {
      "id": "2311644",
      "postDate": "06/21/2023 10:08:29",
      "content": "<p>I did a late sub on my final sub.<br>\nThe results are as follows.</p>\n<table>\n<thead>\n<tr>\n<th></th>\n<th>Overall</th>\n<th>Scatter</th>\n<th>Dot</th>\n<th>Line</th>\n<th>Vertical Bar</th>\n<th>Horizontal Bar</th>\n</tr>\n</thead>\n<tbody>\n<tr>\n<td>public</td>\n<td>0.86</td>\n<td>0.10</td>\n<td>0.00</td>\n<td>0.32</td>\n<td>0.39</td>\n<td>0.05</td>\n</tr>\n<tr>\n<td>private</td>\n<td>0.72</td>\n<td>0.30</td>\n<td>0.01</td>\n<td>0.13</td>\n<td>0.26</td>\n<td>0.01</td>\n</tr>\n</tbody>\n</table>\n<p>It seems that my scatter score was still a little higher than yours.<br>\nI read the solution you published and the method I used was very similar to the 3rd team's method. However, the difference is that I used Deplot to read the label text, did not use the tic information, and made the overall process a bit simpler, so those differences may have just happened to work well with this data.</p>\n<p>For more details, I am currently working hard to write a solution, so I hope you can wait a little longer.</p>",
      "rawMarkdown": "I did a late sub on my final sub.\nThe results are as follows.\n|              | Overall | Scatter | Dot | Line | Vertical Bar | Horizontal Bar |\n| ---        | ---        | ---        | ---  | ---   |  ---               | ---                    |\n| public   | 0.86 | 0.10 | 0.00 | 0.32 | 0.39 | 0.05 |\n| private | 0.72 | 0.30 | 0.01 | 0.13 | 0.26 | 0.01 | \n\n\nIt seems that my scatter score was still a little higher than yours.\nI read the solution you published and the method I used was very similar to the 3rd team's method. However, the difference is that I used Deplot to read the label text, did not use the tic information, and made the overall process a bit simpler, so those differences may have just happened to work well with this data.\n\nFor more details, I am currently working hard to write a solution, so I hope you can wait a little longer.",
      "votes": null
    },
    {
      "id": "2311668",
      "postDate": "06/21/2023 10:49:12",
      "content": "<p>Awesome scatter score indeed ! Our detection models miss markers in 30+% of plots, resulting in a 0 score, the rest of our pipeline is pretty good. My guess is that your detection model is stronger, we should've pushed ours a bit more.</p>",
      "rawMarkdown": "Awesome scatter score indeed ! Our detection models miss markers in 30+% of plots, resulting in a 0 score, the rest of our pipeline is pretty good. My guess is that your detection model is stronger, we should've pushed ours a bit more.",
      "votes": null
    },
    {
      "id": "2311698",
      "postDate": "06/21/2023 11:45:45",
      "content": "<p>I see, yes, that may be true.<br>\nI was using the ICDAR dataset for point detection, so that may have had a big impact.<br>\nAs a test, I checked the results of the model for patterns that did not use the ICDAR dataset for training.</p>\n<p>It is only for reference since the post-processing, etc. is different, the accuracy of the model without ICDAR data was Public 0.09/Private 0.28, which is about the same as your result.</p>",
      "rawMarkdown": "I see, yes, that may be true.\nI was using the ICDAR dataset for point detection, so that may have had a big impact.\nAs a test, I checked the results of the model for patterns that did not use the ICDAR dataset for training.\n\nIt is only for reference since the post-processing, etc. is different, the accuracy of the model without ICDAR data was Public 0.09/Private 0.28, which is about the same as your result.",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 2310686,
      "author_name": "benbla",
      "author_url": "",
      "post_date": "06/20/2023 14:46:31",
      "content": "<p><a href=\"https://www.kaggle.com/crodoc\" target=\"_blank\">@crodoc</a> <a href=\"https://www.kaggle.com/theoviel\" target=\"_blank\">@theoviel</a> </p>\n<p>I added your result here (place 3 solution):</p>\n<table>\n<thead>\n<tr>\n<th></th>\n<th>Overall</th>\n<th>Scatter</th>\n<th>Dot</th>\n<th>Line</th>\n<th>Vertical Bar</th>\n<th>Horizontal Bar</th>\n</tr>\n</thead>\n<tbody>\n<tr>\n<td>public</td>\n<td>0.87</td>\n<td>0.09</td>\n<td>0</td>\n<td>0.33</td>\n<td>0.39</td>\n<td>0.04</td>\n</tr>\n<tr>\n<td>private</td>\n<td>0.71</td>\n<td>0.28</td>\n<td>0.01</td>\n<td>0.13</td>\n<td>0.27</td>\n<td>0.01</td>\n</tr>\n</tbody>\n</table>",
      "votes": null,
      "replies": [
        {
          "id": 2310698,
          "author_name": "theoviel",
          "author_url": "",
          "post_date": "06/20/2023 14:55:45",
          "content": "<p>Thanks ! I think you swapped public and private horizontal bar in your write-up :)</p>\n<p>Would love to get input from <a href=\"https://www.kaggle.com/kashiwaba\" target=\"_blank\">@kashiwaba</a> and <a href=\"https://www.kaggle.com/conjuring92\" target=\"_blank\">@conjuring92</a> on this</p>",
          "votes": null,
          "replies": [
            {
              "id": 2310754,
              "author_name": "kashiwaba",
              "author_url": "",
              "post_date": "06/20/2023 15:28:39",
              "content": "<p>This is certainly an interesting consideration.<br>\nI have not yet been able to check the scores per chart for the final sub.<br>\nI hope to check them in the late sub in the few days and will share the results when they are available!</p>\n<p>By the way, looking at the scores I submitted, the solutions with improved scatter prediction methods had higher Private scores across the board, so I'm pretty sure that scatter is the key to Private scores.</p>",
              "votes": null,
              "replies": [
                {
                  "id": 2310759,
                  "author_name": "theoviel",
                  "author_url": "",
                  "post_date": "06/20/2023 15:32:49",
                  "content": "<p>I'm pretty sure your scatter is stronger that ours and that's why you went up 7 spots :)</p>",
                  "votes": null,
                  "replies": []
                }
              ]
            },
            {
              "id": 2311644,
              "author_name": "kashiwaba",
              "author_url": "",
              "post_date": "06/21/2023 10:08:29",
              "content": "<p>I did a late sub on my final sub.<br>\nThe results are as follows.</p>\n<table>\n<thead>\n<tr>\n<th></th>\n<th>Overall</th>\n<th>Scatter</th>\n<th>Dot</th>\n<th>Line</th>\n<th>Vertical Bar</th>\n<th>Horizontal Bar</th>\n</tr>\n</thead>\n<tbody>\n<tr>\n<td>public</td>\n<td>0.86</td>\n<td>0.10</td>\n<td>0.00</td>\n<td>0.32</td>\n<td>0.39</td>\n<td>0.05</td>\n</tr>\n<tr>\n<td>private</td>\n<td>0.72</td>\n<td>0.30</td>\n<td>0.01</td>\n<td>0.13</td>\n<td>0.26</td>\n<td>0.01</td>\n</tr>\n</tbody>\n</table>\n<p>It seems that my scatter score was still a little higher than yours.<br>\nI read the solution you published and the method I used was very similar to the 3rd team's method. However, the difference is that I used Deplot to read the label text, did not use the tic information, and made the overall process a bit simpler, so those differences may have just happened to work well with this data.</p>\n<p>For more details, I am currently working hard to write a solution, so I hope you can wait a little longer.</p>",
              "votes": null,
              "replies": [
                {
                  "id": 2311668,
                  "author_name": "theoviel",
                  "author_url": "",
                  "post_date": "06/21/2023 10:49:12",
                  "content": "<p>Awesome scatter score indeed ! Our detection models miss markers in 30+% of plots, resulting in a 0 score, the rest of our pipeline is pretty good. My guess is that your detection model is stronger, we should've pushed ours a bit more.</p>",
                  "votes": null,
                  "replies": [
                    {
                      "id": 2311698,
                      "author_name": "kashiwaba",
                      "author_url": "",
                      "post_date": "06/21/2023 11:45:45",
                      "content": "<p>I see, yes, that may be true.<br>\nI was using the ICDAR dataset for point detection, so that may have had a big impact.<br>\nAs a test, I checked the results of the model for patterns that did not use the ICDAR dataset for training.</p>\n<p>It is only for reference since the post-processing, etc. is different, the accuracy of the model without ICDAR data was Public 0.09/Private 0.28, which is about the same as your result.</p>",
                      "votes": null,
                      "replies": []
                    }
                  ]
                }
              ]
            }
          ]
        }
      ]
    }
  ],
  "raw_markdown_by_id": {
    "2310682": "I think, it would be really interesting to compare chart-specific scores. It might be interesting for the competition host as well. I just used generative models for my solution and missed to tune scatter. However, I think the other scores are quite good. Could you guys share the scores of your final submitted solution?\n\n|  | Overall | Scatter | Dot | Line | Vertical Bar | Horizontal Bar\n| --- | --- | --- | --- | --- |\n| public  | 0.79 | 0.03| 0 | 0.3 | 0.38 | 0.04\n| private | 0.46 🤕 | 0.07 | 0 | 0.11 | 0.26 | 0.01\n____\n\n**Key steps for my training pipeline**\n- Autoregressive ensembling within greedy search (2x matcha-base, 2x deplot) (+0.03 in the public lb in comparison to the best single model)\n- Pretraining on synthetic data and just finetuning on the competition data: pretrained first epoch with max_patches 1024 and increased in the second epoch to 1536 (also finetuned with 1536) (+0.02 for a single model). I think it works like a warmup phase to generate more general features.\n- Using the ICDAR dataset (all charts with one data-series, I also manually adjusted the annotations of line charts to fit the competition conventions) at the finetuning stage (+0.02 for a single model)\n- Training different layer of the encoder and decoder with different learning rates (+0.02 for a single model)\n\n**Further ideas**\n- I thought about \"dynamic ensembling\". It was very important to predict the right amount of data points. I thought it might be interesting to use a much larger ensemble in the case greedy search produces an eos-token. If the eos-token is predicted, go a step back, feed the current sequence to xx models and check if the majority of the model is voting for an eos-token, otherwise go on. That would be an option to utilize more than 4 models. Maybe it might makes sense to do this for the first x tokens as well to point the sequence in the right direction\n- I thought about implementing a seperate model for scatterplots like [https://github.com/bloomberg/scatteract](url) but didn´t have time left\n\nIt would be really great if you guys can share your results.",
    "2310686": "crodoc @theoviel \n\nI added your result here (place 3 solution):\n|  | Overall | Scatter | Dot | Line | Vertical Bar | Horizontal Bar\n| --- | --- | --- | --- | --- |\n| public  | 0.87 | 0.09| 0 | 0.33 | 0.39 | 0.04\n| private | 0.71 | 0.28 | 0.01 | 0.13 | 0.27 | 0.01",
    "2310698": "Thanks ! I think you swapped public and private horizontal bar in your write-up :)\n\nWould love to get input from @kashiwaba and @conjuring92 on this",
    "2310754": "This is certainly an interesting consideration.\nI have not yet been able to check the scores per chart for the final sub.\nI hope to check them in the late sub in the few days and will share the results when they are available!\n\nBy the way, looking at the scores I submitted, the solutions with improved scatter prediction methods had higher Private scores across the board, so I'm pretty sure that scatter is the key to Private scores.",
    "2310759": "I'm pretty sure your scatter is stronger that ours and that's why you went up 7 spots :)",
    "2311644": "I did a late sub on my final sub.\nThe results are as follows.\n|              | Overall | Scatter | Dot | Line | Vertical Bar | Horizontal Bar |\n| ---        | ---        | ---        | ---  | ---   |  ---               | ---                    |\n| public   | 0.86 | 0.10 | 0.00 | 0.32 | 0.39 | 0.05 |\n| private | 0.72 | 0.30 | 0.01 | 0.13 | 0.26 | 0.01 | \n\n\nIt seems that my scatter score was still a little higher than yours.\nI read the solution you published and the method I used was very similar to the 3rd team's method. However, the difference is that I used Deplot to read the label text, did not use the tic information, and made the overall process a bit simpler, so those differences may have just happened to work well with this data.\n\nFor more details, I am currently working hard to write a solution, so I hope you can wait a little longer.",
    "2311668": "Awesome scatter score indeed ! Our detection models miss markers in 30+% of plots, resulting in a 0 score, the rest of our pipeline is pretty good. My guess is that your detection model is stronger, we should've pushed ours a bit more.",
    "2311698": "I see, yes, that may be true.\nI was using the ICDAR dataset for point detection, so that may have had a big impact.\nAs a test, I checked the results of the model for patterns that did not use the ICDAR dataset for training.\n\nIt is only for reference since the post-processing, etc. is different, the accuracy of the model without ICDAR data was Public 0.09/Private 0.28, which is about the same as your result."
  },
  "source": "meta"
}