I created a notebook to train an end-to-end model, Donut π©, and it does ok. End-to-end means the model takes the image as input and directly outputs the text. It is just an image encoder and text decoder,
\n\nAfter 1 epoch, here are the validation results (on only extracted figures):
\n{
\n \"val_score\": 0.3284719351382791,
\n \"horizontal_bar_score\": 0.0,
\n \"line_score\": 0.3415160063217807,
\n \"scatter_score\": 0.009214113313302883,
\n \"vertical_bar_score\": 0.48168405671968667,
\n}
It does very poorly at horizontal bar and scatter charts. There are very few horizontal bar figures to train on, but there are a reasonable amount of scatter plots. More investigation required! The one in the notebook was stopped after 1 epoch, but the ones I've trained for 5 epochs do a bit better (0.38).
\nTraining Notebook
\nInference Notebook
There's definitely lots of potential for improvement!
\nDetails about Donut here: https://github.com/clovaai/donut) te)
", "messages": [ { "id": "2201193", "postDate": "03/29/2023 05:58:08", "content": "I created a notebook to train an end-to-end model, Donut π©, and it does ok. End-to-end means the model takes the image as input and directly outputs the text. It is just an image encoder and text decoder,
\n\nAfter 1 epoch, here are the validation results (on only extracted figures):
\n{
\n \"val_score\": 0.3284719351382791,
\n \"horizontal_bar_score\": 0.0,
\n \"line_score\": 0.3415160063217807,
\n \"scatter_score\": 0.009214113313302883,
\n \"vertical_bar_score\": 0.48168405671968667,
\n}
It does very poorly at horizontal bar and scatter charts. There are very few horizontal bar figures to train on, but there are a reasonable amount of scatter plots. More investigation required! The one in the notebook was stopped after 1 epoch, but the ones I've trained for 5 epochs do a bit better (0.38).
\nTraining Notebook
\nInference Notebook
There's definitely lots of potential for improvement!
\nDetails about Donut here: https://github.com/clovaai/donut) te)
", "rawMarkdown": "I created a notebook to train an end-to-end model, Donut π©, and it does ok. End-to-end means the model takes the image as input and directly outputs the text. It is just an image encoder and text decoder, ~~but I can't get the inference to work, so feel free to try it out and see if you can figure out why it fails when submitted.~~\n\n~~## UPDATE: Inference notebook now works. I had to limit it to only predict for line plots and it got an LB score of 0.16. Line plots are about 40% of the total extracted charts, so 0.4*0.4= 0.14 which is very close. The one I submitted had slightly better CV scores than the scores below. See [this thread](https://www.kaggle.com/competitions/benetech-making-graphs-accessible/discussion/398528) for why I think some submissions fail.~~\n\n## Update 2: The scoring method changed so now the LB score is 0.42. See [this discussion for details](https://www.kaggle.com/competitions/benetech-making-graphs-accessible/discussion/398528)\n\n\nAfter 1 epoch, here are the validation results (on only extracted figures): \n\n{\n \"val_score\": 0.3284719351382791,\n \"horizontal_bar_score\": 0.0,\n \"line_score\": 0.3415160063217807,\n \"scatter_score\": 0.009214113313302883,\n \"vertical_bar_score\": 0.48168405671968667,\n}\n\nIt does very poorly at horizontal bar and scatter charts. There are very few horizontal bar figures to train on, but there are a reasonable amount of scatter plots. More investigation required! The one in the notebook was stopped after 1 epoch, but the ones I've trained for 5 epochs do a bit better (0.38).\n\n\n[Training Notebook](https://www.kaggle.com/nbroad/donut-train-benetech)\n[Inference Notebook](https://www.kaggle.com/nbroad/donut-infer-benetech)\n\nThere's definitely lots of potential for improvement!\n\nDetails about Donut here: https://github.com/clovaai/donut) te)", "votes": null }, { "id": "2203632", "postDate": "03/31/2023 02:02:24", "content": "Nice to see you @nbroad providing a working DoNut pipeline. I felt it could work but had not the stamina to deep dive into it.
", "rawMarkdown": "Nice to see you @nbroad providing a working DoNut pipeline. I felt it could work but had not the stamina to deep dive into it.", "votes": null }, { "id": "2203684", "postDate": "03/31/2023 03:34:39", "content": "Image encoder - text decoder models are super cool. I think it will be interesting because I haven't seen generative models do well on Kaggle yet. For the images it doesn't understand, my model outputs nonsense π
", "rawMarkdown": "Image encoder - text decoder models are super cool. I think it will be interesting because I haven't seen generative models do well on Kaggle yet. For the images it doesn't understand, my model outputs nonsense π ", "votes": null }, { "id": "2276250", "postDate": "05/26/2023 15:58:46", "content": "@nbroad Will generating more horizontal_bar and scatter charts to train improve their performances? BTW, is Matcha much more better than Donut in your experiment?
", "rawMarkdown": "nbroad Will generating more horizontal_bar and scatter charts to train improve their performances? BTW, is Matcha much more better than Donut in your experiment?", "votes": null }, { "id": "2276277", "postDate": "05/26/2023 16:34:42", "content": "I got 0.44 with donut and 0.65 with matcha.
\nAdding more horizontal bar and scatter charts will likely be useful
", "rawMarkdown": "I got 0.44 with donut and 0.65 with matcha. \n\nAdding more horizontal bar and scatter charts will likely be useful", "votes": null }, { "id": "2284825", "postDate": "06/02/2023 09:48:22", "content": "The first Friday of June --> π π©π©π©π©π Today is the National Donut Day π π©π©π©π©π Note: Maybe this will not help the score, but who knows
", "rawMarkdown": "The first Friday of June --> π π©π©π©π©π Today is the National Donut Day π π©π©π©π©π *Note: Maybe this will not help the score, but who knows*", "votes": null }, { "id": "2296334", "postDate": "06/11/2023 18:11:38", "content": "Thank you for sharing.
", "rawMarkdown": "Thank you for sharing.", "votes": null } ], "comments": [ { "id": 2203632, "author_name": "kneroma", "author_url": "", "post_date": "03/31/2023 02:02:24", "content": "Nice to see you @nbroad providing a working DoNut pipeline. I felt it could work but had not the stamina to deep dive into it.
", "votes": null, "replies": [ { "id": 2203684, "author_name": "nbroad", "author_url": "", "post_date": "03/31/2023 03:34:39", "content": "Image encoder - text decoder models are super cool. I think it will be interesting because I haven't seen generative models do well on Kaggle yet. For the images it doesn't understand, my model outputs nonsense π
", "votes": null, "replies": [] } ] }, { "id": 2276250, "author_name": "zifencai", "author_url": "", "post_date": "05/26/2023 15:58:46", "content": "@nbroad Will generating more horizontal_bar and scatter charts to train improve their performances? BTW, is Matcha much more better than Donut in your experiment?
", "votes": null, "replies": [ { "id": 2276277, "author_name": "nbroad", "author_url": "", "post_date": "05/26/2023 16:34:42", "content": "I got 0.44 with donut and 0.65 with matcha.
\nAdding more horizontal bar and scatter charts will likely be useful
", "votes": null, "replies": [] } ] }, { "id": 2284825, "author_name": "kirderf", "author_url": "", "post_date": "06/02/2023 09:48:22", "content": "The first Friday of June --> π π©π©π©π©π Today is the National Donut Day π π©π©π©π©π Note: Maybe this will not help the score, but who knows
", "votes": null, "replies": [] }, { "id": 2296334, "author_name": "annguyenquoc", "author_url": "", "post_date": "06/11/2023 18:11:38", "content": "Thank you for sharing.
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "2201193": "I created a notebook to train an end-to-end model, Donut π©, and it does ok. End-to-end means the model takes the image as input and directly outputs the text. It is just an image encoder and text decoder, ~~but I can't get the inference to work, so feel free to try it out and see if you can figure out why it fails when submitted.~~\n\n~~## UPDATE: Inference notebook now works. I had to limit it to only predict for line plots and it got an LB score of 0.16. Line plots are about 40% of the total extracted charts, so 0.4*0.4= 0.14 which is very close. The one I submitted had slightly better CV scores than the scores below. See [this thread](https://www.kaggle.com/competitions/benetech-making-graphs-accessible/discussion/398528) for why I think some submissions fail.~~\n\n## Update 2: The scoring method changed so now the LB score is 0.42. See [this discussion for details](https://www.kaggle.com/competitions/benetech-making-graphs-accessible/discussion/398528)\n\n\nAfter 1 epoch, here are the validation results (on only extracted figures): \n\n{\n \"val_score\": 0.3284719351382791,\n \"horizontal_bar_score\": 0.0,\n \"line_score\": 0.3415160063217807,\n \"scatter_score\": 0.009214113313302883,\n \"vertical_bar_score\": 0.48168405671968667,\n}\n\nIt does very poorly at horizontal bar and scatter charts. There are very few horizontal bar figures to train on, but there are a reasonable amount of scatter plots. More investigation required! The one in the notebook was stopped after 1 epoch, but the ones I've trained for 5 epochs do a bit better (0.38).\n\n\n[Training Notebook](https://www.kaggle.com/nbroad/donut-train-benetech)\n[Inference Notebook](https://www.kaggle.com/nbroad/donut-infer-benetech)\n\nThere's definitely lots of potential for improvement!\n\nDetails about Donut here: https://github.com/clovaai/donut) te)", "2203632": "Nice to see you @nbroad providing a working DoNut pipeline. I felt it could work but had not the stamina to deep dive into it.", "2203684": "Image encoder - text decoder models are super cool. I think it will be interesting because I haven't seen generative models do well on Kaggle yet. For the images it doesn't understand, my model outputs nonsense π ", "2276250": "nbroad Will generating more horizontal_bar and scatter charts to train improve their performances? BTW, is Matcha much more better than Donut in your experiment?", "2276277": "I got 0.44 with donut and 0.65 with matcha. \n\nAdding more horizontal bar and scatter charts will likely be useful", "2284825": "The first Friday of June --> π π©π©π©π©π Today is the National Donut Day π π©π©π©π©π *Note: Maybe this will not help the score, but who knows*", "2296334": "Thank you for sharing." }, "source": "meta" }