{ "id": 405062, "title": "How can I fine-tuning deplot? (I have asked the author and added the results.)", "url": "/competitions/benetech-making-graphs-accessible/discussion/405062", "author_name": "", "post_date": "2023-04-26T05:46:28.521371700Z", "votes": 1, "comment_count": 4, "views": 0, "content": "

I am trying to fine-tune google/deplot with the following link and Notebook.

\n

link: https://huggingface.co/docs/transformers/main/en/model_doc/deplot#finetuning
\nNotebook: https://github.com/huggingface/notebooks/blob/main/examples/image_captioning_pix2struct.ipynb

\n

But, I got terrible modelšŸ˜„

\n

I have two questions about fine-tuning deplot.

\n
\n

First, I got error, ā€œValueError: A header text must be provided for VQA models.ā€ from following code(quoted from image_captioning_pix2struct.ipynb).

\n
encoding = self.processor(images=item[ā€œimageā€], return_tensors=ā€œptā€, add_special_tokens=, max_patches=MAX_PATCHES)\n
\n

For deplot, what should I input as header text? (Maybe, Generate underlying data table of the figure below: ?)
\nā†’230511 update: I have checked with the author. Link. header text should be \"Generate underlying data table of the figure below:\" or empty.

\n
\n

Second, what format do deplot expect for input text?

\n

According to the paper, it expect a table format, but when I try it, it doesn't work.

\n
\n

The table is linearized as a textual sequence (markdown format) with | separating cells and \\n separating rows.

\n
\n

paper: https://arxiv.org/pdf/2212.10505.pdf

\n

ā†’ 230512 update: I have checked with the author. Link

\n
text = \"\"\"Year | Democrats | Republicans | Independents\n2004 | 68.1% | 45.0% | 53.0%\n2006 | 58.0% | 42.0% | 53.0%\n2007 | 59.0% | 38.0% | 45.0%\n2009 | 72.0% | 49.0% | 60.0%\n2011 | 71.0% | 51.2% | 58.0%\n2012 | 70.0% | 48.0% | 53.0%\n2013 | 72.0% | 41.0% | 60.0%\"\"\"\n
\n

example groudtruth table: Link

", "messages": [ { "id": "2235527", "postDate": "04/26/2023 05:46:28", "content": "

I am trying to fine-tune google/deplot with the following link and Notebook.

\n

link: https://huggingface.co/docs/transformers/main/en/model_doc/deplot#finetuning
\nNotebook: https://github.com/huggingface/notebooks/blob/main/examples/image_captioning_pix2struct.ipynb

\n

But, I got terrible modelšŸ˜„

\n

I have two questions about fine-tuning deplot.

\n
\n

First, I got error, ā€œValueError: A header text must be provided for VQA models.ā€ from following code(quoted from image_captioning_pix2struct.ipynb).

\n
encoding = self.processor(images=item[ā€œimageā€], return_tensors=ā€œptā€, add_special_tokens=, max_patches=MAX_PATCHES)\n
\n

For deplot, what should I input as header text? (Maybe, Generate underlying data table of the figure below: ?)
\nā†’230511 update: I have checked with the author. Link. header text should be \"Generate underlying data table of the figure below:\" or empty.

\n
\n

Second, what format do deplot expect for input text?

\n

According to the paper, it expect a table format, but when I try it, it doesn't work.

\n
\n

The table is linearized as a textual sequence (markdown format) with | separating cells and \\n separating rows.

\n
\n

paper: https://arxiv.org/pdf/2212.10505.pdf

\n

ā†’ 230512 update: I have checked with the author. Link

\n
text = \"\"\"Year | Democrats | Republicans | Independents\n2004 | 68.1% | 45.0% | 53.0%\n2006 | 58.0% | 42.0% | 53.0%\n2007 | 59.0% | 38.0% | 45.0%\n2009 | 72.0% | 49.0% | 60.0%\n2011 | 71.0% | 51.2% | 58.0%\n2012 | 70.0% | 48.0% | 53.0%\n2013 | 72.0% | 41.0% | 60.0%\"\"\"\n
\n

example groudtruth table: Link

", "rawMarkdown": "I am trying to fine-tune google/deplot with the following link and Notebook.\n\nlink: https://huggingface.co/docs/transformers/main/en/model_doc/deplot#finetuning\nNotebook: https://github.com/huggingface/notebooks/blob/main/examples/image_captioning_pix2struct.ipynb\n\nBut, I got terrible modelšŸ˜„\n\nI have two questions about fine-tuning deplot.\n\n---\nFirst, I got error, `ā€œValueError: A header text must be provided for VQA models.ā€` from following code(quoted from `image_captioning_pix2struct.ipynb`).\n\n```python\nencoding = self.processor(images=item[ā€œimageā€], return_tensors=ā€œptā€, add_special_tokens=True, max_patches=MAX_PATCHES)\n```\n\nFor deplot, what should I input as header text? (Maybe, `Generate underlying data table of the figure below:` ?)\nā†’230511 update: I have checked with the author. [Link](https://discuss.huggingface.co/t/for-google-deplot-what-should-i-input-as-header-text-for-fine-tuning/37621/5). header text should be `\"Generate underlying data table of the figure below:\"` or empty. \n\n---\nSecond, what format do deplot expect for input text?\n\nAccording to the paper, it expect a table format, but when I try it, it doesn't work.\n\n> The table is linearized as a textual sequence (markdown format) with | separating cells and \\n separating rows.\n\npaper: https://arxiv.org/pdf/2212.10505.pdf\n\nā†’ 230512 update: I have checked with the author. [Link](https://huggingface.co/google/deplot/discussions/3)\n```\ntext = \"\"\"Year | Democrats | Republicans | Independents\n2004 | 68.1% | 45.0% | 53.0%\n2006 | 58.0% | 42.0% | 53.0%\n2007 | 59.0% | 38.0% | 45.0%\n2009 | 72.0% | 49.0% | 60.0%\n2011 | 71.0% | 51.2% | 58.0%\n2012 | 70.0% | 48.0% | 53.0%\n2013 | 72.0% | 41.0% | 60.0%\"\"\"\n```\n\nexample groudtruth table: [Link](https://huggingface.co/spaces/fl399/deplot_plus_llm/blob/a82c90c37bd452250319484a5e6a1208bea31972/app.py#L39)", "votes": null }, { "id": "2235528", "postDate": "04/26/2023 05:52:30", "content": "

I am also struggling with DePlot fine tuning, I was able to run a training notebook, however with terrible terrible results! Most likely the main problem is the data format ! I tried different formats, I also converted the data to the same output format of deplot, but nthn :-( I think we are missing some critical informations to run it smoothly ! Information that the documentation is not yet explaining or showing.

\n

My current score is not based on fine-tuning DePlot !

", "rawMarkdown": "I am also struggling with DePlot fine tuning, I was able to run a training notebook, however with terrible terrible results! Most likely the main problem is the data format ! I tried different formats, I also converted the data to the same output format of deplot, but nthn :-( I think we are missing some critical informations to run it smoothly ! Information that the documentation is not yet explaining or showing. \n\nMy current score is not based on fine-tuning DePlot !", "votes": null }, { "id": "2235535", "postDate": "04/26/2023 06:05:22", "content": "

If there is something that the documentation does not explain, perhaps it would be better to contact the author(huggingface), thanks!

", "rawMarkdown": "If there is something that the documentation does not explain, perhaps it would be better to contact the author(huggingface), thanks!", "votes": null }, { "id": "2241605", "postDate": "05/01/2023 16:36:50", "content": "

Read this thread šŸ˜‰

", "rawMarkdown": "[Read this thread](https://github.com/huggingface/transformers/issues/22903) šŸ˜‰", "votes": null }, { "id": "2255072", "postDate": "05/11/2023 13:40:09", "content": "

Thank you! Very informative!

", "rawMarkdown": "Thank you! Very informative!", "votes": null } ], "comments": [ { "id": 2235528, "author_name": "asalhi", "author_url": "", "post_date": "04/26/2023 05:52:30", "content": "

I am also struggling with DePlot fine tuning, I was able to run a training notebook, however with terrible terrible results! Most likely the main problem is the data format ! I tried different formats, I also converted the data to the same output format of deplot, but nthn :-( I think we are missing some critical informations to run it smoothly ! Information that the documentation is not yet explaining or showing.

\n

My current score is not based on fine-tuning DePlot !

", "votes": null, "replies": [ { "id": 2235535, "author_name": "sinchir0", "author_url": "", "post_date": "04/26/2023 06:05:22", "content": "

If there is something that the documentation does not explain, perhaps it would be better to contact the author(huggingface), thanks!

", "votes": null, "replies": [] } ] }, { "id": 2241605, "author_name": "nbroad", "author_url": "", "post_date": "05/01/2023 16:36:50", "content": "

Read this thread šŸ˜‰

", "votes": null, "replies": [ { "id": 2255072, "author_name": "sinchir0", "author_url": "", "post_date": "05/11/2023 13:40:09", "content": "

Thank you! Very informative!

", "votes": null, "replies": [] } ] } ], "raw_markdown_by_id": { "2235527": "I am trying to fine-tune google/deplot with the following link and Notebook.\n\nlink: https://huggingface.co/docs/transformers/main/en/model_doc/deplot#finetuning\nNotebook: https://github.com/huggingface/notebooks/blob/main/examples/image_captioning_pix2struct.ipynb\n\nBut, I got terrible modelšŸ˜„\n\nI have two questions about fine-tuning deplot.\n\n---\nFirst, I got error, `ā€œValueError: A header text must be provided for VQA models.ā€` from following code(quoted from `image_captioning_pix2struct.ipynb`).\n\n```python\nencoding = self.processor(images=item[ā€œimageā€], return_tensors=ā€œptā€, add_special_tokens=True, max_patches=MAX_PATCHES)\n```\n\nFor deplot, what should I input as header text? (Maybe, `Generate underlying data table of the figure below:` ?)\nā†’230511 update: I have checked with the author. [Link](https://discuss.huggingface.co/t/for-google-deplot-what-should-i-input-as-header-text-for-fine-tuning/37621/5). header text should be `\"Generate underlying data table of the figure below:\"` or empty. \n\n---\nSecond, what format do deplot expect for input text?\n\nAccording to the paper, it expect a table format, but when I try it, it doesn't work.\n\n> The table is linearized as a textual sequence (markdown format) with | separating cells and \\n separating rows.\n\npaper: https://arxiv.org/pdf/2212.10505.pdf\n\nā†’ 230512 update: I have checked with the author. [Link](https://huggingface.co/google/deplot/discussions/3)\n```\ntext = \"\"\"Year | Democrats | Republicans | Independents\n2004 | 68.1% | 45.0% | 53.0%\n2006 | 58.0% | 42.0% | 53.0%\n2007 | 59.0% | 38.0% | 45.0%\n2009 | 72.0% | 49.0% | 60.0%\n2011 | 71.0% | 51.2% | 58.0%\n2012 | 70.0% | 48.0% | 53.0%\n2013 | 72.0% | 41.0% | 60.0%\"\"\"\n```\n\nexample groudtruth table: [Link](https://huggingface.co/spaces/fl399/deplot_plus_llm/blob/a82c90c37bd452250319484a5e6a1208bea31972/app.py#L39)", "2235528": "I am also struggling with DePlot fine tuning, I was able to run a training notebook, however with terrible terrible results! Most likely the main problem is the data format ! I tried different formats, I also converted the data to the same output format of deplot, but nthn :-( I think we are missing some critical informations to run it smoothly ! Information that the documentation is not yet explaining or showing. \n\nMy current score is not based on fine-tuning DePlot !", "2235535": "If there is something that the documentation does not explain, perhaps it would be better to contact the author(huggingface), thanks!", "2241605": "[Read this thread](https://github.com/huggingface/transformers/issues/22903) šŸ˜‰", "2255072": "Thank you! Very informative!" }, "source": "meta" }