\n
![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F2382612%2Ffd74a88a130f0431dac0747e03cb6e77%2Floss.png?generation=1685858982117324&alt=media\")
I'm currently experimenting with the Pix2Struct and MatCha models. Initially, the training process appears as expected, with the training loss swiftly decreasing from 6 to roughly 0.5. However, after a considerable number of update steps, where the loss stabilizes, it unexpectedly begins to increase, sometimes even reaching NaN. This issue has surfaced with both the Pix2Struct and MatCha models.
\nSome things I have tried but did not work:
\nbtw, the batch size is set to 16.
\nAnd I also follow the setting as @nbroad suggested in this thread.
Despite these steps, the problem persists. Any suggestions or insights would be greatly appreciated.
\n
I'm currently experimenting with the Pix2Struct and MatCha models. Initially, the training process appears as expected, with the training loss swiftly decreasing from 6 to roughly 0.5. However, after a considerable number of update steps, where the loss stabilizes, it unexpectedly begins to increase, sometimes even reaching NaN. This issue has surfaced with both the Pix2Struct and MatCha models.
\nSome things I have tried but did not work:
\nbtw, the batch size is set to 16.
\nAnd I also follow the setting as @nbroad suggested in this thread.
Despite these steps, the problem persists. Any suggestions or insights would be greatly appreciated.
\n
Which version of the transformers library are you using?
\nThere was a change in v4.29.0 that significantly improves Pix2Struct models training: https://github.com/huggingface/transformers/releases/tag/v4.29.0 (See first paragraph of Breaking changes)
\nSo if you are using a version before that, this is probably the reason.
I'm using the latest transformers version (By downloading it from github directly) but my loss looks like this too
", "rawMarkdown": "I'm using the latest transformers version (By downloading it from github directly) but my loss looks like this too" }, { "id": 2287888, "postDate": "2023-06-05T02:35:12.070Z", "content": "I aslo install the transformers package from git source, the version of which is 4.30.0.dev0.
", "rawMarkdown": "I aslo install the transformers package from git source, the version of which is 4.30.0.dev0." }, { "id": 2288230, "postDate": "2023-06-05T09:00:35.410Z", "content": "Hrmm then I'm not sure what's wrong then. Probably a bug somewhere in your training code.
\nIf it helps, I assure you I have been able to get better loss curves with your settings but much lower batch size.
How are you doing BS 16 btw? Where are u getting all this VRAM haha
", "rawMarkdown": "Hrmm then I'm not sure what's wrong then. Probably a bug somewhere in your training code.\nIf it helps, I assure you I have been able to get better loss curves with your settings but much lower batch size.\n\nHow are you doing BS 16 btw? Where are u getting all this VRAM haha", "replies": [ { "id": 2288250, "postDate": "2023-06-05T09:19:31.747Z", "content": "I train the models on my local machine.
\nUpdate: Using a smaller learning rate maybe a good trial. I decrease the initial lr from 3e-5 to 2e-5 and the loss seems to be ok. Maybe it is just a luck.😃
\n![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F2382612%2Fae4e88751ef23e67a410dc7bb03e686a%2Floss.png?generation=1685956762622596&alt=media\")
I am also experiencing the same thing. These models are really sensitive to optimizer/scheduler parameters. May I ask, what is your batch size and do you use any gradient accumulation?
\nEdit: Oh I just realized that you mentioned setting it to 16 in your original question. Thanks!
", "rawMarkdown": "I am also experiencing the same thing. These models are really sensitive to optimizer/scheduler parameters. May I ask, what is your batch size and do you use any gradient accumulation?\n\nEdit: Oh I just realized that you mentioned setting it to 16 in your original question. Thanks! ", "votes": 1 }, { "id": 2288629, "postDate": "2023-06-05T14:29:43.467Z", "content": "@snnclsr I think you can also try smaller learning rate. I do not use gradient accumulation, but use cosine scheduler with warmup and AdamW (Adafactor is ok).
\nI train the Deplot model next and its loss is also relatively stable. I entered this competition quite late, I hope everything is not too late.
\n![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F2382612%2F9ba56f7b74555ba1d083637f167da55f%2Floss.png?generation=1685975364162664&alt=media\")
thank you @marbury yes, both of your experiments look quite stable! One thing I noticed is that I think I am currently underfitting given the total number of steps here and also based on the other discussions. So I will train my model much longer as my next trial. Even my current submission/model started learning something so I think this will improve my score.
\nDid you submit your deplot model as well? Which one do you think performs better for you?
", "rawMarkdown": "thank you @marbury yes, both of your experiments look quite stable! One thing I noticed is that I think I am currently underfitting given the total number of steps here and also based on the other discussions. So I will train my model much longer as my next trial. Even my current submission/model started learning something so I think this will improve my score.\n\nDid you submit your deplot model as well? Which one do you think performs better for you?" }, { "id": 2289303, "postDate": "2023-06-06T04:28:48.773Z", "content": "For my experiments, deplot model performs slightly better.
", "rawMarkdown": "For my experiments, deplot model performs slightly better.", "votes": 1 }, { "id": 2289533, "postDate": "2023-06-06T08:06:50.340Z", "content": "Nice! Thank you for your answer. I also managed to improve a bit more for my matcha model by training longer but still not performing as competent as it should IMO. I'm investigating what the missing part is
", "rawMarkdown": "Nice! Thank you for your answer. I also managed to improve a bit more for my matcha model by training longer but still not performing as competent as it should IMO. I'm investigating what the missing part is", "votes": 1 }, { "id": 2289997, "postDate": "2023-06-06T13:39:55.970Z", "content": "Okay figured out the inference as well 🤓
", "rawMarkdown": "Okay figured out the inference as well 🤓 " }, { "id": 2290008, "postDate": "2023-06-06T13:45:24.593Z", "content": "Good Job! @snnclsr
", "rawMarkdown": "Good Job! @snnclsr ", "votes": 1 }, { "id": 2290033, "postDate": "2023-06-06T13:57:24.630Z", "content": "thank you @marbury !
", "rawMarkdown": "thank you @marbury ! " }, { "id": 2307548, "postDate": "2023-06-18T09:08:51.573Z", "content": "@marbury
\nWhat is the difference in using gradient accumulation and directly using batch_size ? I am using Kaggle T4x2 GPU, i could fit only batch_size of 1 (image,text)
I cannot use Adafactor/AdamW (transfomer optimzers) along with pyorch lightning, i am getting and error, grad_fn is not set to the outputs.loss.
\nDo you have any idea about this ?
", "rawMarkdown": "@marbury \nWhat is the difference in using gradient accumulation and directly using batch_size ? I am using Kaggle T4x2 GPU, i could fit only batch_size of 1 (image,text)\n\nI cannot use Adafactor/AdamW (transfomer optimzers) along with pyorch lightning, i am getting and error, grad_fn is not set to the outputs.loss.\n\nDo you have any idea about this ?\n" } ] } ] }, { "id": 2504858, "postDate": "2023-10-30T07:32:07.753Z", "content": "Can we use pix2struct for document parsing? (if yes, can you please! provide me a direct link to source code or information)
", "rawMarkdown": "Can we use pix2struct for document parsing? (if yes, can you please! provide me a direct link to source code or information)" }, { "id": 2291618, "postDate": "2023-06-07T17:24:44.947Z", "content": "Does anywhere here also encounter the problem with deplot where it predicts some extra data points especially in line charts?
", "rawMarkdown": "Does anywhere here also encounter the problem with deplot where it predicts some extra data points especially in line charts?" }, { "id": 2291318, "postDate": "2023-06-07T13:22:37.937Z", "content": "Do you use scatter data? I don't use it and the loss is really smoth. I setup like you with lr 3e-5, bs 4 ![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F2730329%2F371f274d33de1426b4da387ade182495%2FWB%20Chart%206_7_2023%208_21_43%20PM.png?generation=1686144154165162&alt=media\")
I didn't intentionally exclude scatter plot, it seems like you have also successfully trained the model.👍
", "rawMarkdown": "I didn't intentionally exclude scatter plot, it seems like you have also successfully trained the model.👍" } ] }, { "id": 2287522, "postDate": "2023-06-04T15:32:00.643Z", "content": "Do the Nan loss happen in second epoch? R u using the code similar to public notebook on training Pix2Struct?
", "rawMarkdown": "Do the Nan loss happen in second epoch? R u using the code similar to public notebook on training Pix2Struct?", "replies": [ { "id": 2287893, "postDate": "2023-06-05T02:37:54.230Z", "content": "Yes, it happens at the second epoch. I follow the most part of the public notebook you mentioned, but I modify the training part based on my own convention.
", "rawMarkdown": "Yes, it happens at the second epoch. I follow the most part of the public notebook you mentioned, but I modify the training part based on my own convention.", "replies": [ { "id": 2287966, "postDate": "2023-06-05T04:39:25.510Z", "content": "If you are using AMP just as the author did in the notebook without modifications, I suggest you move only the model forward pass only inside autocast context manager.
", "rawMarkdown": "If you are using AMP just as the author did in the notebook without modifications, I suggest you move only the model forward pass only inside autocast context manager." } ] } ] }, { "id": 2287558, "postDate": "2023-06-04T16:37:43.293Z", "rawMarkdown": "", "isDeleted": true } ], "comments": [ { "id": 2287615, "author_name": "Jack Min Ong", "author_url": "", "post_date": "2023-06-04T17:52:35.420000", "content": "Which version of the transformers library are you using?
\nThere was a change in v4.29.0 that significantly improves Pix2Struct models training: https://github.com/huggingface/transformers/releases/tag/v4.29.0 (See first paragraph of Breaking changes)
\nSo if you are using a version before that, this is probably the reason.
I'm using the latest transformers version (By downloading it from github directly) but my loss looks like this too
", "votes": 0, "replies": [] }, { "id": 2287888, "author_name": "Givan", "author_url": "", "post_date": "2023-06-05T02:35:12.070000", "content": "I aslo install the transformers package from git source, the version of which is 4.30.0.dev0.
", "votes": 0, "replies": [] }, { "id": 2288230, "author_name": "Jack Min Ong", "author_url": "", "post_date": "2023-06-05T09:00:35.410000", "content": "Hrmm then I'm not sure what's wrong then. Probably a bug somewhere in your training code.
\nIf it helps, I assure you I have been able to get better loss curves with your settings but much lower batch size.
How are you doing BS 16 btw? Where are u getting all this VRAM haha
", "votes": 0, "replies": [ { "id": 2288250, "author_name": "Givan", "author_url": "", "post_date": "2023-06-05T09:19:31.747000", "content": "I train the models on my local machine.
\nUpdate: Using a smaller learning rate maybe a good trial. I decrease the initial lr from 3e-5 to 2e-5 and the loss seems to be ok. Maybe it is just a luck.😃
\n
I am also experiencing the same thing. These models are really sensitive to optimizer/scheduler parameters. May I ask, what is your batch size and do you use any gradient accumulation?
\nEdit: Oh I just realized that you mentioned setting it to 16 in your original question. Thanks!
", "votes": 1, "replies": [] }, { "id": 2288629, "author_name": "Givan", "author_url": "", "post_date": "2023-06-05T14:29:43.467000", "content": "@snnclsr I think you can also try smaller learning rate. I do not use gradient accumulation, but use cosine scheduler with warmup and AdamW (Adafactor is ok).
\nI train the Deplot model next and its loss is also relatively stable. I entered this competition quite late, I hope everything is not too late.
\n
thank you @marbury yes, both of your experiments look quite stable! One thing I noticed is that I think I am currently underfitting given the total number of steps here and also based on the other discussions. So I will train my model much longer as my next trial. Even my current submission/model started learning something so I think this will improve my score.
\nDid you submit your deplot model as well? Which one do you think performs better for you?
", "votes": 0, "replies": [] }, { "id": 2289303, "author_name": "Givan", "author_url": "", "post_date": "2023-06-06T04:28:48.773000", "content": "For my experiments, deplot model performs slightly better.
", "votes": 1, "replies": [] }, { "id": 2289533, "author_name": "Sinan Calisir", "author_url": "", "post_date": "2023-06-06T08:06:50.340000", "content": "Nice! Thank you for your answer. I also managed to improve a bit more for my matcha model by training longer but still not performing as competent as it should IMO. I'm investigating what the missing part is
", "votes": 1, "replies": [] }, { "id": 2289997, "author_name": "Sinan Calisir", "author_url": "", "post_date": "2023-06-06T13:39:55.970000", "content": "Okay figured out the inference as well 🤓
", "votes": 0, "replies": [] }, { "id": 2290008, "author_name": "Givan", "author_url": "", "post_date": "2023-06-06T13:45:24.593000", "content": "Good Job! @snnclsr
", "votes": 1, "replies": [] }, { "id": 2290033, "author_name": "Sinan Calisir", "author_url": "", "post_date": "2023-06-06T13:57:24.630000", "content": "thank you @marbury !
", "votes": 0, "replies": [] }, { "id": 2307548, "author_name": "Space Time", "author_url": "", "post_date": "2023-06-18T09:08:51.573000", "content": "@marbury
\nWhat is the difference in using gradient accumulation and directly using batch_size ? I am using Kaggle T4x2 GPU, i could fit only batch_size of 1 (image,text)
I cannot use Adafactor/AdamW (transfomer optimzers) along with pyorch lightning, i am getting and error, grad_fn is not set to the outputs.loss.
\nDo you have any idea about this ?
", "votes": 0, "replies": [] } ] } ] }, { "id": 2504858, "author_name": "Abdul Muqtadir", "author_url": "", "post_date": "2023-10-30T07:32:07.753000", "content": "Can we use pix2struct for document parsing? (if yes, can you please! provide me a direct link to source code or information)
", "votes": 0, "replies": [] }, { "id": 2291618, "author_name": "Chaitanya Giri", "author_url": "", "post_date": "2023-06-07T17:24:44.947000", "content": "Does anywhere here also encounter the problem with deplot where it predicts some extra data points especially in line charts?
", "votes": 0, "replies": [] }, { "id": 2291318, "author_name": "datnt114", "author_url": "", "post_date": "2023-06-07T13:22:37.937000", "content": "Do you use scatter data? I don't use it and the loss is really smoth. I setup like you with lr 3e-5, bs 4
I didn't intentionally exclude scatter plot, it seems like you have also successfully trained the model.👍
", "votes": 0, "replies": [] } ] }, { "id": 2287522, "author_name": "Sandy", "author_url": "", "post_date": "2023-06-04T15:32:00.643000", "content": "Do the Nan loss happen in second epoch? R u using the code similar to public notebook on training Pix2Struct?
", "votes": 0, "replies": [ { "id": 2287893, "author_name": "Givan", "author_url": "", "post_date": "2023-06-05T02:37:54.230000", "content": "Yes, it happens at the second epoch. I follow the most part of the public notebook you mentioned, but I modify the training part based on my own convention.
", "votes": 0, "replies": [ { "id": 2287966, "author_name": "Sandy", "author_url": "", "post_date": "2023-06-05T04:39:25.510000", "content": "If you are using AMP just as the author did in the notebook without modifications, I suggest you move only the model forward pass only inside autocast context manager.
", "votes": 0, "replies": [] } ] } ] }, { "id": 2287558, "author_name": "", "author_url": "", "post_date": "2023-06-04T16:37:43.293000", "content": "", "votes": 0, "replies": [] } ], "raw_markdown_by_id": { "2287084": "I'm currently experimenting with the Pix2Struct and MatCha models. Initially, the training process appears as expected, with the training loss swiftly decreasing from 6 to roughly 0.5. However, after a considerable number of update steps, where the loss stabilizes, it unexpectedly begins to increase, sometimes even reaching NaN. This issue has surfaced with both the Pix2Struct and MatCha models.\n\nSome things I have tried but did not work:\n- lower learning: 3e-5\n- using a learing rate scheduler: cosine leaning rate sheduler\n- apply gradient clipping\n\nbtw, the batch size is set to 16.\nAnd I also follow the setting as @nbroad suggested in this [thread](https://www.kaggle.com/competitions/benetech-making-graphs-accessible/discussion/406250).\n\nDespite these steps, the problem persists. Any suggestions or insights would be greatly appreciated.\n\n", "2287615": "Which version of the `transformers` library are you using?\nThere was a change in `v4.29.0` that significantly improves Pix2Struct models training: https://github.com/huggingface/transformers/releases/tag/v4.29.0 (See first paragraph of **Breaking changes**)\nSo if you are using a version before that, this is probably the reason.", "2504858": "Can we use pix2struct for document parsing? (if yes, can you please! provide me a direct link to source code or information)", "2291618": "Does anywhere here also encounter the problem with deplot where it predicts some extra data points especially in line charts?", "2291318": "Do you use scatter data? I don't use it and the loss is really smoth. I setup like you with lr 3e-5, bs 4 ", "2287522": "Do the Nan loss happen in second epoch? R u using the code similar to public notebook on training Pix2Struct?", "2287558": "" } }