![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F3575309%2F90ce092fb07bc284e03991bcf06b5f2e%2FWB%20Chart%205_2_2023%2010_28_00%20PM.png?generation=1683091692082912&alt=media\")
You should read this thread: https://github.com/huggingface/transformers/issues/22903
\ntldr: make sure to set model.config.text_config.is_decoder=True and don't use label smoothing. It should produce 60+ scores then
If you do pip install git+https://github.com/huggingface/transformers.git, then it should work fine without any modifications
You should read this thread: https://github.com/huggingface/transformers/issues/22903
\ntldr: make sure to set model.config.text_config.is_decoder=True and don't use label smoothing. It should produce 60+ scores then
If you do pip install git+https://github.com/huggingface/transformers.git, then it should work fine without any modifications
Thanks for sharing @nbroad !
\nHave you tired inference with Pix2Struct, MatCha or DePlot? If so were the results reasonable?
\nKeep up with the great work🔥💪
", "rawMarkdown": "Thanks for sharing @nbroad !\n\nHave you tired inference with Pix2Struct, MatCha or DePlot? If so were the results reasonable?\n\nKeep up with the great work🔥💪", "votes": 1, "replies": [ { "id": 2301462, "postDate": "2023-06-14T00:59:25.973Z", "content": "My current score 0.65 is using matcha
", "rawMarkdown": "My current score 0.65 is using matcha", "votes": 2, "replies": [ { "id": 2301651, "postDate": "2023-06-14T04:39:49.860Z", "content": "We are using Matcha for our submission as well. Hyperparameter tuning is a massive pain but it can bring good scores
", "rawMarkdown": "We are using Matcha for our submission as well. Hyperparameter tuning is a massive pain but it can bring good scores", "votes": 2 }, { "id": 2309834, "postDate": "2023-06-19T23:28:39.023Z", "content": "Really interesting, @nbroad @cody11null !
\nI have been playing out with DePlot, but have not explored Matcha yet…. Curious to see how you guys approached this challenge!
", "rawMarkdown": "Really interesting, @nbroad @cody11null ! \n\nI have been playing out with DePlot, but have not explored Matcha yet.... Curious to see how you guys approached this challenge!" } ] } ] }, { "id": 2280277, "postDate": "2023-05-30T02:40:18.673Z", "content": "Thanks for your sharing! @nbroad
\ntried matcha models & pix2struct, only got highest 0.6 cv in local.
\ndo you pass any prompts to model? (just like deplot inputs = processor(images=image, text=\"Generate underlying data table of the figure below:\", return_tensors=\"pt\"))
", "rawMarkdown": "Thanks for your sharing! @nbroad \n\ntried matcha models & pix2struct, only got highest 0.6 cv in local.\n- lr 4e-5\n- epochs 2\n- batch size 2\n- Adam\n- no prompt\n\ndo you pass any prompts to model? (just like deplot inputs = processor(images=image, text=\"Generate underlying data table of the figure below:\", return_tensors=\"pt\"))", "votes": 1, "replies": [ { "id": 2301652, "postDate": "2023-06-14T04:40:32.450Z", "content": "You should be able to do more than 2 epochs, though I am surprised that 2 only bring .6 CV
", "rawMarkdown": "You should be able to do more than 2 epochs, though I am surprised that 2 only bring .6 CV", "votes": 1 }, { "id": 2305356, "postDate": "2023-06-16T16:11:45.253Z", "rawMarkdown": "", "votes": 1, "isDeleted": true, "replies": [ { "id": 2305359, "postDate": "2023-06-16T16:14:28.647Z", "rawMarkdown": "", "isDeleted": true } ] } ] }, { "id": 2273592, "postDate": "2023-05-25T09:03:46.760Z", "content": "may I know your MatCha inference speed, my model inference is too slow and always time out.
", "rawMarkdown": "may I know your MatCha inference speed, my model inference is too slow and always time out.", "votes": 1, "replies": [ { "id": 2273890, "postDate": "2023-05-25T13:39:45.480Z", "content": "My training is taking close to 7 hours and my inference on LB is about 2.5 hours I believe.
", "rawMarkdown": "My training is taking close to 7 hours and my inference on LB is about 2.5 hours I believe. ", "votes": 3, "replies": [ { "id": 2279763, "postDate": "2023-05-29T15:53:57.110Z", "content": "training is 6.5 hours, inference is about 2 hours
", "rawMarkdown": "training is 6.5 hours, inference is about 2 hours", "votes": 2 }, { "id": 2281613, "postDate": "2023-05-31T03:10:05.127Z", "rawMarkdown": "", "isDeleted": true }, { "id": 2283762, "postDate": "2023-06-01T13:35:55.540Z", "rawMarkdown": "", "isDeleted": true }, { "id": 2285235, "postDate": "2023-06-02T14:49:52.543Z", "content": "Really? Shouldn't deplot be a better option since it is fine-tuned on matcha itself and it specifically trained for plot-to-table?
", "rawMarkdown": "Really? Shouldn't deplot be a better option since it is fine-tuned on matcha itself and it specifically trained for plot-to-table?" } ] } ] }, { "id": 2265780, "postDate": "2023-05-19T14:01:40.427Z", "content": "how will you download as the submission is without the internet?
", "rawMarkdown": "how will you download as the submission is without the internet?\n", "votes": 1, "replies": [ { "id": 2265819, "postDate": "2023-05-19T14:29:35.510Z", "content": "There is a good discussion here that shows how to do this.
", "rawMarkdown": "There is a good discussion [here](https://www.kaggle.com/competitions/benetech-making-graphs-accessible/discussion/410130) that shows how to do this.", "votes": 1, "replies": [ { "id": 2300943, "postDate": "2023-06-13T14:23:42.070Z", "content": "Thanks for sharing @brendanartley !
", "rawMarkdown": "Thanks for sharing @brendanartley !" } ] } ] }, { "id": 2255915, "postDate": "2023-05-12T05:18:37.637Z", "content": "Hi Nicholas, did you try to submit a notebook that uses the DePlot model? I found that the model is trying to download something from the Internet and this will cause the run to fail. Did you figure out a way to solve this problem?
", "rawMarkdown": "Hi Nicholas, did you try to submit a notebook that uses the DePlot model? I found that the model is trying to download something from the Internet and this will cause the run to fail. Did you figure out a way to solve this problem? ", "votes": 1, "replies": [ { "id": 2257019, "postDate": "2023-05-12T23:40:53.747Z", "content": "deplot downloads the font file with the following code
\nhttps://github.com/huggingface/transformers/blob/cf11493dce0a1d22446efe0d6c4ade02fd928e50/src/transformers/models/pix2struct/image _processing_pix2struct.py#L129
\nHere is the file that is being loaded.
\nhttps://huggingface.co/ybelkada/fonts
If you upload the above file to Datasets, you can use it on Internet Off.
", "rawMarkdown": "deplot downloads the font file with the following code\n\nhttps://github.com/huggingface/transformers/blob/cf11493dce0a1d22446efe0d6c4ade02fd928e50/src/transformers/models/pix2struct/image _processing_pix2struct.py#L129\n\nHere is the file that is being loaded.\nhttps://huggingface.co/ybelkada/fonts\n\nIf you upload the above file to Datasets, you can use it on Internet Off.", "votes": 3, "replies": [ { "id": 2302774, "postDate": "2023-06-14T19:21:14.370Z", "content": "thank you!! I spend an hour attempting to fix the bug…
", "rawMarkdown": "thank you!! I spend an hour attempting to fix the bug..." }, { "id": 2302911, "postDate": "2023-06-15T00:46:58.257Z", "content": "You can pass the path to the file when you call the processor.
\nprocessor(images=images, text=text, font_path=\"path/to/Arial.TTF\")
Hi @jiayoufan666 if you use Pix2StructProcessor.from_pretrained(..., is_vqa=False) it will not load the font since it doesn't need to call the render_header method when preprocessing, which includes the part of code that downloads the .TTF file.
Hi Sarvazyan, Could you give some hints regarding the training using Pix2StructProcessor.from_pretrained().
", "rawMarkdown": "Hi Sarvazyan, Could you give some hints regarding the training using Pix2StructProcessor.from_pretrained()." }, { "id": 2277461, "postDate": "2023-05-27T19:25:20.553Z", "content": "@pksamantaray , this notebook shows how to train pix2struct/matcha/deplot models
", "rawMarkdown": "@pksamantaray , [this notebook](https://www.kaggle.com/code/heyytanay/training-pix2struct-pytorch-amp-w-b) shows how to train pix2struct/matcha/deplot models" } ] } ] }, { "id": 2241852, "postDate": "2023-05-01T21:50:59.397Z", "content": "Thanks for sharing. Have you tried it yet?
", "rawMarkdown": "Thanks for sharing. Have you tried it yet?", "votes": 1, "replies": [ { "id": 2241941, "postDate": "2023-05-02T01:03:22.660Z", "content": "Yes. It works
", "rawMarkdown": "Yes. It works", "votes": 1 } ] }, { "id": 2242142, "postDate": "2023-05-02T04:09:15.427Z", "content": "Thanks. Do these models work better than donut?
", "rawMarkdown": "Thanks. Do these models work better than donut?", "votes": 2, "replies": [ { "id": 2242154, "postDate": "2023-05-02T04:26:21.837Z", "content": "Yes, my donut approach CV was about 0.44 and my CV for matcha was close to 0.7. I didn't submit for matcha, but my leaderboard score for donut was very close to the CV.
", "rawMarkdown": "Yes, my donut approach CV was about 0.44 and my CV for matcha was close to 0.7. I didn't submit for matcha, but my leaderboard score for donut was very close to the CV.", "votes": 4, "replies": [ { "id": 2243684, "postDate": "2023-05-03T04:57:06.707Z", "content": "How about your training loss after fixing the bug? Mine was very close to 0 but only scored 0.
", "rawMarkdown": "How about your training loss after fixing the bug? Mine was very close to 0 but only scored 0.", "votes": 1 }, { "id": 2243701, "postDate": "2023-05-03T05:28:27.360Z", "content": "
I am trying to train deplot/ matcha with or without the fix (The latest version seems to have some of the bugs fixed , I've checked the source code)
\nHowever the loss starts around 30 goes down to near 1 (once it hit almost 0.9 ) but also with very bad results !
\nStill I think there is something wrong !
In my strategy , I pretty much sure I have a puzzle piece missing
", "rawMarkdown": "In my strategy , I pretty much sure I have a puzzle piece missing", "votes": 1 }, { "id": 2244306, "postDate": "2023-05-03T15:05:05.537Z", "content": "Try installing from source pip install git+https://github.com/huggingface/transformers.git
\n\n\n
@nbroad Will you plan to submit your new model? I am curious about how it performs on the test dataset.
", "rawMarkdown": "> \n\n@nbroad Will you plan to submit your new model? I am curious about how it performs on the test dataset.", "votes": 1 }, { "id": 2244954, "postDate": "2023-05-04T03:28:08.567Z", "content": "Sure I can submit
", "rawMarkdown": "Sure I can submit", "votes": 1 }, { "id": 2245835, "postDate": "2023-05-04T16:18:37.360Z", "content": "@nbroad is it 800x2 = 1600 data points or 3/4 data?
", "rawMarkdown": "@nbroad is it 800x2 = 1600 data points or 3/4 data?", "votes": 1 }, { "id": 2246047, "postDate": "2023-05-04T19:47:30.837Z", "content": "@asalhi same here, tried training pix2struct for a couple of epochs, loss went ~1.0 and when checking the generated texts it was always outputting nearly the same text.
", "rawMarkdown": "@asalhi same here, tried training pix2struct for a couple of epochs, loss went ~1.0 and when checking the generated texts it was always outputting nearly the same text.", "votes": 1 }, { "id": 2246051, "postDate": "2023-05-04T19:54:47.207Z", "content": "@seshurajup, what are you referring to?
", "rawMarkdown": "@seshurajup, what are you referring to?" }, { "id": 2246124, "postDate": "2023-05-04T22:32:29.383Z", "content": "i mean from the training loss
Ah. I trained on all data except for scatter plots because the model does poorly on that anyway and I wanted it to go faster.
\nThe step corresponds to the logging step which was 100 times per epoch
", "rawMarkdown": "Ah. I trained on all data except for scatter plots because the model does poorly on that anyway and I wanted it to go faster. \n\nThe step corresponds to the logging step which was 100 times per epoch", "votes": 2 }, { "id": 2246229, "postDate": "2023-05-05T02:14:24.577Z", "rawMarkdown": "", "isDeleted": true }, { "id": 2246231, "postDate": "2023-05-05T02:18:06.753Z", "content": "Try using bf16
", "rawMarkdown": "Try using bf16" }, { "id": 2246246, "postDate": "2023-05-05T02:34:38.347Z", "content": "Thank you. I will try it.
", "rawMarkdown": "Thank you. I will try it." }, { "id": 2246363, "postDate": "2023-05-05T05:11:42.507Z", "content": "\nI keep getting submission errors (submission file is in the wrong format) when I submit 😔
\nThe model outputs on a subset of the training data and on the test set look good though
\nMaybe @ryanholbrook can kindly offer some suggestions for what is going wrong 🙏
\nIt's probably something silly 😅
😣 This happens a lot to me. 80% of my submissions threw exceptions.
\nThe line chart is extremely hard to predict because the number of points is not consistent with the number of x values while it should be. Otherwise, no one knows how many points are in tin the graph.
Sorry to hear you're having trouble. I'll try to take a look if I can find some time. (Not trivial to look it up, unfortunately.) For now, I might suggest trying to validate the actual data type based on the predicted chart type. If the chart type implies that the values should be numeric, make sure the values can actually be parsed as numbers. Maybe something like: try: float(x); except: 0.0.
Since dot plots can be ambiguous, maybe default to numeric on those since numeric can also pass as a string. Unfortunately, dot plots aren't as well represented in the training data which might be why it's passing there but not on submission if that's the culprit.
", "rawMarkdown": "Sorry to hear you're having trouble. I'll try to take a look if I can find some time. (Not trivial to look it up, unfortunately.) For now, I might suggest trying to validate the actual data type based on the predicted chart type. If the chart type implies that the values should be numeric, make sure the values can actually be parsed as numbers. Maybe something like: `try: float(x); except: 0.0`.\n\nSince dot plots can be ambiguous, maybe default to numeric on those since numeric can also pass as a string. Unfortunately, dot plots aren't as well represented in the training data which might be why it's passing there but not on submission if that's the culprit.", "votes": 3 }, { "id": 2246771, "postDate": "2023-05-05T12:59:18.733Z", "rawMarkdown": "", "isDeleted": true }, { "id": 2246957, "postDate": "2023-05-05T15:10:02.590Z", "content": "@zhugds
\nIt looks like models are supposed to choose coordinates corresponding to tick marks on the x-axis for line plots, even though it looks like there are infinite points for some of them.
1) don't worry about looking it up. I'm 95% sure it's a data type issue
\n2) I thought this thread confirmed that the data type didn't matter. If it didn't match, it would just get a score of 0
", "rawMarkdown": "@ryanholbrook ,\n\n1) don't worry about looking it up. I'm 95% sure it's a data type issue\n\n2) I thought [this thread](https://www.kaggle.com/competitions/benetech-making-graphs-accessible/discussion/398528) confirmed that the data type didn't matter. If it didn't match, it would just get a score of 0" }, { "id": 2246960, "postDate": "2023-05-05T15:13:40.637Z", "content": "Yes, from graph conventions:
\nI think annotations are consistent, specially for extracted subset.
", "rawMarkdown": "Yes, from graph conventions:\n- Categorical values always correspond to a tick label on an axis.\n- Line graphs always have categorical x-axis values and numeric y-axis values\n- The values for the x-axis are the x-axis tick labels beneath some portion of the line graph\n\nI think annotations are consistent, specially for extracted subset.", "votes": 3 }, { "id": 2246973, "postDate": "2023-05-05T15:27:15.337Z", "content": "I thought it shouldn't… I'll take a look again.
", "rawMarkdown": "I thought it shouldn't... I'll take a look again." }, { "id": 2247313, "postDate": "2023-05-05T21:39:49.887Z", "content": "@nbroad The error I'm seeing is Submission had an invalid score. Please check your input data for numerical errors (e.g. \"NaN\", division by zero, etc).
This actually looks like it's not a data type problem, which would have given a different error (an Evaluation Exception). I'm thinking the metric is processing the submission okay, but the leaderboard is rejecting it for some reason.
Some things to check:
\nI tested the metric for robustness to the last two, but it couldn't hurt. (Those conditions are never true of the ground truth.)
", "rawMarkdown": "@nbroad The error I'm seeing is `Submission had an invalid score. Please check your input data for numerical errors (e.g. \"NaN\", division by zero, etc)`.\n\nThis actually looks like it's not a data type problem, which would have given a different error (an `Evaluation Exception`). I'm thinking the metric is processing the submission okay, but the leaderboard is rejecting it for some reason.\n\nSome things to check:\n- all numbers are finite (not inf or nan)\n- strings are non-empty\n- numeric predictions are non-constant\n\nI tested the metric for robustness to the last two, but it couldn't hurt. (Those conditions are never true of the ground truth.)", "votes": 1 }, { "id": 2247329, "postDate": "2023-05-05T21:59:21.290Z", "content": "\nThis is very helpful. I didn't consider the possibility of outputting inf or nan.
\nFor number 2, would the following data series create an error?
\n\"0;1;;2;3\"
", "rawMarkdown": "@ryanholbrook,\n\nThis is very helpful. I didn't consider the possibility of outputting inf or nan. \n\nFor number 2, would the following data series create an error?\n\n\"0;1;;2;3\"" }, { "id": 2248122, "postDate": "2023-05-06T14:01:30.307Z", "content": "I don't think so. I tested the metric with entries like ;;; and the Levenshtein metric just treated them as empty strings as it should. My guess is that the first case (inf or nan) is most likely.
I finally got a submission to work. I think it might have predicted something like 13e888 which would turn into inf when converted to a float.
\nMy score is 0.65 which matches CV (0.68) pretty close
", "rawMarkdown": "@zhugds \n\nI finally got a submission to work. I think it might have predicted something like 13e888 which would turn into inf when converted to a float. \n\nMy score is 0.65 which matches CV (0.68) pretty close", "votes": 8 }, { "id": 2248448, "postDate": "2023-05-06T21:16:06.757Z", "content": "Great score
\nI am still trying to train deplot, loss is crazy no matter what I do.
But I will keep digging till I see what I am missing .
", "rawMarkdown": "Great score\nI am still trying to train deplot, loss is crazy no matter what I do.\n\nBut I will keep digging till I see what I am missing .", "votes": 1 }, { "id": 2248464, "postDate": "2023-05-06T21:33:36.177Z", "content": "@nbroad are you eliminate Inf/nan is the solution for the successful submission ? Nice to see CV is sable !!
", "rawMarkdown": "@nbroad are you eliminate Inf/nan is the solution for the successful submission ? Nice to see CV is sable !!", "votes": 1 }, { "id": 2248470, "postDate": "2023-05-06T21:47:15.777Z", "content": "I think I wasn't directly outputting inf or nan, but when the float conversion happened, it turned into inf (e.g. float(3e999))
", "rawMarkdown": "I think I wasn't directly outputting inf or nan, but when the float conversion happened, it turned into inf (e.g. float(3e999))", "votes": 1 }, { "id": 2248482, "postDate": "2023-05-06T22:23:59.897Z", "content": "Awesome! I haven't made it work yet. Not sure what's wrong.
\nLocal validation is good, but not for the leaderboard.
@nbroad It's so weird that my local validation score is 0.6+, but my submission score is 0.42. I split the images 80/20 for training and validation. Do you know what could go wrong?
\nMy initial thought was that the private testing images are different from training images, which was clearly wrong from your case.
\n@asalhi Does pix2struct work for you?
@zhugds
\nThe test set is only extracted images. You might be overfitting to generated images
Thanks for the tip. That just came to my mind. Did you split the images as you did in the donut training?
", "rawMarkdown": "Thanks for the tip. That just came to my mind. Did you split the images as you did in the donut training?" }, { "id": 2249683, "postDate": "2023-05-08T02:37:58.450Z", "content": "I think I was able to train matcha ! , it just finished the trainning with loss chart similar to @nbroad , I tested on solo images and it seems to generate a nice output ! and I directly moved the weights to Kaggle, I am validating now but it seems that it takes a good amount of time to validate !
\n@nbroad can you verify the scoring time ? I guess it will be near 9 hours (unless I play with max_length) ?
![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F6915758%2F387a8ff0e75081b20ec4dd414bddebd4%2FScreen%20Shot%202023-05-08%20at%205.53.58.png?generation=1683514558138367&alt=media\")
Ok so there is a score finally, not the best but it trained !
", "rawMarkdown": "\n\nOk so there is a score finally, not the best but it trained ! ", "votes": 1 }, { "id": 2249808, "postDate": "2023-05-08T06:09:30.040Z", "content": "And leaderboard score is zero 😬
", "rawMarkdown": "And leaderboard score is zero 😬" }, { "id": 2250131, "postDate": "2023-05-08T11:08:33.543Z", "content": "No way. I guess the most important thing for this competition is how to split the data.
", "rawMarkdown": "No way. I guess the most important thing for this competition is how to split the data. " }, { "id": 2250136, "postDate": "2023-05-08T11:13:01.960Z", "content": "I validated on extracted charts only ! I need a day off I guess then I will revision what I did !
", "rawMarkdown": "I validated on extracted charts only ! I need a day off I guess then I will revision what I did ! " }, { "id": 2250370, "postDate": "2023-05-08T14:14:25.130Z", "content": "\nScoring time was about 1 hour
", "rawMarkdown": "@asalhi \n\nScoring time was about 1 hour" }, { "id": 2250558, "postDate": "2023-05-08T16:22:06.883Z", "content": "@nbroad this is fast! My scoring takes 4 hours every time. I would learn about your solution after this ends.
", "rawMarkdown": "@nbroad this is fast! My scoring takes 4 hours every time. I would learn about your solution after this ends." }, { "id": 2250576, "postDate": "2023-05-08T16:42:44.457Z", "content": "@zhugds Check out if you're using correctly eos_token_id inside model.generate.
Can you please elaborate a bit?
\nHere is my code.
output = model.generate(**inputs, \n max_new_tokens=,\n early_stopping=,\n use_cache=,\n pad_token_id=processor.tokenizer.pad_token_id,\n eos_token_id=processor.tokenizer.eos_token_id,\n ) \nThanks.
", "rawMarkdown": "Can you please elaborate a bit?\nHere is my code.\n```python\n output = model.generate(**inputs, \n max_new_tokens=1024,\n early_stopping=True,\n use_cache=True,\n pad_token_id=processor.tokenizer.pad_token_id,\n eos_token_id=processor.tokenizer.eos_token_id,\n ) \n```\nThanks." }, { "id": 2250668, "postDate": "2023-05-08T17:48:33.607Z", "content": "@zhugds It depends, but maybe in training you are not using default eos_token_id from tokanizer (for example </s> for Donut). Maybe in training your last token is for example <Y_END> than eos_token_id should be eos_token_id=processor.tokenizer.convert_tokens_to_ids([\"<Y_END>\"])[0]
Thank you! I will double check my code to make sure I did that.
", "rawMarkdown": "Thank you! I will double check my code to make sure I did that." }, { "id": 2250747, "postDate": "2023-05-08T18:45:37.563Z", "content": "I set my max tokens to 512
", "rawMarkdown": "I set my max tokens to 512", "votes": 1 }, { "id": 2250891, "postDate": "2023-05-08T21:44:36.713Z", "content": "I think I know why I am getting zero as a score, the batch_size was set to 16, which caused the model generator to throw an error (most likely due to out of memory exception), and this then defaulted to a zero value with default vertical bar --> zero score, I fixed this and I just submitted my model for scoring, I will update with the results.
", "rawMarkdown": "I think I know why I am getting zero as a score, the batch_size was set to 16, which caused the model generator to throw an error (most likely due to out of memory exception), and this then defaulted to a zero value with default vertical bar --> zero score, I fixed this and I just submitted my model for scoring, I will update with the results. ", "votes": 1 }, { "id": 2250916, "postDate": "2023-05-08T22:47:38.543Z", "content": "which pretrained model weights are you guys using?
", "rawMarkdown": "which pretrained model weights are you guys using?" }, { "id": 2250942, "postDate": "2023-05-08T23:35:33.793Z", "content": "@alejopaullier We are talking about pix2struct model. https://huggingface.co/google/pix2struct-base
\nNicholas was actually talking about matcha
thanks @zhugds, I've tried training pix2struct base as an image encoder text decoder (but weirdly predicts always the same text). Matcha seems to require a question as it was trained as a VQA model so I wouldnt know how it can fit this competition's problem (e.g: which question should be asked to get data series)
", "rawMarkdown": "thanks @zhugds, I've tried training pix2struct base as an image encoder text decoder (but weirdly predicts always the same text). Matcha seems to require a question as it was trained as a VQA model so I wouldnt know how it can fit this competition's problem (e.g: which question should be asked to get data series)", "votes": 2 }, { "id": 2251029, "postDate": "2023-05-09T02:27:34.837Z", "content": "If you predict the same text, that's what this thread is for.
\nTry pip install git+https://github.com/huggingface/transformers.git as Nicholas mentioned and it should work.
When you create the processor, you can turn vqa off
\nprocessor = AutoProcessor.from_pretrained(\"google/matcha-base\", is_vqa=False)
Hi @ryanholbrook, if you have some time could you kindly have a look at my submissions. I have been implementing many kind of checks for invalid values:
\nAnd yet I keep having Submission Scoring Errors, maybe it's just some simple bug in my code but at this point I would appreciate very much an idea to continue debugging 😅.
", "rawMarkdown": "Hi @ryanholbrook, if you have some time could you kindly have a look at my submissions. I have been implementing many kind of checks for invalid values:\n- check for empty string\n- force convert every value to float to check if they become inf or nan\n- remove ; from generated value\n\nAnd yet I keep having Submission Scoring Errors, maybe it's just some simple bug in my code but at this point I would appreciate very much an idea to continue debugging 😅." }, { "id": 2251502, "postDate": "2023-05-09T12:09:26.840Z", "content": "Hi @qdv206, I'm seeing ERROR: Record 1256 had 2 columns but expected 3. Maybe you forgot to save the index?
@ryanholbrook
\nThat's strange, on the submission generated from the dummy test set it shows 3 columns for me (I ran the notebook on 10000 images of the train set and the generated submission.csv is also 3 columns). Maybe my post processing caused a problem in a particular sample of the hidden test set.
\nThank you very much for your quick reply, it would have taken me a long time to consider this possibility. Now I know where to look 🙏.
Training error is between 0.4 to 0.2 (for 3 epochs), but generated strings are having repeated value or wrong values. how to handle this case? ![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F761268%2Ff416f1d99b09bd36cac646a7847713c4%2FWB%20Chart%2009_05_2023%2021_31_46.png?generation=1683648143994535&alt=media\")
Use bf16 if you can.
\nMy learning rate was 3e-5.
\nI haven't tried using weight decay.
\nI didn't use a scheduler either
Looks like I need to start trying to figure out matcha! I joined the party a little late here, I was able to get my first donut to .46 and I have a second one submitting now but hard to believe it will be a good solution at the end of competition if so many are getting in the .7's already. Never used this so it should be interesting!
", "rawMarkdown": "Looks like I need to start trying to figure out matcha! I joined the party a little late here, I was able to get my first donut to .46 and I have a second one submitting now but hard to believe it will be a good solution at the end of competition if so many are getting in the .7's already. Never used this so it should be interesting! ", "votes": 1 }, { "id": 2253262, "postDate": "2023-05-10T04:28:29.603Z", "content": "Updates regarding Training , I was able to train and infer with pix2struct and matcha , not high scores (0.51, 0.41) but acceptable due the small training I did.
\nInference took at first about 8 hrs then I was able to decrease it to 2.5 hours (most likely I can decrease it more).
", "rawMarkdown": "Updates regarding Training , I was able to train and infer with pix2struct and matcha , not high scores (0.51, 0.41) but acceptable due the small training I did.\n\nInference took at first about 8 hrs then I was able to decrease it to 2.5 hours (most likely I can decrease it more).\n", "votes": 3 }, { "id": 2253646, "postDate": "2023-05-10T11:21:10.367Z", "content": "how did you decrease the scoring time?
", "rawMarkdown": "how did you decrease the scoring time?" }, { "id": 2253822, "postDate": "2023-05-10T13:20:26.583Z", "content": "I dropped the batching and looped over the test images one by one , this dropped the scoring time from 8 hours to 2.5 hours with same score !
\n\n", "rawMarkdown": "I dropped the batching and looped over the test images one by one , this dropped the scoring time from 8 hours to 2.5 hours with same score ! \n\n>```python\nfor im in tqdm(images):\n ids.append(im.stem)\n try:\n output=pix2struct_predict(im, model.cuda(), processor, device)\n #print(output)\n all_generations.append(output)\n except:\n all_generations.append(\"\")\n```\n ", "votes": 7 }, { "id": 2254634, "postDate": "2023-05-11T06:14:54.843Z", "content": "\nim tqdm(images):\n ids.append(im.stem)\n :\n output=pix2struct_predict(im, model.cuda(), processor, device)\n \n all_generations.append(output)\n :\n all_generations.append()\n
Use a single GPU (P100) and fit the max batch size it can accommodate. It should be faster than 2.5 hours.
", "rawMarkdown": "Use a single GPU (P100) and fit the max batch size it can accommodate. It should be faster than 2.5 hours.", "votes": 1 }, { "id": 2262477, "postDate": "2023-05-17T01:26:14.270Z", "content": "could anyone tell me around how many epochs have you train the model?
", "rawMarkdown": "could anyone tell me around how many epochs have you train the model?", "votes": 3 }, { "id": 2262486, "postDate": "2023-05-17T01:42:13.047Z", "content": "I did 10 epochs
", "rawMarkdown": "I did 10 epochs", "votes": 4 }, { "id": 2262493, "postDate": "2023-05-17T01:50:26.977Z", "content": "Did you do that with the full training dataset or some subset for the sake of time?
", "rawMarkdown": "Did you do that with the full training dataset or some subset for the sake of time?", "votes": 1 }, { "id": 2262544, "postDate": "2023-05-17T02:22:31.287Z", "content": "Full data set
", "rawMarkdown": "Full data set ", "votes": 3 }, { "id": 2264953, "postDate": "2023-05-18T22:06:16.377Z", "rawMarkdown": "", "isDeleted": true }, { "id": 2269105, "postDate": "2023-05-22T08:10:16.973Z", "content": "Could you tell us more about this?
", "rawMarkdown": "Could you tell us more about this?" }, { "id": 2277382, "postDate": "2023-05-27T17:06:10.810Z", "content": "Hi @nbroad What is the batch size used for training?
", "rawMarkdown": "Hi @nbroad What is the batch size used for training?", "votes": 1 }, { "id": 2277462, "postDate": "2023-05-27T19:27:14.253Z", "content": "use whatever batch size your GPU can support. Use gradient checkpointing if you want a bigger one. You should be able to get to the same point no matter what your batch size is.
", "rawMarkdown": "use whatever batch size your GPU can support. Use gradient checkpointing if you want a bigger one. [You should be able to get to the same point no matter what your batch size is.](https://github.com/google-research/tuning_playbook#choosing-the-batch-size)", "votes": 3 }, { "id": 2277812, "postDate": "2023-05-28T04:49:54.597Z", "content": "I'm currently using matcha with the new transformers but my loss almost always stays near 6. No matter what I do I tried to change max_patches, fixed some bugs what I thought could've made it better.
\n![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F7210035%2F8884a5cd22fbaca81756980066aa0c1f%2Floss.png?generation=1685249349864560&alt=media\")
Any ideas what I could be doing wrong? Was there a major change you had to adopt when switching from donut to matcha?
", "rawMarkdown": "I'm currently using matcha with the new transformers but my loss almost always stays near 6. No matter what I do I tried to change max_patches, fixed some bugs what I thought could've made it better.\n\n\nAny ideas what I could be doing wrong? Was there a major change you had to adopt when switching from donut to matcha?", "votes": 1 }, { "id": 2277887, "postDate": "2023-05-28T06:33:52.127Z", "content": "Try formatting the text to be like how it was already trained, not like the format my donut notebook
\nSee the public deplot notebook for not details
", "rawMarkdown": "Try formatting the text to be like how it was already trained, not like the format my donut notebook\n\nSee the public deplot notebook for not details", "votes": 1 }, { "id": 2278236, "postDate": "2023-05-28T13:13:29.200Z", "content": "\n\nI dropped the batching and looped over the test images one by one , this dropped the scoring time from 8 hours to 2.5 hours with same score !
\n\n\n\nim tqdm(images):\n ids.append(im.stem)\n :\n output=pix2struct_predict(im, model.cuda(), processor, device)\n \n all_generations.append(output)\n :\n all_generations.append()\n
Hi,@Ali Salhi ,why is looping over the test images one by one faster than batching them?
", "rawMarkdown": "> I dropped the batching and looped over the test images one by one , this dropped the scoring time from 8 hours to 2.5 hours with same score ! \n> \n> >```python\n> for im in tqdm(images):\n> ids.append(im.stem)\n> try:\n> output=pix2struct_predict(im, model.cuda(), processor, device)\n> #print(output)\n> all_generations.append(output)\n> except:\n> all_generations.append(\"\")\n> ```\n\nHi,[@Ali Salhi](https://www.kaggle.com/competitions/benetech-making-graphs-accessible/discussion/406250#2253822) ,why is looping over the test images one by one faster than batching them? " }, { "id": 2278337, "postDate": "2023-05-28T15:35:55.733Z", "content": "From my understanding I did as you said and the ground truth now looks like this
\nTITLE | <chart_title> <0x0A> <axis_title> | <axis_title> <0x0A> y1 | x1 <0x0A> y2 | x2 <0x0A> y3 | x3 ...
And now the loss bumps around 6.5 and 7. I also tried playing with other checkpoints besides base but they perform 2x worse.
\nEdit: Nevermind I found a way to make it work, I removed spaces and changed the text inside the processor, I really had to dig in some discussions and notebooks. My matcha model is running fine with ~ <1 loss but I don't think the loss is very stable, it often varies +- 0.4, is it maybe because I'm using scatterplots too?
", "rawMarkdown": "From my understanding I did as you said and the ground truth now looks like this \n\n```TITLE |Hi,@nbroad,there are 6 checkpoints are available for MatCha. Why did you choose matcha-base for training? It seems that plotqa-v1 or plotqa-v2 is more suitable for the current task.
", "rawMarkdown": "Hi,@[nbroad](https://www.kaggle.com/nbroad),there are 6 checkpoints are available for MatCha. Why did you choose matcha-base for training? It seems that plotqa-v1 or plotqa-v2 is more suitable for the current task.", "votes": 1 }, { "id": 2278872, "postDate": "2023-05-29T03:36:59.847Z", "content": "I think the default task for matcha-base is more similar to this competition than plotqa. I didn't try the other ones, so this is just a hunch.
", "rawMarkdown": "I think the default task for matcha-base is more similar to this competition than plotqa. I didn't try the other ones, so this is just a hunch.", "votes": 1 }, { "id": 2279318, "postDate": "2023-05-29T09:12:36.430Z", "content": "Hi,@chaitanyagiri,thank you for sharing your experience
\nMy loss is also around 6, and I wonder how did you solve it? What does ‘emoved spaces and changed the text inside the processor’ mean?
\n\nuse whatever batch size your GPU can support. Use gradient checkpointing if you want a bigger one. You should be able to get to the same point no matter what your batch size is.
\n
Hi,@nbroad,when i used gradient checkpointing with the following code,an error popped up
\nmodel = Pix2StructForConditionalGeneration.from_pretrained(Config['MODEL_NAME'])\nmodel.gradient_checkpointing_enable()
\nValueError: Pix2StructForConditionalGeneration does not support gradient checkpointing.
\nMay I ask if there is an error in my code?
@areno9516 Look at the deplot public notebook and see the Pix2StructProcessor call. You may have to change the text inside that particular call. Check @evilpsycho42 's comment on this thread.
", "rawMarkdown": "@areno9516 Look at the deplot public notebook and see the Pix2StructProcessor call. You may have to change the text inside that particular call. Check @evilpsycho42 's comment on this thread.", "votes": 1 }, { "id": 2283659, "postDate": "2023-06-01T12:26:39.243Z", "content": "I am training the pix2struct model. The loss is decreasing normally but eventually stabilizes around 0.7, while I see that your training loss stabilizes around 0.3+. Moreover, when I used sample test images for prediction, the results seemed to be very bad. What could be the possible reasons for this?
\n![\"loss\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F2382612%2F66b760cb4545b6f703bf294ff138fc83%2Floss.png?generation=1685622383943646&alt=media\")
Hi,@marbury,when calculating loss, you can ignore padding tokens. After I ignored the padding tokens, the loss has decreased significantly, which may be useful for you. By the way, my model suffered unstable losses during training, and I would like to ask if you have encountered such a situation.
", "rawMarkdown": "Hi,[@marbury](https://www.kaggle.com/marbury),when calculating loss, you can ignore padding tokens. After I ignored the padding tokens, the loss has decreased significantly, which may be useful for you. By the way, my model suffered unstable losses during training, and I would like to ask if you have encountered such a situation.\n\n" }, { "id": 2283985, "postDate": "2023-06-01T16:36:57.607Z", "rawMarkdown": "", "isDeleted": true }, { "id": 2283986, "postDate": "2023-06-01T16:37:52.233Z", "content": "Try this: config.gradient checkpointing=True or model.config.gradient checkpointing=True, I didn't notice much difference. It's possible that I just don't understand something.
", "rawMarkdown": "Try this: config.gradient checkpointing=True or model.config.gradient checkpointing=True, I didn't notice much difference. It's possible that I just don't understand something.", "votes": 1 }, { "id": 2284001, "postDate": "2023-06-01T16:52:02.783Z", "content": "\nmodel.encoder.gradient_checkpointing_enable()
\nmodel.decoder.gradient_checkpointing_enable()
", "rawMarkdown": "@areno9516 \n\nmodel.encoder.gradient_checkpointing_enable()\n\nmodel.decoder.gradient_checkpointing_enable()", "votes": 2 }, { "id": 2284006, "postDate": "2023-06-01T16:58:54.717Z", "content": "@areno9516 @nbroad Do I need to modify the source code in pix2struct transformers to ignore padding tokens when calculate the loss?
\nIt seems that they have already done this in transformers.
\n labels :\n \n labels = labels.to(logits.device)\n loss_fct = nn.CrossEntropyLoss(ignore_index=-, reduction=)\nYou need to set the labels to be -100 to get ignored
", "rawMarkdown": "You need to set the labels to be -100 to get ignored", "votes": 1 }, { "id": 2284596, "postDate": "2023-06-02T07:01:37.843Z", "content": "@nbroad Thanks, I have set the pad_token_id to -100 as you do in your public notebook. However, the loss is still not decreased as expected.
\n![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F2382612%2Fc6dbb72435ecfc3709e3b8b21513f2f4%2Floss.png?generation=1685689292833465&alt=media\")
\n\n\n
Do you include the scatter plot images for training in this train loss?
", "rawMarkdown": "> \n\nDo you include the scatter plot images for training in this train loss?" }, { "id": 2294656, "postDate": "2023-06-10T07:54:12.500Z", "rawMarkdown": "", "isDeleted": true }, { "id": 2294660, "postDate": "2023-06-10T07:57:35.320Z", "rawMarkdown": "", "isDeleted": true }, { "id": 2294663, "postDate": "2023-06-10T07:59:45.023Z", "rawMarkdown": "", "isDeleted": true }, { "id": 2294665, "postDate": "2023-06-10T08:00:44.943Z", "content": "Hi, I was wondering what else have you done besides this method to reduce the scoring time, since my current scoring time is so long that it takes 6 hours
", "rawMarkdown": "Hi, I was wondering what else have you done besides this method to reduce the scoring time, since my current scoring time is so long that it takes 6 hours" }, { "id": 2295823, "postDate": "2023-06-11T09:30:42.597Z", "rawMarkdown": "", "isDeleted": true }, { "id": 2295824, "postDate": "2023-06-11T09:31:27.853Z", "content": "Did you pre-train the model using matcha-plotqa-v2? Is the training labeled xxxyyy or xyxy? The loss never goes down
", "rawMarkdown": "Did you pre-train the model using matcha-plotqa-v2? Is the training labeled xxxyyy or xyxy? The loss never goes down", "votes": 1 }, { "id": 2301417, "postDate": "2023-06-13T22:40:45.200Z", "content": "Hi @areno, I see that you mentioned you had an unstable loss. Do you still have it? If not may I ask what was your workaround for that?
", "rawMarkdown": "Hi @areno, I see that you mentioned you had an unstable loss. Do you still have it? If not may I ask what was your workaround for that?" }, { "id": 2301552, "postDate": "2023-06-14T02:40:46.680Z", "content": "Hi,@chaitanyagiri
\nYou may be able to ignore that. My loss was unstable, with an average value of about 0.2, but fluctuating between 0.05 and 0.6.And the model finally got 0.75 cv and 0.68 lb
My loss fluctuates between 0.8 and 0 and the average is 0.35. I tried increasing lr but it's still the same. I'm using deplot, what about you?
", "rawMarkdown": "My loss fluctuates between 0.8 and 0 and the average is 0.35. I tried increasing lr but it's still the same. I'm using deplot, what about you?", "votes": 1 }, { "id": 2301802, "postDate": "2023-06-14T07:10:17.780Z", "content": "\n\nMy loss fluctuates between 0.8 and 0 and the average is 0.35. I tried increasing lr but it's still the same. I'm using deplot, what about you?
\n
I have tried the Matcha and Deplot models.Their losses are close and Matcha‘s scores are slightly better. My Learning rate is 3e-5
", "rawMarkdown": "> My loss fluctuates between 0.8 and 0 and the average is 0.35. I tried increasing lr but it's still the same. I'm using deplot, what about you?\n\nI have tried the Matcha and Deplot models.Their losses are close and Matcha‘s scores are slightly better. My Learning rate is 3e-5\n", "votes": 1 }, { "id": 2302497, "postDate": "2023-06-14T15:29:06.357Z", "content": "And do you use the competition's dataset or the 500k images one?
", "rawMarkdown": "And do you use the competition's dataset or the 500k images one?" }, { "id": 2302974, "postDate": "2023-06-15T02:10:02.057Z", "content": "\n\nAnd do you use the competition's dataset or the 500k images one?
\n
I used the competition's dataset and 100k images of 500k images
", "rawMarkdown": "> And do you use the competition's dataset or the 500k images one?\n\nI used the competition's dataset and 100k images of 500k images" }, { "id": 2303350, "postDate": "2023-06-15T08:58:33.680Z", "content": "So the 100k images were of scatterplots or stratified the images like 20k from each. And how many epochs do you train for? I kinda notice that my losses doesn't seem to go down after the 2nd
", "rawMarkdown": "So the 100k images were of scatterplots or stratified the images like 20k from each. And how many epochs do you train for? I kinda notice that my losses doesn't seem to go down after the 2nd" }, { "id": 2304446, "postDate": "2023-06-16T02:16:03.080Z", "content": "\n\nSo the 100k images were of scatterplots or stratified the images like 20k from each. And how many epochs do you train for? I kinda notice that my losses doesn't seem to go down after the 2nd
\n
20k for each type of chart. I trained three epochs. Maybe you can try a smaller lr
", "rawMarkdown": "> So the 100k images were of scatterplots or stratified the images like 20k from each. And how many epochs do you train for? I kinda notice that my losses doesn't seem to go down after the 2nd\n\n20k for each type of chart. I trained three epochs. Maybe you can try a smaller lr\n", "votes": 1 } ] }, { "id": 2271435, "postDate": "2023-05-23T22:29:06.937Z", "content": "Thanks everyone for sharing. I have learnt a lot from this discussion.
\n\n\n\nUpdate: I had max patches set to 4..
", "rawMarkdown": "Thanks everyone for sharing. I have learnt a lot from this discussion.\n\n~~After training for 2 epochs, my model just generates a text sequence with a year then float format. Something like this.~~\n\n~~`1949.1 | 100.1 <0x0A> 1950.1 | 100.3 <0x0A> 1952.1 | 100.3 <0x0A> 1954.1 |`...~~\n\n~~I installed from source and tried all the recommendations in this thread. Did anyone else run into this issue?~~\n\nUpdate: I had max patches set to 4.. ", "votes": 1 } ] }, { "id": 2306839, "postDate": "2023-06-17T16:16:25.600Z", "content": "Is there a public code/notebook that is best to use for Matcha model?
\nThanks.
", "rawMarkdown": "Is there a public code/notebook that is best to use for Matcha model? \n\nThanks.\n\n\n" }, { "id": 2254461, "postDate": "2023-05-11T02:36:24.667Z", "rawMarkdown": "", "isDeleted": true } ], "comments": [ { "id": 2300948, "author_name": "Vladimir Simões da Luz Junior", "author_url": "", "post_date": "2023-06-13T14:27:14.523000", "content": "Thanks for sharing @nbroad !
\nHave you tired inference with Pix2Struct, MatCha or DePlot? If so were the results reasonable?
\nKeep up with the great work🔥💪
", "votes": 1, "replies": [ { "id": 2301462, "author_name": "Nicholas Broad", "author_url": "", "post_date": "2023-06-14T00:59:25.973000", "content": "My current score 0.65 is using matcha
", "votes": 2, "replies": [ { "id": 2301651, "author_name": "Cody_Null", "author_url": "", "post_date": "2023-06-14T04:39:49.860000", "content": "We are using Matcha for our submission as well. Hyperparameter tuning is a massive pain but it can bring good scores
", "votes": 2, "replies": [] }, { "id": 2309834, "author_name": "Vladimir Simões da Luz Junior", "author_url": "", "post_date": "2023-06-19T23:28:39.023000", "content": "Really interesting, @nbroad @cody11null !
\nI have been playing out with DePlot, but have not explored Matcha yet…. Curious to see how you guys approached this challenge!
", "votes": 0, "replies": [] } ] } ] }, { "id": 2280277, "author_name": "KKY", "author_url": "", "post_date": "2023-05-30T02:40:18.673000", "content": "Thanks for your sharing! @nbroad
\ntried matcha models & pix2struct, only got highest 0.6 cv in local.
\ndo you pass any prompts to model? (just like deplot inputs = processor(images=image, text=\"Generate underlying data table of the figure below:\", return_tensors=\"pt\"))
", "votes": 1, "replies": [ { "id": 2301652, "author_name": "Cody_Null", "author_url": "", "post_date": "2023-06-14T04:40:32.450000", "content": "You should be able to do more than 2 epochs, though I am surprised that 2 only bring .6 CV
", "votes": 1, "replies": [] }, { "id": 2305356, "author_name": "", "author_url": "", "post_date": "2023-06-16T16:11:45.253000", "content": "", "votes": 1, "replies": [ { "id": 2305359, "author_name": "", "author_url": "", "post_date": "2023-06-16T16:14:28.647000", "content": "", "votes": 0, "replies": [] } ] } ] }, { "id": 2273592, "author_name": "HZM", "author_url": "", "post_date": "2023-05-25T09:03:46.760000", "content": "may I know your MatCha inference speed, my model inference is too slow and always time out.
", "votes": 1, "replies": [ { "id": 2273890, "author_name": "Cody_Null", "author_url": "", "post_date": "2023-05-25T13:39:45.480000", "content": "My training is taking close to 7 hours and my inference on LB is about 2.5 hours I believe.
", "votes": 3, "replies": [ { "id": 2279763, "author_name": "moth", "author_url": "", "post_date": "2023-05-29T15:53:57.110000", "content": "training is 6.5 hours, inference is about 2 hours
", "votes": 2, "replies": [] }, { "id": 2281613, "author_name": "", "author_url": "", "post_date": "2023-05-31T03:10:05.127000", "content": "", "votes": 0, "replies": [] }, { "id": 2283762, "author_name": "", "author_url": "", "post_date": "2023-06-01T13:35:55.540000", "content": "", "votes": 0, "replies": [] }, { "id": 2285235, "author_name": "Chaitanya Giri", "author_url": "", "post_date": "2023-06-02T14:49:52.543000", "content": "Really? Shouldn't deplot be a better option since it is fine-tuned on matcha itself and it specifically trained for plot-to-table?
", "votes": 0, "replies": [] } ] } ] }, { "id": 2265780, "author_name": "pksamantaray", "author_url": "", "post_date": "2023-05-19T14:01:40.427000", "content": "how will you download as the submission is without the internet?
", "votes": 1, "replies": [ { "id": 2265819, "author_name": "Bartley", "author_url": "", "post_date": "2023-05-19T14:29:35.510000", "content": "There is a good discussion here that shows how to do this.
", "votes": 1, "replies": [ { "id": 2300943, "author_name": "Vladimir Simões da Luz Junior", "author_url": "", "post_date": "2023-06-13T14:23:42.070000", "content": "Thanks for sharing @brendanartley !
", "votes": 0, "replies": [] } ] } ] }, { "id": 2255915, "author_name": "jiayoufan666", "author_url": "", "post_date": "2023-05-12T05:18:37.637000", "content": "Hi Nicholas, did you try to submit a notebook that uses the DePlot model? I found that the model is trying to download something from the Internet and this will cause the run to fail. Did you figure out a way to solve this problem?
", "votes": 1, "replies": [ { "id": 2257019, "author_name": "sinchir0", "author_url": "", "post_date": "2023-05-12T23:40:53.747000", "content": "deplot downloads the font file with the following code
\nhttps://github.com/huggingface/transformers/blob/cf11493dce0a1d22446efe0d6c4ade02fd928e50/src/transformers/models/pix2struct/image _processing_pix2struct.py#L129
\nHere is the file that is being loaded.
\nhttps://huggingface.co/ybelkada/fonts
If you upload the above file to Datasets, you can use it on Internet Off.
", "votes": 3, "replies": [ { "id": 2302774, "author_name": "zj yang", "author_url": "", "post_date": "2023-06-14T19:21:14.370000", "content": "thank you!! I spend an hour attempting to fix the bug…
", "votes": 0, "replies": [] }, { "id": 2302911, "author_name": "Nicholas Broad", "author_url": "", "post_date": "2023-06-15T00:46:58.257000", "content": "You can pass the path to the file when you call the processor.
\nprocessor(images=images, text=text, font_path=\"path/to/Arial.TTF\")
Hi @jiayoufan666 if you use Pix2StructProcessor.from_pretrained(..., is_vqa=False) it will not load the font since it doesn't need to call the render_header method when preprocessing, which includes the part of code that downloads the .TTF file.
Hi Sarvazyan, Could you give some hints regarding the training using Pix2StructProcessor.from_pretrained().
", "votes": 0, "replies": [] }, { "id": 2277461, "author_name": "Nicholas Broad", "author_url": "", "post_date": "2023-05-27T19:25:20.553000", "content": "@pksamantaray , this notebook shows how to train pix2struct/matcha/deplot models
", "votes": 0, "replies": [] } ] } ] }, { "id": 2241852, "author_name": "june", "author_url": "", "post_date": "2023-05-01T21:50:59.397000", "content": "Thanks for sharing. Have you tried it yet?
", "votes": 1, "replies": [ { "id": 2241941, "author_name": "Nicholas Broad", "author_url": "", "post_date": "2023-05-02T01:03:22.660000", "content": "Yes. It works
", "votes": 1, "replies": [] } ] }, { "id": 2242142, "author_name": "Homoalways", "author_url": "", "post_date": "2023-05-02T04:09:15.427000", "content": "Thanks. Do these models work better than donut?
", "votes": 2, "replies": [ { "id": 2242154, "author_name": "Nicholas Broad", "author_url": "", "post_date": "2023-05-02T04:26:21.837000", "content": "Yes, my donut approach CV was about 0.44 and my CV for matcha was close to 0.7. I didn't submit for matcha, but my leaderboard score for donut was very close to the CV.
", "votes": 4, "replies": [ { "id": 2243684, "author_name": "Leon", "author_url": "", "post_date": "2023-05-03T04:57:06.707000", "content": "How about your training loss after fixing the bug? Mine was very close to 0 but only scored 0.
", "votes": 1, "replies": [] }, { "id": 2243701, "author_name": "Nicholas Broad", "author_url": "", "post_date": "2023-05-03T05:28:27.360000", "content": "
I am trying to train deplot/ matcha with or without the fix (The latest version seems to have some of the bugs fixed , I've checked the source code)
\nHowever the loss starts around 30 goes down to near 1 (once it hit almost 0.9 ) but also with very bad results !
\nStill I think there is something wrong !
In my strategy , I pretty much sure I have a puzzle piece missing
", "votes": 1, "replies": [] }, { "id": 2244306, "author_name": "Nicholas Broad", "author_url": "", "post_date": "2023-05-03T15:05:05.537000", "content": "Try installing from source pip install git+https://github.com/huggingface/transformers.git
\n\n\n
@nbroad Will you plan to submit your new model? I am curious about how it performs on the test dataset.
", "votes": 1, "replies": [] }, { "id": 2244954, "author_name": "Nicholas Broad", "author_url": "", "post_date": "2023-05-04T03:28:08.567000", "content": "Sure I can submit
", "votes": 1, "replies": [] }, { "id": 2245835, "author_name": "SeshuRaju 🧘♂️", "author_url": "", "post_date": "2023-05-04T16:18:37.360000", "content": "@nbroad is it 800x2 = 1600 data points or 3/4 data?
", "votes": 1, "replies": [] }, { "id": 2246047, "author_name": "moth", "author_url": "", "post_date": "2023-05-04T19:47:30.837000", "content": "@asalhi same here, tried training pix2struct for a couple of epochs, loss went ~1.0 and when checking the generated texts it was always outputting nearly the same text.
", "votes": 1, "replies": [] }, { "id": 2246051, "author_name": "Nicholas Broad", "author_url": "", "post_date": "2023-05-04T19:54:47.207000", "content": "@seshurajup, what are you referring to?
", "votes": 0, "replies": [] }, { "id": 2246124, "author_name": "SeshuRaju 🧘♂️", "author_url": "", "post_date": "2023-05-04T22:32:29.383000", "content": "i mean from the training loss
Ah. I trained on all data except for scatter plots because the model does poorly on that anyway and I wanted it to go faster.
\nThe step corresponds to the logging step which was 100 times per epoch
", "votes": 2, "replies": [] }, { "id": 2246229, "author_name": "", "author_url": "", "post_date": "2023-05-05T02:14:24.577000", "content": "", "votes": 0, "replies": [] }, { "id": 2246231, "author_name": "Nicholas Broad", "author_url": "", "post_date": "2023-05-05T02:18:06.753000", "content": "Try using bf16
", "votes": 0, "replies": [] }, { "id": 2246246, "author_name": "Leon", "author_url": "", "post_date": "2023-05-05T02:34:38.347000", "content": "Thank you. I will try it.
", "votes": 0, "replies": [] }, { "id": 2246363, "author_name": "Nicholas Broad", "author_url": "", "post_date": "2023-05-05T05:11:42.507000", "content": "\nI keep getting submission errors (submission file is in the wrong format) when I submit 😔
\nThe model outputs on a subset of the training data and on the test set look good though
\nMaybe @ryanholbrook can kindly offer some suggestions for what is going wrong 🙏
\nIt's probably something silly 😅
😣 This happens a lot to me. 80% of my submissions threw exceptions.
\nThe line chart is extremely hard to predict because the number of points is not consistent with the number of x values while it should be. Otherwise, no one knows how many points are in tin the graph.
Sorry to hear you're having trouble. I'll try to take a look if I can find some time. (Not trivial to look it up, unfortunately.) For now, I might suggest trying to validate the actual data type based on the predicted chart type. If the chart type implies that the values should be numeric, make sure the values can actually be parsed as numbers. Maybe something like: try: float(x); except: 0.0.
Since dot plots can be ambiguous, maybe default to numeric on those since numeric can also pass as a string. Unfortunately, dot plots aren't as well represented in the training data which might be why it's passing there but not on submission if that's the culprit.
", "votes": 3, "replies": [] }, { "id": 2246771, "author_name": "", "author_url": "", "post_date": "2023-05-05T12:59:18.733000", "content": "", "votes": 0, "replies": [] }, { "id": 2246957, "author_name": "Nicholas Broad", "author_url": "", "post_date": "2023-05-05T15:10:02.590000", "content": "@zhugds
\nIt looks like models are supposed to choose coordinates corresponding to tick marks on the x-axis for line plots, even though it looks like there are infinite points for some of them.
1) don't worry about looking it up. I'm 95% sure it's a data type issue
\n2) I thought this thread confirmed that the data type didn't matter. If it didn't match, it would just get a score of 0
", "votes": 0, "replies": [] }, { "id": 2246960, "author_name": "Raja Biswas", "author_url": "", "post_date": "2023-05-05T15:13:40.637000", "content": "Yes, from graph conventions:
\nI think annotations are consistent, specially for extracted subset.
", "votes": 3, "replies": [] }, { "id": 2246973, "author_name": "Ryan Holbrook", "author_url": "", "post_date": "2023-05-05T15:27:15.337000", "content": "I thought it shouldn't… I'll take a look again.
", "votes": 0, "replies": [] }, { "id": 2247313, "author_name": "Ryan Holbrook", "author_url": "", "post_date": "2023-05-05T21:39:49.887000", "content": "@nbroad The error I'm seeing is Submission had an invalid score. Please check your input data for numerical errors (e.g. \"NaN\", division by zero, etc).
This actually looks like it's not a data type problem, which would have given a different error (an Evaluation Exception). I'm thinking the metric is processing the submission okay, but the leaderboard is rejecting it for some reason.
Some things to check:
\nI tested the metric for robustness to the last two, but it couldn't hurt. (Those conditions are never true of the ground truth.)
", "votes": 1, "replies": [] }, { "id": 2247329, "author_name": "Nicholas Broad", "author_url": "", "post_date": "2023-05-05T21:59:21.290000", "content": "\nThis is very helpful. I didn't consider the possibility of outputting inf or nan.
\nFor number 2, would the following data series create an error?
\n\"0;1;;2;3\"
", "votes": 0, "replies": [] }, { "id": 2248122, "author_name": "Ryan Holbrook", "author_url": "", "post_date": "2023-05-06T14:01:30.307000", "content": "I don't think so. I tested the metric with entries like ;;; and the Levenshtein metric just treated them as empty strings as it should. My guess is that the first case (inf or nan) is most likely.
I finally got a submission to work. I think it might have predicted something like 13e888 which would turn into inf when converted to a float.
\nMy score is 0.65 which matches CV (0.68) pretty close
", "votes": 8, "replies": [] }, { "id": 2248448, "author_name": "Ali", "author_url": "", "post_date": "2023-05-06T21:16:06.757000", "content": "Great score
\nI am still trying to train deplot, loss is crazy no matter what I do.
But I will keep digging till I see what I am missing .
", "votes": 1, "replies": [] }, { "id": 2248464, "author_name": "SeshuRaju 🧘♂️", "author_url": "", "post_date": "2023-05-06T21:33:36.177000", "content": "@nbroad are you eliminate Inf/nan is the solution for the successful submission ? Nice to see CV is sable !!
", "votes": 1, "replies": [] }, { "id": 2248470, "author_name": "Nicholas Broad", "author_url": "", "post_date": "2023-05-06T21:47:15.777000", "content": "I think I wasn't directly outputting inf or nan, but when the float conversion happened, it turned into inf (e.g. float(3e999))
", "votes": 1, "replies": [] }, { "id": 2248482, "author_name": "june", "author_url": "", "post_date": "2023-05-06T22:23:59.897000", "content": "Awesome! I haven't made it work yet. Not sure what's wrong.
\nLocal validation is good, but not for the leaderboard.
@nbroad It's so weird that my local validation score is 0.6+, but my submission score is 0.42. I split the images 80/20 for training and validation. Do you know what could go wrong?
\nMy initial thought was that the private testing images are different from training images, which was clearly wrong from your case.
\n@asalhi Does pix2struct work for you?
@zhugds
\nThe test set is only extracted images. You might be overfitting to generated images
Thanks for the tip. That just came to my mind. Did you split the images as you did in the donut training?
", "votes": 0, "replies": [] }, { "id": 2249683, "author_name": "Ali", "author_url": "", "post_date": "2023-05-08T02:37:58.450000", "content": "I think I was able to train matcha ! , it just finished the trainning with loss chart similar to @nbroad , I tested on solo images and it seems to generate a nice output ! and I directly moved the weights to Kaggle, I am validating now but it seems that it takes a good amount of time to validate !
\n@nbroad can you verify the scoring time ? I guess it will be near 9 hours (unless I play with max_length) ?
Ok so there is a score finally, not the best but it trained !
", "votes": 1, "replies": [] }, { "id": 2249808, "author_name": "Ali", "author_url": "", "post_date": "2023-05-08T06:09:30.040000", "content": "And leaderboard score is zero 😬
", "votes": 0, "replies": [] }, { "id": 2250131, "author_name": "june", "author_url": "", "post_date": "2023-05-08T11:08:33.543000", "content": "No way. I guess the most important thing for this competition is how to split the data.
", "votes": 0, "replies": [] }, { "id": 2250136, "author_name": "Ali", "author_url": "", "post_date": "2023-05-08T11:13:01.960000", "content": "I validated on extracted charts only ! I need a day off I guess then I will revision what I did !
", "votes": 0, "replies": [] }, { "id": 2250370, "author_name": "Nicholas Broad", "author_url": "", "post_date": "2023-05-08T14:14:25.130000", "content": "\nScoring time was about 1 hour
", "votes": 0, "replies": [] }, { "id": 2250558, "author_name": "june", "author_url": "", "post_date": "2023-05-08T16:22:06.883000", "content": "@nbroad this is fast! My scoring takes 4 hours every time. I would learn about your solution after this ends.
", "votes": 0, "replies": [] }, { "id": 2250576, "author_name": "Roman Khomenko", "author_url": "", "post_date": "2023-05-08T16:42:44.457000", "content": "@zhugds Check out if you're using correctly eos_token_id inside model.generate.
Can you please elaborate a bit?
\nHere is my code.
output = model.generate(**inputs, \n max_new_tokens=,\n early_stopping=,\n use_cache=,\n pad_token_id=processor.tokenizer.pad_token_id,\n eos_token_id=processor.tokenizer.eos_token_id,\n ) \nThanks.
", "votes": 0, "replies": [] }, { "id": 2250668, "author_name": "Roman Khomenko", "author_url": "", "post_date": "2023-05-08T17:48:33.607000", "content": "@zhugds It depends, but maybe in training you are not using default eos_token_id from tokanizer (for example </s> for Donut). Maybe in training your last token is for example <Y_END> than eos_token_id should be eos_token_id=processor.tokenizer.convert_tokens_to_ids([\"<Y_END>\"])[0]
Thank you! I will double check my code to make sure I did that.
", "votes": 0, "replies": [] }, { "id": 2250747, "author_name": "Nicholas Broad", "author_url": "", "post_date": "2023-05-08T18:45:37.563000", "content": "I set my max tokens to 512
", "votes": 1, "replies": [] }, { "id": 2250891, "author_name": "Ali", "author_url": "", "post_date": "2023-05-08T21:44:36.713000", "content": "I think I know why I am getting zero as a score, the batch_size was set to 16, which caused the model generator to throw an error (most likely due to out of memory exception), and this then defaulted to a zero value with default vertical bar --> zero score, I fixed this and I just submitted my model for scoring, I will update with the results.
", "votes": 1, "replies": [] }, { "id": 2250916, "author_name": "moth", "author_url": "", "post_date": "2023-05-08T22:47:38.543000", "content": "which pretrained model weights are you guys using?
", "votes": 0, "replies": [] }, { "id": 2250942, "author_name": "june", "author_url": "", "post_date": "2023-05-08T23:35:33.793000", "content": "@alejopaullier We are talking about pix2struct model. https://huggingface.co/google/pix2struct-base
\nNicholas was actually talking about matcha
thanks @zhugds, I've tried training pix2struct base as an image encoder text decoder (but weirdly predicts always the same text). Matcha seems to require a question as it was trained as a VQA model so I wouldnt know how it can fit this competition's problem (e.g: which question should be asked to get data series)
", "votes": 2, "replies": [] }, { "id": 2251029, "author_name": "june", "author_url": "", "post_date": "2023-05-09T02:27:34.837000", "content": "If you predict the same text, that's what this thread is for.
\nTry pip install git+https://github.com/huggingface/transformers.git as Nicholas mentioned and it should work.
When you create the processor, you can turn vqa off
\nprocessor = AutoProcessor.from_pretrained(\"google/matcha-base\", is_vqa=False)
Hi @ryanholbrook, if you have some time could you kindly have a look at my submissions. I have been implementing many kind of checks for invalid values:
\nAnd yet I keep having Submission Scoring Errors, maybe it's just some simple bug in my code but at this point I would appreciate very much an idea to continue debugging 😅.
", "votes": 0, "replies": [] }, { "id": 2251502, "author_name": "Ryan Holbrook", "author_url": "", "post_date": "2023-05-09T12:09:26.840000", "content": "Hi @qdv206, I'm seeing ERROR: Record 1256 had 2 columns but expected 3. Maybe you forgot to save the index?
@ryanholbrook
\nThat's strange, on the submission generated from the dummy test set it shows 3 columns for me (I ran the notebook on 10000 images of the train set and the generated submission.csv is also 3 columns). Maybe my post processing caused a problem in a particular sample of the hidden test set.
\nThank you very much for your quick reply, it would have taken me a long time to consider this possibility. Now I know where to look 🙏.
Training error is between 0.4 to 0.2 (for 3 epochs), but generated strings are having repeated value or wrong values. how to handle this case?
Use bf16 if you can.
\nMy learning rate was 3e-5.
\nI haven't tried using weight decay.
\nI didn't use a scheduler either
Looks like I need to start trying to figure out matcha! I joined the party a little late here, I was able to get my first donut to .46 and I have a second one submitting now but hard to believe it will be a good solution at the end of competition if so many are getting in the .7's already. Never used this so it should be interesting!
", "votes": 1, "replies": [] }, { "id": 2253262, "author_name": "Ali", "author_url": "", "post_date": "2023-05-10T04:28:29.603000", "content": "Updates regarding Training , I was able to train and infer with pix2struct and matcha , not high scores (0.51, 0.41) but acceptable due the small training I did.
\nInference took at first about 8 hrs then I was able to decrease it to 2.5 hours (most likely I can decrease it more).
", "votes": 3, "replies": [] }, { "id": 2253646, "author_name": "Benedikt Droste", "author_url": "", "post_date": "2023-05-10T11:21:10.367000", "content": "how did you decrease the scoring time?
", "votes": 0, "replies": [] }, { "id": 2253822, "author_name": "Ali", "author_url": "", "post_date": "2023-05-10T13:20:26.583000", "content": "I dropped the batching and looped over the test images one by one , this dropped the scoring time from 8 hours to 2.5 hours with same score !
\n\n", "votes": 7, "replies": [] }, { "id": 2254634, "author_name": "Shai", "author_url": "", "post_date": "2023-05-11T06:14:54.843000", "content": "\nim tqdm(images):\n ids.append(im.stem)\n :\n output=pix2struct_predict(im, model.cuda(), processor, device)\n \n all_generations.append(output)\n :\n all_generations.append()\n
Use a single GPU (P100) and fit the max batch size it can accommodate. It should be faster than 2.5 hours.
", "votes": 1, "replies": [] }, { "id": 2262477, "author_name": "moth", "author_url": "", "post_date": "2023-05-17T01:26:14.270000", "content": "could anyone tell me around how many epochs have you train the model?
", "votes": 3, "replies": [] }, { "id": 2262486, "author_name": "Nicholas Broad", "author_url": "", "post_date": "2023-05-17T01:42:13.047000", "content": "I did 10 epochs
", "votes": 4, "replies": [] }, { "id": 2262493, "author_name": "Cody_Null", "author_url": "", "post_date": "2023-05-17T01:50:26.977000", "content": "Did you do that with the full training dataset or some subset for the sake of time?
", "votes": 1, "replies": [] }, { "id": 2262544, "author_name": "Nicholas Broad", "author_url": "", "post_date": "2023-05-17T02:22:31.287000", "content": "Full data set
", "votes": 3, "replies": [] }, { "id": 2264953, "author_name": "", "author_url": "", "post_date": "2023-05-18T22:06:16.377000", "content": "", "votes": 0, "replies": [] }, { "id": 2269105, "author_name": "Zaakcii Ru", "author_url": "", "post_date": "2023-05-22T08:10:16.973000", "content": "Could you tell us more about this?
", "votes": 0, "replies": [] }, { "id": 2277382, "author_name": "Givan", "author_url": "", "post_date": "2023-05-27T17:06:10.810000", "content": "Hi @nbroad What is the batch size used for training?
", "votes": 1, "replies": [] }, { "id": 2277462, "author_name": "Nicholas Broad", "author_url": "", "post_date": "2023-05-27T19:27:14.253000", "content": "use whatever batch size your GPU can support. Use gradient checkpointing if you want a bigger one. You should be able to get to the same point no matter what your batch size is.
", "votes": 3, "replies": [] }, { "id": 2277812, "author_name": "Chaitanya Giri", "author_url": "", "post_date": "2023-05-28T04:49:54.597000", "content": "I'm currently using matcha with the new transformers but my loss almost always stays near 6. No matter what I do I tried to change max_patches, fixed some bugs what I thought could've made it better.
\n
Any ideas what I could be doing wrong? Was there a major change you had to adopt when switching from donut to matcha?
", "votes": 1, "replies": [] }, { "id": 2277887, "author_name": "Nicholas Broad", "author_url": "", "post_date": "2023-05-28T06:33:52.127000", "content": "Try formatting the text to be like how it was already trained, not like the format my donut notebook
\nSee the public deplot notebook for not details
", "votes": 1, "replies": [] }, { "id": 2278236, "author_name": "areno", "author_url": "", "post_date": "2023-05-28T13:13:29.200000", "content": "\n\nI dropped the batching and looped over the test images one by one , this dropped the scoring time from 8 hours to 2.5 hours with same score !
\n\n\n\nim tqdm(images):\n ids.append(im.stem)\n :\n output=pix2struct_predict(im, model.cuda(), processor, device)\n \n all_generations.append(output)\n :\n all_generations.append()\n
Hi,@Ali Salhi ,why is looping over the test images one by one faster than batching them?
", "votes": 0, "replies": [] }, { "id": 2278337, "author_name": "Chaitanya Giri", "author_url": "", "post_date": "2023-05-28T15:35:55.733000", "content": "From my understanding I did as you said and the ground truth now looks like this
\nTITLE | <chart_title> <0x0A> <axis_title> | <axis_title> <0x0A> y1 | x1 <0x0A> y2 | x2 <0x0A> y3 | x3 ...
And now the loss bumps around 6.5 and 7. I also tried playing with other checkpoints besides base but they perform 2x worse.
\nEdit: Nevermind I found a way to make it work, I removed spaces and changed the text inside the processor, I really had to dig in some discussions and notebooks. My matcha model is running fine with ~ <1 loss but I don't think the loss is very stable, it often varies +- 0.4, is it maybe because I'm using scatterplots too?
", "votes": 1, "replies": [] }, { "id": 2278869, "author_name": "areno", "author_url": "", "post_date": "2023-05-29T03:29:08.223000", "content": "Hi,@nbroad,there are 6 checkpoints are available for MatCha. Why did you choose matcha-base for training? It seems that plotqa-v1 or plotqa-v2 is more suitable for the current task.
", "votes": 1, "replies": [] }, { "id": 2278872, "author_name": "Nicholas Broad", "author_url": "", "post_date": "2023-05-29T03:36:59.847000", "content": "I think the default task for matcha-base is more similar to this competition than plotqa. I didn't try the other ones, so this is just a hunch.
", "votes": 1, "replies": [] }, { "id": 2279318, "author_name": "areno", "author_url": "", "post_date": "2023-05-29T09:12:36.430000", "content": "Hi,@chaitanyagiri,thank you for sharing your experience
\nMy loss is also around 6, and I wonder how did you solve it? What does ‘emoved spaces and changed the text inside the processor’ mean?
\n\nuse whatever batch size your GPU can support. Use gradient checkpointing if you want a bigger one. You should be able to get to the same point no matter what your batch size is.
\n
Hi,@nbroad,when i used gradient checkpointing with the following code,an error popped up
\nmodel = Pix2StructForConditionalGeneration.from_pretrained(Config['MODEL_NAME'])\nmodel.gradient_checkpointing_enable()
\nValueError: Pix2StructForConditionalGeneration does not support gradient checkpointing.
\nMay I ask if there is an error in my code?
@areno9516 Look at the deplot public notebook and see the Pix2StructProcessor call. You may have to change the text inside that particular call. Check @evilpsycho42 's comment on this thread.
", "votes": 1, "replies": [] }, { "id": 2283659, "author_name": "Givan", "author_url": "", "post_date": "2023-06-01T12:26:39.243000", "content": "I am training the pix2struct model. The loss is decreasing normally but eventually stabilizes around 0.7, while I see that your training loss stabilizes around 0.3+. Moreover, when I used sample test images for prediction, the results seemed to be very bad. What could be the possible reasons for this?
\n
Hi,@marbury,when calculating loss, you can ignore padding tokens. After I ignored the padding tokens, the loss has decreased significantly, which may be useful for you. By the way, my model suffered unstable losses during training, and I would like to ask if you have encountered such a situation.
", "votes": 0, "replies": [] }, { "id": 2283985, "author_name": "", "author_url": "", "post_date": "2023-06-01T16:36:57.607000", "content": "", "votes": 0, "replies": [] }, { "id": 2283986, "author_name": "Zaakcii Ru", "author_url": "", "post_date": "2023-06-01T16:37:52.233000", "content": "Try this: config.gradient checkpointing=True or model.config.gradient checkpointing=True, I didn't notice much difference. It's possible that I just don't understand something.
", "votes": 1, "replies": [] }, { "id": 2284001, "author_name": "Nicholas Broad", "author_url": "", "post_date": "2023-06-01T16:52:02.783000", "content": "\nmodel.encoder.gradient_checkpointing_enable()
\nmodel.decoder.gradient_checkpointing_enable()
", "votes": 2, "replies": [] }, { "id": 2284006, "author_name": "Givan", "author_url": "", "post_date": "2023-06-01T16:58:54.717000", "content": "@areno9516 @nbroad Do I need to modify the source code in pix2struct transformers to ignore padding tokens when calculate the loss?
\nIt seems that they have already done this in transformers.
\n labels :\n \n labels = labels.to(logits.device)\n loss_fct = nn.CrossEntropyLoss(ignore_index=-, reduction=)\nYou need to set the labels to be -100 to get ignored
", "votes": 1, "replies": [] }, { "id": 2284596, "author_name": "Givan", "author_url": "", "post_date": "2023-06-02T07:01:37.843000", "content": "@nbroad Thanks, I have set the pad_token_id to -100 as you do in your public notebook. However, the loss is still not decreased as expected.
\n
\n\n\n
Do you include the scatter plot images for training in this train loss?
", "votes": 0, "replies": [] }, { "id": 2294656, "author_name": "", "author_url": "", "post_date": "2023-06-10T07:54:12.500000", "content": "", "votes": 0, "replies": [] }, { "id": 2294660, "author_name": "", "author_url": "", "post_date": "2023-06-10T07:57:35.320000", "content": "", "votes": 0, "replies": [] }, { "id": 2294663, "author_name": "", "author_url": "", "post_date": "2023-06-10T07:59:45.023000", "content": "", "votes": 0, "replies": [] }, { "id": 2294665, "author_name": "some_train", "author_url": "", "post_date": "2023-06-10T08:00:44.943000", "content": "Hi, I was wondering what else have you done besides this method to reduce the scoring time, since my current scoring time is so long that it takes 6 hours
", "votes": 0, "replies": [] }, { "id": 2295823, "author_name": "", "author_url": "", "post_date": "2023-06-11T09:30:42.597000", "content": "", "votes": 0, "replies": [] }, { "id": 2295824, "author_name": "allen wang", "author_url": "", "post_date": "2023-06-11T09:31:27.853000", "content": "Did you pre-train the model using matcha-plotqa-v2? Is the training labeled xxxyyy or xyxy? The loss never goes down
", "votes": 1, "replies": [] }, { "id": 2301417, "author_name": "Chaitanya Giri", "author_url": "", "post_date": "2023-06-13T22:40:45.200000", "content": "Hi @areno, I see that you mentioned you had an unstable loss. Do you still have it? If not may I ask what was your workaround for that?
", "votes": 0, "replies": [] }, { "id": 2301552, "author_name": "areno", "author_url": "", "post_date": "2023-06-14T02:40:46.680000", "content": "Hi,@chaitanyagiri
\nYou may be able to ignore that. My loss was unstable, with an average value of about 0.2, but fluctuating between 0.05 and 0.6.And the model finally got 0.75 cv and 0.68 lb
My loss fluctuates between 0.8 and 0 and the average is 0.35. I tried increasing lr but it's still the same. I'm using deplot, what about you?
", "votes": 1, "replies": [] }, { "id": 2301802, "author_name": "areno", "author_url": "", "post_date": "2023-06-14T07:10:17.780000", "content": "\n\nMy loss fluctuates between 0.8 and 0 and the average is 0.35. I tried increasing lr but it's still the same. I'm using deplot, what about you?
\n
I have tried the Matcha and Deplot models.Their losses are close and Matcha‘s scores are slightly better. My Learning rate is 3e-5
", "votes": 1, "replies": [] }, { "id": 2302497, "author_name": "Chaitanya Giri", "author_url": "", "post_date": "2023-06-14T15:29:06.357000", "content": "And do you use the competition's dataset or the 500k images one?
", "votes": 0, "replies": [] }, { "id": 2302974, "author_name": "areno", "author_url": "", "post_date": "2023-06-15T02:10:02.057000", "content": "\n\nAnd do you use the competition's dataset or the 500k images one?
\n
I used the competition's dataset and 100k images of 500k images
", "votes": 0, "replies": [] }, { "id": 2303350, "author_name": "Chaitanya Giri", "author_url": "", "post_date": "2023-06-15T08:58:33.680000", "content": "So the 100k images were of scatterplots or stratified the images like 20k from each. And how many epochs do you train for? I kinda notice that my losses doesn't seem to go down after the 2nd
", "votes": 0, "replies": [] }, { "id": 2304446, "author_name": "areno", "author_url": "", "post_date": "2023-06-16T02:16:03.080000", "content": "\n\nSo the 100k images were of scatterplots or stratified the images like 20k from each. And how many epochs do you train for? I kinda notice that my losses doesn't seem to go down after the 2nd
\n
20k for each type of chart. I trained three epochs. Maybe you can try a smaller lr
", "votes": 1, "replies": [] } ] }, { "id": 2271435, "author_name": "Bartley", "author_url": "", "post_date": "2023-05-23T22:29:06.937000", "content": "Thanks everyone for sharing. I have learnt a lot from this discussion.
\n\n\n\nUpdate: I had max patches set to 4..
", "votes": 1, "replies": [] } ] }, { "id": 2306839, "author_name": "Bo Peng", "author_url": "", "post_date": "2023-06-17T16:16:25.600000", "content": "Is there a public code/notebook that is best to use for Matcha model?
\nThanks.
", "votes": 0, "replies": [] }, { "id": 2254461, "author_name": "", "author_url": "", "post_date": "2023-05-11T02:36:24.667000", "content": "", "votes": 0, "replies": [] } ], "raw_markdown_by_id": { "2241608": "You should read this thread: https://github.com/huggingface/transformers/issues/22903\n\ntldr: make sure to set `model.config.text_config.is_decoder=True` and don't use label smoothing. It should produce 60+ scores then\n\n\nIf you do `pip install git+https://github.com/huggingface/transformers.git`, then it should work fine without any modifications", "2300948": "Thanks for sharing @nbroad !\n\nHave you tired inference with Pix2Struct, MatCha or DePlot? If so were the results reasonable?\n\nKeep up with the great work🔥💪", "2280277": "Thanks for your sharing! @nbroad \n\ntried matcha models & pix2struct, only got highest 0.6 cv in local.\n- lr 4e-5\n- epochs 2\n- batch size 2\n- Adam\n- no prompt\n\ndo you pass any prompts to model? (just like deplot inputs = processor(images=image, text=\"Generate underlying data table of the figure below:\", return_tensors=\"pt\"))", "2273592": "may I know your MatCha inference speed, my model inference is too slow and always time out.", "2265780": "how will you download as the submission is without the internet?\n", "2255915": "Hi Nicholas, did you try to submit a notebook that uses the DePlot model? I found that the model is trying to download something from the Internet and this will cause the run to fail. Did you figure out a way to solve this problem? ", "2241852": "Thanks for sharing. Have you tried it yet?", "2242142": "Thanks. Do these models work better than donut?", "2306839": "Is there a public code/notebook that is best to use for Matcha model? \n\nThanks.\n\n\n", "2254461": "" } }