![\"Example](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F5570735%2Fb48bdb74f5b1bf6ff8d580ef40c0012b%2Ftest.png?generation=1685405715434222&alt=media\")
One common error that I am seeing with the pix2struct model is the first values on the x and y axis are switched. Even though the predictions are mostly correct, the y-values are given a score of 0.
\nFor example,
\nCorrect Labels
\nJANUARY;FEBRUARY;MARCH;APRIL;MAY;JUNE;JULY;AUGUST;SEPTEMBER;OCTOBER;NOVEMBER;DECEMBER\n3;6;9;12;15;17;17;14;10;6;4;5\nModel Labels
\n3;FEBRUARY;MARCH;APRIL;MAY;JUNE;JULY;AUGUST;SEPTEMBER;OCTOBER;NOVEMBER;DECEMBER\nJANUARY;6;9;12;15;17;17;14;10;6;4;5\n
The benetech score on the y-axis would be zero as the array does not pass the data type check. Current solution is to see if all values except the 1st are numerical, and if so replace the first value with the second value in the array.
\nWill update with more findings as I go along.
", "messages": [ { "id": 2280187, "postDate": "2023-05-30T00:20:59.357Z", "content": "One common error that I am seeing with the pix2struct model is the first values on the x and y axis are switched. Even though the predictions are mostly correct, the y-values are given a score of 0.
\nFor example,
\nCorrect Labels
\nJANUARY;FEBRUARY;MARCH;APRIL;MAY;JUNE;JULY;AUGUST;SEPTEMBER;OCTOBER;NOVEMBER;DECEMBER\n3;6;9;12;15;17;17;14;10;6;4;5\nModel Labels
\n3;FEBRUARY;MARCH;APRIL;MAY;JUNE;JULY;AUGUST;SEPTEMBER;OCTOBER;NOVEMBER;DECEMBER\nJANUARY;6;9;12;15;17;17;14;10;6;4;5\n
The benetech score on the y-axis would be zero as the array does not pass the data type check. Current solution is to see if all values except the 1st are numerical, and if so replace the first value with the second value in the array.
\nWill update with more findings as I go along.
", "rawMarkdown": "One common error that I am seeing with the pix2struct model is the first values on the x and y axis are switched. Even though the predictions are mostly correct, the y-values are given a score of 0.\n\nFor example,\n\nCorrect Labels\n```\nJANUARY;FEBRUARY;MARCH;APRIL;MAY;JUNE;JULY;AUGUST;SEPTEMBER;OCTOBER;NOVEMBER;DECEMBER\n3;6;9;12;15;17;17;14;10;6;4;5\n```\n\nModel Labels\n```\n3;FEBRUARY;MARCH;APRIL;MAY;JUNE;JULY;AUGUST;SEPTEMBER;OCTOBER;NOVEMBER;DECEMBER\nJANUARY;6;9;12;15;17;17;14;10;6;4;5\n```\n\n\n\nThe benetech score on the y-axis would be zero as the array does not pass the data type check. Current solution is to see if all values except the 1st are numerical, and if so replace the first value with the second value in the array.\n\nWill update with more findings as I go along.", "votes": 3 }, { "id": 2280836, "postDate": "2023-05-30T12:01:34.330Z", "content": "Hi Bartley,
\nCould you pass on the necessary Pix2Struct code?
\nI could not get any output so far.
python convert_pix2struct_checkpoint_to_pytorch.py --t5x_checkpoint_path PATH_TO_T5X_CHECKPOINTS --pytorch_dump_path PATH_TO_SAVE --use-large --is_vqa
\nI could not give the correct path in the above command.
Also I did not get any output from MatchA model.
\nfilepath = f'/kaggle/input/benetech-making-graphs-accessible/train/images/{id_}.jpg'
\nimage = cv2.imread(filepath)
\nmodel = Pix2StructForConditionalGeneration.from_pretrained(\"google/matcha-plotqa-v2\").to(0)
\nprocessor = AutoProcessor.from_pretrained(\"google/matcha-plotqa-v2\", is_vqa=False)
\ninputs = processor(images=image, return_tensors=\"pt\").to(0)
\npredictions = model.generate(**inputs, max_new_tokens=512)
\nprocessor.decode(predictions[0], skip_special_tokens=True)
Please advice.
\nI am not expert in using transformer. Any hint will be useful.
Hi @pksamantaray, I would recommend clicking copy & edit in the top right in a public notebook. The code above looks fine to me.
This is one uses a pix2struct, and should be a food starting point. Deplot - Trying Out
", "rawMarkdown": "Hi @pksamantaray, I would recommend clicking `copy & edit` in the top right in a public notebook. The code above looks fine to me.\n\nThis is one uses a pix2struct, and should be a food starting point. [Deplot - Trying Out](https://www.kaggle.com/code/gomaki/deplot-trying-out)", "votes": 1, "replies": [ { "id": 2282945, "postDate": "2023-06-01T00:40:13.653Z", "rawMarkdown": "", "isDeleted": true }, { "id": 2283583, "postDate": "2023-06-01T11:17:35.723Z", "content": "Hi Bartley,
\nI have given the following link.
\n[PLEASE Wait: It is taking more than one day in kaggle to update public notebook. On Saturday it should show updated notebook.]
\nhttps://www.kaggle.com/code/pksamantaray/benetech-matcha-model-processor
The notebook is with internet on. The model and processor are downloaded into the working directory.
\nI am checking the output. There is no prediction output for matcha-plotqa-v1 but I am getting predictions for matcha-base.
Also another question, how will I implement matcha-plotqa-v1 directory for my submission file without internet. I tried the way Benedict-pip. But I could not succeed.
", "rawMarkdown": "Hi Bartley,\nI have given the following link.\n[**PLEASE Wait: It is taking more than one day in kaggle to update public notebook. On Saturday it should show updated notebook.**]\nhttps://www.kaggle.com/code/pksamantaray/benetech-matcha-model-processor\n\nThe notebook is with internet on. The model and processor are downloaded into the working directory. \nI am checking the output. There is no prediction output for matcha-plotqa-v1 but I am getting predictions for matcha-base.\n\nAlso another question, how will I implement matcha-plotqa-v1 directory for my submission file without internet. I tried the way Benedict-pip. But I could not succeed." }, { "id": 2283725, "postDate": "2023-06-01T13:08:24.610Z", "content": "Hey @pksamantaray, matcha-base is a pretrained model which can be used to fine-tune on further downstream tasks whereas the matcha-plotqa-v1 is an already fine-tuned checkpoint that is used for question answering from plots. The latter is often used for outputing just a few tokens (like a number or a word). If you want to have a large sequence generated like in this comp you'll have to use matcha-base I believe.
\nYou can just save your model after training and you can either create a dataset with that output or use the notebook itself. Add the dataset/notebook to your submission notebook and now when loading the model and processor use
preprocessor = Pix2StructProcessor.from_pretrained(\"PATH TO DATASET\")\nmodel = Pix2StructForConditionalGeneration.from_pretrained(\"PATH TO DATASET\")\nYou don't need to run the convert_pix2struct_checkpoint_to_pytorch.py
", "rawMarkdown": "You don't need to run the convert_pix2struct_checkpoint_to_pytorch.py", "votes": 1 } ] } ], "comments": [ { "id": 2280836, "author_name": "pksamantaray", "author_url": "", "post_date": "2023-05-30T12:01:34.330000", "content": "Hi Bartley,
\nCould you pass on the necessary Pix2Struct code?
\nI could not get any output so far.
python convert_pix2struct_checkpoint_to_pytorch.py --t5x_checkpoint_path PATH_TO_T5X_CHECKPOINTS --pytorch_dump_path PATH_TO_SAVE --use-large --is_vqa
\nI could not give the correct path in the above command.
Also I did not get any output from MatchA model.
\nfilepath = f'/kaggle/input/benetech-making-graphs-accessible/train/images/{id_}.jpg'
\nimage = cv2.imread(filepath)
\nmodel = Pix2StructForConditionalGeneration.from_pretrained(\"google/matcha-plotqa-v2\").to(0)
\nprocessor = AutoProcessor.from_pretrained(\"google/matcha-plotqa-v2\", is_vqa=False)
\ninputs = processor(images=image, return_tensors=\"pt\").to(0)
\npredictions = model.generate(**inputs, max_new_tokens=512)
\nprocessor.decode(predictions[0], skip_special_tokens=True)
Please advice.
\nI am not expert in using transformer. Any hint will be useful.
Hi @pksamantaray, I would recommend clicking copy & edit in the top right in a public notebook. The code above looks fine to me.
This is one uses a pix2struct, and should be a food starting point. Deplot - Trying Out
", "votes": 1, "replies": [ { "id": 2282945, "author_name": "", "author_url": "", "post_date": "2023-06-01T00:40:13.653000", "content": "", "votes": 0, "replies": [] }, { "id": 2283583, "author_name": "pksamantaray", "author_url": "", "post_date": "2023-06-01T11:17:35.723000", "content": "Hi Bartley,
\nI have given the following link.
\n[PLEASE Wait: It is taking more than one day in kaggle to update public notebook. On Saturday it should show updated notebook.]
\nhttps://www.kaggle.com/code/pksamantaray/benetech-matcha-model-processor
The notebook is with internet on. The model and processor are downloaded into the working directory.
\nI am checking the output. There is no prediction output for matcha-plotqa-v1 but I am getting predictions for matcha-base.
Also another question, how will I implement matcha-plotqa-v1 directory for my submission file without internet. I tried the way Benedict-pip. But I could not succeed.
", "votes": 0, "replies": [] }, { "id": 2283725, "author_name": "Chaitanya Giri", "author_url": "", "post_date": "2023-06-01T13:08:24.610000", "content": "Hey @pksamantaray, matcha-base is a pretrained model which can be used to fine-tune on further downstream tasks whereas the matcha-plotqa-v1 is an already fine-tuned checkpoint that is used for question answering from plots. The latter is often used for outputing just a few tokens (like a number or a word). If you want to have a large sequence generated like in this comp you'll have to use matcha-base I believe.
\nYou can just save your model after training and you can either create a dataset with that output or use the notebook itself. Add the dataset/notebook to your submission notebook and now when loading the model and processor use
preprocessor = Pix2StructProcessor.from_pretrained(\"PATH TO DATASET\")\nmodel = Pix2StructForConditionalGeneration.from_pretrained(\"PATH TO DATASET\")\nYou don't need to run the convert_pix2struct_checkpoint_to_pytorch.py
", "votes": 1, "replies": [] } ] } ], "raw_markdown_by_id": { "2280187": "One common error that I am seeing with the pix2struct model is the first values on the x and y axis are switched. Even though the predictions are mostly correct, the y-values are given a score of 0.\n\nFor example,\n\nCorrect Labels\n```\nJANUARY;FEBRUARY;MARCH;APRIL;MAY;JUNE;JULY;AUGUST;SEPTEMBER;OCTOBER;NOVEMBER;DECEMBER\n3;6;9;12;15;17;17;14;10;6;4;5\n```\n\nModel Labels\n```\n3;FEBRUARY;MARCH;APRIL;MAY;JUNE;JULY;AUGUST;SEPTEMBER;OCTOBER;NOVEMBER;DECEMBER\nJANUARY;6;9;12;15;17;17;14;10;6;4;5\n```\n\n\n\nThe benetech score on the y-axis would be zero as the array does not pass the data type check. Current solution is to see if all values except the 1st are numerical, and if so replace the first value with the second value in the array.\n\nWill update with more findings as I go along.", "2280836": "Hi Bartley,\nCould you pass on the necessary Pix2Struct code?\nI could not get any output so far.\n\n \npython convert_pix2struct_checkpoint_to_pytorch.py --t5x_checkpoint_path PATH_TO_T5X_CHECKPOINTS --pytorch_dump_path PATH_TO_SAVE --use-large --is_vqa\nI could not give the correct path in the above command.\n\nAlso I did not get any output from MatchA model.\nfilepath = f'/kaggle/input/benetech-making-graphs-accessible/train/images/{id_}.jpg'\nimage = cv2.imread(filepath)\nmodel = Pix2StructForConditionalGeneration.from_pretrained(\"google/matcha-plotqa-v2\").to(0)\nprocessor = AutoProcessor.from_pretrained(\"google/matcha-plotqa-v2\", is_vqa=False)\ninputs = processor(images=image, return_tensors=\"pt\").to(0)\npredictions = model.generate(**inputs, max_new_tokens=512)\nprocessor.decode(predictions[0], skip_special_tokens=True)\n\nPlease advice.\nI am not expert in using transformer. Any hint will be useful.\n\n" } }