{ "id": 418323, "title": "14th Place Solution", "url": "/competitions/benetech-making-graphs-accessible/writeups/penguin46-14th-place-solution", "author_name": "", "post_date": "2023-06-20T02:14:03.283Z", "votes": 30, "comment_count": 4, "views": 0, "content": "

First, congratulations to the winning teams! This task was very interesting and challenging. I hope to learn a lot from the top solutions.

\n

My solution consists of chart_type classification, template matching for scatter and pix2struct model for the rest chart_types.

\n

Chart_type Classification

\n

I finetuned resnet18d with about 70,000 images in the ICDAR dataset, resulting in the accuracy of 0.995 for extracted data.

\n

Template matching for scatter

\n

Scatter often requires a large number of datapoints, and long token lengths are needed. I addressed this problem by object detection, OCR and template matching. This approach scores 0.51 on extracted data. The algorithm is as follows:

\n
    \n
  1. object detection of data points and axis texts by YOLOX (im_size=480)
    2. \n
  2. generate a pattern of datapoints using the highest confidence among the detected points
    3. \n
  3. identify the shape of the pattern based on the distance between the RGB values of the pattern and the background color
    4. \n
  4. scan the entire image and calculate the similarity score (L2 distance) with the pattern
    5. \n
  5. select approximately 1000 pixels with the highest similarity score as candidate points
    6. \n
  6. fill the rectangular area containing the data points with background color and use it as a base for reconstruction
    7. \n
  7. take out the candidate points in order from the one with the highest similarity score, and place a pattern if the following two conditions are satisfied
      \n
    1. the L2 error with the original image is smaller when a pattern is placed than when a pattern is not placed
      2. \n
    2. the overlap with the existing pattern does not exceed 30%.
    8. \n
\n

\n

Since simple linear regression is vulnerable to outliers, I tried to use some form of robust regression. Considering that methods such as RANSAC and huber require a parameter on the scale and that the parameters tend to overfit the data set due to small data size, I implemented a simple robust regression without any hyper parameters. All two point pairs are connected by a straight line and the slope and intercept are calculated. The desired regression line is obtained using the median value for all computed slopes and intercepts.

\n

\n

Pix2Struct model for other types

\n", "messages": [ { "id": "2309934", "postDate": "06/20/2023 02:08:21", "content": "

First, congratulations to the winning teams! This task was very interesting and challenging. I hope to learn a lot from the top solutions.

\n

My solution consists of chart_type classification, template matching for scatter and pix2struct model for the rest chart_types.

\n

Chart_type Classification

\n

I finetuned resnet18d with about 70,000 images in the ICDAR dataset, resulting in the accuracy of 0.995 for extracted data.

\n

Template matching for scatter

\n

Scatter often requires a large number of datapoints, and long token lengths are needed. I addressed this problem by object detection, OCR and template matching. This approach scores 0.51 on extracted data. The algorithm is as follows:

\n
    \n
  1. object detection of data points and axis texts by YOLOX (im_size=480)
    2. \n
  2. generate a pattern of datapoints using the highest confidence among the detected points
    3. \n
  3. identify the shape of the pattern based on the distance between the RGB values of the pattern and the background color
    4. \n
  4. scan the entire image and calculate the similarity score (L2 distance) with the pattern
    5. \n
  5. select approximately 1000 pixels with the highest similarity score as candidate points
    6. \n
  6. fill the rectangular area containing the data points with background color and use it as a base for reconstruction
    7. \n
  7. take out the candidate points in order from the one with the highest similarity score, and place a pattern if the following two conditions are satisfied
      \n
    1. the L2 error with the original image is smaller when a pattern is placed than when a pattern is not placed
      2. \n
    2. the overlap with the existing pattern does not exceed 30%.
    8. \n
\n

\n

Since simple linear regression is vulnerable to outliers, I tried to use some form of robust regression. Considering that methods such as RANSAC and huber require a parameter on the scale and that the parameters tend to overfit the data set due to small data size, I implemented a simple robust regression without any hyper parameters. All two point pairs are connected by a straight line and the slope and intercept are calculated. The desired regression line is obtained using the median value for all computed slopes and intercepts.

\n

\n

Pix2Struct model for other types

\n", "rawMarkdown": "First, congratulations to the winning teams! This task was very interesting and challenging. I hope to learn a lot from the top solutions.\n\nMy solution consists of chart_type classification, template matching for scatter and pix2struct model for the rest chart_types.\n\n# Chart_type Classification\nI finetuned resnet18d with about 70,000 images in the ICDAR dataset, resulting in the accuracy of 0.995 for extracted data.\n\n# Template matching for scatter\nScatter often requires a large number of datapoints, and long token lengths are needed. I addressed this problem by object detection, OCR and template matching. This approach scores 0.51 on extracted data. The algorithm is as follows:\n\n1. object detection of data points and axis texts by YOLOX (im_size=480)\n2. generate a pattern of datapoints using the highest confidence among the detected points\n3. identify the shape of the pattern based on the distance between the RGB values of the pattern and the background color\n4. scan the entire image and calculate the similarity score (L2 distance) with the pattern\n5. select approximately 1000 pixels with the highest similarity score as candidate points\n6. fill the rectangular area containing the data points with background color and use it as a base for reconstruction\n7. take out the candidate points in order from the one with the highest similarity score, and place a pattern if the following two conditions are satisfied\n 1. the L2 error with the original image is smaller when a pattern is placed than when a pattern is not placed\n 2. the overlap with the existing pattern does not exceed 30%.\n\n\n\nSince simple linear regression is vulnerable to outliers, I tried to use some form of robust regression. Considering that methods such as RANSAC and huber require a parameter on the scale and that the parameters tend to overfit the data set due to small data size, I implemented a simple robust regression without any hyper parameters. All two point pairs are connected by a straight line and the slope and intercept are calculated. The desired regression line is obtained using the median value for all computed slopes and intercepts. \n\n\n\n## Pix2Struct model for other types\n- labels are in the form xyxy\n- decimal part of numerical is rounded to 4 significant digits\n- augmentation for translation, rotation, noise, hue, etc.\n- pretrained model: MatCha-base\n- hyperparameters\n - max_length: 512\n - max_patches: 2048\n - lr: 1e-5\n - scheduler: get_cosine_schedule_with_warmup\n - num_warmup_steps: 1000\n - weight_decay: 1e-5\n- Training data\n - extracted (1118)\n - generated (999)\n - ICDAR (labeled 1286+ pseudo&hand labeled 2988)", "votes": null }, { "id": "2310054", "postDate": "06/20/2023 05:02:35", "content": "

Congratulations @ryotayoshinobu. I am mostly intrigued by your object detection part, quite original. Will you be sharing any code of this?

", "rawMarkdown": "Congratulations @ryotayoshinobu. I am mostly intrigued by your object detection part, quite original. Will you be sharing any code of this?", "votes": null }, { "id": "2310095", "postDate": "06/20/2023 05:30:53", "content": "

@ryotayoshinobu congrats! Can you share some your code?

", "rawMarkdown": "ryotayoshinobu congrats! Can you share some your code?", "votes": null }, { "id": "2310373", "postDate": "06/20/2023 10:12:00", "content": "

That's a pretty smart pipeline for the scatter plots @ryotayoshinobu . Congrats on your strong finish!

", "rawMarkdown": "That's a pretty smart pipeline for the scatter plots @ryotayoshinobu . Congrats on your strong finish!", "votes": null }, { "id": "2310390", "postDate": "06/20/2023 10:28:14", "content": "

Congrats. I used resnet34 and I never use any extra data. I achieved 99.94% accuracy for both training and testing. I rotated the vertical bars to get more horizontal bar charts since they are only 73. I got 70100 dataset and then split it into 80% and 20%.

", "rawMarkdown": "Congrats. I used resnet34 and I never use any extra data. I achieved 99.94% accuracy for both training and testing. I rotated the vertical bars to get more horizontal bar charts since they are only 73. I got 70100 dataset and then split it into 80% and 20%.", "votes": null } ], "comments": [ { "id": 2310054, "author_name": "alejopaullier", "author_url": "", "post_date": "06/20/2023 05:02:35", "content": "

Congratulations @ryotayoshinobu. I am mostly intrigued by your object detection part, quite original. Will you be sharing any code of this?

", "votes": null, "replies": [] }, { "id": 2310095, "author_name": "serangu", "author_url": "", "post_date": "06/20/2023 05:30:53", "content": "

@ryotayoshinobu congrats! Can you share some your code?

", "votes": null, "replies": [] }, { "id": 2310373, "author_name": "snnclsr", "author_url": "", "post_date": "06/20/2023 10:12:00", "content": "

That's a pretty smart pipeline for the scatter plots @ryotayoshinobu . Congrats on your strong finish!

", "votes": null, "replies": [] }, { "id": 2310390, "author_name": "hamzahabduljalil", "author_url": "", "post_date": "06/20/2023 10:28:14", "content": "

Congrats. I used resnet34 and I never use any extra data. I achieved 99.94% accuracy for both training and testing. I rotated the vertical bars to get more horizontal bar charts since they are only 73. I got 70100 dataset and then split it into 80% and 20%.

", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "2309934": "First, congratulations to the winning teams! This task was very interesting and challenging. I hope to learn a lot from the top solutions.\n\nMy solution consists of chart_type classification, template matching for scatter and pix2struct model for the rest chart_types.\n\n# Chart_type Classification\nI finetuned resnet18d with about 70,000 images in the ICDAR dataset, resulting in the accuracy of 0.995 for extracted data.\n\n# Template matching for scatter\nScatter often requires a large number of datapoints, and long token lengths are needed. I addressed this problem by object detection, OCR and template matching. This approach scores 0.51 on extracted data. The algorithm is as follows:\n\n1. object detection of data points and axis texts by YOLOX (im_size=480)\n2. generate a pattern of datapoints using the highest confidence among the detected points\n3. identify the shape of the pattern based on the distance between the RGB values of the pattern and the background color\n4. scan the entire image and calculate the similarity score (L2 distance) with the pattern\n5. select approximately 1000 pixels with the highest similarity score as candidate points\n6. fill the rectangular area containing the data points with background color and use it as a base for reconstruction\n7. take out the candidate points in order from the one with the highest similarity score, and place a pattern if the following two conditions are satisfied\n 1. the L2 error with the original image is smaller when a pattern is placed than when a pattern is not placed\n 2. the overlap with the existing pattern does not exceed 30%.\n\n\n\nSince simple linear regression is vulnerable to outliers, I tried to use some form of robust regression. Considering that methods such as RANSAC and huber require a parameter on the scale and that the parameters tend to overfit the data set due to small data size, I implemented a simple robust regression without any hyper parameters. All two point pairs are connected by a straight line and the slope and intercept are calculated. The desired regression line is obtained using the median value for all computed slopes and intercepts. \n\n\n\n## Pix2Struct model for other types\n- labels are in the form xyxy\n- decimal part of numerical is rounded to 4 significant digits\n- augmentation for translation, rotation, noise, hue, etc.\n- pretrained model: MatCha-base\n- hyperparameters\n - max_length: 512\n - max_patches: 2048\n - lr: 1e-5\n - scheduler: get_cosine_schedule_with_warmup\n - num_warmup_steps: 1000\n - weight_decay: 1e-5\n- Training data\n - extracted (1118)\n - generated (999)\n - ICDAR (labeled 1286+ pseudo&hand labeled 2988)", "2310054": "Congratulations @ryotayoshinobu. I am mostly intrigued by your object detection part, quite original. Will you be sharing any code of this?", "2310095": "ryotayoshinobu congrats! Can you share some your code?", "2310373": "That's a pretty smart pipeline for the scatter plots @ryotayoshinobu . Congrats on your strong finish!", "2310390": "Congrats. I used resnet34 and I never use any extra data. I achieved 99.94% accuracy for both training and testing. I rotated the vertical bars to get more horizontal bar charts since they are only 73. I got 70100 dataset and then split it into 80% and 20%." }, "source": "meta" }