{
  "id": 418786,
  "title": "1st Place Solution",
  "url": "/competitions/benetech-making-graphs-accessible/discussion/418786",
  "author_name": "YumeNeko",
  "post_date": "2023-06-22T14:27:01.809000",
  "votes": 60,
  "comment_count": 15,
  "views": 0,
  "content": "<p>First of all, I would like to pay tribute to all the participants who worked on this competition.   <br>\nI would also like to thank the hosts for organizing this interesting task competition.    <br>\nThis task was very interesting and I enjoyed working on it because I could think of many different approaches.    <br>\nI am honored to have won first place in this very interesting competition.</p>\n<h1>Overview</h1>\n<p><img src=\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F3823496%2F16283ff691149c87b3b01a49a432f107%2Foverview.jpg?generation=1687442367636860&amp;alt=media\" alt=\"Overview\"></p>\n<p>My solution consists of a two-step pipeline that first classifies chart types using a classification model and then performs data series inference.  <br>\nIn the inference phase of the data series, Bar, Line, and Dot were end-to-end predictions by Deplot trained for each chart type, while Scatter was predicted by an object detection-based approach.  </p>\n<p>The final scores are as follows.  </p>\n<table>\n<thead>\n<tr>\n<th></th>\n<th>Overall</th>\n<th>Scatter</th>\n<th>Dot</th>\n<th>Line</th>\n<th>Vertical Bar</th>\n<th>Horizontal Bar</th>\n</tr>\n</thead>\n<tbody>\n<tr>\n<td>public</td>\n<td>0.86</td>\n<td>0.10</td>\n<td>0.00</td>\n<td>0.32</td>\n<td>0.39</td>\n<td>0.05</td>\n</tr>\n<tr>\n<td>private</td>\n<td>0.72</td>\n<td>0.30</td>\n<td>0.01</td>\n<td>0.13</td>\n<td>0.26</td>\n<td>0.01</td>\n</tr>\n</tbody>\n</table>\n<h1>Dataset</h1>\n<p>I used the following three data sets.  </p>\n<ol>\n<li><p>Competition data set（comp_extracted_dataset/comp_generated_dataset）  </p>\n<ul>\n<li>Using both extracted dataset (comp_extracted_dataset) and generated dataset (comp_generated_dataset).  </li>\n<li>For the generated data, data with noise in the annotations were removed by a simple check (about 100 images).  </li></ul></li>\n<li><p>ICDAR data set（ICDAR_dataset/ICDAR_manualannot_dataset）  </p>\n<ul>\n<li>I used 1406 pieces of data for which annotations were provided (ICDAR_dataset) and 1903 pieces of data for which no annotations were provided (ICDAR_manualannot_dataset).  </li>\n<li>For data for which annotations were provided, I visually rechecked the annotation contents and manually corrected all data that did not follow the competition's annotation rules (e.g., % notation) or contained noise.  </li>\n<li>For the data for which no annotations were provided, I first visually checked the appearance of all the data and selected the data that could be used in this competition. Next, I inferred and assigned pseudo-labels using the Deplot model, visually checked all the results again, and manually corrected all incorrect annotations.  </li></ul></li>\n<li><p>Generated synthetic data set（synthetic_dataset）  </p>\n<ul>\n<li>After reviewing the images in the competition dataset and determining that the comp_generated_dataset alone did not have a sufficient variation to achieve robustness, so I generated about 65k synthetic data myself.  </li>\n<li>I mainly generated synthetic data with features that comp_generated_dataset does not have.  <ul>\n<li>histogram  </li>\n<li>Label contains line breaks  </li>\n<li>Bar chart with error bars  </li>\n<li>Line chart containing x labels not included in the data series</li>\n<li>etc…  </li></ul></li>\n<li>To add diversity, I also included 10k images from the dataset published by <a href=\"https://www.kaggle.com/brendanartley\" target=\"_blank\">@brendanartley</a>.  </li></ul>\n<p>(Example of synthetic data I generated)  <br>\n<img src=\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F3823496%2Fd4df82a41fcaebeda2ffc93fb8ad9159%2Fgenerated_data.jpg?generation=1687442722440643&amp;alt=media\" alt=\"generated_data\"></p></li>\n</ol>\n<h1>Solution pipeline</h1>\n<h2>1. Chart classification</h2>\n<ul>\n<li>There is not much to note, as I just performed a simple classification task.  </li>\n<li>I used two models, convnext_large_384 and swin_large_patch4_window12_384, and integrated their results in a weighted ensemble.  </li>\n</ul>\n<p><strong>Training configs</strong>  </p>\n<ul>\n<li>Dataset (training data and validation data were randomly divided into folds)  <ul>\n<li>Training（78k images）：comp_extracted_dataset, comp_generated_dataset, ICDAR_dataset, synthetic_dataset(horizontal_bar/dot)  </li>\n<li>Validation（4k images）：comp_extracted_dataset, ICDAR_dataset, synthetic_dataset(horizontal_bar/dot)</li></ul></li>\n<li>Hyperparameters<ul>\n<li>Epochs：15  </li>\n<li>Batch size：16  </li>\n<li>Adafactor (lr=3e-5)  </li></ul></li>\n</ul>\n<h2>2. Data series prediction</h2>\n<p>Based on the classification results of Step 1, different approaches are used for scatter and the rest.  <br>\nExcept for scatter, the predictions are made in Deplot, and scatter is an object detection-based prediction.  </p>\n<h3>2-a. Bars &amp; Lines &amp; Dots</h3>\n<ul>\n<li><p>I experimented by switching the model to Deplot based on the Donut-based notebook that <a href=\"https://www.kaggle.com/nbroad\" target=\"_blank\">@nbroad</a> had published. However, I could not train well just by changing the model, so I made some modifications.  </p>\n<ul>\n<li>Ground Truth Format  <br>\nRemoved chart type from GroundTruth and adapted the format to that of the Deplot original.  <br>\n<code>&lt;0x0A&gt; x_value1 | y_value1 &lt;0x0A&gt; x_value2 | y_value2 &lt;0x0A&gt; x_value3 | y_value3 &lt;/s&gt;</code></li>\n<li>x-axis and y-axis swap of horizontal_bar  <br>\nSince the concept of x-axis and y-axis was reversed in the annotation rules of this competition and the original format of Deplot, I trained according to the original concept of Deplot and swapped the values during inference.  </li></ul></li>\n<li><p>Training was conducted in multiple stages. In the first stage, training was conducted using data from all chart types (All Chart-type Train), and using the results of that training as initial weights, one or two additional training sessions (Specific Chart-type Train) were conducted using only specific chart-type images to generate a model specialized for a chart type.</p>\n<ul>\n<li>The scores for vertical_bar and line were slightly improved by this method. (Two Specific Chart-type Train runs were performed for vertical_bar and one for line.)  </li>\n<li>Horizontal_bar had a worse cv after the second stage of training, probably due to the small amount of extracted data, so I decided to use the All Chart-type Train model to make predictions.    </li>\n<li>Dot chart only has generated data and I decided that I could not validate it successfully, so I decided not to train after the second stage and use the All Chart-type Train model to make predictions.  <br>\n<img src=\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F3823496%2F0a86b394e87edd440600ac531288897c%2FDeplot_training.jpg?generation=1687442833429916&amp;alt=media\" alt=\"Deplot_training\"></li></ul></li>\n</ul>\n<p><strong>Training configs</strong>  </p>\n<ol>\n<li><p>All Chart-type Train  </p>\n<ul>\n<li>Dataset  <ul>\n<li>Train（120k images）：comp_extracted_dataset, comp_generated_dataset, ICDAR_dataset, <br>\nsynthetic_dataset  </li>\n<li>Validation（2.5k images）：comp_extracted_dataset, ICDAR_dataset, ICDAR_manualannot_dataset</li></ul></li>\n<li>Hyperparameters<ul>\n<li>Epochs：8</li>\n<li>Batch size：2  </li>\n<li>Adafactor (lr=1e-5)</li>\n<li>cosine scheduler with warmup (warmup_step=4000)</li>\n<li>Augmentation: GaussianBlur, GaussNoise, some color augmentations </li></ul></li></ul></li>\n<li><p>Specific Chart-type Train (vertical_bar)   </p>\n<ul>\n<li><p>Dataset  <br>\n[1st time]  </p>\n<ul>\n<li>Train（6k images）  ：comp_extracted_dataset, ICDAR_dataset, synthetic_dataset  </li>\n<li>Validation（1.3k images）：comp_extracted_dataset, ICDAR_dataset, ICDAR_manualannot_dataset  </li></ul>\n<p>[2nd time]  </p>\n<ul>\n<li>Train（1500 images）：comp_extracted_dataset, ICDAR_dataset, ICDAR_manualannot_dataset   </li>\n<li>Validation（500 images） ：comp_extracted_dataset, ICDAR_dataset, ICDAR_manualannot_dataset</li></ul></li>\n<li><p>Hyperparameters</p>\n<ul>\n<li>cosine scheduler with warmup (warmup_step=0)</li>\n<li>Otherwise, same as All Chart-type Train  </li></ul></li></ul></li>\n<li><p>Specific Chart-type Train (line)  </p>\n<ul>\n<li><p>Dataset  <br>\n[1st time]  </p>\n<ul>\n<li>Train（1150 images）：comp_extracted_dataset, ICDAR_dataset, ICDAR_manualannot_dataset  </li>\n<li>Validation（400 images）：comp_extracted_dataset, ICDAR_dataset, ICDAR_manualannot_dataset  </li></ul></li>\n<li><p>Hyper paramete</p>\n<ul>\n<li>cosine scheduler with warmup (warmup_step=0)</li>\n<li>Otherwise, same as All Chart-type Train  </li></ul></li></ul></li>\n</ol>\n<p><strong>Score</strong>  </p>\n<ul>\n<li><p>All Chart-type Train</p>\n<table>\n<thead>\n<tr>\n<th></th>\n<th>Overall</th>\n<th>Scatter</th>\n<th>Dot</th>\n<th>Line</th>\n<th>Vertical Bar</th>\n<th>Horizontal Bar</th>\n</tr>\n</thead>\n<tbody>\n<tr>\n<td>public</td>\n<td>0.78</td>\n<td>0.06</td>\n<td>0.00</td>\n<td>0.29</td>\n<td>0.38</td>\n<td>0.05</td>\n</tr>\n<tr>\n<td>private</td>\n<td>0.53</td>\n<td>0.13</td>\n<td>0.01</td>\n<td>0.12</td>\n<td>0.26</td>\n<td>0.01</td>\n</tr>\n</tbody>\n</table></li>\n<li><p>Specific Chart-type Train</p>\n<table>\n<thead>\n<tr>\n<th></th>\n<th>Overall</th>\n<th>Scatter</th>\n<th>Dot</th>\n<th>Line</th>\n<th>Vertical Bar</th>\n<th>Horizontal Bar</th>\n</tr>\n</thead>\n<tbody>\n<tr>\n<td>public</td>\n<td>0.81</td>\n<td>0.06</td>\n<td>0.00</td>\n<td>0.32</td>\n<td>0.39</td>\n<td>0.05</td>\n</tr>\n<tr>\n<td>private</td>\n<td>0.55</td>\n<td>0.13</td>\n<td>0.01</td>\n<td>0.13</td>\n<td>0.26</td>\n<td>0.01</td>\n</tr>\n</tbody>\n</table></li>\n</ul>\n<h3>2-b. Scatter</h3>\n<ul>\n<li>Only scatter was not accurate enough with Deplot no matter how I tried, so I adopted an object detection-based approach.  </li>\n<li>Since scatter is guaranteed to have label values of numeric type, it is easy to calculate values from ratios if the \"label position,\" \"label text,\" and \"scatter point position\" are known. I used CACHED to detect \"label position,\" Deplot to read \"label text,\" and YOLOX to detect \"scatter point\".  </li>\n<li>The mapping between label bboxes and label texts is a bit crude, but the smallest label text is mapped to the left-most (top in the case of y-labels) label bbox, and the largest label text is mapped to the right-most (bottom in the case of y-labels) label bbox. It is possible that the total number of detected label bboxes does not match the total number of read label texts, but we did not take any special measures because such a pattern was very rare when we checked the verification data.  </li>\n<li>The above object detection approach resulted in a scatter score of <strong>Pulic:0.10/Private:0.30</strong>, much better than the Deplot-based score (Pulic:0.06/Private:0.13).</li>\n</ul>\n<p><strong>More Details</strong>  <br>\nScatter point detection  </p>\n<ul>\n<li>comp_extracted_dataset, ICDAR_dataset, synthetic_dataset were used for training data (about 12.5k images)</li>\n<li>Somewhat semi-automated using provided annotation data and image processing, but inevitably noisy, so corrected with manual annotation.  </li>\n<li>To increase detection accuracy, only the plot area was cropped for training and inference.</li>\n<li>The model was YOLOX-l, input image size was 1280 for both training and inference, and 50 epochs were trained.</li>\n</ul>\n<p>Label text reading</p>\n<ul>\n<li>comp_extracted_dataset, ICDAR_dataset, comp_generated_dataset were used for training data (about 62k images, all chart types were used, not just scatter)  </li>\n<li>Use the same training configs as in (2-a) All_Chart-type_Train for the training parameters, and change the format of Ground Truth as follows  <br>\n<code>&lt;0x0A&gt; x_label1 | x_label2 | x_label3 &lt;0x0A&gt; y_label1 | y_label2 &lt;/s&gt;</code> </li>\n<li>At first, I thought of using EasyOCR or similar to achieve this, but there were many issues to be addressed, such as support for character rotation, and it seemed that tuning would be very difficult to increase versatility, so I adopted a method of batch reading with Deplot.  </li>\n</ul>\n<hr>\n<p>Finally, I would like to thank once again everyone involved in this competition.  <br>\nThank you very much for taking the time to read this!</p>\n<p>[edit] </p>\n<ul>\n<li>I have published 1st place <a href=\"https://www.kaggle.com/code/kashiwaba/benetech-1st-place-inference\" target=\"_blank\">inference notebook</a>(23/6/28) </li>\n<li>I have published 1st place <a href=\"https://github.com/KASSII/benetech_1st_place_train\" target=\"_blank\">training code</a>(23/7/8) </li>\n</ul>",
  "messages": [
    {
      "id": 2313203,
      "postDate": "2023-06-22T14:27:01.810Z",
      "content": "<p>First of all, I would like to pay tribute to all the participants who worked on this competition.   <br>\nI would also like to thank the hosts for organizing this interesting task competition.    <br>\nThis task was very interesting and I enjoyed working on it because I could think of many different approaches.    <br>\nI am honored to have won first place in this very interesting competition.</p>\n<h1>Overview</h1>\n<p><img src=\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F3823496%2F16283ff691149c87b3b01a49a432f107%2Foverview.jpg?generation=1687442367636860&amp;alt=media\" alt=\"Overview\"></p>\n<p>My solution consists of a two-step pipeline that first classifies chart types using a classification model and then performs data series inference.  <br>\nIn the inference phase of the data series, Bar, Line, and Dot were end-to-end predictions by Deplot trained for each chart type, while Scatter was predicted by an object detection-based approach.  </p>\n<p>The final scores are as follows.  </p>\n<table>\n<thead>\n<tr>\n<th></th>\n<th>Overall</th>\n<th>Scatter</th>\n<th>Dot</th>\n<th>Line</th>\n<th>Vertical Bar</th>\n<th>Horizontal Bar</th>\n</tr>\n</thead>\n<tbody>\n<tr>\n<td>public</td>\n<td>0.86</td>\n<td>0.10</td>\n<td>0.00</td>\n<td>0.32</td>\n<td>0.39</td>\n<td>0.05</td>\n</tr>\n<tr>\n<td>private</td>\n<td>0.72</td>\n<td>0.30</td>\n<td>0.01</td>\n<td>0.13</td>\n<td>0.26</td>\n<td>0.01</td>\n</tr>\n</tbody>\n</table>\n<h1>Dataset</h1>\n<p>I used the following three data sets.  </p>\n<ol>\n<li><p>Competition data set（comp_extracted_dataset/comp_generated_dataset）  </p>\n<ul>\n<li>Using both extracted dataset (comp_extracted_dataset) and generated dataset (comp_generated_dataset).  </li>\n<li>For the generated data, data with noise in the annotations were removed by a simple check (about 100 images).  </li></ul></li>\n<li><p>ICDAR data set（ICDAR_dataset/ICDAR_manualannot_dataset）  </p>\n<ul>\n<li>I used 1406 pieces of data for which annotations were provided (ICDAR_dataset) and 1903 pieces of data for which no annotations were provided (ICDAR_manualannot_dataset).  </li>\n<li>For data for which annotations were provided, I visually rechecked the annotation contents and manually corrected all data that did not follow the competition's annotation rules (e.g., % notation) or contained noise.  </li>\n<li>For the data for which no annotations were provided, I first visually checked the appearance of all the data and selected the data that could be used in this competition. Next, I inferred and assigned pseudo-labels using the Deplot model, visually checked all the results again, and manually corrected all incorrect annotations.  </li></ul></li>\n<li><p>Generated synthetic data set（synthetic_dataset）  </p>\n<ul>\n<li>After reviewing the images in the competition dataset and determining that the comp_generated_dataset alone did not have a sufficient variation to achieve robustness, so I generated about 65k synthetic data myself.  </li>\n<li>I mainly generated synthetic data with features that comp_generated_dataset does not have.  <ul>\n<li>histogram  </li>\n<li>Label contains line breaks  </li>\n<li>Bar chart with error bars  </li>\n<li>Line chart containing x labels not included in the data series</li>\n<li>etc…  </li></ul></li>\n<li>To add diversity, I also included 10k images from the dataset published by <a href=\"https://www.kaggle.com/brendanartley\" target=\"_blank\">@brendanartley</a>.  </li></ul>\n<p>(Example of synthetic data I generated)  <br>\n<img src=\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F3823496%2Fd4df82a41fcaebeda2ffc93fb8ad9159%2Fgenerated_data.jpg?generation=1687442722440643&amp;alt=media\" alt=\"generated_data\"></p></li>\n</ol>\n<h1>Solution pipeline</h1>\n<h2>1. Chart classification</h2>\n<ul>\n<li>There is not much to note, as I just performed a simple classification task.  </li>\n<li>I used two models, convnext_large_384 and swin_large_patch4_window12_384, and integrated their results in a weighted ensemble.  </li>\n</ul>\n<p><strong>Training configs</strong>  </p>\n<ul>\n<li>Dataset (training data and validation data were randomly divided into folds)  <ul>\n<li>Training（78k images）：comp_extracted_dataset, comp_generated_dataset, ICDAR_dataset, synthetic_dataset(horizontal_bar/dot)  </li>\n<li>Validation（4k images）：comp_extracted_dataset, ICDAR_dataset, synthetic_dataset(horizontal_bar/dot)</li></ul></li>\n<li>Hyperparameters<ul>\n<li>Epochs：15  </li>\n<li>Batch size：16  </li>\n<li>Adafactor (lr=3e-5)  </li></ul></li>\n</ul>\n<h2>2. Data series prediction</h2>\n<p>Based on the classification results of Step 1, different approaches are used for scatter and the rest.  <br>\nExcept for scatter, the predictions are made in Deplot, and scatter is an object detection-based prediction.  </p>\n<h3>2-a. Bars &amp; Lines &amp; Dots</h3>\n<ul>\n<li><p>I experimented by switching the model to Deplot based on the Donut-based notebook that <a href=\"https://www.kaggle.com/nbroad\" target=\"_blank\">@nbroad</a> had published. However, I could not train well just by changing the model, so I made some modifications.  </p>\n<ul>\n<li>Ground Truth Format  <br>\nRemoved chart type from GroundTruth and adapted the format to that of the Deplot original.  <br>\n<code>&lt;0x0A&gt; x_value1 | y_value1 &lt;0x0A&gt; x_value2 | y_value2 &lt;0x0A&gt; x_value3 | y_value3 &lt;/s&gt;</code></li>\n<li>x-axis and y-axis swap of horizontal_bar  <br>\nSince the concept of x-axis and y-axis was reversed in the annotation rules of this competition and the original format of Deplot, I trained according to the original concept of Deplot and swapped the values during inference.  </li></ul></li>\n<li><p>Training was conducted in multiple stages. In the first stage, training was conducted using data from all chart types (All Chart-type Train), and using the results of that training as initial weights, one or two additional training sessions (Specific Chart-type Train) were conducted using only specific chart-type images to generate a model specialized for a chart type.</p>\n<ul>\n<li>The scores for vertical_bar and line were slightly improved by this method. (Two Specific Chart-type Train runs were performed for vertical_bar and one for line.)  </li>\n<li>Horizontal_bar had a worse cv after the second stage of training, probably due to the small amount of extracted data, so I decided to use the All Chart-type Train model to make predictions.    </li>\n<li>Dot chart only has generated data and I decided that I could not validate it successfully, so I decided not to train after the second stage and use the All Chart-type Train model to make predictions.  <br>\n<img src=\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F3823496%2F0a86b394e87edd440600ac531288897c%2FDeplot_training.jpg?generation=1687442833429916&amp;alt=media\" alt=\"Deplot_training\"></li></ul></li>\n</ul>\n<p><strong>Training configs</strong>  </p>\n<ol>\n<li><p>All Chart-type Train  </p>\n<ul>\n<li>Dataset  <ul>\n<li>Train（120k images）：comp_extracted_dataset, comp_generated_dataset, ICDAR_dataset, <br>\nsynthetic_dataset  </li>\n<li>Validation（2.5k images）：comp_extracted_dataset, ICDAR_dataset, ICDAR_manualannot_dataset</li></ul></li>\n<li>Hyperparameters<ul>\n<li>Epochs：8</li>\n<li>Batch size：2  </li>\n<li>Adafactor (lr=1e-5)</li>\n<li>cosine scheduler with warmup (warmup_step=4000)</li>\n<li>Augmentation: GaussianBlur, GaussNoise, some color augmentations </li></ul></li></ul></li>\n<li><p>Specific Chart-type Train (vertical_bar)   </p>\n<ul>\n<li><p>Dataset  <br>\n[1st time]  </p>\n<ul>\n<li>Train（6k images）  ：comp_extracted_dataset, ICDAR_dataset, synthetic_dataset  </li>\n<li>Validation（1.3k images）：comp_extracted_dataset, ICDAR_dataset, ICDAR_manualannot_dataset  </li></ul>\n<p>[2nd time]  </p>\n<ul>\n<li>Train（1500 images）：comp_extracted_dataset, ICDAR_dataset, ICDAR_manualannot_dataset   </li>\n<li>Validation（500 images） ：comp_extracted_dataset, ICDAR_dataset, ICDAR_manualannot_dataset</li></ul></li>\n<li><p>Hyperparameters</p>\n<ul>\n<li>cosine scheduler with warmup (warmup_step=0)</li>\n<li>Otherwise, same as All Chart-type Train  </li></ul></li></ul></li>\n<li><p>Specific Chart-type Train (line)  </p>\n<ul>\n<li><p>Dataset  <br>\n[1st time]  </p>\n<ul>\n<li>Train（1150 images）：comp_extracted_dataset, ICDAR_dataset, ICDAR_manualannot_dataset  </li>\n<li>Validation（400 images）：comp_extracted_dataset, ICDAR_dataset, ICDAR_manualannot_dataset  </li></ul></li>\n<li><p>Hyper paramete</p>\n<ul>\n<li>cosine scheduler with warmup (warmup_step=0)</li>\n<li>Otherwise, same as All Chart-type Train  </li></ul></li></ul></li>\n</ol>\n<p><strong>Score</strong>  </p>\n<ul>\n<li><p>All Chart-type Train</p>\n<table>\n<thead>\n<tr>\n<th></th>\n<th>Overall</th>\n<th>Scatter</th>\n<th>Dot</th>\n<th>Line</th>\n<th>Vertical Bar</th>\n<th>Horizontal Bar</th>\n</tr>\n</thead>\n<tbody>\n<tr>\n<td>public</td>\n<td>0.78</td>\n<td>0.06</td>\n<td>0.00</td>\n<td>0.29</td>\n<td>0.38</td>\n<td>0.05</td>\n</tr>\n<tr>\n<td>private</td>\n<td>0.53</td>\n<td>0.13</td>\n<td>0.01</td>\n<td>0.12</td>\n<td>0.26</td>\n<td>0.01</td>\n</tr>\n</tbody>\n</table></li>\n<li><p>Specific Chart-type Train</p>\n<table>\n<thead>\n<tr>\n<th></th>\n<th>Overall</th>\n<th>Scatter</th>\n<th>Dot</th>\n<th>Line</th>\n<th>Vertical Bar</th>\n<th>Horizontal Bar</th>\n</tr>\n</thead>\n<tbody>\n<tr>\n<td>public</td>\n<td>0.81</td>\n<td>0.06</td>\n<td>0.00</td>\n<td>0.32</td>\n<td>0.39</td>\n<td>0.05</td>\n</tr>\n<tr>\n<td>private</td>\n<td>0.55</td>\n<td>0.13</td>\n<td>0.01</td>\n<td>0.13</td>\n<td>0.26</td>\n<td>0.01</td>\n</tr>\n</tbody>\n</table></li>\n</ul>\n<h3>2-b. Scatter</h3>\n<ul>\n<li>Only scatter was not accurate enough with Deplot no matter how I tried, so I adopted an object detection-based approach.  </li>\n<li>Since scatter is guaranteed to have label values of numeric type, it is easy to calculate values from ratios if the \"label position,\" \"label text,\" and \"scatter point position\" are known. I used CACHED to detect \"label position,\" Deplot to read \"label text,\" and YOLOX to detect \"scatter point\".  </li>\n<li>The mapping between label bboxes and label texts is a bit crude, but the smallest label text is mapped to the left-most (top in the case of y-labels) label bbox, and the largest label text is mapped to the right-most (bottom in the case of y-labels) label bbox. It is possible that the total number of detected label bboxes does not match the total number of read label texts, but we did not take any special measures because such a pattern was very rare when we checked the verification data.  </li>\n<li>The above object detection approach resulted in a scatter score of <strong>Pulic:0.10/Private:0.30</strong>, much better than the Deplot-based score (Pulic:0.06/Private:0.13).</li>\n</ul>\n<p><strong>More Details</strong>  <br>\nScatter point detection  </p>\n<ul>\n<li>comp_extracted_dataset, ICDAR_dataset, synthetic_dataset were used for training data (about 12.5k images)</li>\n<li>Somewhat semi-automated using provided annotation data and image processing, but inevitably noisy, so corrected with manual annotation.  </li>\n<li>To increase detection accuracy, only the plot area was cropped for training and inference.</li>\n<li>The model was YOLOX-l, input image size was 1280 for both training and inference, and 50 epochs were trained.</li>\n</ul>\n<p>Label text reading</p>\n<ul>\n<li>comp_extracted_dataset, ICDAR_dataset, comp_generated_dataset were used for training data (about 62k images, all chart types were used, not just scatter)  </li>\n<li>Use the same training configs as in (2-a) All_Chart-type_Train for the training parameters, and change the format of Ground Truth as follows  <br>\n<code>&lt;0x0A&gt; x_label1 | x_label2 | x_label3 &lt;0x0A&gt; y_label1 | y_label2 &lt;/s&gt;</code> </li>\n<li>At first, I thought of using EasyOCR or similar to achieve this, but there were many issues to be addressed, such as support for character rotation, and it seemed that tuning would be very difficult to increase versatility, so I adopted a method of batch reading with Deplot.  </li>\n</ul>\n<hr>\n<p>Finally, I would like to thank once again everyone involved in this competition.  <br>\nThank you very much for taking the time to read this!</p>\n<p>[edit] </p>\n<ul>\n<li>I have published 1st place <a href=\"https://www.kaggle.com/code/kashiwaba/benetech-1st-place-inference\" target=\"_blank\">inference notebook</a>(23/6/28) </li>\n<li>I have published 1st place <a href=\"https://github.com/KASSII/benetech_1st_place_train\" target=\"_blank\">training code</a>(23/7/8) </li>\n</ul>",
      "rawMarkdown": "First of all, I would like to pay tribute to all the participants who worked on this competition.   \nI would also like to thank the hosts for organizing this interesting task competition.    \nThis task was very interesting and I enjoyed working on it because I could think of many different approaches.    \nI am honored to have won first place in this very interesting competition.\n\n# Overview\n![Overview](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F3823496%2F16283ff691149c87b3b01a49a432f107%2Foverview.jpg?generation=1687442367636860&alt=media)\n\nMy solution consists of a two-step pipeline that first classifies chart types using a classification model and then performs data series inference.  \nIn the inference phase of the data series, Bar, Line, and Dot were end-to-end predictions by Deplot trained for each chart type, while Scatter was predicted by an object detection-based approach.  \n\nThe final scores are as follows.  \n| | Overall | Scatter | Dot | Line | Vertical Bar | Horizontal Bar |\n| ---     | ---  | ---  | ---  | ---  |  --- | ---  |\n| public  | 0.86 | 0.10 | 0.00 | 0.32 | 0.39 | 0.05 |\n| private | 0.72 | 0.30 | 0.01 | 0.13 | 0.26 | 0.01 | \n\n# Dataset\nI used the following three data sets.  \n1. Competition data set（comp_extracted_dataset/comp_generated_dataset）  \n  * Using both extracted dataset (comp_extracted_dataset) and generated dataset (comp_generated_dataset).  \n  * For the generated data, data with noise in the annotations were removed by a simple check (about 100 images).  \n\n2. ICDAR data set（ICDAR_dataset/ICDAR_manualannot_dataset）  \n  * I used 1406 pieces of data for which annotations were provided (ICDAR_dataset) and 1903 pieces of data for which no annotations were provided (ICDAR_manualannot_dataset).  \n  * For data for which annotations were provided, I visually rechecked the annotation contents and manually corrected all data that did not follow the competition's annotation rules (e.g., % notation) or contained noise.  \n  * For the data for which no annotations were provided, I first visually checked the appearance of all the data and selected the data that could be used in this competition. Next, I inferred and assigned pseudo-labels using the Deplot model, visually checked all the results again, and manually corrected all incorrect annotations.  \n\n3. Generated synthetic data set（synthetic_dataset）  \n  * After reviewing the images in the competition dataset and determining that the comp_generated_dataset alone did not have a sufficient variation to achieve robustness, so I generated about 65k synthetic data myself.  \n  * I mainly generated synthetic data with features that comp_generated_dataset does not have.  \n      * histogram  \n      * Label contains line breaks  \n      * Bar chart with error bars  \n      * Line chart containing x labels not included in the data series\n      * etc...  \n  * To add diversity, I also included 10k images from the dataset published by @brendanartley.  \n\n  (Example of synthetic data I generated)  \n![generated_data](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F3823496%2Fd4df82a41fcaebeda2ffc93fb8ad9159%2Fgenerated_data.jpg?generation=1687442722440643&alt=media)\n\n# Solution pipeline  \n## 1. Chart classification  \n* There is not much to note, as I just performed a simple classification task.  \n* I used two models, convnext_large_384 and swin_large_patch4_window12_384, and integrated their results in a weighted ensemble.  \n\n**Training configs**  \n* Dataset (training data and validation data were randomly divided into folds)  \n    * Training（78k images）：comp_extracted_dataset, comp_generated_dataset, ICDAR_dataset, synthetic_dataset(horizontal_bar/dot)  \n    * Validation（4k images）：comp_extracted_dataset, ICDAR_dataset, synthetic_dataset(horizontal_bar/dot)\n* Hyperparameters\n    * Epochs：15  \n    * Batch size：16  \n    * Adafactor (lr=3e-5)  \n\n## 2. Data series prediction  \nBased on the classification results of Step 1, different approaches are used for scatter and the rest.  \nExcept for scatter, the predictions are made in Deplot, and scatter is an object detection-based prediction.  \n\n### 2-a. Bars & Lines & Dots  \n* I experimented by switching the model to Deplot based on the Donut-based notebook that @nbroad had published. However, I could not train well just by changing the model, so I made some modifications.  \n    * Ground Truth Format  \n      Removed chart type from GroundTruth and adapted the format to that of the Deplot original.  \n      `<0x0A> x_value1 | y_value1 <0x0A> x_value2 | y_value2 <0x0A> x_value3 | y_value3 </s>`\n    * x-axis and y-axis swap of horizontal_bar  \n      Since the concept of x-axis and y-axis was reversed in the annotation rules of this competition and the original format of Deplot, I trained according to the original concept of Deplot and swapped the values during inference.  \n\n* Training was conducted in multiple stages. In the first stage, training was conducted using data from all chart types (All Chart-type Train), and using the results of that training as initial weights, one or two additional training sessions (Specific Chart-type Train) were conducted using only specific chart-type images to generate a model specialized for a chart type.\n    * The scores for vertical_bar and line were slightly improved by this method. (Two Specific Chart-type Train runs were performed for vertical_bar and one for line.)  \n    * Horizontal_bar had a worse cv after the second stage of training, probably due to the small amount of extracted data, so I decided to use the All Chart-type Train model to make predictions.    \n    * Dot chart only has generated data and I decided that I could not validate it successfully, so I decided not to train after the second stage and use the All Chart-type Train model to make predictions.  \n![Deplot_training](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F3823496%2F0a86b394e87edd440600ac531288897c%2FDeplot_training.jpg?generation=1687442833429916&alt=media)\n\n**Training configs**  \n1. All Chart-type Train  \n    * Dataset  \n        * Train（120k images）：comp_extracted_dataset, comp_generated_dataset, ICDAR_dataset, \n    synthetic_dataset  \n        * Validation（2.5k images）：comp_extracted_dataset, ICDAR_dataset, ICDAR_manualannot_dataset\n    * Hyperparameters\n        * Epochs：8\n        * Batch size：2  \n        * Adafactor (lr=1e-5)\n        * cosine scheduler with warmup (warmup_step=4000)\n        * Augmentation: GaussianBlur, GaussNoise, some color augmentations \n\n2. Specific Chart-type Train (vertical_bar)   \n    * Dataset  \n      [1st time]  \n        * Train（6k images）  ：comp_extracted_dataset, ICDAR_dataset, synthetic_dataset  \n        * Validation（1.3k images）：comp_extracted_dataset, ICDAR_dataset, ICDAR_manualannot_dataset  \n\n      [2nd time]  \n        * Train（1500 images）：comp_extracted_dataset, ICDAR_dataset, ICDAR_manualannot_dataset   \n        * Validation（500 images） ：comp_extracted_dataset, ICDAR_dataset, ICDAR_manualannot_dataset\n\n    * Hyperparameters\n        * cosine scheduler with warmup (warmup_step=0)\n        * Otherwise, same as All Chart-type Train  \n\n3. Specific Chart-type Train (line)  \n    * Dataset  \n      [1st time]  \n        * Train（1150 images）：comp_extracted_dataset, ICDAR_dataset, ICDAR_manualannot_dataset  \n        * Validation（400 images）：comp_extracted_dataset, ICDAR_dataset, ICDAR_manualannot_dataset  \n\n    * Hyper paramete\n        * cosine scheduler with warmup (warmup_step=0)\n        * Otherwise, same as All Chart-type Train  \n\n**Score**  \n* All Chart-type Train\n| | Overall | Scatter | Dot | Line | Vertical Bar | Horizontal Bar |\n| ---     | ---  | ---  | ---  | ---  |  --- | ---  |\n| public  | 0.78 | 0.06 | 0.00 | 0.29 | 0.38 | 0.05 |\n| private | 0.53 | 0.13 | 0.01 | 0.12 | 0.26 | 0.01 | \n\n\n* Specific Chart-type Train\n| | Overall | Scatter | Dot | Line | Vertical Bar | Horizontal Bar |\n| ---     | ---  | ---  | ---  | ---  |  --- | ---  |\n| public  | 0.81 | 0.06 | 0.00 | 0.32 | 0.39 | 0.05 |\n| private | 0.55 | 0.13 | 0.01 | 0.13 | 0.26 | 0.01 | \n\n\n### 2-b. Scatter\n* Only scatter was not accurate enough with Deplot no matter how I tried, so I adopted an object detection-based approach.  \n* Since scatter is guaranteed to have label values of numeric type, it is easy to calculate values from ratios if the \"label position,\" \"label text,\" and \"scatter point position\" are known. I used CACHED to detect \"label position,\" Deplot to read \"label text,\" and YOLOX to detect \"scatter point\".  \n* The mapping between label bboxes and label texts is a bit crude, but the smallest label text is mapped to the left-most (top in the case of y-labels) label bbox, and the largest label text is mapped to the right-most (bottom in the case of y-labels) label bbox. It is possible that the total number of detected label bboxes does not match the total number of read label texts, but we did not take any special measures because such a pattern was very rare when we checked the verification data.  \n* The above object detection approach resulted in a scatter score of **Pulic:0.10/Private:0.30**, much better than the Deplot-based score (Pulic:0.06/Private:0.13).\n\n**More Details**  \nScatter point detection  \n* comp_extracted_dataset, ICDAR_dataset, synthetic_dataset were used for training data (about 12.5k images)\n* Somewhat semi-automated using provided annotation data and image processing, but inevitably noisy, so corrected with manual annotation.  \n* To increase detection accuracy, only the plot area was cropped for training and inference.\n* The model was YOLOX-l, input image size was 1280 for both training and inference, and 50 epochs were trained.\n\nLabel text reading\n* comp_extracted_dataset, ICDAR_dataset, comp_generated_dataset were used for training data (about 62k images, all chart types were used, not just scatter)  \n* Use the same training configs as in (2-a) All_Chart-type_Train for the training parameters, and change the format of Ground Truth as follows  \n  `<0x0A> x_label1 | x_label2 | x_label3 <0x0A> y_label1 | y_label2 </s>` \n* At first, I thought of using EasyOCR or similar to achieve this, but there were many issues to be addressed, such as support for character rotation, and it seemed that tuning would be very difficult to increase versatility, so I adopted a method of batch reading with Deplot.  \n\n---\nFinally, I would like to thank once again everyone involved in this competition.  \nThank you very much for taking the time to read this!\n\n[edit] \n* I have published 1st place [inference notebook](https://www.kaggle.com/code/kashiwaba/benetech-1st-place-inference)(23/6/28) \n* I have published 1st place [training code](https://github.com/KASSII/benetech_1st_place_train)(23/7/8) ",
      "votes": 60
    },
    {
      "id": 2314005,
      "postDate": "2023-06-23T05:29:42.903Z",
      "content": "<p><a href=\"https://www.kaggle.com/kashiwaba\" target=\"_blank\">@kashiwaba</a> congrats with 1st place! Complex and nice solution!<br>\nIf you can please share the code. Thanks!</p>",
      "rawMarkdown": "@kashiwaba congrats with 1st place! Complex and nice solution!\nIf you can please share the code. Thanks!",
      "votes": 1,
      "replies": [
        {
          "id": 2315712,
          "postDate": "2023-06-24T10:44:16.023Z",
          "content": "<p>Thank you.<br>\nI may make it public once I get it sorted out.</p>",
          "rawMarkdown": "Thank you.\nI may make it public once I get it sorted out.",
          "votes": 2
        }
      ]
    },
    {
      "id": 2313607,
      "postDate": "2023-06-22T19:08:29.820Z",
      "content": "<p>Wow, tons of work. Well deserved! <br>\nThanks for the detailed write-up and congratz again on the win! </p>",
      "rawMarkdown": "Wow, tons of work. Well deserved! \nThanks for the detailed write-up and congratz again on the win! ",
      "votes": 1
    },
    {
      "id": 2313586,
      "postDate": "2023-06-22T18:58:51.737Z",
      "content": "<p>Thank you for the thorough write-up, and congratulations on securing the 1st place.</p>\n<p>I have a few questions regarding your approach:</p>\n<ol>\n<li>Have you observed any improvements by applying image augmentation techniques? I attempted to implement them as well but did not observe any noticeable enhancements.</li>\n<li>Could you please share the number of epochs you trained the model for during the second stage, specifically for chart-specific training?</li>\n<li>I would appreciate it if you could provide further clarification on how you established the mapping between label text and its corresponding coordinates.</li>\n</ol>",
      "rawMarkdown": "Thank you for the thorough write-up, and congratulations on securing the 1st place.\n\nI have a few questions regarding your approach:\n1. Have you observed any improvements by applying image augmentation techniques? I attempted to implement them as well but did not observe any noticeable enhancements.\n2. Could you please share the number of epochs you trained the model for during the second stage, specifically for chart-specific training?\n3. I would appreciate it if you could provide further clarification on how you established the mapping between label text and its corresponding coordinates.",
      "votes": 1,
      "replies": [
        {
          "id": 2315730,
          "postDate": "2023-06-24T11:09:51.277Z",
          "content": "<p>Thank you.</p>\n<ol>\n<li><p>I honestly don't know because I haven't pursued it that far, but Augmentation was introduced when I was experimenting with Donut. The score remained the same or increased slightly, but the number of epochs of convergence increased, allowing for longer learning, so I kept using it.<br>\nAlso, the model we used this time was highly expressive and easily overfitted, so we intuitively judged Augmentation to be effective, which is another reason why we used it.</p></li>\n<li><p>All parameters are the same except the learning rate and scheduler. However, the actual number of epochs in which the model used was generated should have been much earlier, since the validation score was output for each epoch and the weights at the best time were used.</p></li>\n</ol>",
          "rawMarkdown": "Thank you.\n\n1. I honestly don't know because I haven't pursued it that far, but Augmentation was introduced when I was experimenting with Donut. The score remained the same or increased slightly, but the number of epochs of convergence increased, allowing for longer learning, so I kept using it.\nAlso, the model we used this time was highly expressive and easily overfitted, so we intuitively judged Augmentation to be effective, which is another reason why we used it.\n\n2. All parameters are the same except the learning rate and scheduler. However, the actual number of epochs in which the model used was generated should have been much earlier, since the validation score was output for each epoch and the weights at the best time were used.",
          "votes": 2,
          "replies": [
            {
              "id": 2315734,
              "postDate": "2023-06-24T11:14:40.200Z",
              "content": "<p>3 . <br>\nAs shown in the figure below, I simply tied the smallest and largest indexed readings output by Deplot to the smallest and largest bbox coordinates detected by the detection model. All other results are completely ignored.<br>\nIn other words, as long as the minimum and maximum values are correct, the other results are not affected at all even if they are wrong, but conversely, if the minimum and maximum values are wrong, the accuracy will decrease even if the other results are correct.<br>\nIn fact, it would be better to take this into account in the implementation, but since there was not enough time and the accuracy was still sufficient, a compromise was made.<br>\nI consider this part to be an area for improvement.<br>\n<img src=\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F3823496%2F50ca73f954168bddf1b6d4512aaedc53%2Fbenetech_answer.jpg?generation=1687605259155019&amp;alt=media\" alt=\"\"></p>",
              "rawMarkdown": "3 . \nAs shown in the figure below, I simply tied the smallest and largest indexed readings output by Deplot to the smallest and largest bbox coordinates detected by the detection model. All other results are completely ignored.\nIn other words, as long as the minimum and maximum values are correct, the other results are not affected at all even if they are wrong, but conversely, if the minimum and maximum values are wrong, the accuracy will decrease even if the other results are correct.\nIn fact, it would be better to take this into account in the implementation, but since there was not enough time and the accuracy was still sufficient, a compromise was made.\nI consider this part to be an area for improvement.\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F3823496%2F50ca73f954168bddf1b6d4512aaedc53%2Fbenetech_answer.jpg?generation=1687605259155019&alt=media)",
              "votes": 2
            },
            {
              "id": 2317687,
              "postDate": "2023-06-25T21:37:29.063Z",
              "content": "<p>Thank you for the reply.</p>",
              "rawMarkdown": "Thank you for the reply."
            }
          ]
        }
      ]
    },
    {
      "id": 2313255,
      "postDate": "2023-06-22T14:58:43.140Z",
      "content": "<p>Great 😍 Congratulations on winning!</p>",
      "rawMarkdown": "Great 😍 Congratulations on winning!",
      "votes": 1
    },
    {
      "id": 2313252,
      "postDate": "2023-06-22T14:57:08.183Z",
      "content": "<p>Thanks for the write-up and congratz again on the win !</p>",
      "rawMarkdown": "Thanks for the write-up and congratz again on the win !",
      "votes": 1
    },
    {
      "id": 2313650,
      "postDate": "2023-06-22T20:01:57.410Z",
      "content": "<p>OMG! What an excellent solution! Well deserved for the first place!<br>\nDo you mind sharing the script generating the synthetic graphs? Did you generate them based on the script shared by <a href=\"https://www.kaggle.com/brendanartley\" target=\"_blank\">@brendanartley</a>?<br>\nCongrats again for the win!</p>",
      "rawMarkdown": "OMG! What an excellent solution! Well deserved for the first place!\nDo you mind sharing the script generating the synthetic graphs? Did you generate them based on the script shared by @brendanartley?\nCongrats again for the win!",
      "votes": 2,
      "replies": [
        {
          "id": 2315715,
          "postDate": "2023-06-24T10:47:11.580Z",
          "content": "<p>Thank you.<br>\nThere are some differences from Bartley's source code as I created the original.<br>\nI may release it once I have it organized to a level that others can read, along with other scripts.</p>",
          "rawMarkdown": "Thank you.\nThere are some differences from Bartley's source code as I created the original.\nI may release it once I have it organized to a level that others can read, along with other scripts.",
          "votes": 1,
          "replies": [
            {
              "id": 2350057,
              "postDate": "2023-07-19T00:05:01.230Z",
              "content": "<p>Thank you for sharing the code!!!</p>",
              "rawMarkdown": "Thank you for sharing the code!!!"
            }
          ]
        }
      ]
    },
    {
      "id": 2361049,
      "postDate": "2023-07-27T06:45:13.313Z",
      "content": "<p>Congratulations!</p>",
      "rawMarkdown": "Congratulations!"
    },
    {
      "id": 2315769,
      "postDate": "2023-06-24T12:01:44.620Z",
      "content": "<p>Congratulations on winning first place in the competition! Your solution pipeline and approach utilizing classification and object detection models are impressive. The use of multiple datasets and synthetic data helped improve the robustness of your model. Well done on your outstanding performance!</p>",
      "rawMarkdown": "Congratulations on winning first place in the competition! Your solution pipeline and approach utilizing classification and object detection models are impressive. The use of multiple datasets and synthetic data helped improve the robustness of your model. Well done on your outstanding performance!"
    },
    {
      "id": 2403436,
      "postDate": "2023-08-22T17:23:35.567Z",
      "rawMarkdown": "",
      "isDeleted": true
    }
  ],
  "comments": [
    {
      "id": 2314005,
      "author_name": "serangu",
      "author_url": "",
      "post_date": "2023-06-23T05:29:42.903000",
      "content": "<p><a href=\"https://www.kaggle.com/kashiwaba\" target=\"_blank\">@kashiwaba</a> congrats with 1st place! Complex and nice solution!<br>\nIf you can please share the code. Thanks!</p>",
      "votes": 1,
      "replies": [
        {
          "id": 2315712,
          "author_name": "YumeNeko",
          "author_url": "",
          "post_date": "2023-06-24T10:44:16.023000",
          "content": "<p>Thank you.<br>\nI may make it public once I get it sorted out.</p>",
          "votes": 2,
          "replies": []
        }
      ]
    },
    {
      "id": 2313607,
      "author_name": "cooleel",
      "author_url": "",
      "post_date": "2023-06-22T19:08:29.820000",
      "content": "<p>Wow, tons of work. Well deserved! <br>\nThanks for the detailed write-up and congratz again on the win! </p>",
      "votes": 1,
      "replies": []
    },
    {
      "id": 2313586,
      "author_name": "TanjiroLL",
      "author_url": "",
      "post_date": "2023-06-22T18:58:51.737000",
      "content": "<p>Thank you for the thorough write-up, and congratulations on securing the 1st place.</p>\n<p>I have a few questions regarding your approach:</p>\n<ol>\n<li>Have you observed any improvements by applying image augmentation techniques? I attempted to implement them as well but did not observe any noticeable enhancements.</li>\n<li>Could you please share the number of epochs you trained the model for during the second stage, specifically for chart-specific training?</li>\n<li>I would appreciate it if you could provide further clarification on how you established the mapping between label text and its corresponding coordinates.</li>\n</ol>",
      "votes": 1,
      "replies": [
        {
          "id": 2315730,
          "author_name": "YumeNeko",
          "author_url": "",
          "post_date": "2023-06-24T11:09:51.277000",
          "content": "<p>Thank you.</p>\n<ol>\n<li><p>I honestly don't know because I haven't pursued it that far, but Augmentation was introduced when I was experimenting with Donut. The score remained the same or increased slightly, but the number of epochs of convergence increased, allowing for longer learning, so I kept using it.<br>\nAlso, the model we used this time was highly expressive and easily overfitted, so we intuitively judged Augmentation to be effective, which is another reason why we used it.</p></li>\n<li><p>All parameters are the same except the learning rate and scheduler. However, the actual number of epochs in which the model used was generated should have been much earlier, since the validation score was output for each epoch and the weights at the best time were used.</p></li>\n</ol>",
          "votes": 2,
          "replies": [
            {
              "id": 2315734,
              "author_name": "YumeNeko",
              "author_url": "",
              "post_date": "2023-06-24T11:14:40.200000",
              "content": "<p>3 . <br>\nAs shown in the figure below, I simply tied the smallest and largest indexed readings output by Deplot to the smallest and largest bbox coordinates detected by the detection model. All other results are completely ignored.<br>\nIn other words, as long as the minimum and maximum values are correct, the other results are not affected at all even if they are wrong, but conversely, if the minimum and maximum values are wrong, the accuracy will decrease even if the other results are correct.<br>\nIn fact, it would be better to take this into account in the implementation, but since there was not enough time and the accuracy was still sufficient, a compromise was made.<br>\nI consider this part to be an area for improvement.<br>\n<img src=\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F3823496%2F50ca73f954168bddf1b6d4512aaedc53%2Fbenetech_answer.jpg?generation=1687605259155019&amp;alt=media\" alt=\"\"></p>",
              "votes": 2,
              "replies": []
            },
            {
              "id": 2317687,
              "author_name": "TanjiroLL",
              "author_url": "",
              "post_date": "2023-06-25T21:37:29.063000",
              "content": "<p>Thank you for the reply.</p>",
              "votes": 0,
              "replies": []
            }
          ]
        }
      ]
    },
    {
      "id": 2313255,
      "author_name": "Md. Hasnain Ali",
      "author_url": "",
      "post_date": "2023-06-22T14:58:43.140000",
      "content": "<p>Great 😍 Congratulations on winning!</p>",
      "votes": 1,
      "replies": []
    },
    {
      "id": 2313252,
      "author_name": "Theo Viel",
      "author_url": "",
      "post_date": "2023-06-22T14:57:08.183000",
      "content": "<p>Thanks for the write-up and congratz again on the win !</p>",
      "votes": 1,
      "replies": []
    },
    {
      "id": 2313650,
      "author_name": "june",
      "author_url": "",
      "post_date": "2023-06-22T20:01:57.410000",
      "content": "<p>OMG! What an excellent solution! Well deserved for the first place!<br>\nDo you mind sharing the script generating the synthetic graphs? Did you generate them based on the script shared by <a href=\"https://www.kaggle.com/brendanartley\" target=\"_blank\">@brendanartley</a>?<br>\nCongrats again for the win!</p>",
      "votes": 2,
      "replies": [
        {
          "id": 2315715,
          "author_name": "YumeNeko",
          "author_url": "",
          "post_date": "2023-06-24T10:47:11.580000",
          "content": "<p>Thank you.<br>\nThere are some differences from Bartley's source code as I created the original.<br>\nI may release it once I have it organized to a level that others can read, along with other scripts.</p>",
          "votes": 1,
          "replies": [
            {
              "id": 2350057,
              "author_name": "june",
              "author_url": "",
              "post_date": "2023-07-19T00:05:01.230000",
              "content": "<p>Thank you for sharing the code!!!</p>",
              "votes": 0,
              "replies": []
            }
          ]
        }
      ]
    },
    {
      "id": 2361049,
      "author_name": "Zacchaeus",
      "author_url": "",
      "post_date": "2023-07-27T06:45:13.313000",
      "content": "<p>Congratulations!</p>",
      "votes": 0,
      "replies": []
    },
    {
      "id": 2315769,
      "author_name": "Pooja Chauhan",
      "author_url": "",
      "post_date": "2023-06-24T12:01:44.620000",
      "content": "<p>Congratulations on winning first place in the competition! Your solution pipeline and approach utilizing classification and object detection models are impressive. The use of multiple datasets and synthetic data helped improve the robustness of your model. Well done on your outstanding performance!</p>",
      "votes": 0,
      "replies": []
    },
    {
      "id": 2403436,
      "author_name": "",
      "author_url": "",
      "post_date": "2023-08-22T17:23:35.567000",
      "content": "",
      "votes": 0,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "2313203": "First of all, I would like to pay tribute to all the participants who worked on this competition.   \nI would also like to thank the hosts for organizing this interesting task competition.    \nThis task was very interesting and I enjoyed working on it because I could think of many different approaches.    \nI am honored to have won first place in this very interesting competition.\n\n# Overview\n![Overview](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F3823496%2F16283ff691149c87b3b01a49a432f107%2Foverview.jpg?generation=1687442367636860&alt=media)\n\nMy solution consists of a two-step pipeline that first classifies chart types using a classification model and then performs data series inference.  \nIn the inference phase of the data series, Bar, Line, and Dot were end-to-end predictions by Deplot trained for each chart type, while Scatter was predicted by an object detection-based approach.  \n\nThe final scores are as follows.  \n| | Overall | Scatter | Dot | Line | Vertical Bar | Horizontal Bar |\n| ---     | ---  | ---  | ---  | ---  |  --- | ---  |\n| public  | 0.86 | 0.10 | 0.00 | 0.32 | 0.39 | 0.05 |\n| private | 0.72 | 0.30 | 0.01 | 0.13 | 0.26 | 0.01 | \n\n# Dataset\nI used the following three data sets.  \n1. Competition data set（comp_extracted_dataset/comp_generated_dataset）  \n  * Using both extracted dataset (comp_extracted_dataset) and generated dataset (comp_generated_dataset).  \n  * For the generated data, data with noise in the annotations were removed by a simple check (about 100 images).  \n\n2. ICDAR data set（ICDAR_dataset/ICDAR_manualannot_dataset）  \n  * I used 1406 pieces of data for which annotations were provided (ICDAR_dataset) and 1903 pieces of data for which no annotations were provided (ICDAR_manualannot_dataset).  \n  * For data for which annotations were provided, I visually rechecked the annotation contents and manually corrected all data that did not follow the competition's annotation rules (e.g., % notation) or contained noise.  \n  * For the data for which no annotations were provided, I first visually checked the appearance of all the data and selected the data that could be used in this competition. Next, I inferred and assigned pseudo-labels using the Deplot model, visually checked all the results again, and manually corrected all incorrect annotations.  \n\n3. Generated synthetic data set（synthetic_dataset）  \n  * After reviewing the images in the competition dataset and determining that the comp_generated_dataset alone did not have a sufficient variation to achieve robustness, so I generated about 65k synthetic data myself.  \n  * I mainly generated synthetic data with features that comp_generated_dataset does not have.  \n      * histogram  \n      * Label contains line breaks  \n      * Bar chart with error bars  \n      * Line chart containing x labels not included in the data series\n      * etc...  \n  * To add diversity, I also included 10k images from the dataset published by @brendanartley.  \n\n  (Example of synthetic data I generated)  \n![generated_data](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F3823496%2Fd4df82a41fcaebeda2ffc93fb8ad9159%2Fgenerated_data.jpg?generation=1687442722440643&alt=media)\n\n# Solution pipeline  \n## 1. Chart classification  \n* There is not much to note, as I just performed a simple classification task.  \n* I used two models, convnext_large_384 and swin_large_patch4_window12_384, and integrated their results in a weighted ensemble.  \n\n**Training configs**  \n* Dataset (training data and validation data were randomly divided into folds)  \n    * Training（78k images）：comp_extracted_dataset, comp_generated_dataset, ICDAR_dataset, synthetic_dataset(horizontal_bar/dot)  \n    * Validation（4k images）：comp_extracted_dataset, ICDAR_dataset, synthetic_dataset(horizontal_bar/dot)\n* Hyperparameters\n    * Epochs：15  \n    * Batch size：16  \n    * Adafactor (lr=3e-5)  \n\n## 2. Data series prediction  \nBased on the classification results of Step 1, different approaches are used for scatter and the rest.  \nExcept for scatter, the predictions are made in Deplot, and scatter is an object detection-based prediction.  \n\n### 2-a. Bars & Lines & Dots  \n* I experimented by switching the model to Deplot based on the Donut-based notebook that @nbroad had published. However, I could not train well just by changing the model, so I made some modifications.  \n    * Ground Truth Format  \n      Removed chart type from GroundTruth and adapted the format to that of the Deplot original.  \n      `<0x0A> x_value1 | y_value1 <0x0A> x_value2 | y_value2 <0x0A> x_value3 | y_value3 </s>`\n    * x-axis and y-axis swap of horizontal_bar  \n      Since the concept of x-axis and y-axis was reversed in the annotation rules of this competition and the original format of Deplot, I trained according to the original concept of Deplot and swapped the values during inference.  \n\n* Training was conducted in multiple stages. In the first stage, training was conducted using data from all chart types (All Chart-type Train), and using the results of that training as initial weights, one or two additional training sessions (Specific Chart-type Train) were conducted using only specific chart-type images to generate a model specialized for a chart type.\n    * The scores for vertical_bar and line were slightly improved by this method. (Two Specific Chart-type Train runs were performed for vertical_bar and one for line.)  \n    * Horizontal_bar had a worse cv after the second stage of training, probably due to the small amount of extracted data, so I decided to use the All Chart-type Train model to make predictions.    \n    * Dot chart only has generated data and I decided that I could not validate it successfully, so I decided not to train after the second stage and use the All Chart-type Train model to make predictions.  \n![Deplot_training](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F3823496%2F0a86b394e87edd440600ac531288897c%2FDeplot_training.jpg?generation=1687442833429916&alt=media)\n\n**Training configs**  \n1. All Chart-type Train  \n    * Dataset  \n        * Train（120k images）：comp_extracted_dataset, comp_generated_dataset, ICDAR_dataset, \n    synthetic_dataset  \n        * Validation（2.5k images）：comp_extracted_dataset, ICDAR_dataset, ICDAR_manualannot_dataset\n    * Hyperparameters\n        * Epochs：8\n        * Batch size：2  \n        * Adafactor (lr=1e-5)\n        * cosine scheduler with warmup (warmup_step=4000)\n        * Augmentation: GaussianBlur, GaussNoise, some color augmentations \n\n2. Specific Chart-type Train (vertical_bar)   \n    * Dataset  \n      [1st time]  \n        * Train（6k images）  ：comp_extracted_dataset, ICDAR_dataset, synthetic_dataset  \n        * Validation（1.3k images）：comp_extracted_dataset, ICDAR_dataset, ICDAR_manualannot_dataset  \n\n      [2nd time]  \n        * Train（1500 images）：comp_extracted_dataset, ICDAR_dataset, ICDAR_manualannot_dataset   \n        * Validation（500 images） ：comp_extracted_dataset, ICDAR_dataset, ICDAR_manualannot_dataset\n\n    * Hyperparameters\n        * cosine scheduler with warmup (warmup_step=0)\n        * Otherwise, same as All Chart-type Train  \n\n3. Specific Chart-type Train (line)  \n    * Dataset  \n      [1st time]  \n        * Train（1150 images）：comp_extracted_dataset, ICDAR_dataset, ICDAR_manualannot_dataset  \n        * Validation（400 images）：comp_extracted_dataset, ICDAR_dataset, ICDAR_manualannot_dataset  \n\n    * Hyper paramete\n        * cosine scheduler with warmup (warmup_step=0)\n        * Otherwise, same as All Chart-type Train  \n\n**Score**  \n* All Chart-type Train\n| | Overall | Scatter | Dot | Line | Vertical Bar | Horizontal Bar |\n| ---     | ---  | ---  | ---  | ---  |  --- | ---  |\n| public  | 0.78 | 0.06 | 0.00 | 0.29 | 0.38 | 0.05 |\n| private | 0.53 | 0.13 | 0.01 | 0.12 | 0.26 | 0.01 | \n\n\n* Specific Chart-type Train\n| | Overall | Scatter | Dot | Line | Vertical Bar | Horizontal Bar |\n| ---     | ---  | ---  | ---  | ---  |  --- | ---  |\n| public  | 0.81 | 0.06 | 0.00 | 0.32 | 0.39 | 0.05 |\n| private | 0.55 | 0.13 | 0.01 | 0.13 | 0.26 | 0.01 | \n\n\n### 2-b. Scatter\n* Only scatter was not accurate enough with Deplot no matter how I tried, so I adopted an object detection-based approach.  \n* Since scatter is guaranteed to have label values of numeric type, it is easy to calculate values from ratios if the \"label position,\" \"label text,\" and \"scatter point position\" are known. I used CACHED to detect \"label position,\" Deplot to read \"label text,\" and YOLOX to detect \"scatter point\".  \n* The mapping between label bboxes and label texts is a bit crude, but the smallest label text is mapped to the left-most (top in the case of y-labels) label bbox, and the largest label text is mapped to the right-most (bottom in the case of y-labels) label bbox. It is possible that the total number of detected label bboxes does not match the total number of read label texts, but we did not take any special measures because such a pattern was very rare when we checked the verification data.  \n* The above object detection approach resulted in a scatter score of **Pulic:0.10/Private:0.30**, much better than the Deplot-based score (Pulic:0.06/Private:0.13).\n\n**More Details**  \nScatter point detection  \n* comp_extracted_dataset, ICDAR_dataset, synthetic_dataset were used for training data (about 12.5k images)\n* Somewhat semi-automated using provided annotation data and image processing, but inevitably noisy, so corrected with manual annotation.  \n* To increase detection accuracy, only the plot area was cropped for training and inference.\n* The model was YOLOX-l, input image size was 1280 for both training and inference, and 50 epochs were trained.\n\nLabel text reading\n* comp_extracted_dataset, ICDAR_dataset, comp_generated_dataset were used for training data (about 62k images, all chart types were used, not just scatter)  \n* Use the same training configs as in (2-a) All_Chart-type_Train for the training parameters, and change the format of Ground Truth as follows  \n  `<0x0A> x_label1 | x_label2 | x_label3 <0x0A> y_label1 | y_label2 </s>` \n* At first, I thought of using EasyOCR or similar to achieve this, but there were many issues to be addressed, such as support for character rotation, and it seemed that tuning would be very difficult to increase versatility, so I adopted a method of batch reading with Deplot.  \n\n---\nFinally, I would like to thank once again everyone involved in this competition.  \nThank you very much for taking the time to read this!\n\n[edit] \n* I have published 1st place [inference notebook](https://www.kaggle.com/code/kashiwaba/benetech-1st-place-inference)(23/6/28) \n* I have published 1st place [training code](https://github.com/KASSII/benetech_1st_place_train)(23/7/8) ",
    "2314005": "@kashiwaba congrats with 1st place! Complex and nice solution!\nIf you can please share the code. Thanks!",
    "2313607": "Wow, tons of work. Well deserved! \nThanks for the detailed write-up and congratz again on the win! ",
    "2313586": "Thank you for the thorough write-up, and congratulations on securing the 1st place.\n\nI have a few questions regarding your approach:\n1. Have you observed any improvements by applying image augmentation techniques? I attempted to implement them as well but did not observe any noticeable enhancements.\n2. Could you please share the number of epochs you trained the model for during the second stage, specifically for chart-specific training?\n3. I would appreciate it if you could provide further clarification on how you established the mapping between label text and its corresponding coordinates.",
    "2313255": "Great 😍 Congratulations on winning!",
    "2313252": "Thanks for the write-up and congratz again on the win !",
    "2313650": "OMG! What an excellent solution! Well deserved for the first place!\nDo you mind sharing the script generating the synthetic graphs? Did you generate them based on the script shared by @brendanartley?\nCongrats again for the win!",
    "2361049": "Congratulations!",
    "2315769": "Congratulations on winning first place in the competition! Your solution pipeline and approach utilizing classification and object detection models are impressive. The use of multiple datasets and synthetic data helped improve the robustness of your model. Well done on your outstanding performance!",
    "2403436": ""
  },
  "complete": true,
  "source": "existing_discussion",
  "competition_id": 43873,
  "competition_slug": "benetech-making-graphs-accessible",
  "team_id": 10186935,
  "solution_url": "/competitions/benetech-making-graphs-accessible/writeups/yumeneko-1st-place-solution"
}