\n
![\"“”\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F9250575%2F93bb591a9717e095b9f422c224b31c02%2Fline61.jpg?generation=1687298963057596&alt=media\")
\n
![\"“”\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F9250575%2F6f24a6ba2660e0440b150164d6390ab6%2Fvbar3.jpg?generation=1687298981955314&alt=media\")
Many thanks to Kaggle and Benetech for this interesting competition where many different approaches are possible and ideas are endless. Given the strong distribution shift in the private test set, we are quite happy with our finish.
\nSummary
\nOur solution consists of 3 main components:
\n1. Synthetic data generation
\nEarly on we realized this competition is unique in the sense that we are not limited to the available dataset. If the model struggles with certain plot patterns, we can generate a large number of plots with these patterns and as the model (matcha) is very strong, it will learn.
\nWe started by building upon this amazing repo: https://github.com/rakutentech/chart-synthesizer. We used both competition’s extracted data and ICDAR data as validation and try to emulate as many patterns from them as possible. For example:
\nSome examples of our generated data:
\n
\n
With each round of training, we would analyze the validation set, locate the patterns that give low score, and add these patterns to our chart generation code. We made the rise from 0.74 to 0.82 public LB simply by repeating this process multiple times.
\nWith our generation code we can create as many bar, line and scatter plots as necessary for each training stage. For dot plot we sampled 10k images from the great dataset provided by @brendanartley
\n2. Three-stage training of matcha models
\nThanks to the kind sharing by @nbroad , we started the competition training matcha-base models and quickly reached a good score.
The training is done in 3 stages:
\nmatcha-base checkpoint as a chart classifier. The model is trained for 5 epochs, using little synthetic data and oversampling extracted data.At inference time, the chart classifier is run first and then each group of charts will be handled by its dedicated model.
\nAt the beginning our process had only step 2 and 3, and we used model from step 2 as classifier. But then we observed that adding a lot of synthetic data decreases the performance of classifying task. As a result, training for the classifier was separated.
\nIt is also possible to perform classifying task with a simple CNN, however from our early experiments we saw that using matcha gave a little better performance, so we sticked with its usage.
\nSome training details:
\n<chart_type><start>x1|y1;…;xn|yn<end>3. Scatter plots processing
\nWhile the mentioned above method helped us reach good results for bar, dot and line charts, we found scatter plots much harder for matcha to handle. As a result, we developed a separate scheme only for scatter:
\nThis process helped us score 0.09 public and 0.28 private for scatter and is the key for us to survive the private test set.
\nThank you for reading and please let us know if you have any questions.
\nEdit: Inference notebook for our solution now available: https://www.kaggle.com/code/qdv206/benetech-5th-place-inference
", "messages": [ { "id": "2311069", "postDate": "06/20/2023 22:18:53", "content": "Many thanks to Kaggle and Benetech for this interesting competition where many different approaches are possible and ideas are endless. Given the strong distribution shift in the private test set, we are quite happy with our finish.
\nSummary
\nOur solution consists of 3 main components:
\n1. Synthetic data generation
\nEarly on we realized this competition is unique in the sense that we are not limited to the available dataset. If the model struggles with certain plot patterns, we can generate a large number of plots with these patterns and as the model (matcha) is very strong, it will learn.
\nWe started by building upon this amazing repo: https://github.com/rakutentech/chart-synthesizer. We used both competition’s extracted data and ICDAR data as validation and try to emulate as many patterns from them as possible. For example:
\nSome examples of our generated data:
\n
\n
With each round of training, we would analyze the validation set, locate the patterns that give low score, and add these patterns to our chart generation code. We made the rise from 0.74 to 0.82 public LB simply by repeating this process multiple times.
\nWith our generation code we can create as many bar, line and scatter plots as necessary for each training stage. For dot plot we sampled 10k images from the great dataset provided by @brendanartley
\n2. Three-stage training of matcha models
\nThanks to the kind sharing by @nbroad , we started the competition training matcha-base models and quickly reached a good score.
The training is done in 3 stages:
\nmatcha-base checkpoint as a chart classifier. The model is trained for 5 epochs, using little synthetic data and oversampling extracted data.At inference time, the chart classifier is run first and then each group of charts will be handled by its dedicated model.
\nAt the beginning our process had only step 2 and 3, and we used model from step 2 as classifier. But then we observed that adding a lot of synthetic data decreases the performance of classifying task. As a result, training for the classifier was separated.
\nIt is also possible to perform classifying task with a simple CNN, however from our early experiments we saw that using matcha gave a little better performance, so we sticked with its usage.
\nSome training details:
\n<chart_type><start>x1|y1;…;xn|yn<end>3. Scatter plots processing
\nWhile the mentioned above method helped us reach good results for bar, dot and line charts, we found scatter plots much harder for matcha to handle. As a result, we developed a separate scheme only for scatter:
\nThis process helped us score 0.09 public and 0.28 private for scatter and is the key for us to survive the private test set.
\nThank you for reading and please let us know if you have any questions.
\nEdit: Inference notebook for our solution now available: https://www.kaggle.com/code/qdv206/benetech-5th-place-inference
", "rawMarkdown": "Many thanks to Kaggle and Benetech for this interesting competition where many different approaches are possible and ideas are endless. Given the strong distribution shift in the private test set, we are quite happy with our finish.\n\n**Summary**\n\nOur solution consists of 3 main components:\n-\tSynthetic data generation.\n-\t3-stage training of matcha models.\n-\tSeparate process to handle scatter plot.\n\n**1. Synthetic data generation**\n\nEarly on we realized this competition is unique in the sense that we are not limited to the available dataset. If the model struggles with certain plot patterns, we can generate a large number of plots with these patterns and as the model (matcha) is very strong, it will learn.\n\nWe started by building upon this amazing repo: https://github.com/rakutentech/chart-synthesizer. We used both competition’s extracted data and ICDAR data as validation and try to emulate as many patterns from them as possible. For example:\n-\tDifferent fonts, tick orientations, tick styles, background colors, grid styles etc.\n-\tAdd error bars to bar and line plots.\n-\tGenerate histogram plots as a separate chart type.\n-\tDifficult line plot patterns, such as when the line starts very close to a tick mark but doesn’t touch it, the model without additional training data will very likely include this tick value in the prediction. \n-\tAdd blur and noise effects to reproduce the look of extracted plots.\n\nSome examples of our generated data:\n\n\n\nWith each round of training, we would analyze the validation set, locate the patterns that give low score, and add these patterns to our chart generation code. We made the rise from 0.74 to 0.82 public LB simply by repeating this process multiple times.\n\nWith our generation code we can create as many bar, line and scatter plots as necessary for each training stage. For dot plot we sampled 10k images from the great dataset provided by @brendanartley \n\n**2. Three-stage training of matcha models**\n\nThanks to the kind sharing by @nbroad , we started the competition training `matcha-base` models and quickly reached a good score.\n\nThe training is done in 3 stages:\n-\tTrain from `matcha-base` checkpoint as a chart classifier. The model is trained for 5 epochs, using little synthetic data and oversampling extracted data.\n-\tUse the weight from stage 1, continue training for 10 epochs to extract data series from all chart types. In this step we add a lot of our own synthetic data (~150k images) to combine with the original dataset.\n-\tUse the weight from stage 2, finetune separate models for 5 epochs with each chart type group:\n + vertical bar/dot (add 50k synthetic vertical bar plots, 5k histograms, 5k dot plots)\n + horizontal bar (add 50k synthetic plots)\n + line (add 200k synthetic plots)\n + scatter (add 30k synthetic plots)\n\nAt inference time, the chart classifier is run first and then each group of charts will be handled by its dedicated model.\n\nAt the beginning our process had only step 2 and 3, and we used model from step 2 as classifier. But then we observed that adding a lot of synthetic data decreases the performance of classifying task. As a result, training for the classifier was separated.\n\nIt is also possible to perform classifying task with a simple CNN, however from our early experiments we saw that using matcha gave a little better performance, so we sticked with its usage.\n\n**Some training details:**\n-\tLearning rate 3e-5 for 1st and 2nd stages, 2e-5 for 3rd stage, using cosine schedule with warmup.\n-\tAdafactor optimizer.\n-\tMax patches 2048, max length 512.\n-\tFreeze first 4 layers of encoder.\n-\tPrediction string: `Nice work!
\nWhat was your CV/LB for each type?
", "rawMarkdown": "Nice work!\n\nWhat was your CV/LB for each type?", "votes": null }, { "id": "2311585", "postDate": "06/21/2023 09:03:08", "content": "A very thorough approach! Great job of finding information and code
", "rawMarkdown": "A very thorough approach! Great job of finding information and code", "votes": null }, { "id": "2312092", "postDate": "06/21/2023 17:00:54", "content": "Thank you very much! Without your sharing on matcha we probably would have had a much harder time with this competition.
\nHere is our result by chart type: (CV/Public/Private)
\nNice work! Congratulations on getting 5th position!! @qdv206
", "rawMarkdown": "Nice work! Congratulations on getting 5th position!! @qdv206", "votes": null }, { "id": "2313947", "postDate": "06/23/2023 04:26:53", "content": "Amazing approach, thank @qdv206 .
", "rawMarkdown": "Amazing approach, thank @qdv206 .", "votes": null }, { "id": "2313964", "postDate": "06/23/2023 04:53:12", "content": "Superb work, Congratulations on achieving the fifth spot! @qdv206
", "rawMarkdown": "Superb work, Congratulations on achieving the fifth spot! @qdv206", "votes": null }, { "id": "2317193", "postDate": "06/25/2023 13:57:22", "content": "nice work, thanks for sharing
", "rawMarkdown": "nice work, thanks for sharing", "votes": null }, { "id": "2319390", "postDate": "06/27/2023 05:26:54", "content": "@qdv206 good solution! Thank you!
", "rawMarkdown": "qdv206 good solution! Thank you!", "votes": null } ], "comments": [ { "id": 2311081, "author_name": "nbroad", "author_url": "", "post_date": "06/20/2023 22:30:14", "content": "Nice work!
\nWhat was your CV/LB for each type?
", "votes": null, "replies": [ { "id": 2312092, "author_name": "qdv206", "author_url": "", "post_date": "06/21/2023 17:00:54", "content": "Thank you very much! Without your sharing on matcha we probably would have had a much harder time with this competition.
\nHere is our result by chart type: (CV/Public/Private)
\nA very thorough approach! Great job of finding information and code
", "votes": null, "replies": [] }, { "id": 2312304, "author_name": "akshitsharma1", "author_url": "", "post_date": "06/21/2023 19:50:43", "content": "Nice work! Congratulations on getting 5th position!! @qdv206
", "votes": null, "replies": [] }, { "id": 2313947, "author_name": "danghieuan", "author_url": "", "post_date": "06/23/2023 04:26:53", "content": "Amazing approach, thank @qdv206 .
", "votes": null, "replies": [] }, { "id": 2313964, "author_name": "pushkar007", "author_url": "", "post_date": "06/23/2023 04:53:12", "content": "Superb work, Congratulations on achieving the fifth spot! @qdv206
", "votes": null, "replies": [] }, { "id": 2317193, "author_name": "joek47", "author_url": "", "post_date": "06/25/2023 13:57:22", "content": "nice work, thanks for sharing
", "votes": null, "replies": [] }, { "id": 2319390, "author_name": "serangu", "author_url": "", "post_date": "06/27/2023 05:26:54", "content": "@qdv206 good solution! Thank you!
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "2311069": "Many thanks to Kaggle and Benetech for this interesting competition where many different approaches are possible and ideas are endless. Given the strong distribution shift in the private test set, we are quite happy with our finish.\n\n**Summary**\n\nOur solution consists of 3 main components:\n-\tSynthetic data generation.\n-\t3-stage training of matcha models.\n-\tSeparate process to handle scatter plot.\n\n**1. Synthetic data generation**\n\nEarly on we realized this competition is unique in the sense that we are not limited to the available dataset. If the model struggles with certain plot patterns, we can generate a large number of plots with these patterns and as the model (matcha) is very strong, it will learn.\n\nWe started by building upon this amazing repo: https://github.com/rakutentech/chart-synthesizer. We used both competition’s extracted data and ICDAR data as validation and try to emulate as many patterns from them as possible. For example:\n-\tDifferent fonts, tick orientations, tick styles, background colors, grid styles etc.\n-\tAdd error bars to bar and line plots.\n-\tGenerate histogram plots as a separate chart type.\n-\tDifficult line plot patterns, such as when the line starts very close to a tick mark but doesn’t touch it, the model without additional training data will very likely include this tick value in the prediction. \n-\tAdd blur and noise effects to reproduce the look of extracted plots.\n\nSome examples of our generated data:\n\n\n\nWith each round of training, we would analyze the validation set, locate the patterns that give low score, and add these patterns to our chart generation code. We made the rise from 0.74 to 0.82 public LB simply by repeating this process multiple times.\n\nWith our generation code we can create as many bar, line and scatter plots as necessary for each training stage. For dot plot we sampled 10k images from the great dataset provided by @brendanartley \n\n**2. Three-stage training of matcha models**\n\nThanks to the kind sharing by @nbroad , we started the competition training `matcha-base` models and quickly reached a good score.\n\nThe training is done in 3 stages:\n-\tTrain from `matcha-base` checkpoint as a chart classifier. The model is trained for 5 epochs, using little synthetic data and oversampling extracted data.\n-\tUse the weight from stage 1, continue training for 10 epochs to extract data series from all chart types. In this step we add a lot of our own synthetic data (~150k images) to combine with the original dataset.\n-\tUse the weight from stage 2, finetune separate models for 5 epochs with each chart type group:\n + vertical bar/dot (add 50k synthetic vertical bar plots, 5k histograms, 5k dot plots)\n + horizontal bar (add 50k synthetic plots)\n + line (add 200k synthetic plots)\n + scatter (add 30k synthetic plots)\n\nAt inference time, the chart classifier is run first and then each group of charts will be handled by its dedicated model.\n\nAt the beginning our process had only step 2 and 3, and we used model from step 2 as classifier. But then we observed that adding a lot of synthetic data decreases the performance of classifying task. As a result, training for the classifier was separated.\n\nIt is also possible to perform classifying task with a simple CNN, however from our early experiments we saw that using matcha gave a little better performance, so we sticked with its usage.\n\n**Some training details:**\n-\tLearning rate 3e-5 for 1st and 2nd stages, 2e-5 for 3rd stage, using cosine schedule with warmup.\n-\tAdafactor optimizer.\n-\tMax patches 2048, max length 512.\n-\tFreeze first 4 layers of encoder.\n-\tPrediction string: `