{ "id": 418382, "title": "Our solution", "url": "/competitions/benetech-making-graphs-accessible/discussion/418382", "author_name": "", "post_date": "2023-06-20T09:43:14.863880900Z", "votes": 8, "comment_count": 4, "views": 0, "content": "

Although we have a little score, we will still share the solution. Our solution contains a classifier, segmentation, OCR and logic code.
\nAlmost the entire solution is based on pix2pix. pix2pix has been redesigned for 512x512
\nAll images were padded to a square. If the side is greater than 512, then a resize was performed. And with inference - reverse resize.
\nThe detection of texts using this method is very accurate, almost >95 correct, rare cases when the spots merge or the captions of the y gets confused with the title, etc.
\nThe advantage of the method is that it takes the areas of the rotated text, remembers the regularity of the fact that the tick labels are lined up in a row, highlights the areas of the associated text into a single spot.
\nDisadvantages - when the ticks are close, they merge.
\nAll the graphs were also transformed using pix2pix into images, which are easy to process further.
\nHere are some examples.
\n\"\"
\n\"\"
\n\"\"
\n\"\"
\n\"\"
\n\"\"
\nThe problem was only with the scatters.
\nWe wanted to apply object detection - but the initial experiments gave unsatisfactory results.
\nSometimes the noise in the picture is detected as a point, sometimes it takes two close points for one, sometimes it takes the text for a point, and so on.
\nIn general, we started doing it, but it seems to me that this is not quite the right approach to the solution.
\nThe use of pix2pix is also a bad fit here. The points merge, as well as a small area of points compared to the area of the graph. The neural network considers the losses insignificant.
\nOCR - easyOCR.
\nA network for image upscaling was also written. It works fine, but the score increased only by 1%
\n\"\"
\nClassifiers were also implemented - histogram/non-histogram, cat/num for dot graphs.
\nThe classifier is the simplest 3-layer CNN
\nBut it turns out that it did not work very accurately due to the fact that there were few horizontal graphs in the data provided, and dots are very similar to each other. Because of this, for example, dot charts were considered vertical bars.
\n5000 graphs were generated (1000 for each type) After that, the classifier was practically not wrong - only 1 time per 300 files.
\nSomewhere in the middle of the competition, I devoted one day to pix2struct. But I realized that this is a huge topic, and decided not to spray and focus on my approach.
\nOur solution gave 0.8 on a local test (all extracted + 2000 generated = 3000 graphs) But the LB could not rise above 0.58. We thought there were graphs in the hidden data where the signatures are arranged differently, or my text detector does not work well on unseen data.
\nNow I think it's about the scatters. We couldn't come up with a good solution for them.
\nThanks to the competition, I gained a lot of experience in image processing, neural networks, in particular CNN.
\nThanks for the answers in the discussions.
\nWhat I found at the end
\n1) A lot of images are saved (and seem to have been resaved many times) in jpeg format. Because of what they have a lot of artifacts.
\nI found that cv2.bilateralFilter perfectly removes jpeg artifacts, while leaving texts and graphic elements intact.
\n2) It turns out that line-type graphs can break off, and it is not necessary for each label on x to have a value on y
\nI discovered this just yesterday.
\nhttps://www.kaggle.com/code/anuarsenk/main-submitter-7?scriptVersionId=134029325

", "messages": [ { "id": "2310342", "postDate": "06/20/2023 09:43:14", "content": "

Although we have a little score, we will still share the solution. Our solution contains a classifier, segmentation, OCR and logic code.
\nAlmost the entire solution is based on pix2pix. pix2pix has been redesigned for 512x512
\nAll images were padded to a square. If the side is greater than 512, then a resize was performed. And with inference - reverse resize.
\nThe detection of texts using this method is very accurate, almost >95 correct, rare cases when the spots merge or the captions of the y gets confused with the title, etc.
\nThe advantage of the method is that it takes the areas of the rotated text, remembers the regularity of the fact that the tick labels are lined up in a row, highlights the areas of the associated text into a single spot.
\nDisadvantages - when the ticks are close, they merge.
\nAll the graphs were also transformed using pix2pix into images, which are easy to process further.
\nHere are some examples.
\n\"\"
\n\"\"
\n\"\"
\n\"\"
\n\"\"
\n\"\"
\nThe problem was only with the scatters.
\nWe wanted to apply object detection - but the initial experiments gave unsatisfactory results.
\nSometimes the noise in the picture is detected as a point, sometimes it takes two close points for one, sometimes it takes the text for a point, and so on.
\nIn general, we started doing it, but it seems to me that this is not quite the right approach to the solution.
\nThe use of pix2pix is also a bad fit here. The points merge, as well as a small area of points compared to the area of the graph. The neural network considers the losses insignificant.
\nOCR - easyOCR.
\nA network for image upscaling was also written. It works fine, but the score increased only by 1%
\n\"\"
\nClassifiers were also implemented - histogram/non-histogram, cat/num for dot graphs.
\nThe classifier is the simplest 3-layer CNN
\nBut it turns out that it did not work very accurately due to the fact that there were few horizontal graphs in the data provided, and dots are very similar to each other. Because of this, for example, dot charts were considered vertical bars.
\n5000 graphs were generated (1000 for each type) After that, the classifier was practically not wrong - only 1 time per 300 files.
\nSomewhere in the middle of the competition, I devoted one day to pix2struct. But I realized that this is a huge topic, and decided not to spray and focus on my approach.
\nOur solution gave 0.8 on a local test (all extracted + 2000 generated = 3000 graphs) But the LB could not rise above 0.58. We thought there were graphs in the hidden data where the signatures are arranged differently, or my text detector does not work well on unseen data.
\nNow I think it's about the scatters. We couldn't come up with a good solution for them.
\nThanks to the competition, I gained a lot of experience in image processing, neural networks, in particular CNN.
\nThanks for the answers in the discussions.
\nWhat I found at the end
\n1) A lot of images are saved (and seem to have been resaved many times) in jpeg format. Because of what they have a lot of artifacts.
\nI found that cv2.bilateralFilter perfectly removes jpeg artifacts, while leaving texts and graphic elements intact.
\n2) It turns out that line-type graphs can break off, and it is not necessary for each label on x to have a value on y
\nI discovered this just yesterday.
\nhttps://www.kaggle.com/code/anuarsenk/main-submitter-7?scriptVersionId=134029325

", "rawMarkdown": "Although we have a little score, we will still share the solution. Our solution contains a classifier, segmentation, OCR and logic code.\nAlmost the entire solution is based on pix2pix. pix2pix has been redesigned for 512x512\nAll images were padded to a square. If the side is greater than 512, then a resize was performed. And with inference - reverse resize.\n\nThe detection of texts using this method is very accurate, almost >95 correct, rare cases when the spots merge or the captions of the y gets confused with the title, etc.\nThe advantage of the method is that it takes the areas of the rotated text, remembers the regularity of the fact that the tick labels are lined up in a row, highlights the areas of the associated text into a single spot.\nDisadvantages - when the ticks are close, they merge.\n\nAll the graphs were also transformed using pix2pix into images, which are easy to process further.\nHere are some examples.\n\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F14442154%2F9978b09c145275c11f8f3249b7ab13de%2F1066.png?generation=1687254102639623&alt=media)\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F14442154%2F6837aa4e14d2f133109a9727b25391f7%2F1067.png?generation=1687254119066174&alt=media)\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F14442154%2Fcddd5eb0b453b126fabb1a434b9ec409%2F5955.png?generation=1687254127105283&alt=media)\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F14442154%2F856e8bc4b0c8db2f3687ce69fbd63889%2F9956.png?generation=1687254136766670&alt=media)\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F14442154%2F13e11c4c03c52d6598607c5f57f0d4ca%2F17316.png?generation=1687254143023887&alt=media)\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F14442154%2F5589d313e6de443ab31fc00334f0ec56%2Fline-25.png?generation=1687254149981134&alt=media)\n\n\nThe problem was only with the scatters.\nWe wanted to apply object detection - but the initial experiments gave unsatisfactory results.\nSometimes the noise in the picture is detected as a point, sometimes it takes two close points for one, sometimes it takes the text for a point, and so on.\nIn general, we started doing it, but it seems to me that this is not quite the right approach to the solution.\nThe use of pix2pix is also a bad fit here. The points merge, as well as a small area of points compared to the area of the graph. The neural network considers the losses insignificant.\n\nOCR - easyOCR.\nA network for image upscaling was also written. It works fine, but the score increased only by 1%\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F14442154%2F295784e1bc6ecf57d7d08777aa1edadc%2Fsdd.png?generation=1687254184601974&alt=media)\n\nClassifiers were also implemented - histogram/non-histogram, cat/num for dot graphs.\n\nThe classifier is the simplest 3-layer CNN\nBut it turns out that it did not work very accurately due to the fact that there were few horizontal graphs in the data provided, and dots are very similar to each other. Because of this, for example, dot charts were considered vertical bars.\n5000 graphs were generated (1000 for each type) After that, the classifier was practically not wrong - only 1 time per 300 files.\nSomewhere in the middle of the competition, I devoted one day to pix2struct. But I realized that this is a huge topic, and decided not to spray and focus on my approach.\n\nOur solution gave 0.8 on a local test (all extracted + 2000 generated = 3000 graphs) But the LB could not rise above 0.58. We thought there were graphs in the hidden data where the signatures are arranged differently, or my text detector does not work well on unseen data.\nNow I think it's about the scatters. We couldn't come up with a good solution for them.\n\nThanks to the competition, I gained a lot of experience in image processing, neural networks, in particular CNN.\nThanks for the answers in the discussions.\n\nWhat I found at the end\n1) A lot of images are saved (and seem to have been resaved many times) in jpeg format. Because of what they have a lot of artifacts.\nI found that cv2.bilateralFilter perfectly removes jpeg artifacts, while leaving texts and graphic elements intact.\n\n2) It turns out that line-type graphs can break off, and it is not necessary for each label on x to have a value on y\nI discovered this just yesterday.\n\nhttps://www.kaggle.com/code/anuarsenk/main-submitter-7?scriptVersionId=134029325", "votes": null }, { "id": "2311267", "postDate": "06/21/2023 04:13:49", "content": "

@anuarsenk thank you for this topic. Very interesting!
\nI have one question. You got 87 place in this competition. It is a bronze. But I not see medal in your profile. Why?

", "rawMarkdown": "anuarsenk thank you for this topic. Very interesting!\nI have one question. You got 87 place in this competition. It is a bronze. But I not see medal in your profile. Why?", "votes": null }, { "id": "2311372", "postDate": "06/21/2023 06:17:48", "content": "

I don't know either, this is my first time participating. I was thinking when the results would be verified
\nI'm about \"The competition has ended. The private leaderboard is preliminary and will be finalized after the results are verified.\"

", "rawMarkdown": "I don't know either, this is my first time participating. I was thinking when the results would be verified\nI'm about \"The competition has ended. The private leaderboard is preliminary and will be finalized after the results are verified.\"", "votes": null }, { "id": "2311403", "postDate": "06/21/2023 06:39:08", "content": "

The medal will appear in your profile when everything is finalized. Usually no more than a week. Sometimes just a couple of days

", "rawMarkdown": "The medal will appear in your profile when everything is finalized. Usually no more than a week. Sometimes just a couple of days", "votes": null }, { "id": "2311546", "postDate": "06/21/2023 08:28:38", "content": "

Thank you!

", "rawMarkdown": "Thank you!", "votes": null } ], "comments": [ { "id": 2311267, "author_name": "serangu", "author_url": "", "post_date": "06/21/2023 04:13:49", "content": "

@anuarsenk thank you for this topic. Very interesting!
\nI have one question. You got 87 place in this competition. It is a bronze. But I not see medal in your profile. Why?

", "votes": null, "replies": [ { "id": 2311372, "author_name": "anuarsenk", "author_url": "", "post_date": "06/21/2023 06:17:48", "content": "

I don't know either, this is my first time participating. I was thinking when the results would be verified
\nI'm about \"The competition has ended. The private leaderboard is preliminary and will be finalized after the results are verified.\"

", "votes": null, "replies": [ { "id": 2311403, "author_name": "nbroad", "author_url": "", "post_date": "06/21/2023 06:39:08", "content": "

The medal will appear in your profile when everything is finalized. Usually no more than a week. Sometimes just a couple of days

", "votes": null, "replies": [ { "id": 2311546, "author_name": "anuarsenk", "author_url": "", "post_date": "06/21/2023 08:28:38", "content": "

Thank you!

", "votes": null, "replies": [] } ] } ] } ] } ], "raw_markdown_by_id": { "2310342": "Although we have a little score, we will still share the solution. Our solution contains a classifier, segmentation, OCR and logic code.\nAlmost the entire solution is based on pix2pix. pix2pix has been redesigned for 512x512\nAll images were padded to a square. If the side is greater than 512, then a resize was performed. And with inference - reverse resize.\n\nThe detection of texts using this method is very accurate, almost >95 correct, rare cases when the spots merge or the captions of the y gets confused with the title, etc.\nThe advantage of the method is that it takes the areas of the rotated text, remembers the regularity of the fact that the tick labels are lined up in a row, highlights the areas of the associated text into a single spot.\nDisadvantages - when the ticks are close, they merge.\n\nAll the graphs were also transformed using pix2pix into images, which are easy to process further.\nHere are some examples.\n\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F14442154%2F9978b09c145275c11f8f3249b7ab13de%2F1066.png?generation=1687254102639623&alt=media)\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F14442154%2F6837aa4e14d2f133109a9727b25391f7%2F1067.png?generation=1687254119066174&alt=media)\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F14442154%2Fcddd5eb0b453b126fabb1a434b9ec409%2F5955.png?generation=1687254127105283&alt=media)\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F14442154%2F856e8bc4b0c8db2f3687ce69fbd63889%2F9956.png?generation=1687254136766670&alt=media)\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F14442154%2F13e11c4c03c52d6598607c5f57f0d4ca%2F17316.png?generation=1687254143023887&alt=media)\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F14442154%2F5589d313e6de443ab31fc00334f0ec56%2Fline-25.png?generation=1687254149981134&alt=media)\n\n\nThe problem was only with the scatters.\nWe wanted to apply object detection - but the initial experiments gave unsatisfactory results.\nSometimes the noise in the picture is detected as a point, sometimes it takes two close points for one, sometimes it takes the text for a point, and so on.\nIn general, we started doing it, but it seems to me that this is not quite the right approach to the solution.\nThe use of pix2pix is also a bad fit here. The points merge, as well as a small area of points compared to the area of the graph. The neural network considers the losses insignificant.\n\nOCR - easyOCR.\nA network for image upscaling was also written. It works fine, but the score increased only by 1%\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F14442154%2F295784e1bc6ecf57d7d08777aa1edadc%2Fsdd.png?generation=1687254184601974&alt=media)\n\nClassifiers were also implemented - histogram/non-histogram, cat/num for dot graphs.\n\nThe classifier is the simplest 3-layer CNN\nBut it turns out that it did not work very accurately due to the fact that there were few horizontal graphs in the data provided, and dots are very similar to each other. Because of this, for example, dot charts were considered vertical bars.\n5000 graphs were generated (1000 for each type) After that, the classifier was practically not wrong - only 1 time per 300 files.\nSomewhere in the middle of the competition, I devoted one day to pix2struct. But I realized that this is a huge topic, and decided not to spray and focus on my approach.\n\nOur solution gave 0.8 on a local test (all extracted + 2000 generated = 3000 graphs) But the LB could not rise above 0.58. We thought there were graphs in the hidden data where the signatures are arranged differently, or my text detector does not work well on unseen data.\nNow I think it's about the scatters. We couldn't come up with a good solution for them.\n\nThanks to the competition, I gained a lot of experience in image processing, neural networks, in particular CNN.\nThanks for the answers in the discussions.\n\nWhat I found at the end\n1) A lot of images are saved (and seem to have been resaved many times) in jpeg format. Because of what they have a lot of artifacts.\nI found that cv2.bilateralFilter perfectly removes jpeg artifacts, while leaving texts and graphic elements intact.\n\n2) It turns out that line-type graphs can break off, and it is not necessary for each label on x to have a value on y\nI discovered this just yesterday.\n\nhttps://www.kaggle.com/code/anuarsenk/main-submitter-7?scriptVersionId=134029325", "2311267": "anuarsenk thank you for this topic. Very interesting!\nI have one question. You got 87 place in this competition. It is a bronze. But I not see medal in your profile. Why?", "2311372": "I don't know either, this is my first time participating. I was thinking when the results would be verified\nI'm about \"The competition has ended. The private leaderboard is preliminary and will be finalized after the results are verified.\"", "2311403": "The medal will appear in your profile when everything is finalized. Usually no more than a week. Sometimes just a couple of days", "2311546": "Thank you!" }, "source": "meta" }