\n
![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F2125251%2F896a1d13b8a90fc50fa8a3b7eb4b1100%2FScreenshot%202023-06-20%20at%208.30.35%20AM.png?generation=1687221055426709&alt=media\")
Congrats to the winners!
\nNow that the private leaderboard is open, and seeing the massive result differences, i wanted to open discussion in what might be the reasons?
\nE.g. our team best public LB was 0.64 but we dropped to as low as 0.36!! I believe scatter plots were a huge issue, was this found by other teams too? Also there seemed to not be many dot plots or scatter plots in general in the public test set. Was this found by someone else?
", "messages": [ { "id": 2309856, "postDate": "2023-06-20T00:11:42.973Z", "content": "Our solution achieves 0.29 for scatter only on private. The score was 0.09 on public.
\nOur scatter CV was 0.67 so private LB has almost 50% of scatter plots which is insanely high
yes, scatter was highly represented in private. I also have exact same score for scatter….
\n
Congrats to the winners!
\nNow that the private leaderboard is open, and seeing the massive result differences, i wanted to open discussion in what might be the reasons?
\nE.g. our team best public LB was 0.64 but we dropped to as low as 0.36!! I believe scatter plots were a huge issue, was this found by other teams too? Also there seemed to not be many dot plots or scatter plots in general in the public test set. Was this found by someone else?
", "rawMarkdown": "Congrats to the winners!\n\nNow that the private leaderboard is open, and seeing the massive result differences, i wanted to open discussion in what might be the reasons?\n\nE.g. our team best public LB was 0.64 but we dropped to as low as 0.36!! I believe scatter plots were a huge issue, was this found by other teams too? Also there seemed to not be many dot plots or scatter plots in general in the public test set. Was this found by someone else?", "votes": 4 }, { "id": 2309877, "postDate": "2023-06-20T00:34:48.257Z", "content": "I’m sure Private LB has a huge set of scatter data. I created a pipeline for scatter plot, and it showed a incerement of 0.05 in public LB. So I got 0.68 with Matcha(0.63) + scatter module(0.05) and was 60th in Public LB, but I got much higher score than others in Private LB.
", "rawMarkdown": "I’m sure Private LB has a huge set of scatter data. I created a pipeline for scatter plot, and it showed a incerement of 0.05 in public LB. So I got 0.68 with Matcha(0.63) + scatter module(0.05) and was 60th in Public LB, but I got much higher score than others in Private LB.", "votes": 1 }, { "id": 2309908, "postDate": "2023-06-20T01:18:22.613Z", "content": "Our approach was quite different from others as our entire submission was based solely on object detection and OCR. It was challenging to compete with deploy/Matcha models using only object detection. Congratulations to all the winners!
", "rawMarkdown": "Our approach was quite different from others as our entire submission was based solely on object detection and OCR. It was challenging to compete with deploy/Matcha models using only object detection. Congratulations to all the winners! ", "votes": 2 }, { "id": 2309861, "postDate": "2023-06-20T00:13:52.743Z", "content": "Scatter is also what matcha/deplot struggles most with. Probably another model/technique was used to get into the gold territory
", "rawMarkdown": "Scatter is also what matcha/deplot struggles most with. Probably another model/technique was used to get into the gold territory", "votes": 2, "replies": [ { "id": 2309863, "postDate": "2023-06-20T00:16:16.167Z", "content": "I think many are likely using object detection or something of that nature?
", "rawMarkdown": "I think many are likely using object detection or something of that nature?", "votes": 2, "replies": [ { "id": 2309867, "postDate": "2023-06-20T00:18:47.357Z", "content": "Yes, I stumbled on this the day before the comp ended, so I wouldn't be surprised if people created synthetic data and then trained an object detection model on it. It seems to work well.
", "rawMarkdown": "Yes, I stumbled on [this](https://github.com/bloomberg/scatteract) the day before the comp ended, so I wouldn't be surprised if people created synthetic data and then trained an object detection model on it. It seems to work well.", "votes": 3 } ] }, { "id": 2309869, "postDate": "2023-06-20T00:26:04.147Z", "content": "Thanks a lot @nbroad for sharing your approach, quality work as always. I only used matcha model, but trained a separate model to handle scatter plots (first train on all data, next only on scatter).
", "rawMarkdown": "Thanks a lot @nbroad for sharing your approach, quality work as always. I only used matcha model, but trained a separate model to handle scatter plots (first train on all data, next only on scatter). ", "votes": 3, "replies": [ { "id": 2309873, "postDate": "2023-06-20T00:31:04.130Z", "content": "How good could your matcha do on scatter?
", "rawMarkdown": "How good could your matcha do on scatter?" }, { "id": 2309875, "postDate": "2023-06-20T00:33:46.447Z", "content": "I did a few scatter only subs, it scored 0.09 in public and 0.29 in private, my CV distribution was -
\nhorizontal bar: 0.95
\nvertical bar: 0.95
\ndot: -1 (no dots in extracted)
\nline: 0.85
\nscatter: 0.56
Great approach, especially the two steps on scatter. Thank you for sharing!
", "rawMarkdown": "Great approach, especially the two steps on scatter. Thank you for sharing!" }, { "id": 2310982, "postDate": "2023-06-20T19:41:00.037Z", "content": "@conjuring92
\nMy CV compared to yours:
horizontal bar: -0.01
\nvertical bar: -0.02
\ndot: ?
\nline: -0.04
\nscatter: -0.18
And private lb difference: -0.24 🤯
\nThe scatter makes such a big difference! I also think my classifier was bad because I made it the day before the comp finished, so that also contributed 😅
", "rawMarkdown": "@conjuring92 \nMy CV compared to yours:\n\nhorizontal bar: -0.01\nvertical bar: -0.02\ndot: ?\nline: -0.04\nscatter: -0.18\n\nAnd private lb difference: -0.24 🤯\n\nThe scatter makes such a big difference! I also think my classifier was bad because I made it the day before the comp finished, so that also contributed 😅" } ] } ] }, { "id": 2309928, "postDate": "2023-06-20T01:55:12.030Z", "content": "I sometimes had the idea that the main purpose (and hidden) of the competition is a scatter.
\nFor me, they were the most difficult. There are no ideas how to solve cases when the points intersect or partially intersect.
\nFirst of all, the method of detecting objects, but they work well on cats, dogs and other objects that have characteristic features.
\nI tried to train the detector a little, but I abandoned it because of the lack of prospects. I think we need a special architecture to successfully solve such a problem.
\nI had >0.8 on all charts, and a scatter of 0.6 on local test data. There was a total of 0.8 on the local test, 0.58 on LB and I could not understand why such a spread.
\nWho solved the scatter, please share, it's very interesting.
It is a “hack scatter” competition :-) and I agree the scatter distribution in private dataset is insanely high!
\nCongratulations to all winners.
Our solution achieves 0.29 for scatter only on private. The score was 0.09 on public.
\nOur scatter CV was 0.67 so private LB has almost 50% of scatter plots which is insanely high
yes, scatter was highly represented in private. I also have exact same score for scatter….
\n
I’m sure Private LB has a huge set of scatter data. I created a pipeline for scatter plot, and it showed a incerement of 0.05 in public LB. So I got 0.68 with Matcha(0.63) + scatter module(0.05) and was 60th in Public LB, but I got much higher score than others in Private LB.
", "votes": 1, "replies": [] }, { "id": 2309908, "author_name": "Aravind S", "author_url": "", "post_date": "2023-06-20T01:18:22.613000", "content": "Our approach was quite different from others as our entire submission was based solely on object detection and OCR. It was challenging to compete with deploy/Matcha models using only object detection. Congratulations to all the winners!
", "votes": 2, "replies": [] }, { "id": 2309861, "author_name": "Nicholas Broad", "author_url": "", "post_date": "2023-06-20T00:13:52.743000", "content": "Scatter is also what matcha/deplot struggles most with. Probably another model/technique was used to get into the gold territory
", "votes": 2, "replies": [ { "id": 2309863, "author_name": "Cody_Null", "author_url": "", "post_date": "2023-06-20T00:16:16.167000", "content": "I think many are likely using object detection or something of that nature?
", "votes": 2, "replies": [ { "id": 2309867, "author_name": "Nicholas Broad", "author_url": "", "post_date": "2023-06-20T00:18:47.357000", "content": "Yes, I stumbled on this the day before the comp ended, so I wouldn't be surprised if people created synthetic data and then trained an object detection model on it. It seems to work well.
", "votes": 3, "replies": [] } ] }, { "id": 2309869, "author_name": "Raja Biswas", "author_url": "", "post_date": "2023-06-20T00:26:04.147000", "content": "Thanks a lot @nbroad for sharing your approach, quality work as always. I only used matcha model, but trained a separate model to handle scatter plots (first train on all data, next only on scatter).
", "votes": 3, "replies": [ { "id": 2309873, "author_name": "Nicholas Broad", "author_url": "", "post_date": "2023-06-20T00:31:04.130000", "content": "How good could your matcha do on scatter?
", "votes": 0, "replies": [] }, { "id": 2309875, "author_name": "Raja Biswas", "author_url": "", "post_date": "2023-06-20T00:33:46.447000", "content": "I did a few scatter only subs, it scored 0.09 in public and 0.29 in private, my CV distribution was -
\nhorizontal bar: 0.95
\nvertical bar: 0.95
\ndot: -1 (no dots in extracted)
\nline: 0.85
\nscatter: 0.56
Great approach, especially the two steps on scatter. Thank you for sharing!
", "votes": 0, "replies": [] }, { "id": 2310982, "author_name": "Nicholas Broad", "author_url": "", "post_date": "2023-06-20T19:41:00.037000", "content": "@conjuring92
\nMy CV compared to yours:
horizontal bar: -0.01
\nvertical bar: -0.02
\ndot: ?
\nline: -0.04
\nscatter: -0.18
And private lb difference: -0.24 🤯
\nThe scatter makes such a big difference! I also think my classifier was bad because I made it the day before the comp finished, so that also contributed 😅
", "votes": 0, "replies": [] } ] } ] }, { "id": 2309928, "author_name": "Anuar-s", "author_url": "", "post_date": "2023-06-20T01:55:12.030000", "content": "I sometimes had the idea that the main purpose (and hidden) of the competition is a scatter.
\nFor me, they were the most difficult. There are no ideas how to solve cases when the points intersect or partially intersect.
\nFirst of all, the method of detecting objects, but they work well on cats, dogs and other objects that have characteristic features.
\nI tried to train the detector a little, but I abandoned it because of the lack of prospects. I think we need a special architecture to successfully solve such a problem.
\nI had >0.8 on all charts, and a scatter of 0.6 on local test data. There was a total of 0.8 on the local test, 0.58 on LB and I could not understand why such a spread.
\nWho solved the scatter, please share, it's very interesting.
It is a “hack scatter” competition :-) and I agree the scatter distribution in private dataset is insanely high!
\nCongratulations to all winners.