![\"shrink](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F548996%2F552409d5a24798a66ae71cf3e382ae19%2F.png?generation=1598018273243440&alt=media\")
This competition was unique, since we experienced private set shakedowns.. lol
\nI was wondering why a large shakeup occurred by the cleaned private set.
\nOur score went down from 0.70 to 0.65, which is extremely large for bounding box correction. (But everyone had a score drop of 0.03-0.05)
\nLittle can be known, since most of our subs were disabled.
\nMystery can be reviled after late subs are open, but would be interesting to discuss on this.
Some cause for this might be
\nWould be helpful if anyone else insights! Thx!
\nLate subs are open!
\nBased on @its7171 's insights, I did some experiments on shrinking bb as post processing.
\nThe model is based on my best single model for fast experiments.
Looks like shrinking 0.96x boosts the private +0.012.
\n| shrink ratio | \n1 | \n0.96 | \n0.94 | \n0.92 | \n0.9 | \n
|---|---|---|---|---|---|
| Public | \n0.756 | \n0.732 | \n0.729 | \n0.687 | \n0.606 | \n
| Private | \n0.642 | \n0.654 | \n0.636 | \n0.625 | \n0.573 | \n
I shrinked boxes of the train data as well.
\nPsuedo Label for 3 epochs with the best single model.
\nNote that the boxes are not shrinked during the final inference.
The ratio is 0.96x, resulting in a +0.027 boost.
\n| Shrink on PL? | \nNo | \nYes(WBF IoU threshold=0.42) | \nYes (WBF IoU threshold=0.5) | \n
|---|---|---|---|
| Public | \n0.7561 | \n0.7378 | \n0.7402 | \n
| Private | \n0.6428 | \n0.6626 | \n0.6700 | \n
This competition was unique, since we experienced private set shakedowns.. lol
\nI was wondering why a large shakeup occurred by the cleaned private set.
\nOur score went down from 0.70 to 0.65, which is extremely large for bounding box correction. (But everyone had a score drop of 0.03-0.05)
\nLittle can be known, since most of our subs were disabled.
\nMystery can be reviled after late subs are open, but would be interesting to discuss on this.
Some cause for this might be
\nWould be helpful if anyone else insights! Thx!
\nLate subs are open!
\nBased on @its7171 's insights, I did some experiments on shrinking bb as post processing.
\nThe model is based on my best single model for fast experiments.
Looks like shrinking 0.96x boosts the private +0.012.
\n| shrink ratio | \n1 | \n0.96 | \n0.94 | \n0.92 | \n0.9 | \n
|---|---|---|---|---|---|
| Public | \n0.756 | \n0.732 | \n0.729 | \n0.687 | \n0.606 | \n
| Private | \n0.642 | \n0.654 | \n0.636 | \n0.625 | \n0.573 | \n
I shrinked boxes of the train data as well.
\nPsuedo Label for 3 epochs with the best single model.
\nNote that the boxes are not shrinked during the final inference.
The ratio is 0.96x, resulting in a +0.027 boost.
\n| Shrink on PL? | \nNo | \nYes(WBF IoU threshold=0.42) | \nYes (WBF IoU threshold=0.5) | \n
|---|---|---|---|
| Public | \n0.7561 | \n0.7378 | \n0.7402 | \n
| Private | \n0.6428 | \n0.6626 | \n0.6700 | \n
Thank you for sharing the experiment results.
\nIt's interesting to shrink bbs for psuedo labeling.
\nScore looks very sensitive to the shrink scale and this results consistent with my experience.
This graph is shrink scale vs score for my 5CV(source split).
\nEven for source split 5 CV, the optimal shrink size is not zero.
\nAnd best scores differed greatly at its optimal shrink scale.
So it was very important to select optimal shrink scale for private, which we never know beforehand. :)
\n
\n\nSo it was very important to select optimal shrink scale for private, which we never know beforehand. :)
\n
@its7171 As you mention, the challenge was that we never know the optimal parameter for shrinking. BTW, how did you set your shrink scale?
\nOne step I could have taken was labeling the test set (or part of the train set) manually and look at the mAP dependency on the shrink size. But I didn't think the shrinking would have done such a impact to PB, so I didn't dig much further (which lead to the shake down.. :(
\nMy teammate also found an interesting solution, where resizing only small boxes can make a further boost.
\nhttps://www.kaggle.com/c/global-wheat-detection/discussion/176732
I set the shrink scale that gives the best public score for one sub, and set 6% smaller shrink scale for the another sub.
\nThis 6% is a kind of wild guess with reference to the figure shown above.
\n\nresizing only small boxes can make a further boost
\n
interesting. 👀
", "rawMarkdown": "I set the shrink scale that gives the best public score for one sub, and set 6% smaller shrink scale for the another sub.\nThis 6% is a kind of wild guess with reference to the figure shown above.\n\n> resizing only small boxes can make a further boost\n\ninteresting. 👀" } ] }, { "id": 979137, "postDate": "2020-08-20T16:42:10.857Z", "content": "I had an interesting experience on this.
\nThese are my 2 final sub scores.
\n| name | \nprivate | \npublic | \n
|---|---|---|
| A | \n0.6707 | \n0.7439 | \n
| B | \n0.6558 | \n0.7677 | \n
Only difference of these 2 models is bb size.
\nI rescaled predicted boxes in the post processing, modelB has 1.06 times bigger bbs than modelA.
\nSo I guess private bbs are smaller than public bbs.
Your model might have predicted relatively large bbs for some reason, like my model B.
", "rawMarkdown": "I had an interesting experience on this.\n\nThese are my 2 final sub scores.\n\n| name|private|public|\n| ---- | ---- | ---- |\n| A |**0.6707**|0.7439|\n| B |0.6558|**0.7677**|\n\nOnly difference of these 2 models is bb size.\nI rescaled predicted boxes in the post processing, modelB has 1.06 times bigger bbs than modelA.\nSo I guess private bbs are smaller than public bbs.\n\nYour model might have predicted relatively large bbs for some reason, like my model B.", "votes": 4, "replies": [ { "id": 979233, "postDate": "2020-08-20T17:50:35.217Z", "content": "It is a really interesting.
\nI also used clean dataset and original dataset.
\n| name | \nprivate | \npublic | \n
|---|---|---|
| A | \n0.6444 | \n0.7545 | \n
| B | \n0.6484 | \n0.7552 | \n
Most of the difference is whether or not to delete wrong labels.
\nI cannot be verified since previous submissions cannot be viewed.
So I only reply with my public/private lb. :)
", "rawMarkdown": "It is a really interesting.\n\nI also used clean dataset and original dataset.\n\n| name | private | public |\n| --- | --- | --- |\n| A | 0.6444 | 0.7545 |\n| B | 0.6484 | 0.7552 |\n\n\nMost of the difference is whether or not to delete wrong labels.\nI cannot be verified since previous submissions cannot be viewed.\n\nSo I only reply with my public/private lb. :)", "votes": 1 }, { "id": 979554, "postDate": "2020-08-21T00:06:03.477Z", "content": "@its7171
\nThanks for sharing the knowledge.
\nI now remember that the hosts said that the boxes will be tight to the wheat heads, which mean that simply shrinking the boxes will benefit.
Since the late subs are open, I will test if shrinking will help me much (will report the results to you guys)
", "rawMarkdown": "@its7171 \nThanks for sharing the knowledge.\nI now remember that the hosts said that the boxes will be tight to the wheat heads, which mean that simply shrinking the boxes will benefit. \n\nSince the late subs are open, I will test if shrinking will help me much (will report the results to you guys)", "votes": 2 }, { "id": 979605, "postDate": "2020-08-21T01:48:48.943Z", "content": "@piantic
\nWe cleaned the mistaken labels too.
\nTop teams performed label cleaning on the train set by psuedo label for the wheats on the edges, thinking that was a good solution.
See Liao and Miras solution for details ;)
\nhttps://www.kaggle.com/c/global-wheat-detection/discussion/175961
\nhttps://www.kaggle.com/c/global-wheat-detection/discussion/172569
Surprised to see people getting 0.71 with late subs.
\nWonder what kind of models are being used..?
\nhttps://kaggledb.com/c/global-wheat-detection/late
Didn't know about kaggledb - thanks for sharing. Possibly these are YoloV5? Would be interesting if people want to share.
", "rawMarkdown": "Didn't know about kaggledb - thanks for sharing. Possibly these are YoloV5? Would be interesting if people want to share." } ] }, { "id": 979596, "postDate": "2020-08-21T01:20:39.133Z", "content": "I think pseudo labeling is useful.The main reason cause to the drop maybe dirty labels. Our team use models to predict the trainning data, average several model's results and ground truth.It helps us to get a stable score(It just like pseudo labeling).
", "rawMarkdown": "I think pseudo labeling is useful.The main reason cause to the drop maybe dirty labels. Our team use models to predict the trainning data, average several model's results and ground truth.It helps us to get a stable score(It just like pseudo labeling).", "votes": 1 }, { "id": 979240, "postDate": "2020-08-20T17:55:02.673Z", "content": "Very likely you overfitted dirty labels, my score increased by around 3% points on both clean public and private and I used a clean dataset. To put this into perspective, I was placed around 760 on the dirty private leaderboard. I made some kind of mistake in my notebook so I wasn't ever able to get Pseudo labeling works so I cannot speak to that.
\n![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F1110596%2F5ac3a9619d8154f90c20a970ec32b001%2Fscores.png?generation=1597945900443084&alt=media\")
Thank you for sharing the experiment results.
\nIt's interesting to shrink bbs for psuedo labeling.
\nScore looks very sensitive to the shrink scale and this results consistent with my experience.
This graph is shrink scale vs score for my 5CV(source split).
\nEven for source split 5 CV, the optimal shrink size is not zero.
\nAnd best scores differed greatly at its optimal shrink scale.
So it was very important to select optimal shrink scale for private, which we never know beforehand. :)
\n
\n\nSo it was very important to select optimal shrink scale for private, which we never know beforehand. :)
\n
@its7171 As you mention, the challenge was that we never know the optimal parameter for shrinking. BTW, how did you set your shrink scale?
\nOne step I could have taken was labeling the test set (or part of the train set) manually and look at the mAP dependency on the shrink size. But I didn't think the shrinking would have done such a impact to PB, so I didn't dig much further (which lead to the shake down.. :(
\nMy teammate also found an interesting solution, where resizing only small boxes can make a further boost.
\nhttps://www.kaggle.com/c/global-wheat-detection/discussion/176732
I set the shrink scale that gives the best public score for one sub, and set 6% smaller shrink scale for the another sub.
\nThis 6% is a kind of wild guess with reference to the figure shown above.
\n\nresizing only small boxes can make a further boost
\n
interesting. 👀
", "votes": 0, "replies": [] } ] }, { "id": 979137, "author_name": "tito", "author_url": "", "post_date": "2020-08-20T16:42:10.857000", "content": "I had an interesting experience on this.
\nThese are my 2 final sub scores.
\n| name | \nprivate | \npublic | \n
|---|---|---|
| A | \n0.6707 | \n0.7439 | \n
| B | \n0.6558 | \n0.7677 | \n
Only difference of these 2 models is bb size.
\nI rescaled predicted boxes in the post processing, modelB has 1.06 times bigger bbs than modelA.
\nSo I guess private bbs are smaller than public bbs.
Your model might have predicted relatively large bbs for some reason, like my model B.
", "votes": 4, "replies": [ { "id": 979233, "author_name": "Heroseo", "author_url": "", "post_date": "2020-08-20T17:50:35.217000", "content": "It is a really interesting.
\nI also used clean dataset and original dataset.
\n| name | \nprivate | \npublic | \n
|---|---|---|
| A | \n0.6444 | \n0.7545 | \n
| B | \n0.6484 | \n0.7552 | \n
Most of the difference is whether or not to delete wrong labels.
\nI cannot be verified since previous submissions cannot be viewed.
So I only reply with my public/private lb. :)
", "votes": 1, "replies": [] }, { "id": 979554, "author_name": "arutema47", "author_url": "", "post_date": "2020-08-21T00:06:03.477000", "content": "@its7171
\nThanks for sharing the knowledge.
\nI now remember that the hosts said that the boxes will be tight to the wheat heads, which mean that simply shrinking the boxes will benefit.
Since the late subs are open, I will test if shrinking will help me much (will report the results to you guys)
", "votes": 2, "replies": [] }, { "id": 979605, "author_name": "arutema47", "author_url": "", "post_date": "2020-08-21T01:48:48.943000", "content": "@piantic
\nWe cleaned the mistaken labels too.
\nTop teams performed label cleaning on the train set by psuedo label for the wheats on the edges, thinking that was a good solution.
See Liao and Miras solution for details ;)
\nhttps://www.kaggle.com/c/global-wheat-detection/discussion/175961
\nhttps://www.kaggle.com/c/global-wheat-detection/discussion/172569
Surprised to see people getting 0.71 with late subs.
\nWonder what kind of models are being used..?
\nhttps://kaggledb.com/c/global-wheat-detection/late
Didn't know about kaggledb - thanks for sharing. Possibly these are YoloV5? Would be interesting if people want to share.
", "votes": 0, "replies": [] } ] }, { "id": 979596, "author_name": "Dapeng", "author_url": "", "post_date": "2020-08-21T01:20:39.133000", "content": "I think pseudo labeling is useful.The main reason cause to the drop maybe dirty labels. Our team use models to predict the trainning data, average several model's results and ground truth.It helps us to get a stable score(It just like pseudo labeling).
", "votes": 1, "replies": [] }, { "id": 979240, "author_name": "Max", "author_url": "", "post_date": "2020-08-20T17:55:02.673000", "content": "Very likely you overfitted dirty labels, my score increased by around 3% points on both clean public and private and I used a clean dataset. To put this into perspective, I was placed around 760 on the dirty private leaderboard. I made some kind of mistake in my notebook so I wasn't ever able to get Pseudo labeling works so I cannot speak to that.
\n