I'm trying to diagnose an issue where I'm getting worse results when ensembling over several TTAs and models at once. Doing TTA with one model works pretty well, and not doing TTA but ensembling over models also works pretty well. Combining the two gives worse results somehow.
\n\nI'm using wbf so I wouldn't expect to see discontinuities in scores as a function of number of predictions sets. I would expect the score to become smoother as more prediction sets are added.
", "messages": [ { "id": "897965", "postDate": "06/23/2020 07:43:00", "content": "I'm trying to diagnose an issue where I'm getting worse results when ensembling over several TTAs and models at once. Doing TTA with one model works pretty well, and not doing TTA but ensembling over models also works pretty well. Combining the two gives worse results somehow.
\n\nI'm using wbf so I wouldn't expect to see discontinuities in scores as a function of number of predictions sets. I would expect the score to become smoother as more prediction sets are added.
", "rawMarkdown": "I'm trying to diagnose an issue where I'm getting worse results when ensembling over several TTAs and models at once. Doing TTA with one model works pretty well, and not doing TTA but ensembling over models also works pretty well. Combining the two gives worse results somehow.\n\nI'm using wbf so I wouldn't expect to see discontinuities in scores as a function of number of predictions sets. I would expect the score to become smoother as more prediction sets are added.", "votes": null }, { "id": "897978", "postDate": "06/23/2020 07:54:04", "content": "Me too. using TTA in 5 fold model ensembles reduced my LB score. \nHowever, I think the results are much better than my single model.
", "rawMarkdown": "Me too. using TTA in 5 fold model ensembles reduced my LB score. \nHowever, I think the results are much better than my single model.", "votes": null }, { "id": "897980", "postDate": "06/23/2020 07:54:51", "content": "I've always found ensembling over multiple models to be worse than just using the single best model out of the bunch.
\n\nThough this makes me think that the model which performs the best may just be overfit to the public LB. And even though it has a lower score, the ensemble may generalize better. Not sure how I can clear up this suspicion though.
", "rawMarkdown": "I've always found ensembling over multiple models to be worse than just using the single best model out of the bunch.\n\nThough this makes me think that the model which performs the best may just be overfit to the public LB. And even though it has a lower score, the ensemble may generalize better. Not sure how I can clear up this suspicion though.", "votes": null }, { "id": "897988", "postDate": "06/23/2020 07:58:43", "content": "@chanhu @matthewmasters thanks for the responses. I had a thought - let me know what you think. If the training set is introducing a bias to our models, then I suppose more averaging will cause a result to pull towards that bias point. Whereas with less averaging it's luck of the draw (enter psychology, where we tend to only place significance on the lucky draws, and therefore think that a bigger ensemble is always worse).
", "rawMarkdown": "chanhu @matthewmasters thanks for the responses. I had a thought - let me know what you think. If the training set is introducing a bias to our models, then I suppose more averaging will cause a result to pull towards that bias point. Whereas with less averaging it's luck of the draw (enter psychology, where we tend to only place significance on the lucky draws, and therefore think that a bigger ensemble is always worse).", "votes": null }, { "id": "898020", "postDate": "06/23/2020 08:29:10", "content": "Yes, I think so. how to deal with the noisy labels might be key point. (large bounding boxes, some wheat does not be labeled and so on). I have tried to fix them by myself, and hopefully they will be fixed by datasets update.
\n\nAlso, the approach in this paper may useful. \nhttp://openaccess.thecvf.com/content_CVPR_2020/papers/Li_Learning_From_Noisy_Anchors_for_One-Stage_Object_Detection_CVPR_2020_paper.pdf
", "rawMarkdown": "Yes, I think so. how to deal with the noisy labels might be key point. (large bounding boxes, some wheat does not be labeled and so on). I have tried to fix them by myself, and hopefully they will be fixed by datasets update.\n\nAlso, the approach in this paper may useful. \nhttp://openaccess.thecvf.com/content_CVPR_2020/papers/Li_Learning_From_Noisy_Anchors_for_One-Stage_Object_Detection_CVPR_2020_paper.pdf", "votes": null }, { "id": "898093", "postDate": "06/23/2020 09:45:30", "content": "we have not tried model ensembles yet. we are expecting some boost from it, sad to hear that.
\n\n[UPDATED] model ensembles works
", "rawMarkdown": "we have not tried model ensembles yet. we are expecting some boost from it, sad to hear that.\n\n[UPDATED] model ensembles works", "votes": null }, { "id": "898109", "postDate": "06/23/2020 10:05:24", "content": "I've got the same problem when I tried it. It is counter-intuitive at some point I doubt having a mistake somewhere in my code x)
", "rawMarkdown": "I've got the same problem when I tried it. It is counter-intuitive at some point I doubt having a mistake somewhere in my code x)", "votes": null }, { "id": "898116", "postDate": "06/23/2020 10:12:41", "content": "let me clarify if we are talking about the same things here. \nWhen you say model ensembles you probably wanted to say models with different architectures. Because if you are doing KFold that's already ensemble IMHO. TTA is also ensemble for me.
", "rawMarkdown": "let me clarify if we are talking about the same things here. \nWhen you say model ensembles you probably wanted to say models with different architectures. Because if you are doing KFold that's already ensemble IMHO. TTA is also ensemble for me.", "votes": null }, { "id": "898237", "postDate": "06/23/2020 11:49:01", "content": "I agree. Even I already used TTA, I am still hoping ensemble over folds or over models will give a a boost.
", "rawMarkdown": "I agree. Even I already used TTA, I am still hoping ensemble over folds or over models will give a a boost.", "votes": null }, { "id": "899231", "postDate": "06/24/2020 05:05:48", "content": "I saw a slight boost in public lb score when going from [single model + tta] to [ensemble + tta]. The improvement was a little disappointing though. I thought it'd be larger. In my case, I'm ensembling fairly \"weak\" models which only score ~0.71 without tta on the public lb.
", "rawMarkdown": "I saw a slight boost in public lb score when going from [single model + tta] to [ensemble + tta]. The improvement was a little disappointing though. I thought it'd be larger. In my case, I'm ensembling fairly \"weak\" models which only score ~0.71 without tta on the public lb.", "votes": null }, { "id": "899422", "postDate": "06/24/2020 08:25:23", "content": "Me too. I just used TTA with single model. But it decreased my LB score. I really wondered how did they get boost with TTA...
", "rawMarkdown": "Me too. I just used TTA with single model. But it decreased my LB score. I really wondered how did they get boost with TTA...", "votes": null }, { "id": "899736", "postDate": "06/24/2020 12:16:33", "content": "I found it very useful to build a test bench to try out various TTA schemes on my holdout set. There are some times when TTA doesn't help, but it looks a lot like random luck of the draw, and modifying the TTA scheme a bit yields good results.
", "rawMarkdown": "I found it very useful to build a test bench to try out various TTA schemes on my holdout set. There are some times when TTA doesn't help, but it looks a lot like random luck of the draw, and modifying the TTA scheme a bit yields good results.", "votes": null }, { "id": "910649", "postDate": "07/01/2020 09:21:07", "content": "TTA in 5 fold model ensembles reduces my score. I only used OOF prediction to fine tune the thresholds. Then single model with pesudo-labelling works wonders for me :p and I put all possible TTA. Time to fine tuning :p...
", "rawMarkdown": "TTA in 5 fold model ensembles reduces my score. I only used OOF prediction to fine tune the thresholds. Then single model with pesudo-labelling works wonders for me :p and I put all possible TTA. Time to fine tuning :p...", "votes": null }, { "id": "913764", "postDate": "07/03/2020 12:01:57", "content": "Me too. using rotation TTA with just one model with 80 train 20 val => LB 73.23\nbut 5 fold ensemble + tta => LB 72.9 with efficient det
", "rawMarkdown": "Me too. using rotation TTA with just one model with 80 train 20 val => LB 73.23\nbut 5 fold ensemble + tta => LB 72.9 with efficient det", "votes": null }, { "id": "917456", "postDate": "07/06/2020 14:28:49", "content": "Thanks. I also got TTA worked. My experience is that the score threshold should be increased when using TTA.
", "rawMarkdown": "Thanks. I also got TTA worked. My experience is that the score threshold should be increased when using TTA.", "votes": null }, { "id": "940183", "postDate": "07/22/2020 18:10:40", "content": "TTA in 5 fold model ensembles reduces my score. I am sad and confused
", "rawMarkdown": "TTA in 5 fold model ensembles reduces my score. I am sad and confused", "votes": null }, { "id": "947581", "postDate": "07/27/2020 11:03:08", "content": "@fangcheng what exactly you did? I am also facing similar issue.
", "rawMarkdown": "fangcheng what exactly you did? I am also facing similar issue.", "votes": null } ], "comments": [ { "id": 897978, "author_name": "chanhu", "author_url": "", "post_date": "06/23/2020 07:54:04", "content": "Me too. using TTA in 5 fold model ensembles reduced my LB score. \nHowever, I think the results are much better than my single model.
", "votes": null, "replies": [] }, { "id": 897980, "author_name": "matthewmasters", "author_url": "", "post_date": "06/23/2020 07:54:51", "content": "I've always found ensembling over multiple models to be worse than just using the single best model out of the bunch.
\n\nThough this makes me think that the model which performs the best may just be overfit to the public LB. And even though it has a lower score, the ensemble may generalize better. Not sure how I can clear up this suspicion though.
", "votes": null, "replies": [] }, { "id": 897988, "author_name": "alexandersoare", "author_url": "", "post_date": "06/23/2020 07:58:43", "content": "@chanhu @matthewmasters thanks for the responses. I had a thought - let me know what you think. If the training set is introducing a bias to our models, then I suppose more averaging will cause a result to pull towards that bias point. Whereas with less averaging it's luck of the draw (enter psychology, where we tend to only place significance on the lucky draws, and therefore think that a bigger ensemble is always worse).
", "votes": null, "replies": [ { "id": 898020, "author_name": "chanhu", "author_url": "", "post_date": "06/23/2020 08:29:10", "content": "Yes, I think so. how to deal with the noisy labels might be key point. (large bounding boxes, some wheat does not be labeled and so on). I have tried to fix them by myself, and hopefully they will be fixed by datasets update.
\n\nAlso, the approach in this paper may useful. \nhttp://openaccess.thecvf.com/content_CVPR_2020/papers/Li_Learning_From_Noisy_Anchors_for_One-Stage_Object_Detection_CVPR_2020_paper.pdf
", "votes": null, "replies": [] } ] }, { "id": 898093, "author_name": "nvnnghia", "author_url": "", "post_date": "06/23/2020 09:45:30", "content": "we have not tried model ensembles yet. we are expecting some boost from it, sad to hear that.
\n\n[UPDATED] model ensembles works
", "votes": null, "replies": [ { "id": 898116, "author_name": "valanm", "author_url": "", "post_date": "06/23/2020 10:12:41", "content": "let me clarify if we are talking about the same things here. \nWhen you say model ensembles you probably wanted to say models with different architectures. Because if you are doing KFold that's already ensemble IMHO. TTA is also ensemble for me.
", "votes": null, "replies": [] }, { "id": 898237, "author_name": "nvnnghia", "author_url": "", "post_date": "06/23/2020 11:49:01", "content": "I agree. Even I already used TTA, I am still hoping ensemble over folds or over models will give a a boost.
", "votes": null, "replies": [] } ] }, { "id": 898109, "author_name": "ffares", "author_url": "", "post_date": "06/23/2020 10:05:24", "content": "I've got the same problem when I tried it. It is counter-intuitive at some point I doubt having a mistake somewhere in my code x)
", "votes": null, "replies": [] }, { "id": 899231, "author_name": "reppic", "author_url": "", "post_date": "06/24/2020 05:05:48", "content": "I saw a slight boost in public lb score when going from [single model + tta] to [ensemble + tta]. The improvement was a little disappointing though. I thought it'd be larger. In my case, I'm ensembling fairly \"weak\" models which only score ~0.71 without tta on the public lb.
", "votes": null, "replies": [] }, { "id": 899422, "author_name": "fangcheng", "author_url": "", "post_date": "06/24/2020 08:25:23", "content": "Me too. I just used TTA with single model. But it decreased my LB score. I really wondered how did they get boost with TTA...
", "votes": null, "replies": [ { "id": 899736, "author_name": "alexandersoare", "author_url": "", "post_date": "06/24/2020 12:16:33", "content": "I found it very useful to build a test bench to try out various TTA schemes on my holdout set. There are some times when TTA doesn't help, but it looks a lot like random luck of the draw, and modifying the TTA scheme a bit yields good results.
", "votes": null, "replies": [] }, { "id": 917456, "author_name": "fangcheng", "author_url": "", "post_date": "07/06/2020 14:28:49", "content": "Thanks. I also got TTA worked. My experience is that the score threshold should be increased when using TTA.
", "votes": null, "replies": [] }, { "id": 947581, "author_name": "iamprateek", "author_url": "", "post_date": "07/27/2020 11:03:08", "content": "@fangcheng what exactly you did? I am also facing similar issue.
", "votes": null, "replies": [] } ] }, { "id": 910649, "author_name": "ahmmadrashid", "author_url": "", "post_date": "07/01/2020 09:21:07", "content": "TTA in 5 fold model ensembles reduces my score. I only used OOF prediction to fine tune the thresholds. Then single model with pesudo-labelling works wonders for me :p and I put all possible TTA. Time to fine tuning :p...
", "votes": null, "replies": [] }, { "id": 913764, "author_name": "deepkim", "author_url": "", "post_date": "07/03/2020 12:01:57", "content": "Me too. using rotation TTA with just one model with 80 train 20 val => LB 73.23\nbut 5 fold ensemble + tta => LB 72.9 with efficient det
", "votes": null, "replies": [] }, { "id": 940183, "author_name": "irinali", "author_url": "", "post_date": "07/22/2020 18:10:40", "content": "TTA in 5 fold model ensembles reduces my score. I am sad and confused
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "897965": "I'm trying to diagnose an issue where I'm getting worse results when ensembling over several TTAs and models at once. Doing TTA with one model works pretty well, and not doing TTA but ensembling over models also works pretty well. Combining the two gives worse results somehow.\n\nI'm using wbf so I wouldn't expect to see discontinuities in scores as a function of number of predictions sets. I would expect the score to become smoother as more prediction sets are added.", "897978": "Me too. using TTA in 5 fold model ensembles reduced my LB score. \nHowever, I think the results are much better than my single model.", "897980": "I've always found ensembling over multiple models to be worse than just using the single best model out of the bunch.\n\nThough this makes me think that the model which performs the best may just be overfit to the public LB. And even though it has a lower score, the ensemble may generalize better. Not sure how I can clear up this suspicion though.", "897988": "chanhu @matthewmasters thanks for the responses. I had a thought - let me know what you think. If the training set is introducing a bias to our models, then I suppose more averaging will cause a result to pull towards that bias point. Whereas with less averaging it's luck of the draw (enter psychology, where we tend to only place significance on the lucky draws, and therefore think that a bigger ensemble is always worse).", "898020": "Yes, I think so. how to deal with the noisy labels might be key point. (large bounding boxes, some wheat does not be labeled and so on). I have tried to fix them by myself, and hopefully they will be fixed by datasets update.\n\nAlso, the approach in this paper may useful. \nhttp://openaccess.thecvf.com/content_CVPR_2020/papers/Li_Learning_From_Noisy_Anchors_for_One-Stage_Object_Detection_CVPR_2020_paper.pdf", "898093": "we have not tried model ensembles yet. we are expecting some boost from it, sad to hear that.\n\n[UPDATED] model ensembles works", "898109": "I've got the same problem when I tried it. It is counter-intuitive at some point I doubt having a mistake somewhere in my code x)", "898116": "let me clarify if we are talking about the same things here. \nWhen you say model ensembles you probably wanted to say models with different architectures. Because if you are doing KFold that's already ensemble IMHO. TTA is also ensemble for me.", "898237": "I agree. Even I already used TTA, I am still hoping ensemble over folds or over models will give a a boost.", "899231": "I saw a slight boost in public lb score when going from [single model + tta] to [ensemble + tta]. The improvement was a little disappointing though. I thought it'd be larger. In my case, I'm ensembling fairly \"weak\" models which only score ~0.71 without tta on the public lb.", "899422": "Me too. I just used TTA with single model. But it decreased my LB score. I really wondered how did they get boost with TTA...", "899736": "I found it very useful to build a test bench to try out various TTA schemes on my holdout set. There are some times when TTA doesn't help, but it looks a lot like random luck of the draw, and modifying the TTA scheme a bit yields good results.", "910649": "TTA in 5 fold model ensembles reduces my score. I only used OOF prediction to fine tune the thresholds. Then single model with pesudo-labelling works wonders for me :p and I put all possible TTA. Time to fine tuning :p...", "913764": "Me too. using rotation TTA with just one model with 80 train 20 val => LB 73.23\nbut 5 fold ensemble + tta => LB 72.9 with efficient det", "917456": "Thanks. I also got TTA worked. My experience is that the score threshold should be increased when using TTA.", "940183": "TTA in 5 fold model ensembles reduces my score. I am sad and confused", "947581": "fangcheng what exactly you did? I am also facing similar issue." }, "source": "meta" }