When we ensemble different models, there are currently two strategies: 1. All models have the same weight; 2. Different models have different weights. Based on the current public score comparison, it is evident that the second method can achieve higher scores. However, please note that we set weights for different models based on their public scores. If there is a significant difference in scores between the public dataset and the final dataset, this may lead to overfitting. So do you have any strategies for model ensemble？Welcome to discuss😁
", "messages": [ { "id": "2538541", "postDate": "11/26/2023 08:42:46", "content": "When we ensemble different models, there are currently two strategies: 1. All models have the same weight; 2. Different models have different weights. Based on the current public score comparison, it is evident that the second method can achieve higher scores. However, please note that we set weights for different models based on their public scores. If there is a significant difference in scores between the public dataset and the final dataset, this may lead to overfitting. So do you have any strategies for model ensemble？Welcome to discuss😁
", "rawMarkdown": "When we ensemble different models, there are currently two strategies: 1. All models have the same weight; 2. Different models have different weights. Based on the current public score comparison, it is evident that the second method can achieve higher scores. However, please note that we set weights for different models based on their public scores. If there is a significant difference in scores between the public dataset and the final dataset, this may lead to overfitting. So do you have any strategies for model ensemble？Welcome to discuss😁", "votes": null }, { "id": "2539204", "postDate": "11/26/2023 20:18:22", "content": "The simplest strategy is to take several different models with similar CV/LB, average their predictions, and get final submission file from it.
", "rawMarkdown": "The simplest strategy is to take several different models with similar CV/LB, average their predictions, and get final submission file from it.", "votes": null }, { "id": "2539373", "postDate": "11/27/2023 02:37:30", "content": "That makes sense but it is slightly more difficult in this competition. Are you doing this with an augmentation of the public notebook pre post-processing on all steps and then post processing?
", "rawMarkdown": "That makes sense but it is slightly more difficult in this competition. Are you doing this with an augmentation of the public notebook pre post-processing on all steps and then post processing?", "votes": null }, { "id": "2539376", "postDate": "11/27/2023 02:46:34", "content": "In fact, as you said, I have performed a simple average ensemble of several models, but blindly merging the models does not improve the score on lb. I think some special skills need to be added to the post-processing, This technique can significantly improve LB scores. I believe the top 20 on the list have discovered this technique. I have not been able to explore it yet.
", "rawMarkdown": "In fact, as you said, I have performed a simple average ensemble of several models, but blindly merging the models does not improve the score on lb. I think some special skills need to be added to the post-processing, This technique can significantly improve LB scores. I believe the top 20 on the list have discovered this technique. I have not been able to explore it yet.", "votes": null }, { "id": "2539569", "postDate": "11/27/2023 06:35:34", "content": "Although the evaluation metric is different from this competition, the technique to maximize AUC is posted by @sirishks in SIIM-ISIC Melanoma Classification competition. If you are outputting the probability for each step, you may want to try it. I was particularly interested in power averaging. But it didn't work for me.
", "rawMarkdown": "Although the evaluation metric is different from this competition, [the technique to maximize AUC](https://www.kaggle.com/c/siim-isic-melanoma-classification/discussion/165653) is posted by @sirishks in [SIIM-ISIC Melanoma Classification](https://www.kaggle.com/c/siim-isic-melanoma-classification) competition. If you are outputting the probability for each step, you may want to try it. I was particularly interested in power averaging. But it didn't work for me.", "votes": null }, { "id": "2539577", "postDate": "11/27/2023 06:49:25", "content": "Maybe \"power averaging\" is a good idea. I looked at the probability distributions of several of my best models and found that their distributions were extremely similar. Thank you for your reply.
", "rawMarkdown": "Maybe \"power averaging\" is a good idea. I looked at the probability distributions of several of my best models and found that their distributions were extremely similar. Thank you for your reply.", "votes": null }, { "id": "2539582", "postDate": "11/27/2023 06:55:54", "content": "I'm still experimenting with single models
", "rawMarkdown": "I'm still experimenting with single models", "votes": null }, { "id": "2539703", "postDate": "11/27/2023 09:20:28", "content": "I think post-processing is quite important in this competition, not only for detecting true positive events more accurately but for eliminating false positive events.
", "rawMarkdown": "I think post-processing is quite important in this competition, not only for detecting true positive events more accurately but for eliminating false positive events.", "votes": null }, { "id": "2539738", "postDate": "11/27/2023 09:34:19", "content": "totally agree with you
", "rawMarkdown": "totally agree with you", "votes": null }, { "id": "2541421", "postDate": "11/28/2023 13:33:52", "content": "Yes, like you, I have also performed a simple averaging of multiple models and have reached a bottleneck. It seems that post-processing is crucial.
", "rawMarkdown": "Yes, like you, I have also performed a simple averaging of multiple models and have reached a bottleneck. It seems that post-processing is crucial.", "votes": null } ], "comments": [ { "id": 2539204, "author_name": "atamazian", "author_url": "", "post_date": "11/26/2023 20:18:22", "content": "The simplest strategy is to take several different models with similar CV/LB, average their predictions, and get final submission file from it.
", "votes": null, "replies": [ { "id": 2539373, "author_name": "cody11null", "author_url": "", "post_date": "11/27/2023 02:37:30", "content": "That makes sense but it is slightly more difficult in this competition. Are you doing this with an augmentation of the public notebook pre post-processing on all steps and then post processing?
", "votes": null, "replies": [ { "id": 2539582, "author_name": "atamazian", "author_url": "", "post_date": "11/27/2023 06:55:54", "content": "I'm still experimenting with single models
", "votes": null, "replies": [] } ] }, { "id": 2539376, "author_name": "gentlezdh", "author_url": "", "post_date": "11/27/2023 02:46:34", "content": "In fact, as you said, I have performed a simple average ensemble of several models, but blindly merging the models does not improve the score on lb. I think some special skills need to be added to the post-processing, This technique can significantly improve LB scores. I believe the top 20 on the list have discovered this technique. I have not been able to explore it yet.
", "votes": null, "replies": [ { "id": 2541421, "author_name": "koalas", "author_url": "", "post_date": "11/28/2023 13:33:52", "content": "Yes, like you, I have also performed a simple averaging of multiple models and have reached a bottleneck. It seems that post-processing is crucial.
", "votes": null, "replies": [] } ] } ] }, { "id": 2539569, "author_name": "itsuki9180", "author_url": "", "post_date": "11/27/2023 06:35:34", "content": "Although the evaluation metric is different from this competition, the technique to maximize AUC is posted by @sirishks in SIIM-ISIC Melanoma Classification competition. If you are outputting the probability for each step, you may want to try it. I was particularly interested in power averaging. But it didn't work for me.
", "votes": null, "replies": [ { "id": 2539577, "author_name": "gentlezdh", "author_url": "", "post_date": "11/27/2023 06:49:25", "content": "Maybe \"power averaging\" is a good idea. I looked at the probability distributions of several of my best models and found that their distributions were extremely similar. Thank you for your reply.
", "votes": null, "replies": [] } ] }, { "id": 2539703, "author_name": "takanashihumbert", "author_url": "", "post_date": "11/27/2023 09:20:28", "content": "I think post-processing is quite important in this competition, not only for detecting true positive events more accurately but for eliminating false positive events.
", "votes": null, "replies": [ { "id": 2539738, "author_name": "gentlezdh", "author_url": "", "post_date": "11/27/2023 09:34:19", "content": "totally agree with you
", "votes": null, "replies": [] } ] } ], "raw_markdown_by_id": { "2538541": "When we ensemble different models, there are currently two strategies: 1. All models have the same weight; 2. Different models have different weights. Based on the current public score comparison, it is evident that the second method can achieve higher scores. However, please note that we set weights for different models based on their public scores. If there is a significant difference in scores between the public dataset and the final dataset, this may lead to overfitting. So do you have any strategies for model ensemble？Welcome to discuss😁", "2539204": "The simplest strategy is to take several different models with similar CV/LB, average their predictions, and get final submission file from it.", "2539373": "That makes sense but it is slightly more difficult in this competition. Are you doing this with an augmentation of the public notebook pre post-processing on all steps and then post processing?", "2539376": "In fact, as you said, I have performed a simple average ensemble of several models, but blindly merging the models does not improve the score on lb. I think some special skills need to be added to the post-processing, This technique can significantly improve LB scores. I believe the top 20 on the list have discovered this technique. I have not been able to explore it yet.", "2539569": "Although the evaluation metric is different from this competition, [the technique to maximize AUC](https://www.kaggle.com/c/siim-isic-melanoma-classification/discussion/165653) is posted by @sirishks in [SIIM-ISIC Melanoma Classification](https://www.kaggle.com/c/siim-isic-melanoma-classification) competition. If you are outputting the probability for each step, you may want to try it. I was particularly interested in power averaging. But it didn't work for me.", "2539577": "Maybe \"power averaging\" is a good idea. I looked at the probability distributions of several of my best models and found that their distributions were extremely similar. Thank you for your reply.", "2539582": "I'm still experimenting with single models", "2539703": "I think post-processing is quite important in this competition, not only for detecting true positive events more accurately but for eliminating false positive events.", "2539738": "totally agree with you", "2541421": "Yes, like you, I have also performed a simple averaging of multiple models and have reached a bottleneck. It seems that post-processing is crucial." }, "source": "meta" }