I got some idea that first train the model with binary targets, and get proba prediction on the trainset, we use that as features or label smoothing for the second stage training with multiclass, is it a valid approach, Am i crazy?
", "messages": [ { "id": "3223350", "postDate": "06/13/2025 07:27:36", "content": "I got some idea that first train the model with binary targets, and get proba prediction on the trainset, we use that as features or label smoothing for the second stage training with multiclass, is it a valid approach, Am i crazy?
", "rawMarkdown": "I got some idea that first train the model with binary targets, and get proba prediction on the trainset, we use that as features or label smoothing for the second stage training with multiclass, is it a valid approach, Am i crazy?", "votes": null }, { "id": "3223407", "postDate": "06/13/2025 08:57:52", "content": "This is kind of stacking and ofc you can try it. You can also try predicting them at the same time with a multi-label modelling approach.
", "rawMarkdown": "This is kind of stacking and ofc you can try it. You can also try predicting them at the same time with a multi-label modelling approach.", "votes": null }, { "id": "3223421", "postDate": "06/13/2025 09:12:12", "content": "I try to use 'sequence_type' as feature and it can increase my cv 0.01-0.015.However,when i train a model to predict 'sequence_type' and add predictions to features to test cv,it decrease 0.01.
", "rawMarkdown": "I try to use 'sequence_type' as feature and it can increase my cv 0.01-0.015.However,when i train a model to predict 'sequence_type' and add predictions to features to test cv,it decrease 0.01.", "votes": null }, { "id": "3223499", "postDate": "06/13/2025 12:06:30", "content": "Interesting, maybe you can change the aaproach. Since in the HCT comp this eventually led to not getting gold.
", "rawMarkdown": "Interesting, maybe you can change the aaproach. Since in the HCT comp this eventually led to not getting gold.", "votes": null }, { "id": "3223525", "postDate": "06/13/2025 12:48:15", "content": "As long as it doesn't take much time for inference you could try it
", "rawMarkdown": "As long as it doesn't take much time for inference you could try it", "votes": null }, { "id": "3223703", "postDate": "06/13/2025 17:53:45", "content": "Your proposed method follows an ensemble strategy known as \"stacking.\" In my view, the effectiveness of this approach relies heavily on the differences between the two-stage models.
\nAs my intuition, if both stages use similar models (such as neural networks with only different classification heads), the benefits of stacking might be limited or even negative. Here are some short ideas:
\nThe second-stage model already has the capability to learn what the first-stage model extracted from the original data. Thus, the output probabilities of the first stage become redundant, failing to provide a truly new perspective. It may introduce noise and bias from the first-stage model, disrupting the learning process of the second-stage model.
\nAlso, stacking can cause overfitting, therefore cross-validation (oof as feature) should be strictly used to generate probability features.
\nTherefore, for this approach to succeed, the first-stage model must provide genuinely valuable and unique information. This is typically achieved by using very different types of models, leveraging their cognitive differences to create information gain.
\nThat said, if you're planning to use stacking with two models, why not directly use multiclass probabilities? This approach has the potential for higher information gains.
", "rawMarkdown": "Your proposed method follows an ensemble strategy known as \"stacking.\" In my view, the effectiveness of this approach relies heavily on the differences between the two-stage models.\n\nAs my intuition, if both stages use similar models (such as neural networks with only different classification heads), the benefits of stacking might be limited or even negative. Here are some short ideas:\n\nThe second-stage model already has the capability to learn what the first-stage model extracted from the original data. Thus, the output probabilities of the first stage become redundant, failing to provide a truly new perspective. It may introduce noise and bias from the first-stage model, disrupting the learning process of the second-stage model.\n\nAlso, stacking can cause overfitting, therefore cross-validation (oof as feature) should be strictly used to generate probability features.\n\nTherefore, for this approach to succeed, the first-stage model must provide genuinely valuable and unique information. This is typically achieved by using very different types of models, leveraging their cognitive differences to create information gain.\n\nThat said, if you're planning to use stacking with two models, why not directly use multiclass probabilities? This approach has the potential for higher information gains.", "votes": null } ], "comments": [ { "id": 3223407, "author_name": "nlztrk", "author_url": "", "post_date": "06/13/2025 08:57:52", "content": "This is kind of stacking and ofc you can try it. You can also try predicting them at the same time with a multi-label modelling approach.
", "votes": null, "replies": [] }, { "id": 3223421, "author_name": "minatoyukinaxlisa", "author_url": "", "post_date": "06/13/2025 09:12:12", "content": "I try to use 'sequence_type' as feature and it can increase my cv 0.01-0.015.However,when i train a model to predict 'sequence_type' and add predictions to features to test cv,it decrease 0.01.
", "votes": null, "replies": [ { "id": 3223499, "author_name": "stefanoclss", "author_url": "", "post_date": "06/13/2025 12:06:30", "content": "Interesting, maybe you can change the aaproach. Since in the HCT comp this eventually led to not getting gold.
", "votes": null, "replies": [] } ] }, { "id": 3223525, "author_name": "bryanpiguave", "author_url": "", "post_date": "06/13/2025 12:48:15", "content": "As long as it doesn't take much time for inference you could try it
", "votes": null, "replies": [] }, { "id": 3223703, "author_name": "ryenhails", "author_url": "", "post_date": "06/13/2025 17:53:45", "content": "Your proposed method follows an ensemble strategy known as \"stacking.\" In my view, the effectiveness of this approach relies heavily on the differences between the two-stage models.
\nAs my intuition, if both stages use similar models (such as neural networks with only different classification heads), the benefits of stacking might be limited or even negative. Here are some short ideas:
\nThe second-stage model already has the capability to learn what the first-stage model extracted from the original data. Thus, the output probabilities of the first stage become redundant, failing to provide a truly new perspective. It may introduce noise and bias from the first-stage model, disrupting the learning process of the second-stage model.
\nAlso, stacking can cause overfitting, therefore cross-validation (oof as feature) should be strictly used to generate probability features.
\nTherefore, for this approach to succeed, the first-stage model must provide genuinely valuable and unique information. This is typically achieved by using very different types of models, leveraging their cognitive differences to create information gain.
\nThat said, if you're planning to use stacking with two models, why not directly use multiclass probabilities? This approach has the potential for higher information gains.
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "3223350": "I got some idea that first train the model with binary targets, and get proba prediction on the trainset, we use that as features or label smoothing for the second stage training with multiclass, is it a valid approach, Am i crazy?", "3223407": "This is kind of stacking and ofc you can try it. You can also try predicting them at the same time with a multi-label modelling approach.", "3223421": "I try to use 'sequence_type' as feature and it can increase my cv 0.01-0.015.However,when i train a model to predict 'sequence_type' and add predictions to features to test cv,it decrease 0.01.", "3223499": "Interesting, maybe you can change the aaproach. Since in the HCT comp this eventually led to not getting gold.", "3223525": "As long as it doesn't take much time for inference you could try it", "3223703": "Your proposed method follows an ensemble strategy known as \"stacking.\" In my view, the effectiveness of this approach relies heavily on the differences between the two-stage models.\n\nAs my intuition, if both stages use similar models (such as neural networks with only different classification heads), the benefits of stacking might be limited or even negative. Here are some short ideas:\n\nThe second-stage model already has the capability to learn what the first-stage model extracted from the original data. Thus, the output probabilities of the first stage become redundant, failing to provide a truly new perspective. It may introduce noise and bias from the first-stage model, disrupting the learning process of the second-stage model.\n\nAlso, stacking can cause overfitting, therefore cross-validation (oof as feature) should be strictly used to generate probability features.\n\nTherefore, for this approach to succeed, the first-stage model must provide genuinely valuable and unique information. This is typically achieved by using very different types of models, leveraging their cognitive differences to create information gain.\n\nThat said, if you're planning to use stacking with two models, why not directly use multiclass probabilities? This approach has the potential for higher information gains." }, "source": "meta" }