{ "id": 545933, "title": "Distilling for catboost", "url": "/competitions/applications-of-deep-learning-wustl-fall-2024/discussion/545933", "author_name": "", "post_date": "2024-11-12T20:57:06.013525500Z", "votes": 1, "comment_count": 11, "views": 0, "content": "

I think my model is going plateau whatever new information or data I inject to it at this point. It might be due to too much information to be put in the same model to catboost. Are there any hints on that? I was trying to do feature selection the other day but the result made the training worse, even for soft selection

", "messages": [ { "id": "3043958", "postDate": "11/12/2024 20:57:06", "content": "

I think my model is going plateau whatever new information or data I inject to it at this point. It might be due to too much information to be put in the same model to catboost. Are there any hints on that? I was trying to do feature selection the other day but the result made the training worse, even for soft selection

", "rawMarkdown": "I think my model is going plateau whatever new information or data I inject to it at this point. It might be due to too much information to be put in the same model to catboost. Are there any hints on that? I was trying to do feature selection the other day but the result made the training worse, even for soft selection", "votes": null }, { "id": "3043984", "postDate": "11/12/2024 22:03:38", "content": "

I tried feature permutation but it wasn't helpful. It might due to the size of our data I assume. How about adding all information from the tabular data into the description, and then transfer it into embeddings? I've never manipulated embeddings before, but viellicht there would be some surprising www

", "rawMarkdown": "I tried feature permutation but it wasn't helpful. It might due to the size of our data I assume. How about adding all information from the tabular data into the description, and then transfer it into embeddings? I've never manipulated embeddings before, but viellicht there would be some surprising www", "votes": null }, { "id": "3044176", "postDate": "11/13/2024 06:14:09", "content": "

Hi @dantingzeng and @kaiyisung,

\n

Broadly speaking, I would not expect that feature selection would meaningfully improve accuracy and converting the tabular data into embeddings likely would not translate into new information that the models can leverage. I will share a couple general hints with you and the class so you can continue learning and testing your abilities in a productive direction.

\n

1) Using an ensemble of machine learning models general improves the generalizability of your model. You will find in Jeff's prior competitions the top solutions combined the predictions from multiple predictive algorithem - this approach helped my team win in the fall 2022. Although CatBoost is very practical for this dataset, because it can handle high-dimensional categorical features, you may still want to experiment with alternatives, such as XGBoost. Additionally, you can also derive multiple Catboost algorithem by changing their hyperparameters, such as having one model that is very deep but does not have many trees and another that is very shallow with many trees. There are many hyperparameters and algorithem to test and evaluate.

\n

2) You can use multimodal models to generate a description of each image of the home and subsequently turn the descriptions into word embeddings that you can add to your model. The effectiveness of this approach will likely come down to how well you can prompt the large language model to describe important elements from the images - that are highly correlate with the home value. I have prompted an example script of how to download and use a multimodal model from huggingfaces. These are powerful models but they require a lot of compute and memory, so running them across all 1200 images might be a challenge. Lastly, focus on engineering features that can help explain the most expensive homes, which are the outliers in this dataset.

", "rawMarkdown": "Hi @dantingzeng and @kaiyisung,\n\nBroadly speaking, I would not expect that feature selection would meaningfully improve accuracy and converting the tabular data into embeddings likely would not translate into new information that the models can leverage. I will share a couple general hints with you and the class so you can continue learning and testing your abilities in a productive direction. \n\n1) Using an ensemble of machine learning models general improves the generalizability of your model. You will find in Jeff's prior competitions the top solutions combined the predictions from multiple predictive algorithem - this approach helped my team win in the fall 2022. Although CatBoost is very practical for this dataset, because it can handle high-dimensional categorical features, you may still want to experiment with alternatives, such as XGBoost. Additionally, you can also derive multiple Catboost algorithem by changing their hyperparameters, such as having one model that is very deep but does not have many trees and another that is very shallow with many trees. There are many hyperparameters and algorithem to test and evaluate.\n\n2) You can use multimodal models to generate a description of each image of the home and subsequently turn the descriptions into word embeddings that you can add to your model. The effectiveness of this approach will likely come down to how well you can prompt the large language model to describe important elements from the images - that are highly correlate with the home value. I have prompted an example script of how to download and use a multimodal model from huggingfaces. These are powerful models but they require a lot of compute and memory, so running them across all 1200 images might be a challenge. Lastly, focus on engineering features that can help explain the most expensive homes, which are the outliers in this dataset.", "votes": null }, { "id": "3044262", "postDate": "11/13/2024 08:01:30", "content": "

Most of what you said is what I have already been doing haha

", "rawMarkdown": "Most of what you said is what I have already been doing haha", "votes": null }, { "id": "3044267", "postDate": "11/13/2024 08:03:25", "content": "

Vision-language models I have been using include the Qwen2 you have suggested

", "rawMarkdown": "Vision-language models I have been using include the Qwen2 you have suggested", "votes": null }, { "id": "3044268", "postDate": "11/13/2024 08:05:32", "content": "

XGBoost and different versions of catboost did not add additional grounds to my model

", "rawMarkdown": "XGBoost and different versions of catboost did not add additional grounds to my model", "votes": null }, { "id": "3044272", "postDate": "11/13/2024 08:08:06", "content": "

There is a multi-modal structure that I have been using that varies from what you said, that can improve the prediction for a single model for a lot though. However, ensembling it with my other models does not gain a lot more grounds than what I already have

", "rawMarkdown": "There is a multi-modal structure that I have been using that varies from what you said, that can improve the prediction for a single model for a lot though. However, ensembling it with my other models does not gain a lot more grounds than what I already have", "votes": null }, { "id": "3044287", "postDate": "11/13/2024 08:17:37", "content": "

This is why I had the language model is all you need discussion

", "rawMarkdown": "This is why I had the language model is all you need discussion", "votes": null }, { "id": "3044290", "postDate": "11/13/2024 08:19:25", "content": "

I think the greatest insight you might be able to provide is how to ensemble better then. I have tried using oof for ridge and hill_climb, but they gave worse result than just averaging for this competition

", "rawMarkdown": "I think the greatest insight you might be able to provide is how to ensemble better then. I have tried using oof for ridge and hill_climb, but they gave worse result than just averaging for this competition", "votes": null }, { "id": "3044291", "postDate": "11/13/2024 08:19:53", "content": "

it only gained 3000 ish in lb and around the same in CV for the ensemble

", "rawMarkdown": "it only gained 3000 ish in lb and around the same in CV for the ensemble", "votes": null }, { "id": "3044299", "postDate": "11/13/2024 08:29:40", "content": "

It went around 8000 lower, to around 86500 though. For a single main model notebook it is impressive.

", "rawMarkdown": "It went around 8000 lower, to around 86500 though. For a single main model notebook it is impressive.", "votes": null }, { "id": "3044317", "postDate": "11/13/2024 08:57:50", "content": "

embeddings actually tells a lot less than the features themselves, just because it would be blended in a lot more

", "rawMarkdown": "embeddings actually tells a lot less than the features themselves, just because it would be blended in a lot more", "votes": null } ], "comments": [ { "id": 3043984, "author_name": "kaiyisung", "author_url": "", "post_date": "11/12/2024 22:03:38", "content": "

I tried feature permutation but it wasn't helpful. It might due to the size of our data I assume. How about adding all information from the tabular data into the description, and then transfer it into embeddings? I've never manipulated embeddings before, but viellicht there would be some surprising www

", "votes": null, "replies": [ { "id": 3044317, "author_name": "dantingzeng", "author_url": "", "post_date": "11/13/2024 08:57:50", "content": "

embeddings actually tells a lot less than the features themselves, just because it would be blended in a lot more

", "votes": null, "replies": [] } ] }, { "id": 3044176, "author_name": "robertjirsaraie", "author_url": "", "post_date": "11/13/2024 06:14:09", "content": "

Hi @dantingzeng and @kaiyisung,

\n

Broadly speaking, I would not expect that feature selection would meaningfully improve accuracy and converting the tabular data into embeddings likely would not translate into new information that the models can leverage. I will share a couple general hints with you and the class so you can continue learning and testing your abilities in a productive direction.

\n

1) Using an ensemble of machine learning models general improves the generalizability of your model. You will find in Jeff's prior competitions the top solutions combined the predictions from multiple predictive algorithem - this approach helped my team win in the fall 2022. Although CatBoost is very practical for this dataset, because it can handle high-dimensional categorical features, you may still want to experiment with alternatives, such as XGBoost. Additionally, you can also derive multiple Catboost algorithem by changing their hyperparameters, such as having one model that is very deep but does not have many trees and another that is very shallow with many trees. There are many hyperparameters and algorithem to test and evaluate.

\n

2) You can use multimodal models to generate a description of each image of the home and subsequently turn the descriptions into word embeddings that you can add to your model. The effectiveness of this approach will likely come down to how well you can prompt the large language model to describe important elements from the images - that are highly correlate with the home value. I have prompted an example script of how to download and use a multimodal model from huggingfaces. These are powerful models but they require a lot of compute and memory, so running them across all 1200 images might be a challenge. Lastly, focus on engineering features that can help explain the most expensive homes, which are the outliers in this dataset.

", "votes": null, "replies": [ { "id": 3044262, "author_name": "dantingzeng", "author_url": "", "post_date": "11/13/2024 08:01:30", "content": "

Most of what you said is what I have already been doing haha

", "votes": null, "replies": [] }, { "id": 3044267, "author_name": "dantingzeng", "author_url": "", "post_date": "11/13/2024 08:03:25", "content": "

Vision-language models I have been using include the Qwen2 you have suggested

", "votes": null, "replies": [ { "id": 3044287, "author_name": "dantingzeng", "author_url": "", "post_date": "11/13/2024 08:17:37", "content": "

This is why I had the language model is all you need discussion

", "votes": null, "replies": [] } ] }, { "id": 3044268, "author_name": "dantingzeng", "author_url": "", "post_date": "11/13/2024 08:05:32", "content": "

XGBoost and different versions of catboost did not add additional grounds to my model

", "votes": null, "replies": [] }, { "id": 3044272, "author_name": "dantingzeng", "author_url": "", "post_date": "11/13/2024 08:08:06", "content": "

There is a multi-modal structure that I have been using that varies from what you said, that can improve the prediction for a single model for a lot though. However, ensembling it with my other models does not gain a lot more grounds than what I already have

", "votes": null, "replies": [ { "id": 3044291, "author_name": "dantingzeng", "author_url": "", "post_date": "11/13/2024 08:19:53", "content": "

it only gained 3000 ish in lb and around the same in CV for the ensemble

", "votes": null, "replies": [] }, { "id": 3044299, "author_name": "dantingzeng", "author_url": "", "post_date": "11/13/2024 08:29:40", "content": "

It went around 8000 lower, to around 86500 though. For a single main model notebook it is impressive.

", "votes": null, "replies": [] } ] }, { "id": 3044290, "author_name": "dantingzeng", "author_url": "", "post_date": "11/13/2024 08:19:25", "content": "

I think the greatest insight you might be able to provide is how to ensemble better then. I have tried using oof for ridge and hill_climb, but they gave worse result than just averaging for this competition

", "votes": null, "replies": [] } ] } ], "raw_markdown_by_id": { "3043958": "I think my model is going plateau whatever new information or data I inject to it at this point. It might be due to too much information to be put in the same model to catboost. Are there any hints on that? I was trying to do feature selection the other day but the result made the training worse, even for soft selection", "3043984": "I tried feature permutation but it wasn't helpful. It might due to the size of our data I assume. How about adding all information from the tabular data into the description, and then transfer it into embeddings? I've never manipulated embeddings before, but viellicht there would be some surprising www", "3044176": "Hi @dantingzeng and @kaiyisung,\n\nBroadly speaking, I would not expect that feature selection would meaningfully improve accuracy and converting the tabular data into embeddings likely would not translate into new information that the models can leverage. I will share a couple general hints with you and the class so you can continue learning and testing your abilities in a productive direction. \n\n1) Using an ensemble of machine learning models general improves the generalizability of your model. You will find in Jeff's prior competitions the top solutions combined the predictions from multiple predictive algorithem - this approach helped my team win in the fall 2022. Although CatBoost is very practical for this dataset, because it can handle high-dimensional categorical features, you may still want to experiment with alternatives, such as XGBoost. Additionally, you can also derive multiple Catboost algorithem by changing their hyperparameters, such as having one model that is very deep but does not have many trees and another that is very shallow with many trees. There are many hyperparameters and algorithem to test and evaluate.\n\n2) You can use multimodal models to generate a description of each image of the home and subsequently turn the descriptions into word embeddings that you can add to your model. The effectiveness of this approach will likely come down to how well you can prompt the large language model to describe important elements from the images - that are highly correlate with the home value. I have prompted an example script of how to download and use a multimodal model from huggingfaces. These are powerful models but they require a lot of compute and memory, so running them across all 1200 images might be a challenge. Lastly, focus on engineering features that can help explain the most expensive homes, which are the outliers in this dataset.", "3044262": "Most of what you said is what I have already been doing haha", "3044267": "Vision-language models I have been using include the Qwen2 you have suggested", "3044268": "XGBoost and different versions of catboost did not add additional grounds to my model", "3044272": "There is a multi-modal structure that I have been using that varies from what you said, that can improve the prediction for a single model for a lot though. However, ensembling it with my other models does not gain a lot more grounds than what I already have", "3044287": "This is why I had the language model is all you need discussion", "3044290": "I think the greatest insight you might be able to provide is how to ensemble better then. I have tried using oof for ridge and hill_climb, but they gave worse result than just averaging for this competition", "3044291": "it only gained 3000 ish in lb and around the same in CV for the ensemble", "3044299": "It went around 8000 lower, to around 86500 though. For a single main model notebook it is impressive.", "3044317": "embeddings actually tells a lot less than the features themselves, just because it would be blended in a lot more" }, "source": "meta" }