Kinda out of topic, I heard that ensembling models is a powerful method in Kaggle competition. However, do we also solve a real-world problem with this method or it's more like a kaggle thing?
", "messages": [ { "id": "3044889", "postDate": "11/13/2024 23:43:10", "content": "Kinda out of topic, I heard that ensembling models is a powerful method in Kaggle competition. However, do we also solve a real-world problem with this method or it's more like a kaggle thing?
", "rawMarkdown": "Kinda out of topic, I heard that ensembling models is a powerful method in Kaggle competition. However, do we also solve a real-world problem with this method or it's more like a kaggle thing?", "votes": null }, { "id": "3045042", "postDate": "11/14/2024 05:41:40", "content": "@kaiyisung ensemble models are very much used in real life scenarios, but project requirements necessitate their usage. We often work on several projects where time constraints, maintenance problems and overall customer demands are skewed. In such cases, a simple single model/ small model works well. Ensembles can be used in cases where budgets and time constraints are relatively less stringent.
", "rawMarkdown": "kaiyisung ensemble models are very much used in real life scenarios, but project requirements necessitate their usage. We often work on several projects where time constraints, maintenance problems and overall customer demands are skewed. In such cases, a simple single model/ small model works well. Ensembles can be used in cases where budgets and time constraints are relatively less stringent.", "votes": null }, { "id": "3045420", "postDate": "11/14/2024 13:22:43", "content": "@ravi20076 Thanks for your sharing. I just started in this field. Can you give me some advices to my workflow? I start with data cleaning, make a model ranking to select the best model, use optuna to find the best hyperparameter, and start to train the model. After that, I might do features permutation and retrain the model again. This is what I have done so far… I think there is more I can do or some details that were ignored by me… Thanks again!!
", "rawMarkdown": "ravi20076 Thanks for your sharing. I just started in this field. Can you give me some advices to my workflow? I start with data cleaning, make a model ranking to select the best model, use optuna to find the best hyperparameter, and start to train the model. After that, I might do features permutation and retrain the model again. This is what I have done so far... I think there is more I can do or some details that were ignored by me... Thanks again!!", "votes": null }, { "id": "3045718", "postDate": "11/14/2024 18:03:42", "content": "Different models can come together. You can try hill climb, ridge or etc. They were not too strong for me but they could work for you.
", "rawMarkdown": "Different models can come together. You can try hill climb, ridge or etc. They were not too strong for me but they could work for you.", "votes": null }, { "id": "3045873", "postDate": "11/14/2024 21:43:35", "content": "I just checked ridge , can I understand it as another regression model?
", "rawMarkdown": "I just checked ridge , can I understand it as another regression model?", "votes": null }, { "id": "3045908", "postDate": "11/14/2024 22:38:56", "content": "yes. It can be used to regress for your oof and select what percentage of each notebook's output to be used in your final submission. You can check this out. https://www.kaggle.com/code/adaubas/pss4e05-1st-place-solution-ensemble-with-ridge
", "rawMarkdown": "yes. It can be used to regress for your oof and select what percentage of each notebook's output to be used in your final submission. You can check this out. https://www.kaggle.com/code/adaubas/pss4e05-1st-place-solution-ensemble-with-ridge", "votes": null }, { "id": "3045911", "postDate": "11/14/2024 22:40:25", "content": "What type of data cleaning did you do? My feature selection tells me every data is important which is a bit strange
", "rawMarkdown": "What type of data cleaning did you do? My feature selection tells me every data is important which is a bit strange", "votes": null }, { "id": "3046699", "postDate": "11/15/2024 18:51:35", "content": "Is it what is working for you now?
", "rawMarkdown": "Is it what is working for you now?", "votes": null }, { "id": "3046826", "postDate": "11/15/2024 23:01:09", "content": "@dantingzeng Jein…? I tried three different ways, but I said the main improvement is achieved by autogluon. It definitely did a far better emsembling models job than me.
\n1st try: performing feature engineering with cat and lgbm(90000)
\n I combine an embeddings(1546 columns) and tabular data(11 columns). After feature engineering, I got a 1200 columns data. Yep, it actually improve my grade from 100000 to 90000.
\n2nd try: Autogluon without feature engineering(77000)
\n I get a very good result as you noticed.
\n3rd try: Autogluon withfeature engineering(70000)
\n Though Autogluon can do that on its own, I trust myself more :P
This is what just happened… However, I'm worried about overfitting when my score get higher and higher.
", "rawMarkdown": "dantingzeng Jein...? I tried three different ways, but I said the main improvement is achieved by autogluon. It definitely did a far better emsembling models job than me.\n\n**1st try: performing feature engineering with cat and lgbm(90000)** \n I combine an embeddings(1546 columns) and tabular data(11 columns). After feature engineering, I got a 1200 columns data. Yep, it actually improve my grade from 100000 to 90000.\n**2nd try: Autogluon without feature engineering(77000)** \n I get a very good result as you noticed.\n**3rd try: Autogluon withfeature engineering(70000)** \n Though Autogluon can do that on its own, I trust myself more :P \n\nThis is what just happened... However, I'm worried about overfitting when my score get higher and higher.", "votes": null }, { "id": "3046998", "postDate": "11/16/2024 06:11:11", "content": "This is impressive! Thanks so much for sharing. You can check your overfitting by holding out some of your data for training and testing them afterwards. People in kaggle usually have both cv and lb score to verify that they are doing a good training. To reciprocate I have shared my base line for image prompt. I will start to experiment with AutoML and I will say that doing autoencoder before merging to catboost is actually what improved my score to what it was. https://aclanthology.org/2021.maiworkshop-1.5/ Almost the same method have used for my conference paper a few years ago. Maybe it could be doing some similar things to the AutoML but not as good haha. Technology improving.
", "rawMarkdown": "This is impressive! Thanks so much for sharing. You can check your overfitting by holding out some of your data for training and testing them afterwards. People in kaggle usually have both cv and lb score to verify that they are doing a good training. To reciprocate I have shared my base line for image prompt. I will start to experiment with AutoML and I will say that doing autoencoder before merging to catboost is actually what improved my score to what it was. https://aclanthology.org/2021.maiworkshop-1.5/ Almost the same method have used for my conference paper a few years ago. Maybe it could be doing some similar things to the AutoML but not as good haha. Technology improving.", "votes": null }, { "id": "3047036", "postDate": "11/16/2024 07:06:21", "content": "This is very impressive work from you both! I have not used AutoML or an autoencoder prior to model development, but I will look more into these methods after seeing your results. Keep up the great work!
\nAnother key benefit of Ensemble ML is that it is a useful way to increase the generalizability of your solution. It is easy to overfit to the public leaderboard, especially if you are using a complex algorithm and iteratively assessing its performance based on your public score. When I was completing in 2022, I used an averaging ensemble approach, which did not yield the most accurate public score relative to my prior submissions, but that set of prediction ranked first on the private leaderboard.
", "rawMarkdown": "This is very impressive work from you both! I have not used AutoML or an autoencoder prior to model development, but I will look more into these methods after seeing your results. Keep up the great work!\n\nAnother key benefit of Ensemble ML is that it is a useful way to increase the generalizability of your solution. It is easy to overfit to the public leaderboard, especially if you are using a complex algorithm and iteratively assessing its performance based on your public score. When I was completing in 2022, I used an averaging ensemble approach, which did not yield the most accurate public score relative to my prior submissions, but that set of prediction ranked first on the private leaderboard.", "votes": null }, { "id": "3047194", "postDate": "11/16/2024 11:05:25", "content": "Thans for sharing! I'm so shocked when I saw your school on the paper www
\nWill spend my afternoon on it.
update: my autoencoder together with the automl improved on the lb score greatly. Starting to worry about overfitting too
", "rawMarkdown": "update: my autoencoder together with the automl improved on the lb score greatly. Starting to worry about overfitting too", "votes": null }, { "id": "3049568", "postDate": "11/19/2024 09:24:07", "content": "I also did feature selection by catboost first and put the features into automl training. I only retain about 800 features after selection of ~6000 features
", "rawMarkdown": "I also did feature selection by catboost first and put the features into automl training. I only retain about 800 features after selection of ~6000 features", "votes": null }, { "id": "3049569", "postDate": "11/19/2024 09:24:58", "content": "I do have an initial oof score but it is rather coarse
", "rawMarkdown": "I do have an initial oof score but it is rather coarse", "votes": null }, { "id": "3049782", "postDate": "11/19/2024 14:07:15", "content": "Perfekt!! As for the overfitting problem… We'll just wait and see www
\nDo you have any idea how to do further improvement?
I think you can also try autoencoder. As for myself I don’t exactly know at this point. Maybe some individual house can also be filtered since they are outliers to some extent. Maybe doing a dozen of automl models and put them together in the end could be helpful to prevent overfitting. Maybe there could be a better way of doing attention that I can come up with than just autoencoder. We will see.
", "rawMarkdown": "I think you can also try autoencoder. As for myself I don’t exactly know at this point. Maybe some individual house can also be filtered since they are outliers to some extent. Maybe doing a dozen of automl models and put them together in the end could be helpful to prevent overfitting. Maybe there could be a better way of doing attention that I can come up with than just autoencoder. We will see.", "votes": null }, { "id": "3053775", "postDate": "11/23/2024 22:17:02", "content": "You can also train separate models with automl and merge them. I boosted another 3000 with that.
", "rawMarkdown": "You can also train separate models with automl and merge them. I boosted another 3000 with that.", "votes": null }, { "id": "3053780", "postDate": "11/23/2024 22:28:43", "content": "Thanks again! I'm struggling with reproducing my best result for now… How did you merge them? By using VotingRegressor…?
", "rawMarkdown": "Thanks again! I'm struggling with reproducing my best result for now... How did you merge them? By using VotingRegressor...?", "votes": null }, { "id": "3054654", "postDate": "11/25/2024 00:15:11", "content": "No. merely averaging ensemble. My best result is very reproducible right now… Hopefully that is a good sign. Good luck though!
", "rawMarkdown": "No. merely averaging ensemble. My best result is very reproducible right now... Hopefully that is a good sign. Good luck though!", "votes": null }, { "id": "3054655", "postDate": "11/25/2024 00:16:18", "content": "I could not store oofs for the automl. It always reports size is too large, so I have to only do averaging
", "rawMarkdown": "I could not store oofs for the automl. It always reports size is too large, so I have to only do averaging", "votes": null }, { "id": "3057148", "postDate": "11/27/2024 18:56:34", "content": "@dantingzeng Done! My score(error) increased 10%, but I think this is acceptible? Seems like I survived from overfitting. Very glad to discuss a lot with you in this competition.
\nHopefully, we'll meet again soon.
\n祝妳在美國一切順利！
Haha. My image feature selection actually overfitted, but the text feature selection made up for it. Test based language embedding is actually all we need to get to 70000… which is hilarious. Solutions will be posted later.
\nThe version with no feature selection on the image data, just feature selection on txt ensembling with image, will get to 68000 in both public and private leaderboard, but that is hindsight. Congratulations Kai for the great result!
\nI will be glad to collaborate with you some time. I think https://www.kaggle.com/competitions/santa-2024 this competition is really fun because it's also submission score based. BTW only if you have time. I know studying in Germany is hard! I heard from Dr. Heaton that you landed an internship. Congratulations on that too!
你也一切顺利！
", "rawMarkdown": "Haha. My image feature selection actually overfitted, but the text feature selection made up for it. Test based language embedding is actually all we need to get to 70000… which is hilarious. Solutions will be posted later.\nThe version with no feature selection on the image data, just feature selection on txt ensembling with image, will get to 68000 in both public and private leaderboard, but that is hindsight. Congratulations Kai for the great result! \nI will be glad to collaborate with you some time. I think https://www.kaggle.com/competitions/santa-2024 this competition is really fun because it's also submission score based. BTW only if you have time. I know studying in Germany is hard! I heard from Dr. Heaton that you landed an internship. Congratulations on that too!\n\n你也一切顺利！", "votes": null }, { "id": "3057184", "postDate": "11/27/2024 20:18:32", "content": "Thanks!! Actually I lost my internship in the end because my manager finally realised I don't speak business German:P
\nAnd Yes! Now jobless Kai definitely has time to join another competition! I proposed that we all spend some times to figure out and make a group when the deadline gets closer!
", "rawMarkdown": "Thanks!! Actually I lost my internship in the end because my manager finally realised I don't speak business German:P\n\nAnd Yes! Now jobless Kai definitely has time to join another competition! I proposed that we all spend some times to figure out and make a group when the deadline gets closer!", "votes": null }, { "id": "3057189", "postDate": "11/27/2024 20:32:54", "content": "Please remove your personal info. I have sent you a message. Oh no about the internship
", "rawMarkdown": "Please remove your personal info. I have sent you a message. Oh no about the internship", "votes": null } ], "comments": [ { "id": 3045042, "author_name": "ravi20076", "author_url": "", "post_date": "11/14/2024 05:41:40", "content": "@kaiyisung ensemble models are very much used in real life scenarios, but project requirements necessitate their usage. We often work on several projects where time constraints, maintenance problems and overall customer demands are skewed. In such cases, a simple single model/ small model works well. Ensembles can be used in cases where budgets and time constraints are relatively less stringent.
", "votes": null, "replies": [ { "id": 3045420, "author_name": "kaiyisung", "author_url": "", "post_date": "11/14/2024 13:22:43", "content": "@ravi20076 Thanks for your sharing. I just started in this field. Can you give me some advices to my workflow? I start with data cleaning, make a model ranking to select the best model, use optuna to find the best hyperparameter, and start to train the model. After that, I might do features permutation and retrain the model again. This is what I have done so far… I think there is more I can do or some details that were ignored by me… Thanks again!!
", "votes": null, "replies": [ { "id": 3045718, "author_name": "dantingzeng", "author_url": "", "post_date": "11/14/2024 18:03:42", "content": "Different models can come together. You can try hill climb, ridge or etc. They were not too strong for me but they could work for you.
", "votes": null, "replies": [ { "id": 3045873, "author_name": "kaiyisung", "author_url": "", "post_date": "11/14/2024 21:43:35", "content": "I just checked ridge , can I understand it as another regression model?
", "votes": null, "replies": [ { "id": 3045908, "author_name": "dantingzeng", "author_url": "", "post_date": "11/14/2024 22:38:56", "content": "yes. It can be used to regress for your oof and select what percentage of each notebook's output to be used in your final submission. You can check this out. https://www.kaggle.com/code/adaubas/pss4e05-1st-place-solution-ensemble-with-ridge
", "votes": null, "replies": [] }, { "id": 3046699, "author_name": "dantingzeng", "author_url": "", "post_date": "11/15/2024 18:51:35", "content": "Is it what is working for you now?
", "votes": null, "replies": [ { "id": 3046826, "author_name": "kaiyisung", "author_url": "", "post_date": "11/15/2024 23:01:09", "content": "@dantingzeng Jein…? I tried three different ways, but I said the main improvement is achieved by autogluon. It definitely did a far better emsembling models job than me.
\n1st try: performing feature engineering with cat and lgbm(90000)
\n I combine an embeddings(1546 columns) and tabular data(11 columns). After feature engineering, I got a 1200 columns data. Yep, it actually improve my grade from 100000 to 90000.
\n2nd try: Autogluon without feature engineering(77000)
\n I get a very good result as you noticed.
\n3rd try: Autogluon withfeature engineering(70000)
\n Though Autogluon can do that on its own, I trust myself more :P
This is what just happened… However, I'm worried about overfitting when my score get higher and higher.
", "votes": null, "replies": [ { "id": 3046998, "author_name": "dantingzeng", "author_url": "", "post_date": "11/16/2024 06:11:11", "content": "This is impressive! Thanks so much for sharing. You can check your overfitting by holding out some of your data for training and testing them afterwards. People in kaggle usually have both cv and lb score to verify that they are doing a good training. To reciprocate I have shared my base line for image prompt. I will start to experiment with AutoML and I will say that doing autoencoder before merging to catboost is actually what improved my score to what it was. https://aclanthology.org/2021.maiworkshop-1.5/ Almost the same method have used for my conference paper a few years ago. Maybe it could be doing some similar things to the AutoML but not as good haha. Technology improving.
", "votes": null, "replies": [ { "id": 3047036, "author_name": "robertjirsaraie", "author_url": "", "post_date": "11/16/2024 07:06:21", "content": "This is very impressive work from you both! I have not used AutoML or an autoencoder prior to model development, but I will look more into these methods after seeing your results. Keep up the great work!
\nAnother key benefit of Ensemble ML is that it is a useful way to increase the generalizability of your solution. It is easy to overfit to the public leaderboard, especially if you are using a complex algorithm and iteratively assessing its performance based on your public score. When I was completing in 2022, I used an averaging ensemble approach, which did not yield the most accurate public score relative to my prior submissions, but that set of prediction ranked first on the private leaderboard.
", "votes": null, "replies": [] }, { "id": 3047194, "author_name": "kaiyisung", "author_url": "", "post_date": "11/16/2024 11:05:25", "content": "Thans for sharing! I'm so shocked when I saw your school on the paper www
\nWill spend my afternoon on it.
update: my autoencoder together with the automl improved on the lb score greatly. Starting to worry about overfitting too
", "votes": null, "replies": [ { "id": 3049568, "author_name": "dantingzeng", "author_url": "", "post_date": "11/19/2024 09:24:07", "content": "I also did feature selection by catboost first and put the features into automl training. I only retain about 800 features after selection of ~6000 features
", "votes": null, "replies": [] }, { "id": 3049569, "author_name": "dantingzeng", "author_url": "", "post_date": "11/19/2024 09:24:58", "content": "I do have an initial oof score but it is rather coarse
", "votes": null, "replies": [ { "id": 3049782, "author_name": "kaiyisung", "author_url": "", "post_date": "11/19/2024 14:07:15", "content": "Perfekt!! As for the overfitting problem… We'll just wait and see www
\nDo you have any idea how to do further improvement?
I think you can also try autoencoder. As for myself I don’t exactly know at this point. Maybe some individual house can also be filtered since they are outliers to some extent. Maybe doing a dozen of automl models and put them together in the end could be helpful to prevent overfitting. Maybe there could be a better way of doing attention that I can come up with than just autoencoder. We will see.
", "votes": null, "replies": [ { "id": 3053775, "author_name": "dantingzeng", "author_url": "", "post_date": "11/23/2024 22:17:02", "content": "You can also train separate models with automl and merge them. I boosted another 3000 with that.
", "votes": null, "replies": [ { "id": 3053780, "author_name": "kaiyisung", "author_url": "", "post_date": "11/23/2024 22:28:43", "content": "Thanks again! I'm struggling with reproducing my best result for now… How did you merge them? By using VotingRegressor…?
", "votes": null, "replies": [ { "id": 3054654, "author_name": "dantingzeng", "author_url": "", "post_date": "11/25/2024 00:15:11", "content": "No. merely averaging ensemble. My best result is very reproducible right now… Hopefully that is a good sign. Good luck though!
", "votes": null, "replies": [] }, { "id": 3054655, "author_name": "dantingzeng", "author_url": "", "post_date": "11/25/2024 00:16:18", "content": "I could not store oofs for the automl. It always reports size is too large, so I have to only do averaging
", "votes": null, "replies": [ { "id": 3057148, "author_name": "kaiyisung", "author_url": "", "post_date": "11/27/2024 18:56:34", "content": "@dantingzeng Done! My score(error) increased 10%, but I think this is acceptible? Seems like I survived from overfitting. Very glad to discuss a lot with you in this competition.
\nHopefully, we'll meet again soon.
\n祝妳在美國一切順利！
Haha. My image feature selection actually overfitted, but the text feature selection made up for it. Test based language embedding is actually all we need to get to 70000… which is hilarious. Solutions will be posted later.
\nThe version with no feature selection on the image data, just feature selection on txt ensembling with image, will get to 68000 in both public and private leaderboard, but that is hindsight. Congratulations Kai for the great result!
\nI will be glad to collaborate with you some time. I think https://www.kaggle.com/competitions/santa-2024 this competition is really fun because it's also submission score based. BTW only if you have time. I know studying in Germany is hard! I heard from Dr. Heaton that you landed an internship. Congratulations on that too!
你也一切顺利！
", "votes": null, "replies": [ { "id": 3057184, "author_name": "kaiyisung", "author_url": "", "post_date": "11/27/2024 20:18:32", "content": "Thanks!! Actually I lost my internship in the end because my manager finally realised I don't speak business German:P
\nAnd Yes! Now jobless Kai definitely has time to join another competition! I proposed that we all spend some times to figure out and make a group when the deadline gets closer!
", "votes": null, "replies": [ { "id": 3057189, "author_name": "dantingzeng", "author_url": "", "post_date": "11/27/2024 20:32:54", "content": "Please remove your personal info. I have sent you a message. Oh no about the internship
", "votes": null, "replies": [] } ] } ] } ] } ] } ] } ] } ] } ] } ] } ] } ] } ] } ] } ] } ] } ] }, { "id": 3045911, "author_name": "dantingzeng", "author_url": "", "post_date": "11/14/2024 22:40:25", "content": "What type of data cleaning did you do? My feature selection tells me every data is important which is a bit strange
", "votes": null, "replies": [] } ] } ] } ], "raw_markdown_by_id": { "3044889": "Kinda out of topic, I heard that ensembling models is a powerful method in Kaggle competition. However, do we also solve a real-world problem with this method or it's more like a kaggle thing?", "3045042": "kaiyisung ensemble models are very much used in real life scenarios, but project requirements necessitate their usage. We often work on several projects where time constraints, maintenance problems and overall customer demands are skewed. In such cases, a simple single model/ small model works well. Ensembles can be used in cases where budgets and time constraints are relatively less stringent.", "3045420": "ravi20076 Thanks for your sharing. I just started in this field. Can you give me some advices to my workflow? I start with data cleaning, make a model ranking to select the best model, use optuna to find the best hyperparameter, and start to train the model. After that, I might do features permutation and retrain the model again. This is what I have done so far... I think there is more I can do or some details that were ignored by me... Thanks again!!", "3045718": "Different models can come together. You can try hill climb, ridge or etc. They were not too strong for me but they could work for you.", "3045873": "I just checked ridge , can I understand it as another regression model?", "3045908": "yes. It can be used to regress for your oof and select what percentage of each notebook's output to be used in your final submission. You can check this out. https://www.kaggle.com/code/adaubas/pss4e05-1st-place-solution-ensemble-with-ridge", "3045911": "What type of data cleaning did you do? My feature selection tells me every data is important which is a bit strange", "3046699": "Is it what is working for you now?", "3046826": "dantingzeng Jein...? I tried three different ways, but I said the main improvement is achieved by autogluon. It definitely did a far better emsembling models job than me.\n\n**1st try: performing feature engineering with cat and lgbm(90000)** \n I combine an embeddings(1546 columns) and tabular data(11 columns). After feature engineering, I got a 1200 columns data. Yep, it actually improve my grade from 100000 to 90000.\n**2nd try: Autogluon without feature engineering(77000)** \n I get a very good result as you noticed.\n**3rd try: Autogluon withfeature engineering(70000)** \n Though Autogluon can do that on its own, I trust myself more :P \n\nThis is what just happened... However, I'm worried about overfitting when my score get higher and higher.", "3046998": "This is impressive! Thanks so much for sharing. You can check your overfitting by holding out some of your data for training and testing them afterwards. People in kaggle usually have both cv and lb score to verify that they are doing a good training. To reciprocate I have shared my base line for image prompt. I will start to experiment with AutoML and I will say that doing autoencoder before merging to catboost is actually what improved my score to what it was. https://aclanthology.org/2021.maiworkshop-1.5/ Almost the same method have used for my conference paper a few years ago. Maybe it could be doing some similar things to the AutoML but not as good haha. Technology improving.", "3047036": "This is very impressive work from you both! I have not used AutoML or an autoencoder prior to model development, but I will look more into these methods after seeing your results. Keep up the great work!\n\nAnother key benefit of Ensemble ML is that it is a useful way to increase the generalizability of your solution. It is easy to overfit to the public leaderboard, especially if you are using a complex algorithm and iteratively assessing its performance based on your public score. When I was completing in 2022, I used an averaging ensemble approach, which did not yield the most accurate public score relative to my prior submissions, but that set of prediction ranked first on the private leaderboard.", "3047194": "Thans for sharing! I'm so shocked when I saw your school on the paper www \nWill spend my afternoon on it.", "3049567": "update: my autoencoder together with the automl improved on the lb score greatly. Starting to worry about overfitting too", "3049568": "I also did feature selection by catboost first and put the features into automl training. I only retain about 800 features after selection of ~6000 features", "3049569": "I do have an initial oof score but it is rather coarse", "3049782": "Perfekt!! As for the overfitting problem... We'll just wait and see www\nDo you have any idea how to do further improvement?", "3049993": "I think you can also try autoencoder. As for myself I don’t exactly know at this point. Maybe some individual house can also be filtered since they are outliers to some extent. Maybe doing a dozen of automl models and put them together in the end could be helpful to prevent overfitting. Maybe there could be a better way of doing attention that I can come up with than just autoencoder. We will see.", "3053775": "You can also train separate models with automl and merge them. I boosted another 3000 with that.", "3053780": "Thanks again! I'm struggling with reproducing my best result for now... How did you merge them? By using VotingRegressor...?", "3054654": "No. merely averaging ensemble. My best result is very reproducible right now... Hopefully that is a good sign. Good luck though!", "3054655": "I could not store oofs for the automl. It always reports size is too large, so I have to only do averaging", "3057148": "dantingzeng Done! My score(error) increased 10%, but I think this is acceptible? Seems like I survived from overfitting. Very glad to discuss a lot with you in this competition. \n\nHopefully, we'll meet again soon.\n祝妳在美國一切順利！", "3057174": "Haha. My image feature selection actually overfitted, but the text feature selection made up for it. Test based language embedding is actually all we need to get to 70000… which is hilarious. Solutions will be posted later.\nThe version with no feature selection on the image data, just feature selection on txt ensembling with image, will get to 68000 in both public and private leaderboard, but that is hindsight. Congratulations Kai for the great result! \nI will be glad to collaborate with you some time. I think https://www.kaggle.com/competitions/santa-2024 this competition is really fun because it's also submission score based. BTW only if you have time. I know studying in Germany is hard! I heard from Dr. Heaton that you landed an internship. Congratulations on that too!\n\n你也一切顺利！", "3057184": "Thanks!! Actually I lost my internship in the end because my manager finally realised I don't speak business German:P\n\nAnd Yes! Now jobless Kai definitely has time to join another competition! I proposed that we all spend some times to figure out and make a group when the deadline gets closer!", "3057189": "Please remove your personal info. I have sent you a message. Oh no about the internship" }, "source": "meta" }