![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F5742832%2Fa2acf24671a17facc7fbf9fb0dd54610%2Fsolution.png?generation=1732737675321877&alt=media\")
Overview of the Approach
\nTo predict home values, our solution integrates structured tabular data, unstructured textual inspection reports, and street-level images. We employ a multimodal pipeline, leveraging advanced feature extraction and machine learning techniques. The process is divided into three main pipelines: Text Pipeline, Image Pipeline, and Tabular Data Integration, followed by an ensemble model to combine predictions.
Text Pipeline
\nEmbedding Extraction: Text from inspection reports is transformed into numerical embeddings using pre-trained language models: BERT, RoBERTa, FastText, and MiniLM, as suggested in the ApplyEmbeddings notebook provided by the TA. https://www.kaggle.com/code/robertjirsaraie/applyembeddings
\nVariational Autoencoder (VAE): The ~3000-dimensional embeddings are reduced via a VAE to focus on the most meaningful features.
\nFeature Selection: By training using CatBoost and assessing feature importance, we retain ~400 features that are most predictive of home values.
\nIntegration and Training: The selected features are combined with tabular data and used to train a Tabular AutoML model (AutoGluon) for 3 hours. The AutoML method was suggested by Kai Sung https://www.kaggle.com/competitions/applications-of-deep-learning-wustl-fall-2024/discussion/546106 initally.
Image Pipeline
\nEmbedding Extraction: Pre-trained vision models (BLIP-2, CLIP, and Qwen2-vl followed by BERT) are used to generate numerical embeddings from street-level images. Prompts that are used for Qwen2-vl include building, surroundings and mood of the image.
\nVariational Autoencoder (VAE): The ~3000-dimensional image embeddings are also refined using a VAE to focus on the most meaningful features.
\nFeature Selection: Similar to the text pipeline, CatBoost is used to select ~400 features from the image embeddings. This step may have some problems because the training without feature selection had a better score in the final submission.
\nIntegration and Training: These features are combined with tabular data and additional tabular data retrieved from text data using ChatGPT (e.g., location, renovation cost) and trained using AutoGluon for 3 hours.
Tabular Data Integration
\nStructured Features: Tabular data includes key features provided initially and additional features retrieved using ChatGPT such as location, renovation cost, and home condition. These are integrated into both the text and image pipelines during training. Interestingly, the location feature contribute greatly to the image data training.
Ensemble Model
\nPredictions from the Text Pipeline and Image Pipeline are averaged using an ensemble approach. This ensures complementary insights are combined to minimize prediction errors.
What Didn’t Work
\nSimple Feature Aggregation for Images and Text:
\nInitially, we tried combining image and text embeddings directly using concatenation. However, this approach led to overfitting due to the high dimensionality of the combined feature space and the lack of targeted feature selection. The model struggled to generalize and produced suboptimal predictions on the validation set.
\nNon-Ensemble Approaches:
\nUsing a single unified model to handle text, image, and tabular data together did not perform as well as the ensemble approach. The lack of modularity made it difficult for the model to capture unique insights from each data modality, leading to reduced predictive accuracy.
\nOther Training Methods on Tabular Data:
\nWe experimented with advanced methods like FTTransformer, which is designed to handle tabular data using transformer-based architectures. While promising in other scenarios, it failed to outperform the AutoGluon-based pipeline due to the lack of robustness in handling highly engineered multimodal features in this dataset.
\nCNN-Based Image Training Methods:
\nWe attempted to train Convolutional Neural Networks (CNNs) from scratch on the street-level images. This approach was computationally expensive and resulted in lower performance compared to pre-trained vision-language models like BLIP-2, CLIP, and Qwen2-vl. The lack of pretraining and domain-specific augmentations likely contributed to its inefficacy.
\nLonger Training sessions with AutoGluton
\nBecause the dataset is relatively small, longer training with AutoGluton than 3 hours have lead to no further improvement in the score if not overfitting.
Validation Strategy
\nA 5-fold cross-validation strategy was employed to ensure the model generalized well across unseen data. We split the data by home IDs to avoid data leakage between training and validation sets.
In the end, language model is all you need!
", "messages": [ { "id": 3057177, "postDate": "2024-11-27T20:02:33.250Z", "content": "
Overview of the Approach
\nTo predict home values, our solution integrates structured tabular data, unstructured textual inspection reports, and street-level images. We employ a multimodal pipeline, leveraging advanced feature extraction and machine learning techniques. The process is divided into three main pipelines: Text Pipeline, Image Pipeline, and Tabular Data Integration, followed by an ensemble model to combine predictions.
Text Pipeline
\nEmbedding Extraction: Text from inspection reports is transformed into numerical embeddings using pre-trained language models: BERT, RoBERTa, FastText, and MiniLM, as suggested in the ApplyEmbeddings notebook provided by the TA. https://www.kaggle.com/code/robertjirsaraie/applyembeddings
\nVariational Autoencoder (VAE): The ~3000-dimensional embeddings are reduced via a VAE to focus on the most meaningful features.
\nFeature Selection: By training using CatBoost and assessing feature importance, we retain ~400 features that are most predictive of home values.
\nIntegration and Training: The selected features are combined with tabular data and used to train a Tabular AutoML model (AutoGluon) for 3 hours. The AutoML method was suggested by Kai Sung https://www.kaggle.com/competitions/applications-of-deep-learning-wustl-fall-2024/discussion/546106 initally.
Image Pipeline
\nEmbedding Extraction: Pre-trained vision models (BLIP-2, CLIP, and Qwen2-vl followed by BERT) are used to generate numerical embeddings from street-level images. Prompts that are used for Qwen2-vl include building, surroundings and mood of the image.
\nVariational Autoencoder (VAE): The ~3000-dimensional image embeddings are also refined using a VAE to focus on the most meaningful features.
\nFeature Selection: Similar to the text pipeline, CatBoost is used to select ~400 features from the image embeddings. This step may have some problems because the training without feature selection had a better score in the final submission.
\nIntegration and Training: These features are combined with tabular data and additional tabular data retrieved from text data using ChatGPT (e.g., location, renovation cost) and trained using AutoGluon for 3 hours.
Tabular Data Integration
\nStructured Features: Tabular data includes key features provided initially and additional features retrieved using ChatGPT such as location, renovation cost, and home condition. These are integrated into both the text and image pipelines during training. Interestingly, the location feature contribute greatly to the image data training.
Ensemble Model
\nPredictions from the Text Pipeline and Image Pipeline are averaged using an ensemble approach. This ensures complementary insights are combined to minimize prediction errors.
What Didn’t Work
\nSimple Feature Aggregation for Images and Text:
\nInitially, we tried combining image and text embeddings directly using concatenation. However, this approach led to overfitting due to the high dimensionality of the combined feature space and the lack of targeted feature selection. The model struggled to generalize and produced suboptimal predictions on the validation set.
\nNon-Ensemble Approaches:
\nUsing a single unified model to handle text, image, and tabular data together did not perform as well as the ensemble approach. The lack of modularity made it difficult for the model to capture unique insights from each data modality, leading to reduced predictive accuracy.
\nOther Training Methods on Tabular Data:
\nWe experimented with advanced methods like FTTransformer, which is designed to handle tabular data using transformer-based architectures. While promising in other scenarios, it failed to outperform the AutoGluon-based pipeline due to the lack of robustness in handling highly engineered multimodal features in this dataset.
\nCNN-Based Image Training Methods:
\nWe attempted to train Convolutional Neural Networks (CNNs) from scratch on the street-level images. This approach was computationally expensive and resulted in lower performance compared to pre-trained vision-language models like BLIP-2, CLIP, and Qwen2-vl. The lack of pretraining and domain-specific augmentations likely contributed to its inefficacy.
\nLonger Training sessions with AutoGluton
\nBecause the dataset is relatively small, longer training with AutoGluton than 3 hours have lead to no further improvement in the score if not overfitting.
Validation Strategy
\nA 5-fold cross-validation strategy was employed to ensure the model generalized well across unseen data. We split the data by home IDs to avoid data leakage between training and validation sets.
In the end, language model is all you need!
", "rawMarkdown": "\n\nOverview of the Approach\nTo predict home values, our solution integrates **structured tabular data, unstructured textual inspection reports, and street-level images**. We employ a multimodal pipeline, leveraging advanced feature extraction and machine learning techniques. The process is divided into three main pipelines: **Text Pipeline, Image Pipeline, and Tabular Data Integration**, followed by an ensemble model to combine predictions.\n\n1. Text Pipeline\nEmbedding Extraction: Text from inspection reports is transformed into numerical embeddings using pre-trained language models: BERT, RoBERTa, FastText, and MiniLM, as suggested in the ApplyEmbeddings notebook provided by the TA. https://www.kaggle.com/code/robertjirsaraie/applyembeddings\nVariational Autoencoder (VAE): The ~3000-dimensional embeddings are reduced via a VAE to focus on the most meaningful features.\nFeature Selection: By training using CatBoost and assessing feature importance, we retain ~400 features that are most predictive of home values.\nIntegration and Training: The selected features are combined with tabular data and used to train a Tabular AutoML model (AutoGluon) for 3 hours. The AutoML method was suggested by Kai Sung https://www.kaggle.com/competitions/applications-of-deep-learning-wustl-fall-2024/discussion/546106 initally.\n\n2. Image Pipeline\nEmbedding Extraction: Pre-trained vision models (BLIP-2, CLIP, and Qwen2-vl followed by BERT) are used to generate numerical embeddings from street-level images. Prompts that are used for Qwen2-vl include building, surroundings and mood of the image.\nVariational Autoencoder (VAE): The ~3000-dimensional image embeddings are also refined using a VAE to focus on the most meaningful features.\nFeature Selection: Similar to the text pipeline, CatBoost is used to select ~400 features from the image embeddings. This step may have some problems because the training without feature selection had a better score in the final submission.\nIntegration and Training: These features are combined with tabular data and additional tabular data retrieved from text data using ChatGPT (e.g., location, renovation cost) and trained using AutoGluon for 3 hours.\n\n3. Tabular Data Integration\nStructured Features: Tabular data includes key features provided initially and additional features retrieved using ChatGPT such as location, renovation cost, and home condition. These are integrated into both the text and image pipelines during training. Interestingly, the location feature contribute greatly to the image data training.\n\n4. Ensemble Model\nPredictions from the Text Pipeline and Image Pipeline are averaged using an ensemble approach. This ensures complementary insights are combined to minimize prediction errors.\n\nWhat Didn’t Work\nSimple Feature Aggregation for Images and Text:\nInitially, we tried combining image and text embeddings directly using concatenation. However, this approach led to overfitting due to the high dimensionality of the combined feature space and the lack of targeted feature selection. The model struggled to generalize and produced suboptimal predictions on the validation set.\nNon-Ensemble Approaches:\nUsing a single unified model to handle text, image, and tabular data together did not perform as well as the ensemble approach. The lack of modularity made it difficult for the model to capture unique insights from each data modality, leading to reduced predictive accuracy.\nOther Training Methods on Tabular Data:\nWe experimented with advanced methods like FTTransformer, which is designed to handle tabular data using transformer-based architectures. While promising in other scenarios, it failed to outperform the AutoGluon-based pipeline due to the lack of robustness in handling highly engineered multimodal features in this dataset.\nCNN-Based Image Training Methods:\nWe attempted to train Convolutional Neural Networks (CNNs) from scratch on the street-level images. This approach was computationally expensive and resulted in lower performance compared to pre-trained vision-language models like BLIP-2, CLIP, and Qwen2-vl. The lack of pretraining and domain-specific augmentations likely contributed to its inefficacy.\nLonger Training sessions with AutoGluton\nBecause the dataset is relatively small, longer training with AutoGluton than 3 hours have lead to no further improvement in the score if not overfitting.\n\nValidation Strategy\nA 5-fold cross-validation strategy was employed to ensure the model generalized well across unseen data. We split the data by home IDs to avoid data leakage between training and validation sets.\n\nIn the end, language model is all you need!\n", "votes": 2 } ], "comments": [], "raw_markdown_by_id": { "3057177": "\n\nOverview of the Approach\nTo predict home values, our solution integrates **structured tabular data, unstructured textual inspection reports, and street-level images**. We employ a multimodal pipeline, leveraging advanced feature extraction and machine learning techniques. The process is divided into three main pipelines: **Text Pipeline, Image Pipeline, and Tabular Data Integration**, followed by an ensemble model to combine predictions.\n\n1. Text Pipeline\nEmbedding Extraction: Text from inspection reports is transformed into numerical embeddings using pre-trained language models: BERT, RoBERTa, FastText, and MiniLM, as suggested in the ApplyEmbeddings notebook provided by the TA. https://www.kaggle.com/code/robertjirsaraie/applyembeddings\nVariational Autoencoder (VAE): The ~3000-dimensional embeddings are reduced via a VAE to focus on the most meaningful features.\nFeature Selection: By training using CatBoost and assessing feature importance, we retain ~400 features that are most predictive of home values.\nIntegration and Training: The selected features are combined with tabular data and used to train a Tabular AutoML model (AutoGluon) for 3 hours. The AutoML method was suggested by Kai Sung https://www.kaggle.com/competitions/applications-of-deep-learning-wustl-fall-2024/discussion/546106 initally.\n\n2. Image Pipeline\nEmbedding Extraction: Pre-trained vision models (BLIP-2, CLIP, and Qwen2-vl followed by BERT) are used to generate numerical embeddings from street-level images. Prompts that are used for Qwen2-vl include building, surroundings and mood of the image.\nVariational Autoencoder (VAE): The ~3000-dimensional image embeddings are also refined using a VAE to focus on the most meaningful features.\nFeature Selection: Similar to the text pipeline, CatBoost is used to select ~400 features from the image embeddings. This step may have some problems because the training without feature selection had a better score in the final submission.\nIntegration and Training: These features are combined with tabular data and additional tabular data retrieved from text data using ChatGPT (e.g., location, renovation cost) and trained using AutoGluon for 3 hours.\n\n3. Tabular Data Integration\nStructured Features: Tabular data includes key features provided initially and additional features retrieved using ChatGPT such as location, renovation cost, and home condition. These are integrated into both the text and image pipelines during training. Interestingly, the location feature contribute greatly to the image data training.\n\n4. Ensemble Model\nPredictions from the Text Pipeline and Image Pipeline are averaged using an ensemble approach. This ensures complementary insights are combined to minimize prediction errors.\n\nWhat Didn’t Work\nSimple Feature Aggregation for Images and Text:\nInitially, we tried combining image and text embeddings directly using concatenation. However, this approach led to overfitting due to the high dimensionality of the combined feature space and the lack of targeted feature selection. The model struggled to generalize and produced suboptimal predictions on the validation set.\nNon-Ensemble Approaches:\nUsing a single unified model to handle text, image, and tabular data together did not perform as well as the ensemble approach. The lack of modularity made it difficult for the model to capture unique insights from each data modality, leading to reduced predictive accuracy.\nOther Training Methods on Tabular Data:\nWe experimented with advanced methods like FTTransformer, which is designed to handle tabular data using transformer-based architectures. While promising in other scenarios, it failed to outperform the AutoGluon-based pipeline due to the lack of robustness in handling highly engineered multimodal features in this dataset.\nCNN-Based Image Training Methods:\nWe attempted to train Convolutional Neural Networks (CNNs) from scratch on the street-level images. This approach was computationally expensive and resulted in lower performance compared to pre-trained vision-language models like BLIP-2, CLIP, and Qwen2-vl. The lack of pretraining and domain-specific augmentations likely contributed to its inefficacy.\nLonger Training sessions with AutoGluton\nBecause the dataset is relatively small, longer training with AutoGluton than 3 hours have lead to no further improvement in the score if not overfitting.\n\nValidation Strategy\nA 5-fold cross-validation strategy was employed to ensure the model generalized well across unseen data. We split the data by home IDs to avoid data leakage between training and validation sets.\n\nIn the end, language model is all you need!\n" }, "complete": true, "source": "existing_discussion", "competition_id": 85540, "competition_slug": "applications-of-deep-learning-wustl-fall-2024", "team_id": 12811447, "solution_url": "/competitions/applications-of-deep-learning-wustl-fall-2024/writeups/danting-zeng-solution-writeup-first-place-70816-lb" }