{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"name":"python","version":"3.10.14","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"},"kaggle":{"accelerator":"none","dataSources":[{"sourceId":85540,"databundleVersionId":9725394,"sourceType":"competition"}],"dockerImageVersionId":30786,"isInternetEnabled":true,"language":"python","sourceType":"notebook","isGpuEnabled":false}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"**Introduction to Embedding Models**\n\nAs covered in [Module 6 on Retrieval-Augmented Generation](https://github.com/jeffheaton/app_generative_ai/blob/a4ceb5dd0e6714cdd905d3db3edf2a688706328c//t81_559_class_06_3_embeddings.ipynb), embedding models are an efficient way to capture the semantic meaning for a given chunk of text and find similarities across a large database of files. Embedding models work representing words or phrases as vectors in a high-dimensional space. For each segment of text that are input into the model, a single vector is returned. Vectors that are numerically closer together indicate that the corresponding segments of text have a similar semantic meaning. We demonstrated previously that these models are useful for providing LLMs with relevant information that would otherwise be unknown to the foundational model. Here, we can extend the potential use cases of embedding models to derive novel features based on the home inspection reports, which in turn may help us better predict home values.\n\nThrough this process we will also explore alternative embedding models that are open-source and free to use, as opposed to the *text-embedding-3-large* and *text-embedding-3-small* models from OpenAI. You may want to experiment with which embedding model yeilds the best results, as each one comes with a unique set of advantages and pitfalls. Let's start by loading the relevant software packages and the housing data.","metadata":{}},{"cell_type":"code","source":"#Install Software Packages\n!pip install wget -q\n!pip install fasttext -q\n!pip install sentence_transformers -q\n\n#Load Software Packages\nfrom sentence_transformers import SentenceTransformer\nfrom concurrent.futures import ProcessPoolExecutor\nimport pandas as pd, numpy as np, os, glob, warnings\nimport requests, zipfile, wget, gzip, shutil\nimport fasttext, fasttext.util, textwrap, re\nfrom fasttext import load_model\n\n#Load Kaggle Database \nROOT='/kaggle/input/applications-of-deep-learning-wustl-fall-2024/'\ntest=pd.read_csv(f\"{ROOT}/df-test_200x11.csv\", dtype={'homeid': str})\ntrain=pd.read_csv(f\"{ROOT}/df-train_1000x12.csv\", dtype={'homeid': str})\nreports=glob.glob(f\"{ROOT}/SimReports/SimReports/Home-*_Split-*.txt\")\nimages=glob.glob(f\"{ROOT}/SimPictures/SimPictures/Home-*_Split-*.txt\")\n             \n#Read in a Single Inspection Report\nwith open(reports[0], 'r') as file:\n    inspection = file.read()\n\nprint(inspection)","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**Evaluating the Inspection Reports & Exploring potential Models**\n\nYou can see above that each report contains three sections, including a general description of the home, key findings from the inspection, and a comprehensive estimate of the total repairs. Given that each of these segments contains a unique perspective about the quality of the home, it is possible that applying the embedding model to a specific subsection of text may lead to features that better map onto home values. You may want to conduct some tests to find the vectors that are contain the most predictive power, for now we will proceed with applying embedding models to the entire insepection report. \n\n\nI will show you how to access four modes below, but there are a wide range of options available. The [Huggingfaces Platform contains 6,413 models](https://huggingface.co/models?pipeline_tag=sentence-similarity&sort=trending) that can be used for feature extraction or sentense simularity tasks. There are an endless number of options available to you, I would encourage you to go beyond the models covered below.","metadata":{}},{"cell_type":"markdown","source":"\n**FastText** is an efficient word representation and text classification library developed by Meta. It improves upon Word2Vec by using subword information, allowing it to generate embeddings for out-of-vocabulary words and morphologically complex languages. Although this library was design to yeild embeddings for individual words rather than sentences/pharagraphs, I have found that these word embeddings can be useful for training page classification algorithems.","metadata":{}},{"cell_type":"code","source":"#Downloading fasttext model\nurl = \"https://dl.fbaipublicfiles.com/fasttext/vectors-crawl/cc.en.300.bin.gz\"\nwget.download(url, out=\"cc.en.300.bin.gz\")\n\n#Decompress Model\nwith gzip.open(\"cc.en.300.bin.gz\", 'rb') as f_in:\n    with open(\"cc.en.300.bin\", 'wb') as f_out:\n        shutil.copyfileobj(f_in, f_out)\n\n#Apply Model\nmodel = fasttext.load_model('./cc.en.300.bin')\nword_vector = model.get_word_vector(inspection)\nprint(f\"Length of Vectors Produced by fasttext: {len(word_vector)}\")","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**RoBERTa (Robustly Optimized BERT Pretraining Approach)** is a transformer-based model that was deigned to be applied across a larger sequence of words and larger batch sizes. This approach uses contextual embeddings to dynamically adjusting word meanings based on sentence context. As such, a larger number of vectors are produced by this model, which may better capture the nauances that are missed by smaller word embedding models. Consequently, applying RoBERTa requires more compute resources and processing time to accomedate its larger model size and extensive pretraining. This is the largest embedding model of all ones cover in this guide.","metadata":{}},{"cell_type":"code","source":"roberta = SentenceTransformer(f'sentence-transformers/nli-roberta-base-v2'); vector=roberta.encode(inspection)\nprint(f\"Length of Vectors Produced by Roberta: {len(vector)}\")","metadata":{"execution":{"iopub.status.busy":"2024-10-03T20:17:39.012214Z","iopub.execute_input":"2024-10-03T20:17:39.012676Z","iopub.status.idle":"2024-10-03T20:17:39.037185Z","shell.execute_reply.started":"2024-10-03T20:17:39.012636Z","shell.execute_reply":"2024-10-03T20:17:39.035768Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**PMPNet (Phrase-based Multilingual Pretrained Network)** focuses on learning embeddings for multilingual phrases, enhancing performance on cross-lingual NLP tasks. It is optimized for phrase-level rather than word-level understanding. As such, it likely is the best suited to be applied on larger bodies of text, as is the case for the inspection report herein this kaggle compeition. Yet, its multilingual benefits are not releveant in this context and applying this relatively large model will also require more compute power and processing time.","metadata":{}},{"cell_type":"code","source":"pmpnet = SentenceTransformer(f'sentence-transformers/paraphrase-mpnet-base-v2'); vector=pmpnet.encode(inspection)\nprint(f\"Length of Vectors Produced by pmpnet: {len(vector)}\")","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**MiniML** is a lightweight word embedding model designed for efficient inference with reduced computational and memory requirements. It balances performance and resource efficiency, making it suitable for low-resource environments. However, its emphrasis on speed and efficiency may lead to reduced accuracy relative to the larger models shown above.","metadata":{}},{"cell_type":"code","source":"minilm = SentenceTransformer(f'sentence-transformers/all-MiniLM-L12-v2'); vector=minilm.encode(inspection)\nprint(f\"Length of Vectors Produced by MiniLM: {len(vector)}\")","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**Restructuring Embedding Vectors into Pandas Dataframe**\n\nAfter obtaining vectors you will need to convert them into a pandas database format. You can apply the code below for each home in the database using a for-loop or custom-built function. This is an essential step before these features can be combined with the training/testing data or further processed using unsupervised clusting algorithems.","metadata":{}},{"cell_type":"code","source":"df = pd.DataFrame(vector).T\ndf.columns = [f'EmbeddingVec_{i+1}' for i in range(len(df.columns))]\ndf['homeid'] = os.path.basename(reports[0]).split('_')[0].split('-')[1]\ndf=df[['homeid'] + [col for col in df.columns if col != 'homeid']]","metadata":{},"execution_count":null,"outputs":[]}]}